Throughout the animal kingdom, the brains of male and female individuals become specialized during pre- and postnatal development for the division of behavioral tasks that deal with reproduction and the rearing of offspring. These differences reflect the consequences of intrinsically produced signals provided primarily by the embryo itself. In mammals, the strongest determinant of these differences is the initial differentiation of gonadal tissues, under the control of the masculinizing transcription factor SRY. SRY determines an individual’s genetic sex, and usually the phenotypic sex as well, but it is not expressed in the brain. The influence of SRY on the nervous system is thus indirect: SRY-mediated masculinization leads to differential production of gonadal tissues during fetal development and thus to sex-specific levels of circulating gonadal hormones (estrogen and testosterone in particular over the course of development). These hormones profoundly influence the development of brain structures that subserve the peripheral structures (genitalia, mammary glands) directly related to reproduction and parenting. Some of these dimorphisms reflect trophic regulation of cell survival and death in relevant structures based on parallel development of peripheral organs that these cells innervate or regulate (male and female genitalia, mammary glands in females, and related muscles). Differences in size, presence or absence of muscles or glandular tissue results in distinct levels of target derived trophic support. The origins, existence, and functional significance of dimorphisms related to distinctions in gender identity and sexual orientation remain controversial. These differences in the brains of women and men must ultimately be due to brain organization and function. Whether they arise via learned behaviors that are used to define gender roles in society or intrinsic developmental mechanisms is not known.