Discerning a speaker's gender from their voice is a basic and crucial aspect of human communication. Voice pitch height, the perceptual correlate of fundamental frequency, is higher in females and provides a cue for gender discrimination. However, male and female voices are also differentiated by multiple other spectral and temporal characteristics, including mean formant frequency and spectral flux. The robust perceptual segregation of male and female voices is thought to result from processing the combination of discriminating features, which in neural terms may correspond to early sound object analysis occurring in non-primary auditory cortex. However, the specific mechanism for gender perception has been unclear. Here, using functional magnetic resonance imaging, we show that discrete sites in non-primary auditory cortex are differentially activated by male and female voices, with female voices consistently evoking greater activation in the upper bank of the superior temporal sulcus and posterior superior temporal plane. This finding was observed at the individual subject-level in all 24 subjects. The neural response was highly specific: no auditory regions were more activated by male than female voices. Further, the activation associated with female voices was 1) larger than can be accounted for by a sole effect of fundamental frequency, 2) not due to psychological attribution of female gender and 3) unaffected by listener gender. These results demonstrate that male and female voices are represented as distinct auditory objects in the human brain, with the mechanism for gender discrimination being a gender-dependent activation-level cue in non-primary auditory cortex.