Previous studies showed that older adults have difficulties processing fine structure (FS) speech cues such as voice onset time (VOT). In our study, differences between younger (YA) and older (OA) adults in FS perception and their relation to cognitive abilities were investigated behaviorally and electrophysiologically. YA and OA categorized and discriminated syllables (/da/ and /ta/) with varying VOTs. OA had significantly higher error rates than YA both when the difference in VOT was small (discrimination within the same syllable type) and large (discrimination across the /da/-/ta/ boundary), indicating that OA have more problems discriminating VOT in general and especially across the syllable border. Also, OA showed less specificity in the categorization task, suggesting that difficulties in categorization are related to discriminatory abilities across the syllable border. OA had significantly longer latencies and significantly smaller amplitudes of the P2 ERP component evoked by the syllables than YA, reflecting slower and more inaccurate stimulus encoding. Latency of the P2 component was related to perceptual speed in both groups, suggesting that latency reflects perceptual speed rather than encoding effort. Attentional and inhibitory abilities were associated with better discrimination across the syllable border only in OA, showing a beneficial influence of cognitive ability.