Retina ganglion cells extract specific features from natural scenes and send this information to the brain. In particular, they respond to local light increase (ON responses), and/or decrease (OFF). However, it is unclear if this ON-OFF selectivity, characterized with synthetic stimuli, is maintained when they are stimulated with natural scenes. Here we recorded the responses of ganglion cells of mice and axolotls to stimuli composed of natural images slightly perturbed by patterns of random noise to determine their selectivity during natural stimulation. The ON-OFF selectivity strongly depended on the natural image. A single ganglion cell can signal luminance increase for one natural image, and luminance decrease for another. Modeling and experiments showed that this was due to the non-linear combination of different pathways of the retinal circuit. Despite the versatility of the ON-OFF selectivity, a systematic analysis demonstrated that contrast was reliably encoded in these responses. Our perturbative approach thus uncovers the selectivity of retinal ganglion cells to more complex features than initially thought during natural scene stimulation.