Distinguishing mirror from glass: A 'big data' approach to material perception

Visually identifying materials is crucial for many tasks, yet materialperception remains poorly understood. Distinguishing mirror from glass isparticularly challenging as both materials derive their appearance from theirsurroundings, yet we rarely experience difficulties telling them apart. Here wetook a 'big data' approach to uncovering the underlying visual cues andprocesses, leveraging recent advances in neural network models of vision. Wetrained thousands of convolutional neural networks on >750,000 simulated mirrorand glass objects, and compared their performance with human judgments, as wellas alternative classifiers based on 'hand-engineered' image features. Forrandomly chosen images, all classifiers and humans performed with highaccuracy, and therefore correlated highly with one another. To tease the modelsapart, we then painstakingly assembled a diagnostic image set for which humansmake highly systematic errors, allowing us to decouple accuracy from human-likeperformance. A large-scale, systematic search through feedforward neuralarchitectures revealed that relatively shallow networks predicted humanjudgments better than any other models. However, surprisingly, no networkcorrelated better than 0.6 with humans (below inter-human correlations). Thus,although the model sets new standards for simulating human vision in achallenging material perception task, the results cast doubt on recent claimsthat such architectures are generally good models of human vision.