Graph View

Every Encoder Loves Its Own Images

Seven vision models predict the brain equally well, then disagree about what the brain wants to see.

5 min read

PageRank 853rd of 884

Starting interactive Python…
Initializing...
 r = 0.300.400.500.60CLIP ResNet50CLIP ResNet50: 0.61SigLIPSigLIP: 0.59CLIPCLIP: 0.58DINOv2DINOv2: 0.58ResNet50ResNet50: 0.58Robust ResNet50Robust ResNet50: 0.56UntrainedUntrained: 0.32 made by ↓ judged by →CLIPSigLIPDINOv2ResNet50CLIP ResNet50Robust ResNet50UntrainedCLIPCLIP's images, judged by CLIP: +3.743.7CLIP's images, judged by SigLIP: +1.541.5CLIP's images, judged by DINOv2: +1.291.3CLIP's images, judged by ResNet50: +1.241.2CLIP's images, judged by CLIP ResNet50: +1.431.4CLIP's images, judged by Robust ResNet50: +1.101.1CLIP's images, judged by Untrained: +0.310.3SigLIPSigLIP's images, judged by CLIP: +1.581.6SigLIP's images, judged by SigLIP: +4.134.1SigLIP's images, judged by DINOv2: +1.361.4SigLIP's images, judged by ResNet50: +1.351.4SigLIP's images, judged by CLIP ResNet50: +1.521.5SigLIP's images, judged by Robust ResNet50: +1.141.1SigLIP's images, judged by Untrained: +0.400.4DINOv2DINOv2's images, judged by CLIP: +0.961.0DINOv2's images, judged by SigLIP: +1.061.1DINOv2's images, judged by DINOv2: +3.633.6DINOv2's images, judged by ResNet50: +1.061.1DINOv2's images, judged by CLIP ResNet50: +1.061.1DINOv2's images, judged by Robust ResNet50: +0.790.8DINOv2's images, judged by Untrained: +0.240.2ResNet50ResNet50's images, judged by CLIP: +0.961.0ResNet50's images, judged by SigLIP: +0.920.9ResNet50's images, judged by DINOv2: +0.940.9ResNet50's images, judged by ResNet50: +3.753.7ResNet50's images, judged by CLIP ResNet50: +1.081.1ResNet50's images, judged by Robust ResNet50: +0.981.0ResNet50's images, judged by Untrained: +0.240.2CLIP ResNet50CLIP ResNet50's images, judged by CLIP: +0.760.8CLIP ResNet50's images, judged by SigLIP: +0.760.8CLIP ResNet50's images, judged by DINOv2: +0.710.7CLIP ResNet50's images, judged by ResNet50: +0.880.9CLIP ResNet50's images, judged by CLIP ResNet50: +2.582.6CLIP ResNet50's images, judged by Robust ResNet50: +0.680.7CLIP ResNet50's images, judged by Untrained: +0.250.3Robust ResNet50Robust ResNet50's images, judged by CLIP: +1.711.7Robust ResNet50's images, judged by SigLIP: +1.591.6Robust ResNet50's images, judged by DINOv2: +1.741.7Robust ResNet50's images, judged by ResNet50: +1.841.8Robust ResNet50's images, judged by CLIP ResNet50: +1.841.8Robust ResNet50's images, judged by Robust ResNet50: +5.365.4Robust ResNet50's images, judged by Untrained: +1.471.5UntrainedUntrained's images, judged by CLIP: +0.550.5Untrained's images, judged by SigLIP: +0.550.6Untrained's images, judged by DINOv2: +0.570.6Untrained's images, judged by ResNet50: +0.590.6Untrained's images, judged by CLIP ResNet50: +0.590.6Untrained's images, judged by Robust ResNet50: +0.640.6Untrained's images, judged by Untrained: +8.989.0SDs above unsteered, averaged over the four regions 0.000.100.200.30how far the other models follow, per step of the steering modelwith diffusionpixels onlyRobust ResNet50Robust ResNet50: 0.28Robust ResNet50: 0.32SigLIPSigLIP: 0.16SigLIP: 0.13ResNet50ResNet50: 0.13ResNet50: 0.15CLIP ResNet50CLIP ResNet50: 0.13CLIP ResNet50: 0.14DINOv2DINOv2: 0.13DINOv2: 0.14CLIPCLIP: 0.13CLIP: 0.13UntrainedUntrained: 0.03Untrained: 0.01
coarsefineimage power by spatial frequency, relative to the same noise unsteered×1×2×4Robust ResNet50UntrainedCLIPCLIP ResNet50DINOv2SigLIPResNet500.00.51.01.5extra coarse power in its images (percentage points)gradient spreadRobust ResNet50Robust ResNet50: 1.310DINOv2DINOv2: 0.780CLIPCLIP: 0.6100CLIP ResNet50CLIP ResNet50: 0.696ResNet50ResNet50: 0.671UntrainedUntrained: 0.591SigLIPSigLIP: 0.497
 natural photossteering, judged by modelsunusual images, real brains0.13CLIP0.58CLIP0.280.15SigLIP0.59SigLIP0.340.14DINOv20.58DINOv20.310.14ResNet500.58ResNet500.280.14CLIP ResNet500.61CLIP ResNet500.360.30Robust ResNet500.56Robust ResNet500.240.02Untrained0.32Untrained0.07

Bibliography

  • Henderson, M. M., Luo, A. F., Park, S., Tarr, M. J., & Wehbe, L. (2026). Diffusion-Based Stimulus Optimization Reveals Functional Organization across Higher Visual Cortex (p. 2026.05.12.724119). bioRxiv. https://doi.org/10.64898/2026.05.12.724119
  • Prince, J. S., Wang, B., Fel, T., Jagadeesh, A. V., Vaziri, P. A., Alvarez, G. A., Livingstone, M. S., & Konkle, T. (2026). Parametric Neural Control Differentiates Top Neural Network Models of Primate Visual Cortex (p. 2026.08.16.745063). bioRxiv. https://doi.org/10.64898/2026.08.16.745063