To understand a system is to see it under perturbation. A systems most ordinary behavior conceals the structure responsible, understanding only comes when looking for circumstances where where behavior changes, breaks down, or diverges from prediction. Merleau-Ponty discusses how neurological pathologies reveal the organization of embodied cogntion. Karl Popper notes importance of falsification as a source of knowledge, a sundial is best understood when it doesn’t work without sun. Norbert Wiener, through Cybernetics, exposes system organization through the disturbance-feedback cycle. Failure is the instrument of understanding, and the most important information about a system is often the least common.
- Every Encoder Loves Its Own Images (Prince et al., 2026) - predictive accuracy leaves mechanism undertermined, two models will explain the same observations while encoding fundamentally different representations.
Differentiate between prediction and discrimination. The most information rich experiment is not necessarily the one yielding the strongest response, but the one which most sharply distinguishes competing hypotheses.
In every regard when humanity has gained a scalpel for a field, new levels of understanding beyond anything beforehand emerge. I believe deep learning gives us a new set of scalpels into Representation as a whole, for the brain, the mind, and the model. Neuroimaging and cognition traces have already given us the microscope, only the scalpel has been missing.
- Mechanistic Interpretability and Stimulus Optimization give us a scalpel for the construction of exploration rich circumstances, seeking maximal disagreement, the exposition of hidden dependence, and separation of competing explanations. Bayesian Optimization for seeking the most oil rich patch of land when little is known.
- model Interpretability largely consists of this: seeing where a system fails or changes. Ablation studies, neuron forcing, feature forcing through SAEs, etc.
- Obviously Computational Neuroscience through Stimulus Optimization and In Silico experimentation extends this domain back to interp.
- what perturbations make a model’s internal organization legible.?
- Compression is Intelligence and Information Theory: the most important information is the least likely information?
Very Fristonian-pilled perchannce?
Is the perfect system the one that can perturb itself for maximal understanding and evolution? Goal of Self Configuration and RSI?
Bibliography
- Prince, J. S., Wang, B., Fel, T., Jagadeesh, A. V., Vaziri, P. A., Alvarez, G. A., Livingstone, M. S., & Konkle, T. (2026). Parametric Neural Control Differentiates Top Neural Network Models of Primate Visual Cortex (p. 2026.08.16.745063). bioRxiv. https://doi.org/10.64898/2026.08.16.745063