Useful and common tool for Interpretability work, particularly with Self Supervised Learning methods like Barlow Twins and JEPA.
- it is good for Representation probing because if it achieves high accuracy on a prediction task off of an embedding, that means that the feature is already well organized in the embeddings and seperate using a straight boundary in that space
- one can also train linear probes on a layer by layer basis for activations, such that people can track where a network constructs specific concepts
- very important for self supervised approaches in tasks like that of ImageNet, where self supervised models dont learn to classify the images of the dataset through their labels, only building a good representation of the features of an image that make them identifiable. Linear probes reveal that these SSL models can distinguish the images to their labels with distinct clusters and thresholds.
