English (United Kingdom)

https://curated-unify.zendy.io/wp-json/zendy-region/v1/featured_content/oa?rat=en

https://curated-unify.zendy.io/wp-json/zendy-region/v1/highlighted_journal/

Global: Zendy Plus annual

Presents the access of premium content as premium feature

Premium Content

Presents the keyphrase highlighting as premium feature

Keyphrase Highlighting

Presents the summarisation as premium feature

Summarisation

Insights

Presents the pdf analysis as premium feature

PDF Analysis

Presents the zaia usage as premium feature

ZAIA

Global: Zendy Plus monthly

Global: Zendy Tools annual

Global: Zendy Tools monthly

Global: Zendy Free Plan

How do neural networks extract patterns from pixels? Feature visualizationsattempt to answer this important question by visualizing highly activatingpatterns through optimization. Today, visualization methods form the foundationof our knowledge about the internal workings of neural networks, as a type ofmechanistic interpretability. Here we ask: How reliable are featurevisualizations? We start our investigation by developing network circuits thattrick feature visualizations into showing arbitrary patterns that arecompletely disconnected from normal network behavior on natural input. We thenprovide evidence for a similar phenomenon occurring in standard, unmanipulatednetworks: feature visualizations are processed very differently from standardinput, casting doubt on their ability to "explain" how neural networks processnatural images. This can be used as a sanity check for feature visualizations.We underpin our empirical findings by theory proving that the set of functionsthat can be reliably understood by feature visualization is extremely small anddoes not include general black-box neural networks. Therefore, a promising wayforward could be the development of networks that enforce certain structures inorder to ensure more reliable feature visualizations.

Don't trust your eyes: on the (un)reliability of feature visualizations