BluelightAI has released the first-ever Cross-Layer Transcoders for the Qwen3 model family, starting with Qwen3-0.6B and Qwen3-1.7B versions. These tools enable examination of how Qwen3 encodes concepts, propagates information, and builds meaning across its layers.
The release includes an interactive dashboard — the Qwen3 Explorer — for studying discovered features, tracing activation flows, and visualizing the model through topological data analysis via Cobalt. Together, these resources position "Qwen3 as one of the most interpretable open-source model families available."
Why This Release Matters
Modern language models perform impressive tasks yet remain difficult to explain mechanistically. Mechanistic interpretability research aims to bridge this gap by discovering and understanding the mechanisms AI models use to perform tasks — from syntactic operations to algorithmic reasoning to high-level decision-making.
Cross-layer transcoders, recently introduced by Anthropic, decompose activations into "sparse and interpretable features" that illuminate how models process information and construct meaning.
What Are Cross-Layer Transcoders?
A cross-layer transcoder is an interpreter model trained to decompose internal activations into thousands of sparse features. Each feature ideally corresponds to an interpretable concept: syntactic roles, factual associations, contextual cues, or semantic abstractions the model uses internally.
Operational structure:
- An encoder maps inputs from every model layer into sparse feature activations
- A decoder reconstructs outputs of all future layers using these activations
- Sparse routing enables analysis of how specific features influence later layers and final outputs
Contents of the Release
Cross-Layer Transcoders for Qwen3-0.6B-Base and Qwen3-1.7B-Base are available on Hugging Face and compatible with the open-source circuit-tracer library.
Technical specifications:
- 20,480 features extracted per layer
- Approximately 573,000 total features
- Trained on roughly 750 million tokens from web text, books, code, and mathematical sources
Example Features
The following screenshots illustrate features discovered in Qwen3's intermediate layers through the interactive explorer:
Topological Analysis Using Cobalt
Cobalt builds multiresolution topological graphs from identified features, revealing larger-scale internal representation structures. Separate graphs were generated using encoder vectors, decoder vectors, and coactivation patterns.
All features are explorable at qwen3.bluelightai.com.