tl;dr: About 80% of released J-lenses target the final layer. On DeepSeek-V3, though, that gives a J-lens dominated by one direction inherited from the final block. It shifts English-vs-Chinese readouts and also inflates one eval. Anthropic's J-lens paper had suggested the final block may specialize in calibrating the next-token prediction. That could make it the block most likely to carry a direction like this, meaning the penultimate layer may be a better default. More generally, this is a case study of how a strong downstream direction can dominate a J-lens and change what earlier layers appear to represent.Overview A J-lens lets you peek inside a model by translating its hidden states into words (a "readout"). It is defined relative to a target layer. Specifically, it asks how a nudge at an earlier layer would change the representation at that target, averaged over many prompts, then reads the result through the model's own unembedding. On DeepSeek-V3, changing the target layer changes what the lens shows you. With the final layer as the target, the J-lens is dominated by a single direction inherited from the last transformer block. That direction shifts the language of the readouts between Chinese and English.[1] This dominant direction arises because in DeepSeek-V3, the final block pushes down all the Chinese tokens when the text is English. This barely changes what the model predicts since those tokens already had almost no probability, but it's a large change to the residual stream. Because the direction is strongly amplified by the last block, it becomes the top Jacobian direction at every earlier layer. Halfway through DeepSeek-V3, on English text, about half the readouts from a final-layer lens come out mostly Chinese tokens. If you switch to the penultimate-layer, the lens is only 3% majority Chinese, matching the logit lens readouts. Changing the target also flips the sign of one of the statistics in Anthropic’s original J-lens paper, excess kurtosis. Additionally, on the order-of-operations prompts, one of the paper's tests, targeting the final layer lifts every number token by about a standard deviation and mistakenly makes J-lens look better than the logit lens on the eval. Figure 1. The final-layer target puts a dominant language-separating direction into DeepSeek-V3's readouts, and the penultimate target removes it. All panels use 2,400 held-out English activations. (a) Chinese-character tokens among each half-depth readout's top 20; 5 of 20 would match the vocabulary. (b) Median share of readout variance explained by the language split. (c) Each layer's top singular direction: its cosine with the top direction of the last layer's Jacobian J₅₉→₆₀, and how well it separates Chinese-character from Latin-alphabet unembedding rows. Shading marks Sonnet 4.5's workspace depths; the ten-prompt lenses are noisy below 30% depth. DeepSeek-V3’s last block amplifies a language-separating direction twelve times more than a typical direction. When the J-lens differentiates through that block, this direction dominates the Jacobian even for much earlier source layers. Targeting the penultimate layer excludes that block and reduces the language effect by roughly twentyfold. This same choice explained several observations I had initially treated as unrelated: Chinese-heavy readouts, anomalous kurtosis, a number-token bias that inflates one evaluation, and a handful of fit prompts with Jacobian norms hundreds of times above typical prompts. The important lesson is not that DeepSeek-V3's J-lens inherits a direction that shifts readouts between English and Chinese. It is that a strong downstream direction can dominate a J-lens and change what earlier layers appear to represent. To avoid an output-specialized final block, the penultimate layer may be a better default. This matters since so far most lenses seem to just default to last layer. Of the 78 publicly released J-lenses I checked, 80% target the final layer, 17% the penultimate layer, and 3% something else. Gurnee et al. used a penultimate-layer target for their Sonnet experiments, but the released library defaults to the final layer. For anyone fitting or using a J-lens, it is worth checking what directions the lens might inherit based on the choice of target layer. Take the top direction of each layer’s Jacobian and check whether it aligns with a dominant direction introduced by the final block, and whether changing the target removes it. Separately, I also tested how the original J-lens results transfer to DeepSeek-V3. I replicate the depth curves the paper uses to locate its workspace, but the plain logit lens reproduces them too. On the paper's evals the J-lens beats the logit lens on two sets and slightly loses on two.Fitting a J-lens for DeepSeek-V3 A J-lens shows which words a hidden state is pushing the model toward, by sending it through an average of what the rest of the network does to that state. Briefly summarized: Pick a layer ℓ. Compute the Jacobian of some late residual stream (the target layer, often the last layer or penultimate layer) with respect to the layer-ℓ residual stream, average it over token positions and a corpus of prompts, and call it J_ℓ. To read out a state h, multiply by J_ℓ, apply the model's final norm and unembedding, and look at the top tokens. The logit lens is the same thing with J_ℓ set to the identity. This post explores the how the choice of target layer affects the lens. The reference library defaults to the final layer, with a docstring note that the penultimate layer "can give a better-conditioned" lens. Among publicly released J-lenses, the final layer is by far the most common. I checked 78 publicly released fits, including Neuronpedia (38), OLMo-3 (11), workspace-lenses (8), Prabodha (8), Andy Xu (5), LAMM-MIT (3), Praxagent (2), Qwen3.5 image lenses (2), and Stanley Li (1). Mirrors were deduplicated (so numbers may look lower than expected). Of these, 63 target the final layer (80.8%), 13 the penultimate layer (16.7%), and 2 other intermediate layers (2.6%). In the original Anthropic J-lens paper, the method is presented with the final layer as the target while the appendix says the lens used throughout the paper targets the penultimate layer because "including the last layer can sometimes increase the number of noisy artifacts in lens-readouts" possibly due to specialization for calibrating next-token predictions.[2] I fit on DeepSeek-V3-0324 (671B MoE, 4-bit AWQ weights): 100 WikiText prompts, all 60 source layers, about 40 minutes per prompt on three H200s, so roughly 80 hours of wall time for the 112 prompts I fit, of which 100 went into the released lens (the pre-filter removed twelve and I fit replacements). The fit keeps running sums, so I also have two disjoint 50-prompt lenses, which help check convergence. They pick the same top token 13% of the time at a quarter of the depth, 58% at half depth and 90% at 83%, so I don't trust the first third of the network. Readouts are on 100 held-out paragraphs. For the cross-model comparison I pulled the same held-out text through Qwen3.5-122B-A10B and Qwen3.5-397B-A17B and ran their public lenses through the same code. Pre-filtering prompts was necessary. Without it, a plain average over my first 100 prompts did not produce a useful lens because four prompts carried 84% of the summed Jacobian norm and the resulting lens read out a space or a hyphen as the top token for most inputs. To fix this, I applied a pre-filter that drops prompts whose norm is more than five times the median (the paper's appendix suggests a pre-filter of this kind) and added additional prompts until n=100 was reached. DeepSeek-V3 is a mixture-of-experts model, so single-prompt Jacobians depend on what else is in the batch, because expert routing does. Between a batch of one and the fit's batch of 64, 38% of token positions get a different set of experts (8.5% of individual expert slots change). Averaged readouts don't care (about 95% top-1 agreement either way).[3]Where the language shift comes from My released lens targets the final layer, the library default, and everything in this section is that lens unless I say otherwise. Halfway into the network, on English text, its readouts strongly split into two languages. 46% of them are majority Chinese in their top 20 tokens, 19% are entirely Chinese, and 36% contain no Chinese character at all. The logit lens on the same activations puts a median of 3 Chinese characters in its top 20, close to the 5 that the vocabulary's 27% Chinese share would give, and only 4% of its readouts are majority Chinese (Fig 1a). On a model trained on Chinese and English that isn't immediately shocking, and for a day or two I read it as a bilingual workspace that the logit lens was partially missing. Figure 1 panels (b) and (c) show that is not the case. It was in fact due to a dominant direction inherited by the last block. The logit lens in panel (b) shows that the residual stream carries no language shift that the unembedding can see until the last layer, where the share jumps from 0.012 to 0.42. The model makes this shift once, at the end.[4] The last layer's own Jacobian, J₅₉→₆₀, is close to the identity apart from a couple of directions. The largest of them is amplified about twelve times more than a typical direction, and it separates Chinese-character from Latin-alphabet tokens with an AUC of 0.997 (Fig 1c).[5] A lens that targets the final layer differentiates through that layer for every source layer. So every J_ℓ inherits the direction, which is what panel (c) shows: it is the top direction of every layer's Jacobian (AUC 0.99–1.00 throughout, cosine 0.83–0.89 with the last layer's), and it holds 43% of the Jacobian's squared norm at layer 0. Project that one direction out of the released lens and the language share drops from 0.2–0.5 to 0.015–0.05. A lens that targets the penultimate layer never sees the last layer. Its top direction barely overlaps the last layer's, cosine 0.02–0.22 against 0.83–0.89, and is unrelated to language for the first quarter of the network, after which it climbs to an AUC of 0.89 by 87% depth. That's the orange line in panel (c): the earlier blocks do carry a weak version of the same direction, and what the last layer supplies is the size. The orange line in panel (b) is what that means for readouts: same ten prompts, same settings, same 24 layers, language share 0.52 / 0.38 / 0.23 → 0.03 / 0.015 / 0.010 at a third, a half and three-quarters of the depth.[6] Could there be real bilingual content underneath the shift? If there were, something should survive when the shift is removed, and almost nothing does: both ways of taking it away, projecting the direction out and moving the target, leave readouts with barely any language structure left. Additionally, the shift moves the whole Chinese block together, rather than the words for whatever is being read. And through about 60% of the depth the shift is the same whether the model is reading English or Chinese, and it keeps the same sign after that, while the model's own logits flip from 1.4 standard deviations against Chinese to 1.5 in favor of it. It isn't that the states lack the information: a single direction in them tells Chinese text from English at an AUC of 0.78–1.00 at every depth, where the lens's language offset manages 0.48–0.62 through three-quarters of the network and only climbs (0.69, 0.81, 0.94 at layers 50, 55 and 58) as the source layer approaches the target. Figure 2. The final-layer J-lens amplifies Chinese tokens at early layers and English tokens at late layers. Its language offset largely not depend on the language of the input until late layers. Median over activations of each readout's mean Chinese-character logit minus its mean Latin-alphabet logit, in units of the readout's standard deviation: the released J-lens on English text (black) and on Chinese text (magenta), and the logit lens on the same states (dotted, same colors), which runs to the output, where it is the model's own logits. Why does the shift point toward Chinese early and away from it late? The shift comes almost entirely from the lens's top direction. That direction is a pair of vectors: an input direction, which the lens reads from the state, and an output direction, which it writes into the readout. The output direction barely changes with depth, sitting at a cosine of about 0.5 with the Chinese-minus-Latin axis of the unembedding at every layer. What flips is the sign of the state's component along the input direction. The average state is within a few degrees of perpendicular to the input direction, so small drifts flip that sign. The share of states whose component pushes Chinese up falls from 84% at layer 18 to 54% at layer 32 and 26% at layer 45, and the push turns from Chinese-up to Chinese-down between layers 32 and 35. The state and the input direction both drift a little, and at that angle a little is enough. The model's final norm caps the result, so no readout's offset goes much past ±1.8 standard deviations, though near the crossing, readouts spread across that whole range.Three more effects from varying the target layer Kurtosis. The original J-lens paper, Gurnee et al., tracks four workspace statistics across depth, one of which is excess kurtosis. Kurtosis measures how much of a distribution’s spread comes from a few extreme values. If a few tokens rank far above the rest, kurtosis is high. In Sonnet, kurtosis is near zero early and begins rising around one-third depth, which is one of the signals the authors interpret as the start of the workspace. On DeepSeek-V3, the final-target lens gives a negative median kurtosis through the middle of the network, though. Kurtosis is negative from 17% to 63% of depth in the 100-prompt fit. Figure 3 shows why. It shows one typical English readout a third of the way into the network, with its 129,280 token scores plotted by language. Under the final-layer lens they split into two groups, divided by language. This causes negative kurtosis. Refit the same ten prompts with the penultimate layer as target and the split is gone. Figure 3. Kurtosis goes negative because the final-layer J-lens gives bimodal readouts. One typical held-out English activation at layer 20, a third of the way through DeepSeek-V3, read through a lens fit with the final layer as target (a), through a lens fit on the same ten prompts with the penultimate layer as target (b), and through the logit lens (c). Each panel shows how the readout's 129,280 token scores are spread, in standard deviations from its mean, with the vocabulary stacked by language. With the penultimate target the median never goes negative and it rises through the second half of the network, to about the level of Sonnet's median, though with a far smaller upper tail. Number tokens leading to a misleading eval. The original J-lens paper scores the lens on how well it surfaces hidden intermediates, taking each intermediate's best rank over all layers. On the order-of-operations set, DeepSeek-V3's J-lens beat the logit lens. But this is misleading — it is actually just because the final-target lens lifts all number tokens in the first two-thirds of the network. If you rank the right number among the integers alone, so that lifting all of them together can't help, then the final-target lens picks it out no more often than the logit lens does (49% against 53% in those layers). Is this due to a single inherited direction, like the language shift? No. The last layer does amplify an integer-sorting direction: its second largest, amplified about 10x, separates integers from the rest of the vocabulary at an AUC of 0.963. But when you project that direction out of every readout, the wrong-integer rate only falls from 29% to 24%, and it takes the true number down with it, 62% to 44%. Projecting the last layer's top four directions out gets to 22%, still six times the logit lens's 4%, while fitting one layer down gets to 7%. So the number lift is from the final block, like the language shift, but no single direction carries it.[7] Few prompts dominating the average. Among the first 100 prompts, four held 84% of the norm (they were excluded after I introduced a filter). I refit all four of these prompts with the penultimate target to see if their norms remained anomalously high. Their norms dropped significantly, from 5,038, 3,947, 2,984 and 1,065 to 5.5, 6.0, 21 and 13, below the pre-filter's cutoff of 40 and within a factor of 1.5 of the largest ordinary prompt. For the largest one the ratio grows with backward depth, 31× one layer below the target and 913× at layer 0. So on these prompts the blow-up is the last layer's derivative, multiplied into every layer. This doesn't show whether the penultimate-target fits have a heavy tail of their own since I only fit 10 prompts.Comparing with Qwen Is the same language shift seen in Qwen, another model family trained with significant amounts of both English and Chinese text? (Qwen's tokenizer is 22% Chinese and DeepSeek's is 27% Chinese.) Two Qwen3.5 models have public lenses that also target the final layer, so I ran the same analysis on them: Last layer's language shift on English text Language share in the lens's readouts Top direction separates languages (AUC) DeepSeek-V3 −1.43 sd 0.2–0.5 0.99–1.00 Qwen3.5-397B −0.45 sd ≤ 0.07 0.77–0.83 Qwen3.5-122B −0.27 sd ≤ 0.02 0.50–0.65 Same code, and the logit lens is near zero in all three. The 397B shows a faint version of DeepSeek-V3's shift and the 122B almost none. Language isn't the only direction a last layer could amplify, so I also compared each layer's top direction with the last layer's. Neither Qwen lens is dominated by its last layer the way DeepSeek-V3's is. Their top directions line up only partly with it (cosine 0.35–0.6 at most depths, against 0.84–0.89). This holds even though the 397B's last layer amplifies its top direction eight times more than a typical direction, two-thirds as much as DeepSeek-V3's does. So a last layer that strongly amplifies a direction doesn't by itself mean the lens inherits that direction. To find out, compare each layer's top direction with the last layer's, as in Figure 1(c). This is about how the bulk of the logit vector is arranged, not about whether the models stay in one language. All three put under 0.05% of their probability on Chinese tokens when reading English. On English text, DeepSeek-V3's last layer also suppresses about thirty-five thousand Chinese tokens it was never going to pick. That barely changes what the model says, since those tokens already had almost no probability. But a Jacobian doesn't go through the softmax; every logit counts the same whatever its probability, so shifting thirty-five thousand of them together is one of the largest things the last layer does, and the lens inherits it. I also ran two logit-lens controls on template positions from my filler-token paper, where DeepSeek-V3's own output shift reads −0.81 sd rather than the −1.43 sd on prose above. DeepSeek-V3 in FP8 gives the same shift as the 4-bit model, slightly larger (−0.96 sd). Kimi K2 and K2.5, which use a similar architecture to DeepSeek-V3's, have no Chinese-suppressing shift at the output (−0.04 and +0.22 sd). So the language shift may come from DeepSeek's training rather than its architecture.How to check a lens A lens inherits every block between the layer you read and its target. If the lens includes the final block, this may include output-specific artifacts like the dominant language direction seen with DeepSeek-V3. The first three checks below look for a direction like that. The last two apply to fitting and scoring any lens.If your lens targets the final layer, refit a few prompts at the penultimate layer. This is the most computationally expensive check, but ten prompts with each target is enough to compare the readouts. For DeepSeek-V3, this check caught artifacts that were not removed by projections, such as the number lift.Look at each J_ℓ's top output direction. What does it read out as, how much of the norm does it hold, and what changes when you project it out? A block's direction only shows up at layers below that block, so a direction that stays on top all the way to your target, as in Figure 1(c), comes from the last block. Projecting it out fixes an effect carried by one direction, like the language shift here, but not one spread across many, like the number lift.Compare readouts with a reference. Read the same activations with your lens and with a reference (the logit lens and/or your lens refit one layer down), standardize each readout, subtract, and average the difference over many activations. Sort the tokens by that difference and read both ends. If the lens moves a group of tokens together, it will show up there. On DeepSeek-V3, one end of the list is Chinese characters and Chinese punctuation. A group as big as a language also splits a single readout's histogram, as in Figure 3.Put a wrong-answer control on any best-rank-over-layers metric. Score answers of the same kind that aren't the item's, and subtract. If the right answer belongs to a class the lens might lift as a whole (numbers, say), also rank it among its own class: that separates a lift of the whole class from what the model actually prefers.Keep track of per-prompt norms, and compare lenses fit on two halves of your prompts. The norms show whether a few prompts dominate the average; in DeepSeek-V3, four prompts were 84% of the lens until I filtered them out. The halves help show which layers are fit well enough to read; where they disagree, fit more prompts or don't trust single readouts there. In DeepSeek-V3 they disagreed most in the early layers, where the Jacobian passes through the most blocks. R-lens, described here[link], may be a better option for early layers.The J-lens paper's claims with DeepSeek-V3 Unless specified otherwise, everything below uses the final-target lens with a hundred prompts since I did not run a full fit with the penultimate target. The language shift is still present, so wherever it could change a result, I ran a control or say so. The depth curves generally match, and so does the logit lens. Next-token accuracy and autocorrelation have Sonnet's shape in DeepSeek-V3, and the ten-prompt penultimate lens gives nearly the same curves (accuracy within three points, autocorrelation 7.4 against 7.5 at the peak). Dimensionality has Sonnet's shape too, though its knee is partly the language axis — the ten-prompt penultimate lens needs 24% of the dimensions at a quarter of the depth, where the same ten prompts with the final target need 2% and the released lens 5%. So does the kurtosis median, once the language block is controlled for, though it is much smaller and has no rising tail.[8] But every one of these that has a logit-lens version looks the same under the logit lens. It's ahead on next-token accuracy at every layer but one, it has about 80% of the autocorrelation with the same shape, and its kurtosis rises at the same depth. Readout entropy falls together for both lenses too, at layers 30–35, under either target. Figure 4. The paper's Figure 28 for DeepSeek-V3 with the released lens and with the logit lens. Next-token accuracy, autocorrelation and dimensionality have Sonnet's shapes, though the dimensionality knee is partly the language axis. Kurtosis differs in three ways. The negative median through the middle of the network is the language direction (explained above). The 99th-percentile spikes in the first third are a final-target effect: on the same ten prompts they are 22–39 with the final target and under 4 with the penultimate one. And no lens on DeepSeek-V3 reproduces Sonnet's rising upper tail. Evaluating the final-layer J-lens. I scored five of the paper's six eval sets on DeepSeek-V3.[9] With the wrong-answer control from check 4 applied (twenty intermediates from other items, scored the same way), released lens minus logit lens at pass@10: Set True intermediates Wrong answers What the lens reads Association +0.12 ± 0.04 +0.00 +0.12 Typo +0.13 ± 0.04 +0.00 +0.12 Order of operations +0.06 ± 0.02 +0.05 ± 0.01 +0.01 Multihop −0.04 ± 0.03 −0.01 −0.03 Multilingual −0.04 ± 0.02 +0.00 −0.04 Only the association and typo evals show genuine uplift compared to the logit lens. The order of operations eval is not true uplift; it is from the number lift artifact as seen by the 'wrong answers' control. The wrong answers are other questions' numbers, so the control can't cleanly separate a lift of all numbers from the model's real preference for numbers near the answer; ranking the right number among the integers alone does, and shows no gain over the logit lens (see the section on number tokens above). Multihop is a small loss at pass@10 and a larger one at pass@5. Multilingual is a small but consistent loss. Neither comes from the target. With the same ten prompts, the final-target and penultimate-target lenses trail the logit lens by the same amount on both sets. The penultimate target on the paper's evals, at ten prompts. The original J-lens paper found the penultimate target "a small improvement in extracting intermediates" on Claude. I can only test this with ten prompts on DeepSeek-V3 and do not find any improvement. The penultimate lens loses about a tenth at pass@10 on association and typo, the two sets where the J-lens beats the logit lens, and matches the final target on the other three. That could be because the last layer carrying something real or it could be that the penultimate lens is underfit. Ten prompts are enough for the final target to reproduce the released lens's eval results, but that target is mostly one large direction that a few prompts pin down, and the penultimate lens has no such anchor. To truly test the penultimate-layer lens, more prompts would need to be fit. The same evals on Qwen3.5-122B. Qwen's logit-lens readouts are nearly uniform until the last tenth of the network, so if a J-lens can read what the unembedding can't, Qwen is theoretically where it should show. So I extracted the eval states there too and scored its public lens the same way (layers 0–46, which is what the lens holds; Qwen's tokenizer splits multi-digit numbers into digits, so 22 of the order-of-operations intermediates and 9 of the multihop ones have no single-token form and are skipped): Set True intermediates Wrong answers What the lens reads Association +0.02 ± 0.02 +0.00 +0.02 Typo −0.08 ± 0.05 +0.00 −0.08 Order of operations +0.16 ± 0.05 +0.38 −0.21 Multihop +0.08 ± 0.05 +0.06 +0.01 Multilingual +0.04 ± 0.02 +0.00 +0.04 The Qwen lens does not beat the logit lens on these evals. Its only true uplift is on the multilingual eval, by four points. On order of operations eval, and the number lift is stronger than in DeepSeek-V3's lens. A wrong digit reaches the top ten at some layer 42% of the time, against 4% for the logit lens. Ranked among the digits alone (the only numbers that are single tokens on Qwen), it picks the right one about as often as the logit lens (70% against 64%, ±6 points, 33 items). So both public final-target lenses I've scored inflate this eval the same way. The metric takes each intermediate's best rank over all layers, so it doesn't say where a lens reads. On four of the five sets, the Qwen logit lens gets most of its credit, 60–95%, from the last five layers. On typo it gets it from the middle of the network, where its readouts are nearly uniform on ordinary text, and typo is the set where it beats the J-lens. Translation pairs are mostly the unembedding. Mid-network, DeepSeek-V3's Chinese and English readout tokens are very often translations of each other: 80–87% of activations have a dictionary pair in their top ten of each language. Neel Nanda's review worried that this could come from the vocabulary matrices rather than the representation, and it does. The unembedding alone picks the right Chinese translation of an English token 52% of the time, and if you replace each readout's Chinese tokens with the unembedding's nearest neighbours of its English ones, you get a higher pair rate (87–94%) than the one observed. Almost any residual that points at " felt" also points at 覺得, which is also why the Chinese tokens in a mid-network readout being translations of the English ones says nothing either way about bilingual content.[10]Limitations The final-layer artifacts are all shown for only one model, DeepSeek-V3. The Qwen lenses inherit their last layers only partly and show little to no language shift, and I have not checked what the last blocks of other models amplify. The recommendation in the title rests on DeepSeek-V3 plus a general concern that the final block could be output-specialized in a way that can cause artifacts. What I showed is that DeepSeek-V3's final block has a dominant direction, a language shift, that barely moves the output probabilities but dominates the final-layer J-lens. The fit prompts and the held-out text are English Wikipedia. The lens carries the last block's English-text behaviour onto Chinese text, and I didn't try a fit on mixed-language prompts. The number lift shows the artifacts depend on the input, so nothing here says what the readouts do on code or chat text. The penultimate lens was fit with only ten prompts. That's enough for the paired test, and it's where the paper's prompt sweep says gains become modest, but it's far fewer than the hundred in my released final-target lens or the thousand in Anthropic's Sonnet lens. The ten-prompt lenses aren't reliable in the first third of the network. That's measured on the final-target one, against the hundred-prompt lens with the same target, and the penultimate one has no reason to converge faster. I did not fit a hundred prompts for the penultimate J-lens due to computational limitations, so the original paper's claims were tested with the final-target lens. The evals are coarse. They ask whether an intermediate is recoverable at any layer, which is what the Anthropic J-lens paper needed, and its margins on Sonnet were large enough for fifty-item sets to show it. Mine are a few points on about a hundred items at one k, which is within noise, and the any-layer rule means the metric can't say where a lens reads. No causal or steering tests, so the Anthropic paper's strongest evidence for the J-lens isn't addressed. The Qwen comparisons use one lens per model, fit on different prompts, and the Qwen evals are on the 122B only. The DeepSeek fit used 4-bit weights. The last block's language gate is in the FP8 model too, slightly larger, so the source isn't quantization, but no Jacobian was computed at full precision, so the lens numbers are 4-bit numbers, and the released lens belongs on states from the same checkpoint. DeepSeek-V3 is also a mixture-of-experts model, and the stored Jacobians match finite differences to 0.5% at layer 59 and 2% at layer 50, but at layer 30 they differ by 13–49%, partly expert routing, which averaged readouts hide.Release The released 100-prompt lens and its two halves, the paired ten-prompt lenses (labelled by target and prompt count), the per-prompt norms, and all code: https://huggingface.co/kbrauer/deepseek-v3-jacobian-lens^ I say "language" throughout for readability. What I measure exactly is the writing system a token is in, Chinese-character tokens against Latin-alphabet ones. Japanese kanji count as Chinese characters here, and Latin covers English along with every other Latin-script language. The "language split" in the figures is three-way: Chinese characters, Latin letters, and everything else. Everything else includes digits, punctuation, whitespace and the other writing systems, about 16% of the vocabulary. The share of readout variance it explains is a one-way analysis of variance over the 129,280 token scores in a readout.^ The J-lens paper's appendix (A.7) says the Sonnet lens used throughout the paper takes the gradient at the penultimate layer, "omitting the last transformer block from the backward pass", because "including the last layer can sometimes increase the number of noisy artifacts in lens-readouts. This may be because the final block is heavily specialized for calibrating next token predictions and contains less semantic content." Their ablation compared the two targets and found the methods "fairly consistent among these design choices, though mean aggregation of penultimate is a small improvement in extracting intermediates." The same paper's Figure 4 caption and pseudocode say the final layer, and the released library's `target_layer=None` resolves to the final layer, with a docstring note that the penultimate one "can give a better-conditioned" lens. Neel Nanda’s Qwen3.6 replication targeted the penultimate layer; both public Qwen3.5 lenses target the final layer, as do 38 of the 39 J-lenses on Neuronpedia as of September 2026 (the one penultimate layer lens is a mirror from workspace-lenses).^ I checked with finite differences of the 4-bit forward pass on one prompt. At layer 59 the stored Jacobian matches to 0.5%, at layer 50 to about 2%; at layer 30 the mismatch is partly routing: matching the batch brings it from 34–49% down to 13–29%, and freezing the routing on a batch of one still leaves 26–40%.^ The logit-lens language share is 0.002–0.003 through layer 55, 0.007 at layer 58, 0.012 at layer 59, and 0.415 at the output.^ The next direction is amplified almost as much (10.0 against 12.1) and separates the languages almost as well (AUC 0.993), so what the last layer amplifies is a two-dimensional language subspace rather than a single direction. The lens's own top direction sits mostly along the first of the two: cosine 0.83–0.89 with it, 0.32–0.54 with the second.^ I fixed the criterion before running the fits. The language share had to fall to 0.03 or below at 33, 50 and 75% depth. At 33% the ten-prompt result is 0.031, just above that line, so by my own rule the effect there is reduced rather than removed.^ Projecting the language direction out instead nudges the wrong-integer rate the other way, 29% to 33%, which is what a push that moves non-Chinese tokens down would do. The contrast with the language shift is the point. One projection removes most of the language share, 0.2–0.5 down to 0.015–0.05, and does almost nothing to the number lift.^ Nothing I checked fully reproduces Sonnet's rising tail. Sonnet's 99th-percentile kurtosis climbs to 18 through the band; DeepSeek-V3's stays at 1–4 under either target and under the logit lens. The Qwen lenses come closest: inside the band their 99th percentile runs 2.5–6 (122B) and 2.4–9.5 (397B), peaking at 88–92% depth, with medians that are positive but flat from layer 0. I have not explored why.^ The sixth, poetry, is read at the end of the couplet's first line, a position I didn't cache.^ A matched-state test says a bit more. For 581 concepts that are single tokens in English, Chinese and Japanese, the shared component of an English–Chinese pair of states is 34–41% of the variance through the middle of DeepSeek-V3 (45–54% in Qwen), and the nearest Chinese state to an English one is its translation 80–84% of the time, but it's already 82% at layer 0, and the embedding rows alone give 77%. The network inherits the alignment from its vocabulary matrices. Discuss