Claude Code runs A/B tests on your workflow, and one variable turns them off

One day Claude Code’s output changed shape on me. I hadn’t asked for an update or changed how I worked, but the answers came back structured differently from the day before. So I asked Claude why. Between us we traced it to a family of settings with names like tengu_plank_river_frost, cached in a JSON file in my home directory and set by Anthropic’s servers, not by me.

Tengu is Claude Code’s internal codename. This morning my ~/.claude.json held 680 of those remote feature values and 11 live experiment assignments, each tagged with the variation I had been put in. None of them appears in a release note.

This post is for the people who run AI coding tools every day and the people who sign off on them. It shows how to see your own experiments, the one environment variable that switches them off, what switching off costs, and why I think vendors owe us a label on the box.

Claude Code uses GrowthBook, a feature-flag and experimentation service. At start-up it asks Anthropic’s servers which features are on for you and which experiment arms you are in, then caches the answer in ~/.claude.json. Run this to see yours:

jq -r '.cachedExperimentData // {} | to_entries[]
  | "\(.key)\tvariation \(.value.variationId)"' ~/.claude.json
jq '.cachedGrowthBookFeatures | length' ~/.claude.json
jq -r '.cachedGrowthBookFeaturesAt / 1000 | todate' ~/.claude.json

The first command lists the experiments you are enrolled in. On my machine it printed 11 lines, two of them on variation 1:

tengu_read_dedup_killswitch   variation 1
tengu_plank_river_frost       variation 0
tengu_ultrareview_awareness   variation 1
...

The second counts the feature values, 680 for me. The third tells you when they were last refreshed.

The names are deliberately meaningless. tengu_amber_prism and tengu_ochre_kite tell you nothing about what they change, so a leaked key name gives nothing away. That’s fair from the vendor’s side. From the user’s side it means the list tells you that you are in an experiment and nothing about what the experiment does.

Most of what people know about individual flags comes from reverse engineering, helped along when Claude Code’s full source shipped by accident in an npm source map on 31 March 2026. Anthropic called that “a release packaging issue caused by human error, not a security breach”, which is accurate, and it also put the flag names in front of anyone who wanted to read them.

Three public cases, each with an Anthropic engineer on the record:

• Shorter plans. In March 2026 a developer paying $200 a month noticed plan mode had started capping plans at 40 lines. They wrote it up as Please Do Not A/B Test My Workflow. The engineer who ran the test replied on Hacker News. They had expected shorter plans to cut rate-limit hits, early results showed little effect, and they ended the experiment.
• Effort levels. In August 2026 a Hacker News thread claimed Anthropic was A/B testing reduced effort levels. A Claude Code team member answered: “We sometimes test API serving configs in Claude Code before rolling them out”, and said a running test mapped the numerical effort value differently, but “the effort you selected is the effort you’re getting”.
• Project rules. Until version 2.1.281 on 24 September 2026, a flag called tengu_agents_md_mod decided whether Claude Code read a project’s AGENTS.md at all. A blogger found that with telemetry off, it never did. More on that one below, because it bites the people who opt out.

I take the engineers at their word. The plan-length test was a reasonable hypothesis, measured and switched off. The effort answer came with a claim of evals behind it. None of this is sinister, and the people running these experiments answered in public, which plenty of vendors wouldn’t do.

That’s also the problem. Each of these came to light because a user noticed something odd and went digging. Nobody was told in advance.

Every change-management process I have run rests on the same assumption: a tool behaves the same today as it did yesterday unless someone changed it, and that change left a record. A new Claude Code version leaves a record, in the changelog and in the version string. A flag flip leaves nothing. The binary is identical, the version is identical, and the output is different.

For a practitioner that means a result you can’t reproduce. Two engineers on the same version can get different plans for the same prompt because they sit in different arms, and neither can see which arm the other is in. You debug the prompt, the repository and the model before it occurs to you to check the tool.

For a leader it’s an audit question with no good answer. If an AI agent writes code that ships, the tool that wrote it is part of your software supply chain. “Which configuration produced this change?” should be answerable. Today the honest answer is “the version we can tell you, the experiment arms we can’t, and the cache that would have told us is overwritten on the next refresh.”

I had already set DISABLE_TELEMETRY, DISABLE_ERROR_REPORTING and DO_NOT_TRACK in my ~/.claude/settings.json, on the belief that it stopped the experiments. So I checked, in the 2.1.283 binary on my laptop.

It does, but only two of the three. Claude Code works out a privacy level from the environment:

function Emt(){if(process.env.CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC)return"essential-traffic";if(process.env.DISABLE_TELEMETRY)return"no-telemetry";if(Le(process.env.DO_NOT_TRACK))return"no-telemetry";return"default"}

Anything other than default disables GrowthBook. Every flag lookup then returns the default compiled into the client:

if(!this.deps.isEnabled()&&!this.deps.isDiskCacheReadableWhileDisabled())return{value:n,source:"disabled"};

The old cache on disk is ignored too, so a stale experiment arm doesn’t linger. Anthropic’s data usage page confirms it: setting DISABLE_TELEMETRY or CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC “also disables the feature-flag evaluation” that Remote Control depends on. DISABLE_ERROR_REPORTING doesn’t; it only stops crash reports.Sidenote: The check is if(process.env.DISABLE_TELEMETRY), so any non-empty value switches telemetry off, including "0" and "false". DO_NOT_TRACK goes through a parser and needs a true-ish value.

My own cache agrees. cachedGrowthBookFeaturesAt reads 20 September 2026 at 12:02 SAST, and a week of daily sessions since has not moved it.

So the setting is one line:

{
  "env": {
    "DISABLE_TELEMETRY": "1"
  }
}

Opting out doesn’t give you a neutral Claude Code. It gives you whatever Anthropic compiled in as the default, and there are three costs.

Defaults still change with every release. Each new version can ship new defaults, so behaviour still moves, just on a schedule you can see. If reproducibility matters, pin the version as well: DISABLE_AUTOUPDATER stops the background update check, and "autoUpdatesChannel": "stable" in settings trails the latest release. Both are in the setup docs.

Some features stop working. Remote Control depends on flag evaluation and goes with it. Channels are gated on tengu_harbor, which defaults to off when flags can’t be fetched.

Opting out can hide changes too. The AGENTS.md case is the sharpest. The flag existed so Anthropic could switch the feature off remotely if it broke something. With telemetry off the flag fell back to its default, off, and Claude Code silently ignored project rules for everyone who had opted out. An Anthropic engineer called it “a rollout artifact, we needed a way to turn this off remotely via feature flags if it broke something, and with telemetry off you don’t get those.” It was fixed in 2.1.281. The people who opted out of silent changes got one anyway, and it hit the file that tells an agent what it may and may not do.

I still keep telemetry off. A default I can pin beats an arm I can’t see. But it’s a trade, and “I set the variable, so the tool is stable” is not true.

The best argument against me is that A/B testing is how good products get built. Anthropic can’t evaluate a plan-length change on synthetic prompts alone; it needs real users on real work, and asking every user to opt in shrinks the sample and skews it towards enthusiasts. Anthropic has also been more open than most. Its September 2025 postmortem says “we never reduce model quality due to demand, time of day, or server load”, and its engineers answer in public threads.

I accept all of that. My ask is smaller than “stop testing”:

• Show the cohort. A line in /status listing the experiments this session is enrolled in, by name and arm. The data is already in ~/.claude.json; it just isn’t shown.
• Describe the experiment. A short public note for any experiment that changes output a user will act on: plans, code, permission prompts, which instruction files get read. UI copy and nudges can stay quiet.
• Opt in for anything that touches the work product. A flag that changes how long a plan is changes what an engineer ships. That deserves a yes from the person paying.
• Log it where the user can keep it. Write the arm into the session transcript, so the record survives the next cache refresh.

None of this is expensive. Most of it is a display of data Claude Code already holds.

Run the three jq commands above on your own machine and on one teammate’s. If the arms differ, you have your answer to “why does Claude do that for them and not for me?”

Then decide as a team, not per laptop. If reproducibility matters more than the newest behaviour, set DISABLE_TELEMETRY and DISABLE_AUTOUPDATER in the managed settings file so nobody’s local settings override them. On macOS it lives at /Library/Application Support/ClaudeCode/managed-settings.json, on Linux at /etc/claude-code/managed-settings.json. Record the Claude Code version in your change log next to the model. If you keep experiments on, write that down as an accepted risk, and check the jq output first when a result can’t be reproduced.

I only went looking because the output changed shape. Most people won’t notice, and nothing tells them.