Chad Norman: writing

Claude Code runs A/B tests on your workflow, and one variable turns them off

Sat, 26 Sep 2026 00:00:00 GMT

One day Claude Code’s output changed shape on me. I hadn’t asked for an update or changed how I worked, but the answers came back structured differently from the day before. So I asked Claude why. Between us we traced it to a family of settings with names like tengu_plank_river_frost, cached in a JSON file in my home directory and set by Anthropic’s servers, not by me.

Tengu is Claude Code’s internal codename. This morning my ~/.claude.json held 680 of those remote feature values and 11 live experiment assignments, each tagged with the variation I had been put in. None of them appears in a release note.

This post is for the people who run AI coding tools every day and the people who sign off on them. It shows how to see your own experiments, the one environment variable that switches them off, what switching off costs, and why I think vendors owe us a label on the box.

Claude Code fetches its behaviour from a flag service

Claude Code uses GrowthBook, a feature-flag and experimentation service. At start-up it asks Anthropic’s servers which features are on for you and which experiment arms you are in, then caches the answer in ~/.claude.json. Run this to see yours:

jq -r '.cachedExperimentData // {} | to_entries[]
  | "\(.key)\tvariation \(.value.variationId)"' ~/.claude.json
jq '.cachedGrowthBookFeatures | length' ~/.claude.json
jq -r '.cachedGrowthBookFeaturesAt / 1000 | todate' ~/.claude.json

The first command lists the experiments you are enrolled in. On my machine it printed 11 lines, two of them on variation 1:

tengu_read_dedup_killswitch   variation 1
tengu_plank_river_frost       variation 0
tengu_ultrareview_awareness   variation 1
...

The second counts the feature values, 680 for me. The third tells you when they were last refreshed.

The names are deliberately meaningless. tengu_amber_prism and tengu_ochre_kite tell you nothing about what they change, so a leaked key name gives nothing away. That’s fair from the vendor’s side. From the user’s side it means the list tells you that you are in an experiment and nothing about what the experiment does.

Most of what people know about individual flags comes from reverse engineering, helped along when Claude Code’s full source shipped by accident in an npm source map on 31 March 2026. Anthropic called that “a release packaging issue caused by human error, not a security breach”, which is accurate, and it also put the flag names in front of anyone who wanted to read them.

Experiments have already changed people’s work

Three public cases, each with an Anthropic engineer on the record:

Shorter plans. In March 2026 a developer paying $200 a month noticed plan mode had started capping plans at 40 lines. They wrote it up as Please Do Not A/B Test My Workflow. The engineer who ran the test replied on Hacker News. They had expected shorter plans to cut rate-limit hits, early results showed little effect, and they ended the experiment.
Effort levels. In August 2026 a Hacker News thread claimed Anthropic was A/B testing reduced effort levels. A Claude Code team member answered: “We sometimes test API serving configs in Claude Code before rolling them out”, and said a running test mapped the numerical effort value differently, but “the effort you selected is the effort you’re getting”.
Project rules. Until version 2.1.281 on 24 September 2026, a flag called tengu_agents_md_mod decided whether Claude Code read a project’s AGENTS.md at all. A blogger found that with telemetry off, it never did. More on that one below, because it bites the people who opt out.

I take the engineers at their word. The plan-length test was a reasonable hypothesis, measured and switched off. The effort answer came with a claim of evals behind it. None of this is sinister, and the people running these experiments answered in public, which plenty of vendors wouldn’t do.

That’s also the problem. Each of these came to light because a user noticed something odd and went digging. Nobody was told in advance.

An unannounced experiment is an unmanaged change

Every change-management process I have run rests on the same assumption: a tool behaves the same today as it did yesterday unless someone changed it, and that change left a record. A new Claude Code version leaves a record, in the changelog and in the version string. A flag flip leaves nothing. The binary is identical, the version is identical, and the output is different.

For a practitioner that means a result you can’t reproduce. Two engineers on the same version can get different plans for the same prompt because they sit in different arms, and neither can see which arm the other is in. You debug the prompt, the repository and the model before it occurs to you to check the tool.

For a leader it’s an audit question with no good answer. If an AI agent writes code that ships, the tool that wrote it is part of your software supply chain. “Which configuration produced this change?” should be answerable. Today the honest answer is “the version we can tell you, the experiment arms we can’t, and the cache that would have told us is overwritten on the next refresh.”

One environment variable switches the experiments off

I had already set DISABLE_TELEMETRY, DISABLE_ERROR_REPORTING and DO_NOT_TRACK in my ~/.claude/settings.json, on the belief that it stopped the experiments. So I checked, in the 2.1.283 binary on my laptop.

It does, but only two of the three. Claude Code works out a privacy level from the environment:

function Emt(){if(process.env.CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC)return"essential-traffic";if(process.env.DISABLE_TELEMETRY)return"no-telemetry";if(Le(process.env.DO_NOT_TRACK))return"no-telemetry";return"default"}

Anything other than default disables GrowthBook. Every flag lookup then returns the default compiled into the client:

if(!this.deps.isEnabled()&&!this.deps.isDiskCacheReadableWhileDisabled())return{value:n,source:"disabled"};

The old cache on disk is ignored too, so a stale experiment arm doesn’t linger. Anthropic’s data usage page confirms it: setting DISABLE_TELEMETRY or CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC “also disables the feature-flag evaluation” that Remote Control depends on. DISABLE_ERROR_REPORTING doesn’t; it only stops crash reports.Sidenote: The check is if(process.env.DISABLE_TELEMETRY), so any non-empty value switches telemetry off, including "0" and "false". DO_NOT_TRACK goes through a parser and needs a true-ish value.

My own cache agrees. cachedGrowthBookFeaturesAt reads 20 September 2026 at 12:02 SAST, and a week of daily sessions since has not moved it.

So the setting is one line:

{
  "env": {
    "DISABLE_TELEMETRY": "1"
  }
}

Switching off has its own costs

Opting out doesn’t give you a neutral Claude Code. It gives you whatever Anthropic compiled in as the default, and there are three costs.

Defaults still change with every release. Each new version can ship new defaults, so behaviour still moves, just on a schedule you can see. If reproducibility matters, pin the version as well: DISABLE_AUTOUPDATER stops the background update check, and "autoUpdatesChannel": "stable" in settings trails the latest release. Both are in the setup docs.

Some features stop working. Remote Control depends on flag evaluation and goes with it. Channels are gated on tengu_harbor, which defaults to off when flags can’t be fetched.

Opting out can hide changes too. The AGENTS.md case is the sharpest. The flag existed so Anthropic could switch the feature off remotely if it broke something. With telemetry off the flag fell back to its default, off, and Claude Code silently ignored project rules for everyone who had opted out. An Anthropic engineer called it “a rollout artifact, we needed a way to turn this off remotely via feature flags if it broke something, and with telemetry off you don’t get those.” It was fixed in 2.1.281. The people who opted out of silent changes got one anyway, and it hit the file that tells an agent what it may and may not do.

I still keep telemetry off. A default I can pin beats an arm I can’t see. But it’s a trade, and “I set the variable, so the tool is stable” is not true.

Vendors should tell you which cohort you are in

The best argument against me is that A/B testing is how good products get built. Anthropic can’t evaluate a plan-length change on synthetic prompts alone; it needs real users on real work, and asking every user to opt in shrinks the sample and skews it towards enthusiasts. Anthropic has also been more open than most. Its September 2025 postmortem says “we never reduce model quality due to demand, time of day, or server load”, and its engineers answer in public threads.

I accept all of that. My ask is smaller than “stop testing”:

Show the cohort. A line in /status listing the experiments this session is enrolled in, by name and arm. The data is already in ~/.claude.json; it just isn’t shown.
Describe the experiment. A short public note for any experiment that changes output a user will act on: plans, code, permission prompts, which instruction files get read. UI copy and nudges can stay quiet.
Opt in for anything that touches the work product. A flag that changes how long a plan is changes what an engineer ships. That deserves a yes from the person paying.
Log it where the user can keep it. Write the arm into the session transcript, so the record survives the next cache refresh.

None of this is expensive. Most of it is a display of data Claude Code already holds.

What to do on Monday

Run the three jq commands above on your own machine and on one teammate’s. If the arms differ, you have your answer to “why does Claude do that for them and not for me?”

Then decide as a team, not per laptop. If reproducibility matters more than the newest behaviour, set DISABLE_TELEMETRY and DISABLE_AUTOUPDATER in the managed settings file so nobody’s local settings override them. On macOS it lives at /Library/Application Support/ClaudeCode/managed-settings.json, on Linux at /etc/claude-code/managed-settings.json. Record the Claude Code version in your change log next to the model. If you keep experiments on, write that down as an accepted risk, and check the jq output first when a result can’t be reproduced.

I only went looking because the output changed shape. Most people won’t notice, and nothing tells them.

Just-in-time access shrinks an agent's window, not its reach

Fri, 25 Sep 2026 00:00:00 GMT

Picture a support agent that can refund customers up to R500. An attacker opens a ticket with instructions buried in it, the agent reads them, and it refunds R50,000.

The vendor pitch for this failure is just-in-time access: no standing privileges, a fresh token for every call, expiring in seconds. Now run the same ticket again. The agent asks for a token scoped to “issue refunds”, gets one that lives for 30 seconds, and issues the R50,000 refund in the first two.

Just-in-time credentials shrink how long an agent holds access, not what it can do with it. For an agent, the control that matters is a hard limit enforced outside the model: the refund API that rejects anything over R500. JIT versus standing access comes second. It’s worth doing, but after the limit, not instead of it.

This is for the people wiring agents into business systems and the people signing off on them, at firms where the security team might be one person. It picks up where the least-autonomy post left off: once you’ve decided a step needs an agent, what actually fences it in.

A 30-second token still issues the refund

Security people have two ways to limit the damage a credential can do. One limits it in time: the token expires, so a thief gets a short window. The other limits it in scope: the token, or the system behind it, only allows certain actions with certain values.

Prompt injection doesn’t steal the credential. It borrows the agent, which already holds the credential legitimately, at the moment it’s using it. However short the token’s life, the attacker’s instruction runs inside it. Only a scope limit changes the outcome.

Control	Limits	Stops the injected R50,000 refund?
Token expires in seconds	Time	No. The refund happens inside the window
No standing privileges, token issued per task	Time	No. The task is “issue a refund”
Token scoped to the refund tool only	Which tool	No. It’s the right tool
Refund API rejects amounts over R500	Value	Yes
Refunds only to the card that paid	Recipient	Limits it to the original customer
At most five refunds an hour per agent	Rate	Caps the total loss

OWASP’s LLM06 Excessive Agency entry says it directly: implement authorisation in downstream systems rather than relying on an LLM to decide whether an action is allowed. The R500 cap in the prompt is a request. The R500 cap in the refund API is a control.

A policy engine sees the request, never the intent

The next layer of the pitch is “intent-based” authorisation: a policy engine that checks whether the agent’s action matches what it meant to do. It sounds like the answer to injection. It isn’t, because there is no intent to check.

A policy engine sees a request: the caller, the endpoint, the parameters. When a product says it evaluates intent, the intent is text the model wrote, usually a field like reason: "customer requested refund for damaged goods". The model wrote that after reading the attacker’s ticket. It’s attacker-controllable text, the same as everything else the model produces, and an injected ticket can tell the model what reason to give.

What a policy engine can check reliably is the request itself:

Parameters. The amount is under R500. The date range is under 90 days. The record count is under 50.
Recipients. The email goes to an address already on the client’s file. The payment goes to an account on the allowlist.
Rates. No more than a set number of these actions per hour, per agent and per customer.
Pairing. The refund matches an order that exists, for no more than its value.

All four are deterministic. The same request gives the same answer every time, and an attacker can’t talk them round.

A second model reads the same poisoned ticket

Another common design splits the agent into roles: a planner, an inspector that reviews the plan, and an executor that holds the credentials. It helps only when the split removes something the attacker needs.

If the inspector is another language model, it reads the same ticket the planner read, instructions and all. Models follow instructions in content, and they can’t reliably tell yours from an attacker’s. An injection that fooled the planner can carry a second paragraph addressed to the inspector. Two models in series aren’t two controls. They’re one control that runs twice. The inspector has to be code.

The split that works is the one that takes a leg off the lethal trifecta. Simon Willison described it in 2023 as the dual LLM pattern: a quarantined model reads the untrusted content and has no tools, and a privileged model holds the tools and never sees that content directly. The component that reads the customer’s ticket should hold no credentials at all. It passes structured fields (an order number, a category, an amount) to code, and code decides what happens next.

An agent never holds more than the person it works for

Most business agents act for someone: the customer on the chat, the adviser who asked for a summary, the claims clerk working a queue. The service-account habit is to give the agent one broad token that can reach every customer’s records, and then trust the model to only look at the right one.

Turn that round. The agent gets a token derived from the user it’s acting for, and never more than that user could do. Usually it gets less, because the agent needs read access to one client’s file, not the adviser’s full book. OWASP’s LLM06 guidance says the same: actions taken for a user should run on downstream systems in that user’s context.

OAuth has a standard for this. RFC 8693, OAuth 2.0 Token Exchange (January 2020) defines how a service swaps one token for another, including delegation. With delegation the agent keeps its own identity, separate from the user’s, and the token’s act claim records that the agent is acting for the user. An injected instruction to read another client’s file then fails at the API, because the token doesn’t reach it. The log shows both names: which agent did it, and for whom.

Approval gates work only when they’re rare

Every design in this area ends with a human approval step for risky actions, and OWASP recommends one for high-impact actions. I agree, with one condition: it has to fire rarely.

An approval prompt that appears forty times a day trains the approver to click yes. The nearest analogy I know is MFA push fatigue, where enough prompts turn a security check into noise and a tired user eventually accepts one. An agent’s approval queue drifts the same way, and nobody has to attack it for that to happen.

Keep approvals for what can’t be undone or is unusual: a payment over the cap, a first-time recipient, deleting records, anything a client will see before anyone else does. Everything routine sits under a hard limit in code and needs no click. If a gate fires often enough that people stop reading it, move the limit, not the person.

Short-lived credentials still earn second place

Here’s the best argument for JIT, and it’s a good one. Agents leak credentials. A token ends up in an environment variable that gets printed to a log, a debug trace, a support bundle, or a compromised MCP server the agent talks to. A token that expired four minutes after it was issued is worth very little to whoever finds it. A standing key that works until someone rotates it is worth a great deal.

Better still is having no secret to leak. Google Cloud’s Workload Identity Federation exists to replace service account keys, which Google itself calls powerful credentials that are a risk when mismanaged, with short-lived access tokens.

Per-task tokens also fix a real audit problem. When every action runs under one shared service token, the log says the service did it. When each task gets its own token, tied to a user and a ticket, the log answers who asked for what.

So do it. JIT addresses a real threat: the stolen credential. Prompt injection is a different threat, and the agent doesn’t need to lose its credential for the attacker to use it. Order the work by which threat you’re more likely to meet this year.

A five-person FSP needs vendor limits, not a policy engine

Everything above assumes you run the agent’s code. A five-person financial services provider running one agent inside a vendor’s SaaS has no policy engine, no token broker and no developer to build either. That doesn’t leave it with nothing. The proportionate version uses what the vendor already enforces:

The narrowest OAuth scopes the integration offers: read-only where the agent only reads, one mailbox instead of all of them.
Per-tool limits on the vendor’s side: spend caps, recipient allowlists, a daily action ceiling.
A separate account for the agent, never a person’s login shared with it.
Logs that record which user the agent acted for, and someone who reads them.

A vendor that can’t offer any of these is telling you something about how its product will fail.

An agent’s credential is a service account, whatever the vendor calls it. POPIA section 19 already asks you to take appropriate, reasonable technical and organisational measures against unlawful access to personal information.Sidenote: My reading of POPIA as a practitioner, not legal advice. Your compliance officer decides how it applies to your firm.

Write the limit down before you shorten the token

Expect a blank next to at least one tool on the first pass. Start with the tool that can move money or send data out of the firm, and put a number on it.

The R50,000 refund in the opening needed a token for two seconds. A R500 cap in the refund API would have stopped it at the first request.

Cloudflare's free plan is where your first project should live

Fri, 25 Sep 2026 00:00:00 GMT

The site you’re reading costs R0 a month to host. It’s a static build served from Cloudflare’s edge, with a strict Content Security Policy (CSP), HSTS, a security.txt, cookieless analytics and email on its own domain. None of that needed a server, and none of it needed a credit card beyond the domain.

If you’re a developer shipping your first real project, or the security person a developer asks for advice, this is my answer: put it on Cloudflare’s free plan. You get a custom domain, TLS and DDoS protection that you would otherwise build and patch yourself on a VPS. Below is how this blog is set up, what went wrong, what you still own, and where the argument stops holding.

A static site on Workers costs nothing to serve

As of September 2026, Cloudflare’s docs say it plainly: “Requests to static assets are free and unlimited.” A Worker script on the free plan gets 100,000 requests a day. An assets-only Worker has no script, so that limit never comes into it.

That is why this site costs nothing to serve. Astro builds every page to a file in dist/, and Cloudflare serves those files. There is no Worker code, no server-side rendering and nothing that runs per request. The config is short enough to read in one go:

{
  "name": "chadnorman-fyi",
  "account_id": "<your account id>",
  "compatibility_date": "2026-09-23",
  "assets": {
    "directory": "./dist/",
    "html_handling": "auto-trailing-slash",
    "not_found_handling": "404-page"
  },
  "routes": [{ "pattern": "chadnorman.fyi", "custom_domain": true }],
  "workers_dev": false,
  "preview_urls": true
}

The custom_domain route does more than it looks. When the zone is active on Cloudflare, “Cloudflare will create DNS records and issue necessary certificates on your behalf.” No certbot cron job, no renewal to forget.

The only fixed cost is the .fyi domain, renewed yearly through Cloudflare Registrar, which charges “what is charged by registries and ICANN” with no markup.

What the free plan does for you before you write a line

Here is what came with the zone on the Free plan, and what I would otherwise be running myself:

You get, on Free	On a VPS, you would
Universal SSL: certificates issued and renewed by default	Run and monitor certbot
“Standard, unmetered DDoS protection (layers 3-7)”	Hope, or pay someone
The Cloudflare Free Managed Ruleset for “high-impact and widely exploited vulnerabilities”	Install and tune a WAF
Email Routing, which forwards `hello@` and `security@chadnorman.fyi` to my inbox	Run a mail server, or pay for a mailbox
Web Analytics, which “does not use any client-side state, such as cookies or localStorage”	Self-host Plausible or Umami, or put up a cookie banner
A `_headers` file for security headers on every asset	Edit nginx config

And the part that matters most for a security blog: there is no server for me to patch. No SSH port, no OS, no runtime with a CVE next Tuesday. The attack surface of a static site on someone else’s edge is the build pipeline, the account and the content. That’s a much shorter list than a Linux box.

How this site gets from git to the edge

The pipeline has five steps, all in the repo:

I write a post as an MDX file and open a pull request.
GitHub Actions runs type checks, unit tests, and Playwright with axe for accessibility.
It builds the site. The last step of pnpm build, scripts/csp.ts, hashes every inline script and style in the HTML and writes one CSP into dist/_headers. Markup that would need 'unsafe-inline' fails the build.
Lighthouse runs against the build. A pull request gets a preview URL from wrangler versions upload.
A merge to main runs wrangler deploy with an API token scoped to editing Workers scripts on one account. A daily scheduled run rebuilds main, so a post dated in the future goes live on its day.

The git log tells the rest. The first commit landed at 01:03 SAST on 25 September 2026, and the launch-hardening phase (headers, CSP, security.txt) merged at 03:47 the same morning.

What went wrong

Three things cost me time. None of them are in the getting-started guides.

Automatic analytics cost two Lighthouse points. Web Analytics’ automatic setup “will inject the JS snippet on all pages” at the edge. That beacon loads from a second origin before first paint, and Lighthouse counts it. The live home page sat at 98 on performance while the same build scored 100 on a preview URL, because previews get no injection. The fix was to switch automatic setup off in the dashboard and load Cloudflare’s JS snippet myself, after the load event, only on the production host. Performance went back to 100. The cost is that a visitor who leaves before the page finishes loading isn’t counted. When a live score trails a preview’s, check the live HTML for scripts you didn’t put there.

Some steps are dashboard-only. Wrangler’s OAuth login deploys Workers fine, but its token cannot edit DNS records or zone rules. The www to apex redirect, Email Routing and the Web Analytics setup were all clicks in the dashboard. That’s fine for a one-person blog, but it means part of the site’s config lives outside git. Write those steps down somewhere you will find them in a year.

The headers file has a line limit. Each line in _headers has a 2,000 character limit. A hashed CSP is one long line, and each inline script adds a 50-odd character hash. The line holds about 30 hashes, and this site uses 22 of them. The build script fails loudly past the limit rather than ship a truncated policy, which I would rather find in CI than in a browser console.

The defaults you get, and the ones you still own

Cloudflare handles the transport and the edge. Everything above that is still yours.

Your account is the whole estate. Domain, DNS, hosting, email and analytics all hang off one Cloudflare login. Turn on two-factor authentication, with a security key if you have one. If someone takes that account, they take your domain and your email with it.
Scope your API tokens. The token in CI should do one job. Cloudflare’s tokens can be limited by permission, by account or zone, by client IP and by expiry. Use the global API key for nothing.
Secrets go in wrangler secret put, not in config. Cloudflare’s own docs say “Do not use vars to store sensitive information”. Keep local secrets in .dev.vars and keep that file out of git.
Headers and CSP are yours to write. The _headers file applies to static assets only; once you add Worker code, its responses need their own headers. Nobody sets a CSP for you.
Pin your dependencies. This site pins every npm package exactly, with no ^, and pins each GitHub Action to a commit SHA. The supply chain is the part of a static site that still runs code, at build time, with your deploy token in reach.

If you go on to build AI agents on Workers AI, read my note on the lethal trifecta before you give one a tool.

Where the free plan stops holding

I would still pick it, but go in knowing where it ends.

Dynamic code has a ceiling. A Worker script on the free plan gets 100,000 requests a day, resetting at midnight UTC. If you route asset requests through your script with run_worker_first, those count too, and past the limit they get a 429 instead of the file. Static stays free; logic doesn’t.

Size limits. A Worker version on the free plan holds 20,000 static files, each up to 25 MiB. A blog will never notice. A site with a large image or video archive will.

Email only comes in. Email Routing forwards mail; it doesn’t send it. Replying from your own domain needs another service.

The free WAF is narrow. The Free Managed Ruleset covers widely exploited vulnerabilities. The fuller Cloudflare Managed Ruleset and the OWASP Core Ruleset are on paid plans.

Lock-in is real, and it’s mostly the domain. Cloudflare Registrar requires Cloudflare’s nameservers: “all domains on Cloudflare Registrar use Cloudflare nameservers”. The site itself is portable, because dist/ is plain files that any static host can serve and the posts are MDX in git. The dashboard-only config is not.

One provider is one point of failure. Registrar, DNS, hosting, email and analytics on one account means one outage or one compromised login takes all of them. For a personal blog I accept that trade. For a business, I’d at least keep the registrar and the mail provider somewhere else.

Ship it this week

Here’s the first step, sized for a weekend:

Build your project to static files, whatever the framework.
Add a wrangler.jsonc with an assets block and a custom_domain route, and run wrangler deploy.
In the dashboard, turn on 2FA for your account, check the Free Managed Ruleset is deployed, and switch on Web Analytics.
Add a _headers file with HSTS, X-Content-Type-Options: nosniff and a CSP, then check the result with your browser’s developer tools.
Give CI a token scoped to one account and one job.

That’s a real, secure, fast site for the price of a domain. Small bill. No server.

Hello world

Fri, 25 Sep 2026 00:00:00 GMT

This is a sample post. It shows how each part of a post renders on this site, from the title and date down to the footnotes, and it will be replaced by a real post.

It says nothing about security, GRC or GenAI. Each section below uses one kind of element, so the page doubles as a visual test.

Text and links

A paragraph of body text sits at a comfortable line length. It can hold a link to the home page, some bold text, a little italic text and inline code. A keyboard shortcut looks like Ctrl + K.

A sidenote adds an aside without breaking the flow of the paragraph it belongs to. On a wide screen it sits in the right margin; on a narrow one it appears as a box in the text.Sidenote: This is a sidenote. It holds a short aside that would clutter the main text.

Lists

An unordered list:

The first item in a list
A second item, long enough to wrap onto another line on a narrow screen so the indent shows
A third item
A bold lead-in. Then a short line.

An ordered list:

Write the post as an MDX file
Commit it and push
Read it on the site

Code

A code block with a file name shows the name in its window bar. Every code block has a copy button.

const WORDS_PER_MINUTE = 230;

/** Minutes to read `text`: words / 230, rounded up, at least 1. */
export function readingTime(text: string): number {
  const words = text.split(/\s+/).filter(Boolean).length;
  return Math.max(1, Math.ceil(words / WORDS_PER_MINUTE));
}

A code block without a file name shows its language instead:

pnpm install
pnpm dev

Long lines

A long line scrolls inside its block rather than widening the page:

This line is deliberately long so that it runs past the edge of the code block on a narrow screen and has to scroll sideways.

Callouts

Callouts come in four types. Each one has an icon and a label.

Figures and tables

A figure is an image with a caption, in an outlined card.

A post starts as an MDX file, and the build turns it into an HTML page.

A table scrolls sideways on a narrow screen:

Element	Written as	Rendered as
Heading	`## Heading`	A heading with an anchor link
Callout	`<Callout type="tip">`	A coloured box with an icon and a label
Sidenote	`<Sidenote>`	A note in the margin, or a box in the text
Footnote	`[^1]`	A numbered note at the end of the post

Quotes and footnotes

A blockquote renders as a pull quote:

A pull quote repeats one line from the post in large type.

A footnote marker sits in the text¹ and links to its note at the end of the post. Each note links back to its marker.²

This is the first footnote. ↩
This is the second footnote. It is a little longer, to show how a footnote wraps. ↩

Give your AI workflows the least autonomy that works

Fri, 25 Sep 2026 00:00:00 GMT

In July 2025, SaaStr founder Jason Lemkin told Replit’s AI agent, in plain language, to freeze all changes to the app he was building. It deleted his production database anyway, then told him a rollback was impossible. It wasn’t.

Replit’s fix a few days later was not a better prompt. It separated development and production databases automatically, so the agent could no longer reach live data at all.

The freeze lived in the prompt, and a prompt is a request. The fix lived in the permissions, and a permission is a control. That’s least privilege, and it applies to models the same way it applies to people and service accounts. Every step in an AI workflow should get the least autonomy that does the job.

This post is for the people building those workflows and the people signing off on them. It sets out five levels of autonomy, what goes wrong at each, and how to pick the lowest level each step can live with.

Autonomy is a permission, not a feature

Anthropic’s own guidance on building effective agents draws the line cleanly. In a workflow, code decides the path and calls a model along the way. In an agent, the model decides its own next step and which tools to use. Their advice is to find the simplest solution that works and add complexity only when it earns its keep. That’s advice from the people selling the model.

OWASP makes the same point from the security side. Its LLM06 Excessive Agency entry names three root causes: too much functionality, too many permissions and too much autonomy. All three are decisions someone made when wiring the thing up.

So for each step in a workflow, ask two questions:

Who decides what happens next? Code, a person or the model.
What can the model touch? Which data it reads, which systems it writes to, and whether anything leaves the building.

The answers place the step on a ladder.

Each step up hands the model more of the decisions. Levels 1 to 4 belong in production; level 5 belongs in a sandbox.

Level 1: code decides, and nothing guesses

Plain code, webhooks, or the fixed branches of an n8n or Zapier flow. No model anywhere.

It’s predictable, cheap per run, fast, and an auditor can read it. It’s also brittle. When a supplier renames a field in their API, the flow breaks.

People count that as the weakness. I count it as the strength: level 1 fails loudly. You find out the day it breaks, not three weeks later when someone reconciles the books.

Payments, access provisioning, database sync and anything a regulator will ask you to explain belong here. If you can write the rule as a plain if statement, write it that way. A model adds cost and a new way to be wrong, and it takes away the one thing you had.

Level 2: the model advises, a person acts

Someone pastes a policy, a contract or a log into a chat assistant, reads what comes back, and decides what to do with it. The model has no tools and no credentials. Nothing runs unless a person runs it.

This level often gets drawn above “code plus AI” on autonomy ladders, because it feels more like AI. It’s lower. Nothing executes on its own, so the model decides nothing.

The risks are real but different in kind:

Data leaves. Whatever gets pasted in goes to the provider. Know which tier of which product your staff use, and what its terms say about retention and training.
Review turns into rubber-stamping. A person who approves forty drafts a day stops reading draft forty-one. Human review is a control only while the human is actually reviewing.

It’s the right level for policy analysis, first drafts, code review and thinking out loud. It doesn’t scale, and that’s fine: plenty of the work at a small firm happens a few times a week, not a few thousand.

Level 3: code runs the flow, the model does one narrow job

This is where most production AI should live. Code owns the flow, the routing and every write. It calls the model for one bounded task: pull the fields out of an invoice, classify a support email, summarise a call note, draft a reply. Then code checks the result before the next step runs.

When the model gets it wrong, the error stops at that step. The output fails schema validation, or a business rule catches it (the invoice total doesn’t match the line items), or low confidence sends it to a person’s queue.

Two failure modes to design for:

Valid but wrong. A schema check passes a well-formed JSON object with the wrong amount in it. Validate the values against something you trust, not just the shape.
The output is the action. When a model’s words reach a client without review, the words are what you did. Air Canada learnt this when its chatbot invented a bereavement refund policy; the tribunal held the airline responsible for everything on its website, chatbot included. No tool access was needed for that to cost money.

Prompt injection still matters here. A supplier’s PDF can carry instructions, and the model may follow them. But at level 3 the worst it can do is corrupt one output, which code then checks. It can’t pick a new tool or a new recipient, because it doesn’t choose either.

Level 4: the model decides, inside a fence

Now the model chooses its own next step: which document to search, which query to run, whether it has enough to answer. That’s what makes it an agent, and it’s worth it for work where the steps can’t be written down in advance, such as a multi-step investigation or a support case that branches differently every time.

The fence is what makes this level safe enough for production, and all of it lives outside the model:

Narrow tools. A search tool that searches, not a shell. OWASP’s first mitigation is to remove every tool the task doesn’t need.
Split permissions. One agent reads documents. Another drafts SQL. Only a third, or better, only code, can write to the database. That’s segregation of duties, the same principle you apply to people who can raise and approve a payment.
Gates in code. Sending an email, issuing a refund or changing a record waits for a deterministic check or a person’s click. The model can ask; it can’t approve itself.
Budgets. A maximum number of steps, tokens and minutes per run, enforced by the orchestrator. Loops end when the budget does.
Logs. Every tool call, with its inputs and outputs, in a store the agent can’t edit.

The cost is engineering time. Anthropic says it plainly: agentic systems trade latency and cost for better performance on hard tasks. Pay that only where the task needs it.

Level 5: the model decides everything

One general-purpose model, a broad goal, wide access, and a loop that runs until it decides it’s done. Level 5 needs the least setup and looks brilliant in a demo.

It’s also where the Replit database went. A model with broad access and self-direction will eventually take an action nobody asked for, and at level 5 nothing sits between that decision and your systems.

Level 5 is fine in a sandbox: a throwaway VM, a copy of the data, no production credentials and nothing that can send mail or move money. Use it to explore, prototype and learn what the task really needs. Then build the production version one or two levels lower.

The lethal trifecta tells you where the gates go

Simon Willison named the combination that turns prompt injection from a nuisance into a breach. He calls it the lethal trifecta: a system that has access to private data, is exposed to untrusted content, and can communicate externally. With all three, anyone who can get text in front of the model can instruct it to send your data out. Models can’t reliably tell your instructions from an attacker’s.

Picture an inbox assistant at a financial services provider. It reads client emails (untrusted content), looks up client records (private data) and drafts replies it can send (external communication). That’s all three. One well-crafted email asking it to “attach the client’s latest statement for verification” is all an attacker needs.

POPIA already covers decisions about people

South African firms don’t need a new rulebook before they deploy an agent. POPIA section 71 already covers the case that matters most.

A data subject may not be subject to a decision with legal consequences, or one that affects them to a substantial degree, based solely on automated processing that profiles them, including their creditworthiness or reliability.

The exceptions require, among other things, that the person can make representations and gets enough information about the logic to do so. An agent that declines a claim or flags a client as high risk, with no person in the loop, is a section 71 question before it’s an engineering one.Sidenote: This is a practitioner’s reading of the text, not legal advice. Take a real automated decision to your compliance officer or attorney.

Start low and climb with a reason

Take one AI workflow you run today, or the one you’re about to build. List its steps, and for each one write down:

Question	Write down
Who decides the next step?	Code, a person or the model
What can it read?	The data sets, and whether any are personal information
What can it change or send?	Systems written to; anything that leaves the firm
Trifecta?	Private data, untrusted content, external communication: how many of the three
Level	1 to 5

Then apply three rules. Anything at level 5 moves down or into a sandbox. Anything at level 4 with all three legs of the trifecta gets a gate in code. Anything that makes a decision about a client gets a person, or a documented section 71 exception.

Most steps will land at level 1 or 3. That isn’t a failure of ambition. Predictable is what a system your clients rely on should be.

Replit’s freeze was a sentence in a prompt. The control that would have saved the database was a permission the agent never had. Put your controls there.

Chad Norman: writing

Claude Code runs A/B tests on your workflow, and one variable turns them off

Claude Code fetches its behaviour from a flag service

Experiments have already changed people’s work

An unannounced experiment is an unmanaged change

One environment variable switches the experiments off

Switching off has its own costs

Vendors should tell you which cohort you are in

What to do on Monday

Just-in-time access shrinks an agent's window, not its reach

A 30-second token still issues the refund

A policy engine sees the request, never the intent

A second model reads the same poisoned ticket

An agent never holds more than the person it works for

Approval gates work only when they’re rare

Short-lived credentials still earn second place

A five-person FSP needs vendor limits, not a policy engine

Write the limit down before you shorten the token

Cloudflare's free plan is where your first project should live

A static site on Workers costs nothing to serve

What the free plan does for you before you write a line

How this site gets from git to the edge

What went wrong

The defaults you get, and the ones you still own

Where the free plan stops holding

Ship it this week

Hello world

Text and links

Lists

Code

Long lines

Callouts

Figures and tables

Quotes and footnotes

Footnotes

Give your AI workflows the least autonomy that works

Autonomy is a permission, not a feature

Level 1: code decides, and nothing guesses

Level 2: the model advises, a person acts

Level 3: code runs the flow, the model does one narrow job

Level 4: the model decides, inside a fence

Level 5: the model decides everything

The lethal trifecta tells you where the gates go

POPIA already covers decisions about people

Start low and climb with a reason