pulumi

With ESC Approvals, you can propose, review, and apply environment changes with change requests. Until now, an environment could only have one open change request at a time. If a teammate’s change was waiting for review, others were blocked from opening change requests. Today, multiple people can open change requests on the same environment at the same time. A platform engineer can rotate a database credential while an app team adds a new feature flag, and each change goes through its own review. You no longer need to wait on your teammates’ changes! ## How it works ### Creating and viewing change requests in the Pulumi Cloud console Creating environment change requests works the same as before - just edit an environment that has approvals enabled, and click “Create Draft” once ready, which opens a change request for your teammates to review. You can view all change requests for the environment from either the version dropdown or the Approvals tab. ### Updating to the latest revision If another change request gets applied first, yours is now based on an older revision. This is nothing to worry about - just click “Update to latest revision” and your changes are reapplied on top of the latest revision (using a 3-way merge). If there is a merge conflict, the console will show which lines conflict. You can resolve this by either editing the draft or recreating the change request. ## Get started Concurrent change requests are available today in Pulumi Cloud on the Pro and Enterprise editions, for any environment with approvals enabled. To get started, see the Approvals documentation. Let us know what you think in the Pulumi Community Slack!

TL;DR: we are releasing a new major version of the Google Cloud provider. To migrate, ask Neo or your coding agent to run the Pulumi provider-upgrade skill (how to use it) or follow the v10 migration guide. We are happy to announce the next major version of the Pulumi Google Cloud provider. This release is based on the new v8 major version of terraform-provider-google (see also the upstream v8 upgrade guide). It ships with a migration guide that covers every breaking change in depth. Here are a few links to help you get started if you are new to Pulumi: - Getting Started: a guided walkthrough for creating your first project - Setup & Install: how to install the Google Cloud provider - How-to guides: how to provision specific resources with the Google Cloud provider - Pulumi Neo: ask Pulumi Neo to help you with your project ### Upgrading The migration guide accounts for every resource, type and function that changed in the new version. For the most popular resources, it gives a description of the change, a risk and impact section, a way to check whether you are affected, and step by step migration snippets. ### Use an agent to assist with your upgrade We have updated the provider-upgrade skill in Pulumi Agent Skills to cover the upgrade from v9. You can use it from Pulumi Neo or from any popular coding agent. Before it bumps the provider, it scans your stack and code for the changes that affect you, and it stops to ask before anything that would replace or delete live infrastructure. Then it upgrades the dependency, runs pulumi preview until the diff is explained, and leaves pulumi up to you. In Neo, ask: “Upgrade this project to pulumi-gcp v10.” Neo picks the provider-upgrade skill for you; there is nothing to install or name. Neo plans the upgrade, starting with a scan of your stack while it is still on v9. Before it edits any code, it tells you which resources are affected and what could be replaced, and waits for your answer. Finally, Neo opens a pull request with the changes and the preview results. You review it and run pulumi up when you are ready. In Claude Code, add the Pulumi marketplace and install the pulumi plugin: bash /plugin marketplace add pulumi/agent-skills /plugin install pulumi Then ask: “Upgrade this project to pulumi-gcp v10.” You can also run /provider-upgrade directly. In OpenAI Codex, register the marketplace: bash codex plugin marketplace add pulumi/agent-skills Then run codex, open /plugins, and install pulumi. Ask Codex to upgrade the provider to v10, or invoke the skill with $provider-upgrade. In Pi, and in any other agent that supports the Agent Skills standard, install the skill with the universal installer: bash npx skills add pulumi/agent-skills/pulumi --skill provider-upgrade -a pi Then ask Pi to upgrade the provider to v10, or trigger the skill with /skill:provider-upgrade. ### Why upgrade Moving to the latest version means you keep getting the latest updates from Google Cloud. GCP is covered by our provider support policy: with this release, v9 receives security updates for up to 12 months (or until v11 ships), while new features, bug fixes, and upstream updates land only in v10. Staying on the latest version also means you keep getting new features. Google Cloud offers capabilities you will not find on other clouds, such as TPUs, accelerators Google designed for training and serving AI models. For example, here is how you could create a Cloud TPU v6e (Trillium) slice with the new major version, using gcp.tpu.V2Vm: typescript import * as gcp from "@pulumi/gcp"; // A Cloud TPU v6e (Trillium) slice with 8 chips, on Spot capacity. const tpu = new gcp.tpu.V2Vm("trillium", { zone: "us-east1-d", runtimeVersion: "v2-alpha-tpuv6e", acceleratorConfig: { type: "V6E", topology: "2x4", }, schedulingConfig: { spot: true, }, labels: { workload: "training", }, }); export const tpuName = tpu.name; export const tpuWorkers = tpu.networkEndpoints.apply(eps => eps.map(ep => ep.ipAddress)); python import pulumi import pulumi_gcp as gcp # A Cloud TPU v6e (Trillium) slice with 8 chips, on Spot capacity. tpu = gcp.tpu.V2Vm( "trillium", zone="us-east1-d", runtime_version="v2-alpha-tpuv6e", accelerator_config={ "type": "V6E", "topology": "2x4", }, scheduling_config={ "spot": True, }, labels={ "workload": "training", }, ) pulumi.export("tpu_name", tpu.name) pulumi.export( "tpu_workers", tpu.network_endpoints.apply(lambda eps: [ep.ip_address for ep in eps]), ) go package main import ( "github.com/pulumi/pulumi-gcp/sdk/v10/go/gcp/tpu" "github.com/pulumi/pulumi/sdk/v3/go/pulumi" ) func main() { pulumi.Run(func(ctx *pulumi.Context) error { // A Cloud TPU v6e (Trillium) slice with 8 chips, on Spot capacity. vm, err := tpu.NewV2Vm(ctx, "trillium", &tpu.V2VmArgs{ Zone: pulumi.String("us-east1-d"), RuntimeVersion: pulumi.String("v2-alpha-tpuv6e"), AcceleratorConfig: &tpu.V2VmAcceleratorConfigArgs{ Type: pulumi.String("V6E"), Topology: pulumi.String("2x4"), }, SchedulingConfig: &tpu.V2VmSchedulingConfigArgs{ Spot: pulumi.Bool(true), }, Labels: pulumi.StringMap{ "workload": pulumi.String("training"), }, }) if err != nil { return err } ctx.Export("tpuName", vm.Name) ctx.Export("tpuWorkers", vm.NetworkEndpoints.ApplyT(func(eps []tpu.V2VmNetworkEndpoint) []string { ips := make([]string, len(eps)) for i, ep := range eps { ips[i] = *ep.IpAddress } return ips }).(pulumi.StringArrayOutput)) return nil }) } csharp using System.Collections.Generic; using System.Linq; using Pulumi; using Gcp = Pulumi.Gcp; return await Deployment.RunAsync(() => { // A Cloud TPU v6e (Trillium) slice with 8 chips, on Spot capacity. var tpu = new Gcp.Tpu.V2Vm("trillium", new() { Zone = "us-east1-d", RuntimeVersion = "v2-alpha-tpuv6e", AcceleratorConfig = new Gcp.Tpu.Inputs.V2VmAcceleratorConfigArgs { Type = "V6E", Topology = "2x4", }, SchedulingConfig = new Gcp.Tpu.Inputs.V2VmSchedulingConfigArgs { Spot = true, }, Labels = { { "workload", "training" }, }, }); return new Dictionary<string, object?> { ["tpuName"] = tpu.Name, ["tpuWorkers"] = tpu.NetworkEndpoints.Apply(eps => eps.Select(ep => ep.IpAddress).ToList()), }; }); java package myproject; import com.pulumi.Pulumi; import com.pulumi.gcp.tpu.V2Vm; import com.pulumi.gcp.tpu.V2VmArgs; import com.pulumi.gcp.tpu.inputs.V2VmAcceleratorConfigArgs; import com.pulumi.gcp.tpu.inputs.V2VmSchedulingConfigArgs; import java.util.Map; import java.util.stream.Collectors; public class App { public static void main(String[] args) { Pulumi.run(ctx -> { // A Cloud TPU v6e (Trillium) slice with 8 chips, on Spot capacity. var tpu = new V2Vm("trillium", V2VmArgs.builder() .zone("us-east1-d") .runtimeVersion("v2-alpha-tpuv6e") .acceleratorConfig(V2VmAcceleratorConfigArgs.builder() .type("V6E") .topology("2x4") .build()) .schedulingConfig(V2VmSchedulingConfigArgs.builder() .spot(true) .build()) .labels(Map.of("workload", "training")) .build()); ctx.export("tpuName", tpu.name()); ctx.export("tpuWorkers", tpu.networkEndpoints().applyValue(eps -> eps.stream() .map(ep -> ep.ipAddress().orElse(null)) .collect(Collectors.toList()))); }); } } yaml resources: # A Cloud TPU v6e (Trillium) slice with 8 chips, on Spot capacity. trillium: type: gcp:tpu:V2Vm properties: zone: us-east1-d runtimeVersion: v2-alpha-tpuv6e acceleratorConfig: type: V6E topology: 2x4 schedulingConfig: spot: true labels: workload: training outputs: tpuName: ${trillium.name} tpuWorkers: ${trillium.networkEndpoints} Gemini Enterprise Agent Platform (formerly Vertex AI), Google’s platform for building with Gemini models, also gets new resources. New in v10, gcp.vertex.AiRagCorpus brings its RAG Engine under Pulumi: declare the corpus that grounds your Gemini applications in your own documents, with its embedding model and vector store, in the same program as the rest of your infrastructure. It joins the Agent Runtime and Model Garden resources added during v9. In the provider, these resources keep the gcp.vertex module name. You can find more about this release in the v10 release notes. We hope your transition goes smoothly, and as always we are happy to hear your feedback in our Community Slack or through support if you are a paying Pulumi customer. Happy hacking!

Not every message you send to Claude Code needs the most capable model. A quick question about a Git command runs on the same model as a refactor across three services, unless you remember to switch models first. Jev, a new model from TypeSafe AI, can make that decision for you in a few hundred milliseconds and for a fraction of a cent. So I built jev-router, an open source router that asks Jev which Claude model each of your messages needs, and sends it there. ## What Jev is > After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? > > I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev > > • 20-200x faster > • 40-400x… pic.twitter.com/JSybNG2BKJ > > — Diogo Almeida (@CompleteSkeptic) September 15, 2026 TypeSafe AI released Jev in September 2026. Its founder, Diogo Almeida, co-authored OpenAI’s InstructGPT paper, the research behind ChatGPT. Jev isn’t a large language model, and it can’t write you a sentence. TypeSafe calls it a System One model, a term borrowed from Daniel Kahneman’s Thinking, Fast and Slow. LLMs like Claude and GPT work like System Two: they write their answer token by token, take seconds to minutes, and can say anything. Jev works like System One. You describe a situation, ask a few typed questions, and it answers all of them in one parallel pass, in 70–500 milliseconds end to end, according to TypeSafe’s launch post. Every question takes one of three shapes. A Noul asks whether a statement is true and returns a probability. A Choice picks from a list of options you provide, with a probability for each one. A Score places the situation on a scale. Jev never answers outside your options, so there’s no malformed JSON to parse and no made-up category to handle. TypeSafe trains Jev with a method it calls reinforcement learning for calibrated decisions (RLCD), and says the probabilities are calibrated: a higher confidence should mean a higher accuracy. It charges $0.042 per million input tokens and calls output tokens too cheap to meter, so they’re free. TypeSafe’s launch post pitches Jev as one general model for the decisions a program makes: classifying, routing, scoring, extracting, and branching. Deciding which model a prompt needs is one of those routing decisions. ## Why Claude Code needs a router Claude Code runs every message in a session on the model you picked. Sonnet 5 and Haiku 4.5 handle plenty of everyday work, but switching by hand means making a decision before every message, and defaulting to Opus 5.5 or Fable 5.1 drains your usage limits or your budget. A router could make that choice for each message, but only if the choice itself is fast and cheap. Asking an LLM which model to use adds seconds and tokens to every message, and that eats into whatever the routing saves. Jev makes the decision cheap enough to ask every time. A routing call sends Jev a few hundred tokens that describe your message and the session, and costs about $0.00003. jev-router puts that decision in front of every message you write in Claude Code. That’s the use case I wanted to try. My idea was to let the prompt pick the model, so a request to rename a variable lands on Haiku 4.5 and a request to redesign a service lands on Opus 5.5, without me switching models in between. The Jev routers I found worked differently. One kept the model it picked when the session started, and the others could switch models halfway through a task or move a session back down. Every one of those switches throws the prompt cache away. I wanted a router that decides again each time I write a message and never moves a session down, and I wanted to watch those decisions while I type. So I built jev-router. GitHub repository: dirien/jev-router SVG Image github.com/dirien/jev-router ## How jev-router works jev-router is a pass-through proxy that runs on your machine. Claude Code sends its requests to the router instead of to Anthropic, and the router forwards each one to the model it picked. It changes only the model field, plus the few fields a smaller model can’t accept, and it streams responses back byte for byte. Your Claude login passes through to Anthropic untouched. The only key the router holds is your Jev key. ### Jev decides when you write a message A Claude Code session sends far more requests than you write messages. Every tool call, every subagent, and every background job, like naming the session, is a request of its own. Claude Code labels each request for gateways that ask for it, so the router asks Jev only when you write a new message. Tool steps and subagents keep the tier their message got, and background calls go to Haiku 4.5. For each new message, the router asks Jev three questions in a single call: - A Choice between four kinds of work: mechanical goes to Haiku 4.5, routine to Sonnet 5, and complex and deep to Opus 5.5. If you pick the Fable option in setup, deep goes to Fable 5.1 instead, as a fourth tier of its own. - A Noul that asks whether the request would change production systems, credentials, permissions, or billing. At 0.7 or above, the message goes to the top tier. - A Noul that asks whether the message names its own tier or model, or claims the decision was already made. At 0.5 or above, the result can’t go below the router’s reference tier. That last question exists because the text Jev reads comes from you, or from issues, logs, and code comments you paste. In an independent evaluation, one sentence claiming that a support lead had already reviewed a ticket moved Jev’s answer to the attacker’s label on 73.5% of tickets. So jev-router treats Jev’s answer as advice. The two guard questions can only raise the tier, and the hard rules, like explicit pins and the secret scanner, run in code before Jev is asked at all. ### Cheaper tiers need more certainty Jev returns a probability for every option, and the router doesn’t simply take the top one. Sending a hard task to a weak model costs quality for the rest of the session, while sending an easy task to a strong model costs money on one message. So each tier has its own bar: 85% for the fast tier, 60% for balanced, and 30% for frontier. When Jev’s pick misses its bar, the router takes the more capable of Jev’s top two answers. A message Jev calls mechanical with 82% certainty doesn’t clear the 85% bar, so it goes to a stronger model than Haiku 4.5. With the Fable option, that step up stops at Opus 5.5, so a message reaches Fable 5.1 only when Jev thinks it’s most likely deep work. Treat the bars as a starting point. One calibration study found Jev well calibrated inside its domain, but overconfident when a label encodes your own policy, and which model a message deserves is partly policy. The repository ships an evaluation set of 58 labeled prompts so you can tune the bars on your own messages. ### Within a session, the tier only moves up Once a session reaches Opus 5.5, a quick “thanks!” doesn’t send it back to Haiku 4.5. A new message can raise the tier, but it never lowers it. Anthropic’s prompt cache belongs to one model, so a switch starts the next request cold, and an agent’s requests are mostly cache reads. Handing half-done work to another model doesn’t pay off either: in an AWS study of 500 SWE-bench Verified tasks, handing a Haiku 4.5 transcript to Opus 4.7 recovered only 47% of the quality gap, at $1.61 per task against $0.72 for starting on Opus. A session gets a fresh decision after 10 minutes without traffic, after Claude Code compacts it, or when you start over with /clear. You can still steer. Switching to another model family with /model pins that family’s tier, and #fast, #balanced, or #frontier (or #max for Fable 5.1) as the first or last word of a message asks for a tier. A scanner also checks every message for secrets before anything leaves your machine, and a hit keeps the session on trusted models, whatever Jev says. ## Watch every decision The router serves a live view on http://127.0.0.1:4100. Each message you write shows up with the category Jev picked, the probability behind it, and the tier and model the router chose. Select a request to see why it went where it went. The view reads the router’s log, which never contains your prompts or keys. jev-router report sums up requests, spend, and savings from the same log. ## Try jev-router You need Node.js 22 or newer, Claude Code, and a Jev key from TypeSafe or OpenRouter. Then run: bash npx @ediri/jev-router setup claude Setup asks which models to use. The default is Claude only; the second option adds Fable 5.1 for deep work, the way the live view above routes the redesign. Then it asks for your Jev key and checks it with one call that costs about $0.00003. It saves the key in a file only you can read, installs jev-router globally with npm because a background service can’t run from npx’s cache, starts the router as a launchd agent on macOS or a systemd user service on Linux, and points Claude Code at it. To switch models later, run setup again. Afterwards, jev-router doctor checks the setup. jev-router uninstall removes the service and takes the router back out of Claude Code’s settings. If your team keeps API keys in Pulumi ESC, setup can take the Jev key from an environment that exports TYPESAFE_API_KEY instead of asking you for it. With --yes, setup asks nothing and takes its defaults; add --models fable for the Fable option: bash pulumi env run <your-org>/<your-project>/<your-environment> -- npx --yes @ediri/jev-router setup --yes Setup keeps Claude Code on Claude models by default, because Anthropic doesn’t support pointing Claude Code at other models through a gateway. The router also works with the Codex CLI, which it routes across OpenAI and Ollama Cloud models. Try it on a day of real work and watch where your messages land. If one lands somewhere it shouldn’t, open an issue with its line from the live view, which shows the model and the reason but never your prompt. The code is on GitHub under the Apache 2.0 license.

Pulumi Neo works on infrastructure the way an engineer does: it clones repositories, edits files, installs dependencies, runs previews, and produces intermediate work along the way. A task is not only a conversation with a model. It is also a working directory that has to survive long enough for the agent to keep making progress. Imagine asking an agent to upgrade a Pulumi provider version across a repository, regenerate SDK code, run previews and tests, inspect the failures, and open a pull request. Halfway through, the runtime restarts. Conversation replay can recover what the agent said and which tools it called, but it does not recover the generated files, dependency state, local commits, or scratch output the next step may depend on. Without workspace continuity, the agent can wake up remembering the plan but missing the room it was working in. For early hosted Neo tasks, workspace recovery started with Git. We persisted enough information to reconstruct repositories later: remotes, branches, commits, and local diffs. This was about the hosted runtime workspace, not a user’s local checkout when running pulumi neo, where local filesystem and shell tools execute on the user’s machine. The Git-based approach worked for simple hosted-runtime cases, but it was the wrong abstraction for long-running agent work. We needed to recover the workspace Neo had actually used. This post walks through how we tackled workspace continuity for long-running agent tasks, what we learned along the way, and how the same techniques can help you build more resilient agentic systems of your own. The core primitive we chose was Kopia, an open source backup tool that creates encrypted, incremental, content-addressed snapshots and stores them in backends such as object storage. ## Git was the wrong recovery boundary The original persistence model was repository-oriented. After a turn, the hosted agent runtime scanned its own working directory for Git repositories, recorded each remote, branch, commit, and local diff, and stored that representation through our task state API. On a cold start, a new runtime fetched that state, cloned the repositories again, checked out the saved revisions, and reapplied patches. That was a reasonable first design because most infrastructure work starts in Git. It also kept persisted state relatively small when local changes were simple. But agent workspaces are not only Git repositories. They contain generated files, scratch files, local commits, package-manager state, tool output, and sometimes files outside any repository. The reconstruction path also depended on everything still being fetchable later: the remote repository had to exist, credentials had to work, the saved revision had to be available, and the patch still had to apply cleanly. In practice, that showed up in a few painful ways: - Resumed runtimes could start with incomplete working directories even though the conversation history replayed successfully. - Tasks could fail to resume with clone errors when a repository was deleted after the task started and no longer existed at its original remote. - Large diffs could outgrow the payload limit for persisted task state. - Intentionally skipped directories, such as dependency caches and build output, sometimes contained context the agent had just created. The distinction that finally clarified the design was simple: > Replay events to restore what Neo remembers. Restore a snapshot to recover what Neo changed. Conversation memory and workspace state are separate continuity planes. Event replay rebuilds messages, tool calls, approvals, and compaction state. Filesystem snapshots rebuild the workspace. ## The thing we were not willing to make customers run The deciding question was not “what can back up a directory?” Lots of things can back up a directory. The deciding question was “what are we willing to make every Pulumi Cloud and self-hosted customer operate so Neo can resume a task?” That ruled out more than you might expect. EFS plus AWS Backup is lovely if your answer to every deployment question is “be on AWS.” Ours is not. Self-hosted Pulumi has to work in customer environments, and those environments already have a portable storage requirement: blob storage. The snapshot system needed to ride on that, not introduce an AWS-shaped trapdoor. A Git server had a similar problem. Neo workspaces often include Git repositories, but the workspace is not itself a Git repository. Turning recovery into “run a Git service, maybe with many repositories per task, plus enough conventions to capture non-Git files” would have given us another product to operate and another thing for self-hosted customers to debug. Direct object storage sync looked better from the portability angle, but it was still the wrong abstraction. A sync can copy files, but it does not naturally give us point-in-time restore, efficient rename-heavy history, retention, or a clean recovery model after partially failed writes. Block-volume snapshots had the opposite problem: strong recovery semantics, but the wrong deployment and cost model for task-scoped workspaces. That left us looking for a boringly useful shape: an encrypted, task-scoped snapshot repository over generic object storage. Kopia gave us the closest match to the unit of recovery: the task workspace. It supports incremental snapshots, point-in-time restore, encryption, content-addressed storage, object storage backends, and practical cleanup. Renames and moves generally change metadata rather than rewriting all bytes. Frequent temporary-file churn still matters, which is why ignores and retention policy are part of the design rather than afterthoughts. The repository layout is intentionally task-scoped: text s3://<task-backup-bucket>/ tasks/ <task-id>/ kopia/ ...encrypted repository data... That gives up cross-task deduplication, but it aligns the storage boundary with the security and lifecycle boundary. A task ID names the storage prefix; it is not the access-control boundary by itself. The service still has to authenticate the task before minting temporary credentials, those credentials are scoped to that task’s prefix, and the Kopia repository is encrypted with a per-task password. One task’s runtime must never be able to read another task’s snapshot history. ## Split the control plane from the data plane The next decision was where the bytes should move. We wanted the service to authorize recovery, not become the storage proxy for every dependency tree, generated artifact, and scratch file an agent might create. Our service remains the control plane. It authenticates the task, decides whether snapshotting is disabled, shadow, or enabled, and mints temporary credentials scoped to that task’s storage prefix. The runtime remains the data plane for its own local workspace: it restores and checkpoints directly against object storage instead of streaming large filesystem payloads through the service. That split is also the multi-tenant security boundary. The runtime gets only the storage authority for the task it is currently serving, and the encrypted repository it opens is protected by that task’s repository password. A different task would need both a different storage grant and a different repository password to read its snapshots. flowchart LR Events[Persisted task events] -->|replay memory| Runtime[Neo runtime] Service[Service control plane] -->|snapshot mode| Runtime Service -->|short-lived task-scoped credentials| Runtime Runtime -->|restore and checkpoint| Kopia[Kopia repository for one task] Kopia --> S3[S3 task prefix] Cleanup[Trusted cleanup path] -->|expire and maintain| Kopia This split matters. If the service were the only storage principal, every snapshot and restore would have to flow through the application service. That would turn the service into a high-volume data path for dependency trees, generated artifacts, and arbitrary workspace state. Instead, the service authorizes access, while the runtime moves bytes directly to and from storage. Isolation has several layers: - The service only issues credentials after authenticating the task. - Credentials are short-lived and scoped to a single task prefix, not just a bucket. - Runtime credentials can read and write only that task’s snapshots, but cannot delete durable history. - Each task has its own Kopia repository password, so object access and repository decryption are separate checks. - Cleanup runs through a trusted path with delete authority. Kopia’s retention model also helped us keep delete permission out of the runtime. Snapshot expiration removes references, and maintenance later reclaims unreferenced encrypted blobs. The agent-controlled runtime can create recoverable history without being able to erase it. ## Roll out the escape hatch first We did not want a single switch that made a new persistence layer authoritative on day one. The rollout used three modes: | Mode | Legacy repo reconstruction | Snapshot writes | Snapshot restores | | ---------- | -------------------------- | --------------- | ----------------- | | disabled | Authoritative | No | No | | shadow | Authoritative | Yes | No | | enabled | Fallback | Yes | Yes | shadow mode was the key. It let us write snapshots, observe latency and storage behavior, and validate that repositories could be initialized and updated without changing task recovery. Only after that did enabled make snapshots the primary cold-start restore path. Even in enabled mode, we continued writing the legacy Git repository state. That gave us a real rollback path instead of a ceremonial one. If we needed to turn snapshot restores back off, existing tasks would still have the old recovery data available, so rollback would not degrade into “resume, but with half the workspace missing.” The restore path is deliberately asymmetric: flowchart TD Start[Cold start] --> Mode{Snapshot restore enabled?} Mode -->|No| Legacy[Clone repositories and apply saved patches] Mode -->|Yes| Restore[Attempt latest Kopia snapshot] Restore -->|Success| Workspace[Use restored workspace] Restore -->|Missing or failed| Legacy Workspace --> Replay[Replay persisted events] Legacy --> Replay Replay --> Ready[Resume task] If a snapshot restore fails, Neo can still fall back to the old repository reconstruction path while that path exists. Snapshot failure is observable, but it does not automatically make a task unrecoverable. Checkpointing is also best effort. The runtime returns the final response for the turn, then starts a background checkpoint. If that checkpoint fails, the completed response is still completed. The tradeoff is a bounded durability gap: the conversation may contain a completed turn whose latest filesystem changes are newer than the latest successful snapshot. On the next cold start, Neo restores the latest known-good snapshot rather than pretending a partial checkpoint exists. ## Snapshot the work, not the noise An agent workspace has both signal and noise. The signal is user-relevant work: source changes, generated configuration, local commits, tool output, and files the agent may need on the next turn. The noise is everything a package manager, compiler, or test runner can recreate. The runtime snapshots existing workspace roots such as: - /workspace - /tmp-workspace - /var-workspace Regenerable caches and short-lived dependency or build trees are ignored where appropriate. This is an engineering tradeoff for the team operating the agent. Snapshot too much and storage plus request volume become noisy. Snapshot too little and the workspace stops representing what the agent actually saw. The boundary we cared about was user-relevant continuity. Source changes, generated configuration, local commits, and tool output should be there after recovery. A dependency cache or build directory that can be recreated may not be. In practice, that means a recovered task might spend extra time reinstalling packages or rebuilding generated artifacts, but it should not lose the work the agent produced or the files it needs to decide what to do next. ## Pay for isolation on purpose The big cost decision was choosing one repository per task. A shared repository might deduplicate identical content across tasks, but it would also create the wrong cleanup and isolation model. We chose to pay some extra storage cost so the security and lifecycle boundary matched the product boundary: one task, one encrypted snapshot repository. With a task-scoped repository, deletion, retention, credential scope, and auditability all line up. That means cost management depends on four practical controls: 1. Incremental snapshots. Kopia avoids re-uploading unchanged content inside a task repository. 2. Ignore policy. High-churn generated trees should not dominate every checkpoint. 3. Retention windows. The initial design assumed short default retention for completed tasks, with room for longer retention by policy. 4. Trusted maintenance. Expiring snapshots removes references; repository maintenance later reclaims unreferenced physical data. In the planning model, blob-backed Kopia repositories were dramatically cheaper than direct object storage sync for our benchmark workload. The reason was request shape more than raw storage. A sync-based mirror looks simple, but rename-heavy and small-file-heavy workloads turn into large numbers of write, list, and delete operations. Snapshot repositories have their own metadata and maintenance costs, but they preserve history and behave much better under repeated workspace churn. ## Make failures diagnosable, not mysterious Snapshotting added a new storage system, so we instrumented recovery as a sequence of named phases instead of one “persistence failed” bucket. For each task, we wanted the logs and metrics to answer: - Was the invocation warm or a cold start? - Was the task in disabled, shadow, or enabled mode? - Did task authorization and credential issuance succeed? - Did Kopia connect to the expected task repository? - Which workspace roots existed, and which completed? - Did restore fall back to repository reconstruction? - Was the restored snapshot older than the latest completed turn? - Did event replay succeed independently of filesystem recovery? - Did cleanup expire snapshots or reclaim physical blobs? The takeaway is to make recovery observable at the same boundaries where it can fail. A task can restore files but fail to replay model history. It can replay history but fall back to a stale or reconstructed workspace. Those are different incidents with different remediations, so the logs and metrics need to preserve the distinction. ## Workers made ordering matter Our initial Kopia snapshot path recovered a task when a runtime cold-started. The next step was making the same continuity model work as Neo moved tool execution into managed workers. Worker mode was a separate change to how Neo executes tools: instead of every tool running inside the runtime process, a managed worker can own the workspace and execute the tool calls. Because recovery had already moved from “rebuild Git repositories” to “restore the task workspace,” the migration did not need a second persistence model. The main new problem was ordering: making sure the tool result and the snapshot pointer advanced together. In worker mode, a separate process owns the task workspace, runs tool calls against that workspace, snapshots it after each completed tool call, and reports both the tool result and the latest recovery pointer back to the service. A recovery pointer is small metadata: effectively the Kopia snapshot object ID for one logical workspace root. We store the pointer in the service database, while the encrypted snapshot data stays in object storage. The important invariant is: > Only the current worker can commit the tool result and the snapshot pointer that becomes the next restore source. The service uses a worker lease to keep snapshot pointers ordered. At any point, one worker lease is current for the task. If that worker stalls, dies, or stops heartbeating, the service can assign a replacement worker with a newer lease. If the old worker later wakes up and tries to report a tool result with an older snapshot pointer, the stale lease prevents that pointer from becoming the next restore source. That matters because the result and the snapshot describe the same moment in the task. If Neo records “the command finished” but restores the next worker from an older snapshot, the agent can continue from a workspace that does not contain the files produced by the completed tool call. The lease check makes the persisted tool result and the authoritative recovery pointer advance together. Worker snapshots also have failure fuses. A restore failure resets the workspace and marks the next result so the agent knows local filesystem continuity was lost. Repeated snapshot failures disable further snapshot attempts for that worker process rather than blocking tool execution. ## What users got back The user-visible goal was not “backups.” It was continuity. During our internal stress testing for agent tasks, legacy recovery often took around two minutes and repeatedly reported failed repository recovery for extremely large repositories with extensive history. The same tasks restored from Kopia snapshots in roughly half that time with a 100% success rate. More importantly, the snapshot path avoided failure modes the old model could not solve, such as deleted remotes, oversized diffs, and non-repository workspace state. Restores became faster and much more stable, but the write path did get slower. Kopia checkpoints cost seconds, not milliseconds, because they preserve the workspace rather than a reconstruction recipe. We mitigated that by making turn-level checkpoints non-blocking: the task can return its completed response, then checkpoint the workspace in the background. For our tasks, a few seconds of asynchronous durability work was the right tradeoff for reliable recovery. ## What’s worth stealing from this design If you are building long-running agents, the storage question is easy to underestimate. It can look like an implementation detail: clone the repository again, replay the conversation, rehydrate some state, and keep going. But agents do not only produce messages. They produce working directories. The most useful lesson for us was to persist the abstraction the product depends on. Neo does not resume into a Git diff. It resumes into a workspace. Once we made the workspace the persistence boundary, the rest of the design became clearer: task-scoped repositories, short-lived credentials, runtime-owned data transfer, trusted cleanup, and an incremental rollout. The second lesson was to separate memory from files. Event replay and filesystem restore are both required for a healthy resume, but they are not the same recovery operation. A task can remember the conversation and lose local filesystem continuity. It can restore files and still fail to rebuild model history. Treating those as separate planes made the system easier to reason about, easier to observe, and easier to repair. The final lesson was to make rollback part of the design, not an emergency plan. shadow mode let us learn before relying on snapshots. Fallback kept early restore issues from becoming task-ending incidents. Continuing to write legacy Git state meant enabled mode was still reversible. The storage engine mattered, but the rollout shape mattered just as much. The durable takeaway is this: agent continuity is a product feature, not a backup feature. Users do not care whether a checkpoint succeeded in the abstract. They care whether the agent comes back with the same context, the same files, and the same ability to keep working. Kopia gave us the snapshot primitive, but the real design was choosing the right boundary for recovery.