Notes on AI agents, platform engineering, and running things on hardware I own.https://blog.kellerstein.org/https://blog.kellerstein.org/blog/point-dont-paste/https://blog.kellerstein.org/blog/point-dont-paste/Reviewing a document with an AI still means copying a paragraph into a chat, explaining where it came from, and carrying the answer back by hand. REX puts the conversation on the document instead — every comment is a thread, the thread stays attached to its passage, and nothing reaches your file until you approve it.Mon, 14 Sep 2026 12:00:00 GMThttps://blog.kellerstein.org/blog/a-wrong-place-is-worse-than-no-place/https://blog.kellerstein.org/blog/a-wrong-place-is-worse-than-no-place/A broken anchor does not fail. It lands somewhere, reports `ok`, and paints a highlight that looks exactly like a correct one. How REX keeps a comment on its passage while the document is edited, the three times it lied anyway, and the rule that came out of it: orphan before you guess.Mon, 14 Sep 2026 11:00:00 GMThttps://blog.kellerstein.org/blog/a-comment-is-a-conversation-that-waits-for-you/https://blog.kellerstein.org/blog/a-comment-is-a-conversation-that-waits-for-you/Every REX comment is its own thread with its own agent session, stored on the document rather than in a chat window. What is stored and what is derived, how four SDKs resume a conversation four different ways, and the resume that was silently broken until a debug report asked the filesystem instead of the SDK.Mon, 14 Sep 2026 10:00:00 GMThttps://blog.kellerstein.org/blog/a-question-cannot-change-your-file/https://blog.kellerstein.org/blog/a-question-cannot-change-your-file/REX makes one safety promise: ASK cannot write, ACT edits a copy, and only Approve touches your file. Three times it broke without a sound — a line gone under "Applied to 0 file(s)", a committed file deleted under a notice saying it was put back, a copy removed while the agent was reading it — and what an allowlist, four SDKs and a repository's own settings file taught me about where a boundary actually is.Mon, 14 Sep 2026 09:00:00 GMThttps://blog.kellerstein.org/blog/binary-files-differ/https://blog.kellerstein.org/blog/binary-files-differ/The ACT loop ends in a diff you approve, and on a .pptx that diff is two words. So the agent writes a plan, REX performs it on a copy, and the checker has to prove not only that the change happened but that nothing else did — plus the three numbers I measured wrong on the way.Mon, 14 Sep 2026 08:00:00 GMThttps://blog.kellerstein.org/blog/one-client-multiple-llm-providers/https://blog.kellerstein.org/blog/one-client-multiple-llm-providers/Every project that calls a model carries three constants: a base URL, a key, and a model id — and each vendor spells the request differently, so the swap is a rewrite, not an edit. A gateway turns all of it into one client and one name. The same image request, written four ways, and what the indirection buys beyond convenience.Mon, 31 Aug 2026 12:00:00 GMThttps://blog.kellerstein.org/blog/the-model-got-better-and-i-stopped-reading-it/https://blog.kellerstein.org/blog/the-model-got-better-and-i-stopped-reading-it/Opus 5 wins the benchmarks and lost me somewhere in paragraph three. Two output styles later — one that fixes the words, one that fixes the shape — and a measurement of which colours a terminal will actually let a model use.Tue, 25 Aug 2026 12:00:00 GMThttps://blog.kellerstein.org/blog/drag-a-div-get-a-diff/https://blog.kellerstein.org/blog/drag-a-div-get-a-diff/VEX turns browser gestures into source-code changes. Three hard problems hide in that sentence — here is what each one taught me.Tue, 14 Jul 2026 08:29:44 GMThttps://blog.kellerstein.org/blog/it-works-from-the-terminal/https://blog.kellerstein.org/blog/it-works-from-the-terminal/The app ran perfectly when I launched it from a shell and failed with ENOENT when a user double-clicked it. A GUI app does not have your PATH, and the only thing that knows your PATH is your shell.Sun, 21 Jun 2026 13:12:32 GMThttps://blog.kellerstein.org/blog/six-hundred-lines-or-it-does-not-ship/https://blog.kellerstein.org/blog/six-hundred-lines-or-it-does-not-ship/A 2,313-line orchestrator with a 1,073-line function is maintainable by a human with an IDE and unmaintainable by an agent. The refactor whose acceptance criterion was an agent behaviour, the CI guard that enforced it — and the fact that the guard is red as I write this.Tue, 26 May 2026 12:00:00 GMThttps://blog.kellerstein.org/blog/i-built-a-time-machine-then-deleted-half-of-it/https://blog.kellerstein.org/blog/i-built-a-time-machine-then-deleted-half-of-it/A retrospective on the deletions. Four timeline verbs, parallel variants in git worktrees, a record mode and four branch families — two cleanup specs removed almost all of it, including, awkwardly, the feature whose existence justified the first round.Tue, 12 May 2026 12:00:00 GMThttps://blog.kellerstein.org/blog/driving-the-claude-agent-sdk/https://blog.kellerstein.org/blog/driving-the-claude-agent-sdk/A stage is one query() call. The five options that define what an autonomous agent is allowed to do, the four hooks that make an unattended run watchable, and an honest account of what bypassPermissions actually means.Tue, 05 May 2026 12:00:00 GMThttps://blog.kellerstein.org/blog/three-counters-three-answers/https://blog.kellerstein.org/blog/three-counters-three-answers/Naming a source of truth is the easy half. The hard half is naming the moment every other view gets rebuilt from it — and the run where a UI, a state file and a git history gave three different answers to "how much of this is built".Tue, 28 Apr 2026 12:00:00 GMThttps://blog.kellerstein.org/blog/git-as-the-database/https://blog.kellerstein.org/blog/git-as-the-database/Every stage commits, the run is reconstructable from refs alone, and the app owns exactly two branch families. Why an autonomous builder should store its work in the versioning system its users already have — and what that choice costs at review time.Fri, 24 Apr 2026 12:00:00 GMThttps://blog.kellerstein.org/blog/your-test-suite-bills-by-the-token/https://blog.kellerstein.org/blog/your-test-suite-bills-by-the-token/Twenty minutes and real dollars to verify a two-line UI change. A pluggable agent backend, a mock that refuses to improvise, and checkpoint replay took it to sixty seconds and zero — plus an honest account of what the mock does not buy you.Tue, 21 Apr 2026 12:00:00 GMThttps://blog.kellerstein.org/blog/prose-is-a-wire-protocol/https://blog.kellerstein.org/blog/prose-is-a-wire-protocol/When one agent hands work to the next, prose is the wire protocol — and a catastrophically bad one. The manifest that made feature selection cost zero tokens, and the line between what a schema pins down and what genuinely needs judgement.Mon, 20 Apr 2026 12:00:00 GMThttps://blog.kellerstein.org/blog/nine-stages-nine-amnesiacs/https://blog.kellerstein.org/blog/nine-stages-nine-amnesiacs/Dex never lets a session get long enough to rot. Every stage is a brand new agent with an empty context window, state travels through the filesystem, and the unit of everything — execution, resume, storage, the UI — is the stage. Including what that costs.Sun, 19 Apr 2026 12:00:00 GMThttps://blog.kellerstein.org/blog/the-agent-spawned-an-agent-and-i-could-not-see-it/https://blog.kellerstein.org/blog/the-agent-spawned-an-agent-and-i-could-not-see-it/A subagent is a black box that spends your money. Making it inspectable took no new instrumentation at all — the events were already firing, into nothing.Tue, 14 Apr 2026 23:45:19 GMThttps://blog.kellerstein.org/blog/why-three-pillars/https://blog.kellerstein.org/blog/why-three-pillars/Dex is not a new idea. It is three existing ideas — a bash loop, a spec workflow, and an SDK — each of which is most of an autonomous coding system and none of which is all of it. The interesting part is the shape of the holes.Sun, 12 Apr 2026 12:00:00 GMThttps://blog.kellerstein.org/blog/six-polling-loops-and-a-spinning-fan/https://blog.kellerstein.org/blog/six-polling-loops-and-a-spinning-fan/VEX was idle and my laptop was audible. The fix was not "poll slower" — it was noticing that a UI which asks "anything new?" sixty times a minute is a UI with no idea what happened.Tue, 07 Apr 2026 17:11:56 GMThttps://blog.kellerstein.org/blog/the-field-that-was-never-populated/https://blog.kellerstein.org/blog/the-field-that-was-never-populated/Continuing a conversation with an agent that already finished turned out to be one line of SDK config and three days of admitting my data model thought agents were disposable.Sun, 05 Apr 2026 19:56:25 GMThttps://blog.kellerstein.org/blog/the-model-was-the-easy-part/https://blog.kellerstein.org/blog/the-model-was-the-easy-part/A retrospective on OpenOperator. What I would keep, what I would redo, and why a commit called "remove hardcoded API keys" landing a year late is the most honest thing in the history.Fri, 20 Mar 2026 12:00:00 GMThttps://blog.kellerstein.org/blog/benchmarking-ai-agents-at-scale/https://blog.kellerstein.org/blog/benchmarking-ai-agents-at-scale/Five workload classes with five different scaling shapes, a bounded queue that fans tests into Kubernetes Jobs, and the reason your autoscaler makes terrible decisions about 30 GB virtual machines.Wed, 04 Jun 2025 12:00:00 GMThttps://blog.kellerstein.org/blog/one-proxy-pod-per-vm/https://blog.kellerstein.org/blog/one-proxy-pod-per-vm/A design that is obviously right at n=1 and obviously wrong at n=20, with no warning in between. The commit that deleted it is the best one in the repository.Thu, 01 May 2025 12:00:00 GMThttps://blog.kellerstein.org/blog/streaming-helm-to-a-browser/https://blog.kellerstein.org/blog/streaming-helm-to-a-browser/Provisioning a VM takes three minutes. A spinner is not an acceptable answer. An async generator, a path-keyed WebSocket relay, and one design decision I would not repeat.Wed, 30 Apr 2025 12:00:00 GMThttps://blog.kellerstein.org/blog/volumesnapshots-as-golden-images/https://blog.kellerstein.org/blog/volumesnapshots-as-golden-images/One Helm chart, two completely different storage topologies behind a single boolean. Plus the two bugs it cost — a StatefulSet naming convention and a substring match that matched too much.Tue, 22 Apr 2025 12:00:00 GMThttps://blog.kellerstein.org/blog/reading-teams-telemetry-microsoft-bond/https://blog.kellerstein.org/blog/reading-teams-telemetry-microsoft-bond/Ground truth from an application with no test hooks. mitmproxy local mode against a single process, a hand-written Bond Compact Binary decoder, and the one-line AKS bug that killed a day.Sun, 20 Apr 2025 12:00:00 GMThttps://blog.kellerstein.org/blog/shipping-a-6gb-iso-to-every-pod/https://blog.kellerstein.org/blog/shipping-a-6gb-iso-to-every-pod/An initContainer that pulls once, one PVC projected into eleven mount points via subPath, and a three-month migration from a storage key to workload identity.Wed, 16 Apr 2025 12:00:00 GMThttps://blog.kellerstein.org/blog/kvm-in-a-kubernetes-pod/https://blog.kellerstein.org/blog/kvm-in-a-kubernetes-pod/hostPath /dev/kvm, privileged: true, and the Deployment→StatefulSet migration. Plus the annotation that stops the autoscaler evicting a forty-minute Windows install.Tue, 08 Apr 2025 12:00:00 GMThttps://blog.kellerstein.org/blog/mcp-for-mouse-and-keyboard/https://blog.kellerstein.org/blog/mcp-for-mouse-and-keyboard/Six tools, one resource, and a design decision I still think is underrated — the screenshot is not a tool, it is a resource. Plus the reason we composite a cursor into every capture.Fri, 28 Mar 2025 12:00:00 GMThttps://blog.kellerstein.org/blog/it-looked-like-it-worked/https://blog.kellerstein.org/blog/it-looked-like-it-worked/The least trustworthy signal in the system is the agent reporting success. Here is the evaluator we built instead — and the four-word commit that forced a whole second repository into existence.Wed, 26 Mar 2025 12:00:00 GMThttps://blog.kellerstein.org/blog/tasks-as-config-not-code/https://blog.kellerstein.org/blog/tasks-as-config-not-code/A JSON scenario format with lifecycle hooks, so an agent test starts from the same desktop every time — and so the person writing the test does not have to be the person who wrote the agent.Tue, 25 Mar 2025 12:00:00 GMThttps://blog.kellerstein.org/blog/four-agents-in-four-weeks/https://blog.kellerstein.org/blog/four-agents-in-four-weeks/The old_agents/ directory still holds all the corpses. What each generation got wrong, and why most of a computer-use agent is not acting — it is checking whether the last action worked.Sun, 23 Mar 2025 12:00:00 GMThttps://blog.kellerstein.org/blog/bootstrapping-a-windows-guest/https://blog.kellerstein.org/blog/bootstrapping-a-windows-guest/No SSH, no cloud-init, no config management. Just a .bat file, an SMB share the hypervisor provides, and one Task Scheduler flag that decides whether your screenshots are black.Fri, 21 Mar 2025 12:00:00 GMThttps://blog.kellerstein.org/blog/omniparser-as-a-service/https://blog.kellerstein.org/blog/omniparser-as-a-service/Turning a screenshot into a numbered list of clickable things — YOLOv8 for detection, Florence-2 for captions, OCR for text. And why it had to be a server, not a library.Mon, 17 Mar 2025 12:00:00 GMThttps://blog.kellerstein.org/blog/every-vision-model-we-could-get/https://blog.kellerstein.org/blog/every-vision-model-we-could-get/Magma, Llama 3.2 Vision, Gemma 3, Phi-4, GPT-4o, Claude — including feeding a screen recording instead of stills. What actually moved the needle was not the model.Fri, 07 Mar 2025 12:00:00 GMThttps://blog.kellerstein.org/blog/watching-the-robot-work/https://blog.kellerstein.org/blog/watching-the-robot-work/The day we put noVNC in front of the guest, our mean time to diagnosis dropped by an order of magnitude. Failures in computer use are visual.Tue, 04 Mar 2025 12:00:00 GMThttps://blog.kellerstein.org/blog/qemu-in-docker-a-real-windows-vm/https://blog.kellerstein.org/blog/qemu-in-docker-a-real-windows-vm/It is not Windows in a container. It is QEMU in a container with a Windows ISO and an unattended answer file — and the difference is three lines of compose.yml.Mon, 24 Feb 2025 12:00:00 GMThttps://blog.kellerstein.org/blog/the-api-is-the-screen/https://blog.kellerstein.org/blog/the-api-is-the-screen/Why we built an agent that uses Microsoft Teams the way a person does — and why the model turned out to be the easy part.Sat, 22 Feb 2025 12:00:00 GMT