# Token reduction is not cost reduction | PointFive Canonical: https://www.pointfive.co/blog/token-reduction-is-not-cost-reduction A PointFive study of Claude Code configurations found that reducing delivered tool-output tokens did not reliably reduce billed cost. By: Sarel Weinberger, Amir Hozez Published: 2026-08-06 [All articles](https://www.pointfive.co/blog) Sarel Weinberger & Amir Hozez PointFive Research August 6, 2026 3 min read Updated September 17, 2026 against [version 5 of the paper](https://arxiv.org/abs/2607.12161v5) . ## What the experiment measured PointFive ran four Claude Code configurations across 103 tasks, seven repositories, and three models. The campaign executed 2,908 runs; 2,848 were included in the primary analysis. | Configuration | Delivered tool-output token change | Pooled billed-cost change | | --- | --- | --- | | Unmodified Claude Code | Baseline | Baseline | | RTK v0.44.1 | -1.3% | -2.7% | | RTK-ML, PointFive experimental build | -38.4% | +6.8% | | Headroom v0.27.0 | Not measured | +48.4% | These are paired billed-cost changes against the baseline, not cost per successful task. RTK's pooled estimate was below zero, but its holdout-only confidence interval crossed zero. The paper reports the intervals, exclusions, and success metrics separately. ## What this means for an evaluation Token reduction alone is insufficient evidence of a lower bill. Compare provider charges and task outcomes across representative workloads. Include retries, caching, additional model calls, and any service fees. A useful experiment holds the task and acceptance criteria constant and checks whether the work still succeeds. ## Scope and disclosure The results apply to the tested configurations. They do not establish a universal savings ceiling or describe every compression tool. RTK-ML was an experimental PointFive build; it was not unmodified RTK. This is PointFive-authored research, not an independent evaluation of PointFive's products. ## Benchmark and data The [AI Efficiency Benchmark](https://github.com/PointFiveLabs/ai-efficiency-benchmark) contains the task manifests, retained campaign datasets, analysis scripts, and frozen methodology behind the study. It records provider-reported cost and checks task success with deterministic judges. Inspect the [campaign data](https://github.com/PointFiveLabs/ai-efficiency-benchmark/tree/main/deliverables) , [analysis scripts](https://github.com/PointFiveLabs/ai-efficiency-benchmark/tree/main/analysis) , and [methodology](https://github.com/PointFiveLabs/ai-efficiency-benchmark/tree/main/docs) . Re-analyzing the retained data is offline; running a new campaign incurs provider charges. Meet the [PointFive Labs team](https://www.pointfive.co/labs) and read [Introducing PointFive Labs](https://www.pointfive.co/blog/introducing-pointfive-labs) . ## Read the paper Read the version 5 paper on this page Source: [arXiv, version 5](https://arxiv.org/abs/2607.12161v5) . [Open the PDF](https://arxiv.org/pdf/2607.12161v5) . ## Read the evidence - [Version 5 paper and methodology](https://arxiv.org/abs/2607.12161v5) - [Results and confidence intervals, Table 8](https://arxiv.org/html/2607.12161v5#S7.T8) - [AI Efficiency Benchmark](https://github.com/PointFiveLabs/ai-efficiency-benchmark) - [Download the corrected research summary](https://www.pointfive.co/downloads/token-reduction-field-guide.pdf) ## Related reading [All articles](https://www.pointfive.co/blog) - ### [The fourth layer: why coding agents are about to become your biggest AI bill](https://www.pointfive.co/blog/the-fourth-layer-coding-agents-biggest-ai-bill) June 24, 2026 6 min read - ### [Best Tools to Track Claude Code, Cursor, and Copilot Spend in 2026](https://www.pointfive.co/blog/how-to-track-coding-agent-spend-across-your-team) October 9, 2026 9 min read - ### [Prompt-induced waste in coding agents](https://www.pointfive.co/blog/prompt-induced-waste-in-coding-agents) October 6, 2026 --- Source: the public page above. Product screenshots and illustrative interfaces are examples, not live customer data.