We Found $456 of Cloud Cost That Did Not Exist

On 8 September 2026 we found $456.21 of cloud cost in our own data that did not exist. Not a mis-attribution — money that had been counted twice, against different resources, for as long as the feature had been running.

This is the write-up, including the parts that make us look slow.

What happened

A cloud bill arrives as a set of service-level totals: so much for Lambda, so much for Glue. Some of it also arrives broken down per resource, because the provider can identify which function or which database incurred the charge. Much of it does not, because the provider simply does not publish that breakdown for everything it bills you for.

So a cost platform does two things. Where the provider gives a per-resource figure, it uses it. Where the provider does not, it models one — typically by taking the service's total and distributing it across the resources of that service.

Our distribution step was splitting the service's gross total. It never subtracted what the provider had already attributed directly.

So for any service where some resources had a real per-resource figure and others did not, the dollars already explained by the first group stayed in the pool and were spread across the second. The same money, attributed twice, to different resources.

Why nothing broke, which is the interesting part

This was never a double write. The allocator has always kept a set of resources that already carry a figure and excluded them from the distribution, so no single resource was ever given two costs. There was no duplicate row, no constraint violation, no conflicting update.

The per-resource arithmetic was also correct throughout: take a 30-day pool, divide by the number of resources, divide by 30, write one day. Every step of that is right.

Only the numerator was wrong. And a wrong numerator produces no error, no log line and no failing test — it produces a plausible number, slightly high, on every screen that reads it.

That is the shape of the expensive bugs in cost tooling. They do not crash. They agree with themselves.

Glue: the clearest case

AWS Glue is where this stopped being abstract.

Over the 30 days to 8 September, Glue billed $98.72. Cost Explorer had attributed $98.72 of it to named resources — 100% of the service, to the cent. There was nothing left to estimate. The answer was complete.

Our model then distributed a further $50.68 across the Glue databases that happened not to have a direct row.

Five of them each carried exactly $8.15. Identical amounts are the signature of an even split, not of measured cost — and had anyone looked at that table with the bill beside it, the shape would have given it away immediately. Nobody had, because nothing was asking them to.

The four services, measured

Over the same 30 days, on one production account:

Lambda is the worst in proportion: Cost Explorer had attributed 98.8% of it directly, and the model added another $339.66 on top of that.

ECS is the one worth understanding, because it looks fine and is the same bug. Only $99.70 of $978.91 had been attributed directly, so there genuinely was a large pool left to distribute — and the model distributed $894.51 of it, which is close to right. It is 1.7% over rather than 87% over purely because of how little the provider had already explained. The defect was identical; the visible symptom depended entirely on the shape of the service.

That is why this could not have been found by looking at whether the numbers seemed reasonable. Three of the four seemed reasonable.

The fix, and the one thing it deliberately refuses to count

The pool is now net: the service's total minus what direct and calculated allocation already attributed to that service's resource types, over the same window, clamped so it can never go below zero. A service the provider attributed in full leaves nothing to distribute, which is the correct outcome and was previously the exact moment we invented money.

The subtraction counts two of the three allocation tiers and deliberately not the third: it never subtracts a previous run of the distribution itself.

The reason is worth stating because the alternative is seductive and wrong. If yesterday's estimate counted as "already attributed", today's pool would be smaller, and tomorrow's smaller again. The estimator would quietly shrink itself to zero over a couple of weeks, no error, nothing on any screen saying so — a feature that switches itself off while continuing to report success. We have shipped that class of failure before, and the whole point of a rule is to be applied to the case that resembles the one you got wrong.

Services with nothing left to distribute are now logged by name, with both figures. A pass that stops producing rows and a pass that never ran look identical in a log that says nothing.

The arithmetic is pinned by a test, and the test was verified the only way a test is worth anything: by reverting the fix and confirming it fails, then restoring it and confirming it passes. A test written after a fix and never seen to fail is a test that proves nothing.

Cleaning what was already written

Fixing the code stops new rows being wrong. It does nothing about the ones already in the database, because per-resource cost rows for past dates are never rewritten.

$456.21 across 1,889 rows, corrected. Each row was scaled by a single factor per service and day, bringing each day's total down to that day's net pool. Scaled rather than deleted, because the allocator's own distribution was an even split and scaling an even split by a constant is still an even split — deleting outright would have destroyed roughly $885 of reasonable ECS, SQS, Secrets Manager and ECR estimate in order to remove about $37 of phantom.

Glue and Load Balancing scaled to exactly zero and their rows were deleted. The provider had explained both services in full.

Every row was written to a backup file first, an audit entry recording the rule, the window and the dollar total by service was written before the first statement rather than after it, and every corrected row now carries its scale factor and its original amount. A cost correction you cannot reverse is a second bug waiting to be found.

Why a tool that blends billed and modelled cost cannot find this

Here is the part that generalises beyond us.

This bug is only visible if you can ask the question "how much of this service did the provider already explain?" — which requires having kept the answer. A platform that stores one cost per resource per day, with no record of whether that figure came from the provider's billing or from its own model, has thrown away the only evidence that makes the discrepancy detectable.

It would not be a worse-engineered platform for it. It would be a platform whose data model cannot represent the question. And the failure mode is not that it reports an error — it is that it reports a number, confidently, forever.

We caught this because every per-resource cost row in our system records its own provenance: billed, calculated, proportionally allocated. That distinction exists to be shown to customers, which is the reason we built it. Catching our own arithmetic was a side effect, and it is the second time in a month that it has paid for itself.

The uncomfortable corollary: we do not know how many tools have this bug, and neither do they.

What to ask your own tooling

One question, and it takes a minute:

For a given month, what share of my bill do you attribute to specific resources from the provider's own per-resource billing, and what share from your own model?

Three possible answers, and all three are informative:

• A number, split by basis. Good. Now compare the per-service totals against the service-level bill and check nothing exceeds 100%.
• "All of it is real billing." Almost certainly not true — none of the three major providers publishes per-resource cost for everything they charge for. Worth finding out what the answer becomes under pressure.
• No answer. That is not a worse tool than one that answers. It is a tool that cannot be audited, which is a different property and a more expensive one.

For what our own coverage looks like, measured and split by basis, see Multi-Cloud Cost Management: Why Cost Accuracy Matters — including the section on which of our numbers are the provider's and which are ours.