Our AI Tag Suggestions Analyzed One Resource Type, Every Time

We shipped an AI tag-suggestion feature, considered it done, and moved on. A few days later, reviewing a smaller follow-on fix against a real customer account instead of our own demo data, we found that on an account with hundreds of untagged resources, the AI had never once looked at twenty-plus of the resource types sitting in it. Not under-weighted. Never sampled.

This is the write-up, including the part where our own demo data was too small to ever have caught it.

What happened

The feature's job is simple: take an account's untagged resources, hand a batch of them to an LLM, and come back with suggested team / env / name tags a customer can review and apply. It had shipped, passed review, and worked correctly against the accounts we tested it on.

Those accounts were our demo environment's three seeded tenants — Nexus Digital, Meridian Health Systems, and Atlas Analytics Platform — each with a few dozen to about a hundred untagged resources. Real customer accounts run several times larger. When we checked the feature against one of those, on Nexus Digital's production AWS account, the candidate query had ordered everything alphabetically by resource name. With several hundred untagged resources, the first 100 in alphabetical order were 100% one resource type — CloudWatch log groups, because their names begin with a forward slash and a slash sorts before every letter. Two dozen other resource types on that same account — snapshots, buckets, security groups, clusters — were sitting right there and were never once handed to the model.

Why our own demo never showed it

This is the uncomfortable part to publish, so we are publishing it directly: nothing about this bug is subtle once you know where to look, and we had already shipped a related fix without finding it. Our demo accounts are deliberately small and clean — fast to seed, easy to reason about, good for showing the product. They are also 2.5 to 6.5 times smaller than a real account tends to be, and a 100-row alphabetical sample looks fine on a dataset with only 26 to 100 rows in total, because there usually isn't a 101st row to lose.

We only found it because we went and looked at a real account immediately after the related fix merged, specifically because the demo had felt too easy to validate against. That instinct — that a clean, synthetic dataset proves a feature works on the dataset, not on the problem — is the whole reason this post exists.

Bug one: alphabetical sampling

getUntaggedResources ordered its candidates ORDER BY resource_name. It is a completely reasonable default for a list a human scrolls through. It is a biased sample the instant something downstream only looks at the first page of it — which is exactly what the suggestion route did, capping the candidate set before handing it to the LLM.

The practical effect on Nexus Digital's account: a resource type that happened to sort first consumed the entire sample, every single time the feature ran, regardless of how little that type actually cost or how much of the untagged estate it represented.

Bug two: the batch that timed out

The suggestion route sent up to 100 resources' metadata to the LLM in one call. On Meridian Health Systems' Azure account, with roughly two hundred untagged resources, that batch exceeded our own 60-second timeout. Server logs confirmed it to the millisecond: a request that returned after 60,023ms with {"suggestions":[]} — an exact hit of the configured ceiling, not a slow response that happened to land near it.

The user-visible symptom was the worst kind: the panel loaded, showed no suggestions, and gave no indication that anything had gone wrong. A feature that fails loudly gets noticed. A feature that returns an empty, valid-looking response gets assumed to have nothing to say.

Bug three: a count that lied by omission

Both the display route and the suggestion route capped their result (at 100 and 50 respectively) and reported that capped number as if it were the total. An account with a couple hundred untagged resources would see "50 resources" with no hint that the true figure was several times larger. One of our own users looked at that number and said, in effect, "this makes me think there might be even more than that" — and was right, which is a bad way to find out your own UI is technically true and substantively misleading.

The fix

Three changes, all shipped together because they were one failure mode wearing three disguises:

• Cost-descending ordering, not alphabetical. The candidate query now orders by the resource's own cost, highest first — mirroring a pattern already proven in a sibling query rather than inventing a new one. The highest-dollar-impact resources get analyzed first, which is also the order a human would choose by hand.
• Smaller batches. The LLM batch shrank from 100 resources to 50, with headroom under the timeout instead of running against it.
• An honest count. Both routes now return the true total alongside the capped result, and the UI copy reads "Showing the 50 highest-cost of 214" instead of a bare number that happened to also be the cap.

Re-run against Nexus Digital's real production account after the fix, the top 50 cost-ordered candidates spanned eight distinct resource types — snapshots, volumes, buckets, a cluster or two — instead of one. That spread, not a specific count, is the actual proof the fix works: a sample that reflects the estate instead of its sort order.

The near-miss a final review caught

The part of this we almost shipped broken is the part worth telling on ourselves hardest.

Suggestions are cached for 24 hours, keyed only by which credential they came from — with no record of which candidate query produced them. Changing the candidate ordering and shrinking the batch size, as the fix above did, meant a cache written under the old alphabetical, 100-wide query no longer matched the new cost-ordered, 50-wide one. We checked a real cached row against the new logic directly: 100 cached suggestions, filtered against the new candidate set, zero survived.

Deployed as first written, that is not a bug that shows up in a test — it is an empty panel for up to 24 hours on exactly the accounts the fix was meant to help, because the cache would return "nothing new to suggest" instead of falling through to a fresh call. We only caught it because the last step before shipping was reviewing the whole change as one diff, not task by task — a stale cache intersecting to zero against a changed query is invisible from inside any single commit and obvious the moment you look at before-and-after side by side.

The fix treats an empty post-filter intersection, when candidates genuinely exist, as a stale cache rather than a real "nothing to suggest" — it falls through to a fresh LLM call exactly like a true cache miss would. We also cleared the existing poisoned cache entries on both environments at deploy time rather than waiting out the 24-hour window.

What to ask your own tooling

This bug has a shape that generalizes past us: any feature that samples a subset of a large collection before doing expensive work on it can be silently biased by whatever ordering the sample happens to inherit, and a small test dataset is often too small to ever expose the bias. Two questions worth asking about anything that works this way, in your own stack or anyone else's:

• What order is the sample taken in, and does that order correlate with anything that matters? Alphabetical, insertion-order, and "whatever the database returned" all correlate with something, and it is rarely the thing you'd choose on purpose.
• If the collection is bigger than the sample, does the UI say so? A capped count that looks like a total is a quiet way to make an incomplete answer feel complete.

We found this one by taking a feature we had already called finished and pointing it at an account big enough to disagree with us. That is still the best test we know.