Four Things That Broke: Shipping an Agent Tool on Public Data

There is a version of this post where I describe a tidy agent pipeline, show one screenshot, and let you assume it went smoothly. That post would be useless to you, so this is the other one.

The tool is real and you can use it right now: Tender Scout takes a company website, works out what that company sells and where, scans every public tender published in Germany in the last fourteen days, and hands back the ones worth a look with the reason, the deadline and a link to the original notice. Four agents, two model calls, about eight seconds. The data comes from the federal Datenservice Öffentlicher Einkauf, published under CC0, roughly a thousand notices a day.

The agents were the easy part. Four things broke between "works on my machine" and "works for a stranger", and not one of them was the model being stupid. Each was caught by a different kind of check, and that is the part worth your time.

The shape of the thing

Two of the four steps never call a model at all. That matters for cost, for speed, and for how much of the result you can explain to a client afterwards.

Break one: the provider quietly changed the deal

Every model call started coming back with too many requests. Not a spike at a busy moment: every call, all day, at a volume of about one request per minute. A rate limit that fires at one request per minute is not a rate limit.

The first useful move was not to read my own code. It was to call a second thing that shares the same key, the chatbot that has been sitting on this site for months:

curl -s -X POST https://www.halacli.com/api/ask \
  -H 'content-type: application/json' --data '{"question":"hi"}'

{"ok":false,"error":"I'm getting a lot of questions right now — try again in a minute."}

That one command moved the fault out of the new code and into the account, and it took twenty seconds. The provider had changed its free plan; free traffic now runs on leftover capacity. Nothing in my code was wrong, and no amount of retrying would have fixed it.

So the repair was not "find a better provider". It was to stop having exactly one. Every model call on the site now goes through one small function that walks a list until something answers.

Break two: a 400 that was not a 400

With the new provider wired in, small requests worked and real ones failed with a plain 400. A 400 means your request is malformed, so I went hunting for a malformed request: the JSON mode, the token cap, the headers. All fine. I would have kept holding that wrong idea for another hour, because the provider's explanation never reached me. My own code logged it and returned a tidy sentence to the browser.

The cheap repair was to make failure say more. Every attempt now records which engine was tried and what it answered, and that path travels back on the response:

curl -sD - -X POST https://www.halacli.com/api/tenders/profile \
  -H 'content-type: application/json' --data '{"url":"https://www.materna.de"}' -o /dev/null

x-engines: groq:400, mistral:429

With the provider's own words in the log, the real cause appeared, and it had nothing to do with malformed requests:

{"error":{"message":"Failed to validate JSON. Please adjust your prompt.
See 'failed_generation' for more details.","type":"invalid_request_error"}}

The model was answering correctly and my output cap was cutting it off mid-object. The provider validated the truncated JSON, found it broken, and blamed the request. A bigger cap and an instruction to keep fields short fixed it in one line each.

The general lesson is not about this provider. It is that an error message you swallow is an hour you will spend later. Errors should name the component and quote the upstream, at least into your logs.

Break three: the edge undid my compression

The tender index is stored compressed, because it is about a megabyte. The endpoint sent those bytes with a header saying so. That is correct, and it worked in every test I ran.

In a real browser, against the deployed site, it failed:

await (await fetch('/api/tenders')).json()
// SyntaxError: Unexpected token '\u001f'

Byte 0x1f is the first byte of a gzip file. The browser was being handed compressed bytes and told they were plain JSON, because the network in between had its own opinion about that header and did not pass it through as sent.

The fix was to stop being clever: unpack the data in the function and let the network compress it on the way out, the way it wants to. The lasting fix is different and cost nothing: one check that uses the deployed URL the way a stranger would. Unit tests cannot see a CDN. Nor can they see a cached copy of a wrong answer, which is the other thing I learned that afternoon, when the corrected response kept being served from an hour-old cache until I gave the cache key a version.

Break four: half the data was missing a field

The results looked right and read wrong: many of them said Unnamed buyer. My parser looked in one place for the buyer's name, guided by a day of sample files. The official format allows two layouts, and a large share of real notices use the other one.

# what I wrote, from reading one day of samples
buyer = root.find(".//{*}Organizations/{*}Organization")

# what 48% of a real day actually uses
buyer = root.find("./{*}ContractingParty/{*}Party")

Six lines of fallback took the gap from 48% of notices to under 1%. Nothing was wrong with the code, the tests, or the model. The sample was wrong, and no amount of testing against that sample would ever have said so. The only thing that catches this is counting the real data:

curl -s https://www.halacli.com/api/tenders | python3 -c "
import sys, json
n = json.load(sys.stdin)['notices']
print(sum(1 for x in n if not x['b']), 'of', len(n), 'without a buyer name')"

# before:  2,965 of 6,158 without a buyer name
# after:       1 of 6,099 without a buyer name

What caught what

Four failures, four different kinds of check, and the pattern is the point: none of them was a machine reviewing its own work.

What broke	What found it	What would never have found it
Provider changed its free plan	Calling a second system that shared the key	Any test of this codebase
Truncated JSON reported as a bad request	Making the error name the engine and quote the upstream	Reading the code again
Compression header rewritten in transit	Using the deployed URL in a real browser	Unit tests, all of which passed
Buyer name absent in half the records	Counting the field across the whole live dataset	Tests built from the same sample

What it looks like when it works

This is a real run, unedited, against a mid-sized German IT services firm. Two model calls, eight seconds, ten results — each with the buyer, the deadline, the category and a link to the original notice.

The editable chips under the profile matter more than they look. If the model misreads the company, the visitor removes a category and the shortlist recomputes instantly, with no further model call. Correcting a machine should not cost the user a round trip.

The part I would actually sell you

Agents wrote most of this code. Agents wrote most of the tests. The tests passed.

Not one of the four failures above was caught by an agent checking its own work, and none of them would have been caught by a stricter prompt or a smarter model. They were caught by comparing against a second system, by making failures say more, by using the deployed thing like a stranger, and by counting the real data.

That is the division of labour I bring to teams: let machines do every mechanical check, tirelessly — the linters, the scanners, the pinned dependencies, the gates in the pipeline (the blast-radius post is the long version) — and keep the four checks above in human hands, because each one requires knowing what the system is for. If you want to see the same idea pointed at your own code instead of mine, the Repo Reality Check reads a public repository and tells you which of these habits it can actually find evidence for, and the AI Coding Risk Audit does the same for a team's working practices.

---

The Bottom Line: The agents were never the risk. The risk lives in everything between the agent and the user — a provider's terms, an error that tells you the wrong story, a header a network refuses to honour, and a public data format with two spellings of the same field. None of that is fixed by a better prompt, and all of it is fixed by someone who still checks.

Try it: Tender Scout. The tender data comes from the Datenservice Öffentlicher Einkauf under CC0; contact details are stripped before anything is stored, and nothing you type is kept.