- engineering
- integrations
- debugging
I lost an afternoon once to a call that worked perfectly on my machine and returned a permission error in production. Same code, same request, same headers. The only difference was where it ran from.
The service in question fingerprints the network connection itself. Not the user agent, not the headers, not anything I was sending — the shape of the connection. Which meant no amount of local testing could ever have found it, because from my machine everything was fine, forever.
The class of bug this belongs to
Anything that touches a third party can fail for reasons that have nothing to do with your code:
- Where the request came from. Data centre address ranges get treated differently from home connections.
- Whether the platform recognises your runtime.
- Rate limits that only apply once you are past a volume you never hit locally.
- Certificates and time. A clock that is slightly wrong breaks things in ways that look like authentication bugs.
- Regional blocks that are invisible from where you happen to be sitting.
Your local machine is not a small version of production. It is a different environment with a different identity, and for anything involving another company's servers, identity is most of what determines the answer.
The worst part is the symptom
If these failures announced themselves it would be fine. They do not.
The typical presentation is silence. A status indicator that never updates. A number that stays at zero. A feature that appears to be working because nothing on the screen says otherwise. Errors get caught and logged somewhere nobody is reading, and the page renders its empty state, which looks exactly like "there is nothing to show" rather than "this has been broken since Tuesday".
What to do instead
- Test integrations in the deployed environment, from the first day, not at the end. Ship a route that does nothing but call the third party and print the raw result. Look at it on the real domain.
- Make failure loud. An empty state and an error state must never look the same. If the fetch failed, say so on the page, even if only to yourself.
- Log the outbound call and the raw response, not just your parsed version.
- When something works locally and not in production, stop debugging the code. The code is the same. Debug the environment.
- Have a plan for moving the call somewhere else. Sometimes the only fix is to make the request from a different kind of host, and if that is possible, it is much cheaper than fighting the block.
The general rule
Anything that depends on another company's servers has to be verified where it will actually run, on the real domain, at least once, deliberately.
"It worked when I tested it" is only true if you tested it in the place it lives.
← All writing