Finally, you’re ready to launch, you’ve integrated the API, you’ve decoded the documentation, you have passing integration tests, and FAT passed against their test server, with their test data. You go live. And everything breaks.
Strawman testing
Sometimes the test data or the API out of sync with live, and you find you’ve been testing the wrong thing. Maybe the test server is a version ahead of the live API (if they’ve ignored the advice not to version HTTP APIs).
More frustrating is when the data is plain wrong. When the returned record has { Firstname = “Lincoln”, Lastname = “Abraham” }, or an invalid postcode (did you know, The final two letters do not use the letters CIKMOV) which means the test site may require a postcode that your validator will reject.
Tests doomed to succeed
Sometimes the test data is correct, but incomplete. Imagine a postcode lookup where test addresses are all SW… which means you can’t test for:
- Flat x/y addresses
- BFPO addresses
- Irish or other EU addresses
Or imagine a payment processor test stub that always returns success – so you can’t test for declined cards, or 3DS cards, or check that you only accept debit cards.
If your API supports it, put it in your test, and make it available to users for their tests. If it’s not tested, it’s not complete.
Craig Nicol 