News for hackers. Stuff that matters.
Re:Mocking Is an Anti-Pattern by whack on Monday June 17, 2024 @01:04PM (#40708041)
> If you're writing a custom frontend for GitHub using the GitHub API and don't bother writing a decent set of mocks for how you expect the GitHub API to behave, your app will quickly require either full manual QA at best or become untestable at worst. Some APIs are very stable, and testing against the API itself can hit rate limiting, bans, and other anti-abuse mechanisms that introduce all kinds of instability to your test suite.

I've been doing E2E testing using 3rd-party APIs for a decade now, and this has yet to be a significant problem. The majority of my APIs had a dedicated sandbox environment to avoid "rate limiting, bans, and other anti-abuse mechanisms". The remainder were simple enough that the provider didn't care about users exploring on the live API, and were usually read-only as well.

Did I run into the occasional flaky failure, or API stability issues? Sure. But it was very rare and easy to workaround. It never devolved into becoming "untestable" or "full manual QA"

My other teams that relied on mocks suffered from far worse problems - a ton of time being spent on manual-QA, and bugs that leaked into production, because of mock-reality mismatches.

Re:Mocking Is an Anti-Pattern by jph00 on Monday June 17, 2024 @03:08PM (#40709752)
I have a custom frontend for GitHub using the GitHub API (https://github.com/fastai/ghapi/) and don't use mocks - I test using the real API. I've had very occasional, but not enough to ever cause any real issues.

I don't find mocks for this kind of thing very helpful, because what you're really testing for are things like changes to how an API changes over time -- you need real API calls to see this.

Re:Mocking Is an Anti-Pattern by simonw on Monday June 17, 2024 @02:26PM (#40709209)
Outside of the payments industry I haven't encountered many sandbox APIs that don't have rate-limits, what are some good ones you've seen of those?
Re:Mocking Is an Anti-Pattern by chii on Monday June 17, 2024 @01:30PM (#40708415)
> because of mock-reality mismatches.

you also need to test the mocks against the real thing separately.

Re:Mocking Is an Anti-Pattern by switchbak on Monday June 17, 2024 @01:44PM (#40708626)
I would prefer that we all try to use this language consistently: https://www.martinfowler.com/articles/mocksArentStubs.html

What you're describing sounds like a fake to me.

Re:Mocking Is an Anti-Pattern by drewcoo on Monday June 17, 2024 @02:01PM (#40708886)
Can't tell from the context that's here.

It's important to have a contract testing layer in place to make sure your test doubles are still behaving like the real thing, though.

Re:Mocking Is an Anti-Pattern by ffsm8 on Monday June 17, 2024 @04:16PM (#40710570)
Contract testing is still a thing?

I thought interested mostly fizzled out over the years after the initial hype died down

Re:Mocking Is an Anti-Pattern by Too on Tuesday June 18, 2024 @01:27AM (#40714324)
Good luck with that. Any org I’ve seen can’t even consistently agree what is a unit test, component test, black box test vs integration test.
Re:Mocking Is an Anti-Pattern by hot_gril on Monday June 17, 2024 @04:58PM (#40710990)
Yeah, even if there's no sandbox mode, a separate sandbox account will usually do. Sometimes this catches misuse that would've caused rate-limiting in prod. And if a service makes this hard, maybe you shouldn't use it in prod either.
Re:Mocking Is an Anti-Pattern by ljm on Monday June 17, 2024 @05:14PM (#40711143)
There are plenty of libraries out there, like VCR, that can set up a test and then save the response for future test runs. You don't really have to renew them that often either.

That was always the go-to for me when testing against 3rd party services, especially because the tests would then survive the offboarding of the engineer who set them up with their personal credentials.

If your test suite relies on live Github PATs or user-specific OAuth access tokens, then you can either figure out how to manage some kind of service account with a 'bot' user, or live with things breaking every time someone leaves the org.

Services that incur a per-request charge, or consume account credits, are another problem. Especially if they don't have sandboxes.

Re:Mocking Is an Anti-Pattern by KingOfCoders on Tuesday June 18, 2024 @07:19AM (#40716404)
Same here, I think Customer.IO had the nices sandbox experience yet.