I can never understand why enforcing (sane and secure) TLS and adding a (shared) secret request header isn't the usual simple answer.

As far as I know, TLS prevents all these security issues, except authentication of the webhook caller, hence the shared secret which your webhooks should check.

I've verified in the TLS specs that every issue I've seen raised by people is actually covered, yet people still think (and write) that TLS doesn't sign traffic (it does), or TLS doesn't prevent replay attacks (it does), etc.

Usually I give them pointers to the specs, and they say "it doesn't hurt to add even more security" but they're really rolling out their own encryption, which you shouldn't do.

I'd say enforce TLS and authenticate the webhook caller...

I'm the developer of Diahook (we do webhooks as a service), and I agree. It's just so annoying that we have to implement signing ourselves instead of just using client TLS certificates. Though the problem is that there are so many places in the stack that interact with your request before it hits user code (load balancer, web server, etc), that getting anything like this to work is wishful thinking...

TLS with certificate pinning could probably even take care of the authentication issue.

Hey HN - I'm the author of this post. (Founder of WorkOS)

Lots of people have written about how to scale event-based systems or how to design distributed infrastructure for queue management. But what's often overlooked is the subtle-yet-important product decisions that go into designing webhooks for a great developer experience.

We don't actually offer webhooks-as-a-service at WorkOS, but we provide webhooks to developers and I've built this feature twice now, previously at Nylas and now at WorkOS. Hopefully this article helps the next person who builds them. :)

For anyone curious how we do it at WorkOS, here's our webhooks API reference: https://workos.com/docs/reference/webhooks/connection

SSRF is a tremendous problem with webhooks. Stripe also solves this: https://fly.io/blog/practical-smokescreen-sanitizing-your-ou...

Agreed, I should add something on this. (Haven't updated the article since posting it many months ago.)

This is a pretty good article about preventing SSRF including DNS rebinding-based attacks in Go https://www.agwa.name/blog/post/preventing_server_side_reque...

Yeah, it's one of the thing that people miss when implementing webhooks naively, and I bet it's one of the things people miss when consuming them!

Something else Stripe and most other "serious" webhook senders do is only send Webhooks from a predefined set of IP addresses [1].

This isn't the security guarantee that signing, etc. are but it's a substantial help in adding some Security In Depth.

1 - https://stripe.com/docs/ips

Maybe a stupid question but... why not using an MQTT server instead of webhooks? Clients could be notified in RT, TLS is already backed in the protocol with client auth... the client doesn't need to expose an endpoint, only to connect to a server...

AWS EventBridge announced API destinations recently [1]. I can't find anywhere in their docs about SSRF, but given it's a managed service running outside of your network it sounds like a good option. Anyone care to comment if this would be a good solution for webhooks?

1 - https://aws.amazon.com/blogs/compute/using-api-destinations-...

Hehe, just did a show HN[1] for Diahook (webhooks as a service) earlier today. I wonder if that's what triggered this posting. :P

Anyhow, I wish more people had better webhooks, it's such an important part of any API, so please read this blog post, understand the challenges, and provide a great webhooks experience!

[1] https://news.ycombinator.com/item?id=26399672

Wow Diahook looks great!

My company is in the process of updating our webhooks, so we will be sure to check it out.

Thanks! Feel free to reach out tom @ the domain. :)

Kind of wild that there's no mention of SSRF. A quick search shows it's a pretty frequent security issue in Webhooks: https://www.google.com/search?q=ssrf+webhook

This would be a good addition: make sure that your HTTP library POSTing the webhook won't follow redirects and is blocked from accessing internal domains.

Not just logging the outbound webhooks to STDOUT and hope they get ingested into logstash or something, but actually log them in your SoR/RDBMS so that you can present that data to a debugging developer.

And make sure to impose a max size. Getting a 1GB gzip bomb persisted to your DB is never fun.

This was a great read – thanks for sharing!

I've always been a huge fan of how Stripe handles their webhooks. They are pretty much doing everything listed in your article.

What people totally omit when thinking about webhooks is the parallels to web sockets and client side updates. In the last round of our system updates, we have "attached" the web sockets and webhooks to the same message bus which has simplified the overall design of the system.

Yeah, they share a lot of the same challenges as they are both unlike the rest of the web stack (which is just request -> response). There are a lot of similarities, but a lot of different challenges too.

At FastComments we send an your API token in the webhook request, and if your API doesn't return a 401 with an invalid token the integration won't activate.

I can see why you might do that, but it seems mildly hostile to me and would have broken things for me on at least one or two occasions. I know I’ve set up very simple hook arrangements before where I don’t care about authentication, at least in part because the URL is secret and it’s probably just about dumping the hook call contents for now anyway.

But at the same time, perhaps doing this would solve more problems than it creates.

It's not hostile. It's ensuring you properly authenticate the request, and that you own the target endpoint. :)

This was an interesting read! If you ever decide to update this article & include tools for working with webhooks, Hookdeck(https://hookdeck.io) should be at the top of the list. Hookdeck is a complete webhook infrastructure that makes working with webhooks painless. we pride ourselves in reliability. We provide tooling for easy monitoring and troubleshooting of webhooks

democratize access to webhooks -- love dat energy