Launch HN: Convoy (YC W22) – Open-source cloud-native webhooks service
Some time ago I built an API for local fintech companies alongside Emmanuel. While trying to sell this product, we were asked several times for webhooks. We didn't find a great tool that was easy to use, language-agnostic and cloud-native; something that could scale independently of our core service. We decided to build it ourselves.
Webhooks are the glue of the modern internet. They are used for asynchronous communication by API providers to notify apps/integrators about events that have occurred in their account. I like to say it as: if you make a payment on Shopify with Stripe’s checkout and Stripe fails to send a webhook event to Shopify’s servers about your payment, you won’t get a delivery of your package even though your card has been debited. This explains the importance of webhook events to many consumer apps.
Today, implementing a webhooks delivery system is a highly fragmented problem. Stripe's system is designed for security (with features like replay attacks, rolling secrets etc), Twilio for performance (with features like connection override), Pagerduty for flexibility (with features like payload filtering) and many more. We'd love to make these features available to everyone with Convoy. We're already helping a few companies solve problems they didn't know they had because of the visibility Convoy gives.
API providers need to push webhook events reliably. It’s a pain to build this from scratch. At first glance, it seems like good old HTTP POST, but with time it becomes more tricky as the requirements pile up: dealing with bad endpoints, pushing events to multiple endpoints as configured by your user, rate-limiting of customer endpoints, security of delivery, UX for managing retries. We take care of all that and give you one less infrastructure to worry about.
Convoy is open-source software that exposes a REST API. Developers push events to it using the REST API (we aim to support other means of ingesting data in the future for high volume environments – this is a feature Shopify provides today) then we sign the payload, publish the events, apply retries, etc. We also provide a management UI to manually retry events to user endpoints.
Convoy consists of 3 core components—in addition to the REST API, there is a job queue and a storage layer. The REST API server is used to create applications, endpoints and events. The job queue currently supports 2 backends: in-memory and Redis. The storage layer currently supports 2 backends: on-disk and MongoDB. All events pushed to Convoy are saved to storage and enqueued on the job queue for workers to deliver.
Most of our users use our product to push webhook events to their customers’ endpoints. Customers also use Convoy as a broker for inter-service communication.
We make money by providing a managed service. We've been running the managed service for a while now and charge $1/5k events, but those details are not on our website yet.
The repository is available at https://github.com/frain-dev/convoy and a getting started doc exists here: https://getconvoy.io/docs/guide. Anyone can download the binary or Docker image from here: https://getconvoy.io/download and run it wherever they choose - AWS, Azure, GCP, etc.
We’d love you to try it out and hear your thoughts! We’d also love to hear about your experiences building custom webhook software and the challenges you’ve faced?
1) For something specifically advertised "cloud-native" I would have hoped to see direct integration to e.g. EventBridge/SNS/SQS (or their GCP/Azure equivalents) instead of plain REST API.
2) It is bit weird that you need both Redis and MongoDB here. Is there particular reason why redis alone would not be sufficient?
3) I couldn't at a glance find any docs on the API for receiving the webhooks beyond that small ruby example snippet.
4) You clearly are aware of the security things Stripe has done with their webhook implementation, so it is surprising that you have fair bit simpler model from the looks of it
Some specific security aspects that I noted:
a) Not having some sort of keyids or multiple signatures makes key rotation probably annoying
b) Keys are application-wide instead of per-endpoint
c) Only shared-secret available, no public/private keypair option?
d) No timestamping for replay protection?
Not trying to be too negative about your solution, but its just that I have been thinking exactly these problems myself then they are kinda on top of my head.
There is a draft IETF specification for request signing, that seems relevant here. Maybe you could adopt it in Convoy too? https://datatracker.ietf.org/doc/draft-ietf-httpbis-message-...
1) Yes, it's on our roadmap.
2) It was/is a design choice, we aren't so familiar with using redis as a persistence store. The goal was to enabled people switch backends depending on what they're comfortable with.
3) Thanks for bringing this up, there are a ton of things we should document.
4) We're working on improving things from a security perspective.
On application-wide keys, we're considering making this configurable, so you can choose what works best for you, we only started with app-wide keys, next is endpoint-keys. :)
On public/private keypair, this is new to me as well! we haven't found anyone doing this in prod. But yes, I can see how this can be useful in specific environments, definitely something we'd like to support.
I’d definitely appreciate a Postgres options for the queue backend. We currently have an “everything in Postgres” approach for operational simplicity. It works great, and it would fantastic if we could slot this in there too.
We have it planned for a storage backend but not for queueing at the moment.
Yes, I am putting this on the roadmap! :)
What we care deeply about today is simplifying and democrastising webhooks with Convoy. Under the MPL 2.0 License as I understand it, any modifications to Convoy must be made open-source as well which helps us reach our initial goal.
On building a viable business, we currently run a managed infrastructure, where users pay for our infrastructure and other collaborative features (team membership, rbac etc) to make it easier to work with the core.
Nevertheless, we are open to transitioning to a stricter license in the future that restricts other cloud platforms from competing with us with our software.
It's a great question, and there's no one definitive answer. Some context: I've been an open source developer for all my life, my previous business was also open source, and I spoke with a lot of people over the last year about the right license for us.
The main choices are: copyleft like AGPL (you want AGPL, not GPL for server applications), permissive like MIT (which we chose[0]), or so called "fair-code" which are the not-quite-open-source variants like the SSPL.
While AGPL and SSPL are "better for business", MIT is more aligned with our mission, and is better for fostering a community. You can read more about the thinking on our Show HN[1].
Either way though, all of what the stricter licenses do is make it harder for competitors to use your code in their own service. Which is important, but not the end of the world as competitors can always just copy your API, landing page, and everything else.
[0] https://github.com/svix/svix-webhooks
[1] https://news.ycombinator.com/item?id=30347858
Do you have any plans to support PostgreSQL or an S3 compatible target as storage backends?
Yes, we have plans to support PostgreSQL as a storage backend. At the moment, queue tuning parameters are not exposed as configuration parameters, they're based off a shared queue and we expose prometheus metrics over the shared queue. What use-cases do you have for tuning the queue?
So essentially Convoy's queue mechanism works as a delayed job queue (basically how retries in webhooks work e.g. Stripe will send the event to you every hour for the next 72 hours). When the endpoint is unaccessible it backs up and schedules the retries at a future time on the queue.
The implementation under the hood is a combination of redis zset + redis streams. :) When it maxs out retries the events are still stored in the DB and you can batch retry them once the endpoints become accessible.
https://github.com/svix/svix-webhooks
We're designing a tool to be flexible and simplify webhooks from a binary perspective. We already support multiple queuing & storage backends, we already support an in-memory queue and on-disk database, so you can run convoy as a single binary without third-party dependencies.
The thesis is if we can simplify webhooks like this, we can do more with a tool like this vs. just for API providers.
Although, I learned they recently open sourced some parts of their backend, but I think that's how we're fundamentally different.
I don't know if I'd call us SaaS first, though we definitely spend a lot of time, money and attention making sure our SaaS offering is as reliable as it gets if that's what you mean.
FWIW, I've been an OSS developer and maintainer for all my life (my previous business was also open source), and open sourcing Svix was always the plan. It was literally one of the first tickets I opened for the team, and our first public commit was in Feb 2021.
We've indeed recently announced the open sourcing of our webhooks dispatcher[0]. It's written in Rust, so also a single binary with no third party dependencies. We do NOT however offer in memory storage. This is on purpose, as persistence is a requirement for robustness.
[0] https://news.ycombinator.com/item?id=30347858
Oooo shots fired. :)
I can see arguments both ways. You're totally correct in that queue robustness requires persistence. That being said, if it's an open source tool then it can make sense to offer in memory storage since they may have persistence in a different part of the stack.
Also, even if the persistence is elsewhere in the stack, it's not obvious to me how disaster recovery will actually work. Same with accounting (e.g. marking a message as sent). I'm not saying it's not possible, it's just not immediately obvious to me how you can make that both simple and robust.
> We've indeed recently announced the open sourcing of our webhooks dispatcher
So which one came first? SaaS or OSS?
Also how are you different than Convoy?
> Http Failure Response For Https://Saas-Api.Getconvoy.Io/Api/V1/Users: 400 OK
Looking at the HTTP response in DevTools shows the following error:
> Validation failed: Password Password should be minimum of eight characters, including at least 1 alphabet and one number
The password should be valid according to the error message.
If you can please paste the error here, so we can look into it.
Convoy is largely OSS based. With a simple brew install you can publish events with Convoy from your local machine. :)
Very cool product! But this part threw me off a bit, wouldn't a distributed message broker like Kafka be a much better option for this use-case?
Avoid blasting too much events to a single endpoint at once with flexible rate limiting controls
I would write this as:
Avoid blasting too many events
PostHog, Supabase, Wasmer, Mattermost, Chatwoot and of course Svix, just to name a few!
YC investment in OSS companies have grown considerably in the last years. They are doing a terrific work identifying companies with potential on the space. I'm incredibly hopeful seeing where things are heading :)
Edit: 2021-09-06, most likely: https://whois.domaintools.com/getconvoy.io