Re:Why Zapier Doesn't Use Gevent Yet
by obeattie on Wednesday January 13, 2016 @03:51AM
(#10893088)
The outbound dispatcher here seems like a prime candidate to write in a different language (Go for instance). Presumably it just makes HTTP requests so can be very simple -- building the request payload for instance could still happen in Python. And as you use Celery you already have a message queue over which it could communicate with the rest of your application.
Re:Why Zapier Doesn't Use Gevent Yet
by sametmax on Wednesday January 13, 2016 @05:38AM
(#10893440)
There is not need to overdo it. Asyncio + threads will do nicely. 500 req/seq is not the end of the world. I have similar load on purely threaded systems and they work fine.
Crossbar.io would help though, but is not even necessary.
Re:Why Zapier Doesn't Use Gevent Yet
by robgolding on Wednesday January 13, 2016 @06:19AM
(#10893556)
We have considered this actually, and I think it will be something we look into more as our scale increases. As you say, we already have the communication infrastructure in place via Celery/RabbitMQ.
Re:Why Zapier Doesn't Use Gevent Yet
by mateo411 on Wednesday January 13, 2016 @11:17AM
(#10895358)
I think that is what they are doing with celery. Each server has 92 celery workers. Each worker runs in its own process. Each worker takes tasks off of a queue and completes them.
Re:Why Zapier Doesn't Use Gevent Yet
by miseg on Wednesday January 13, 2016 @04:23AM
(#10893182)
As a complete tangeant, I find that Zapier's UI is what holds back my usage of it.
They added folders which is fantastic.
But editing a Zap is slow in the UI. I can't open a Zap into a new tab by Ctrl+Clicking the Zap title. Lots of Zap management is checking one zap's setup, and adding that configuration in turn to another zap.
Re:Why Zapier Doesn't Use Gevent Yet
by miseg on Wednesday January 13, 2016 @04:53AM
(#10893275)
When editing an existing zap (in Firefox anyway), and if there is a filter selected, and if it has a long name, then I can't see the full filter's name. (Sorry, I'm not logged in right now, so I can't see the UI's exact wording).
For example, if the source trigger is Sendowl. Add a filter to the zap, like product name. The name of the filter is longer than the drop-down list. When coming back to edit the zap, I can't see what trigger is selected.
Re:Why Zapier Doesn't Use Gevent Yet
by miseg on Wednesday January 13, 2016 @04:59AM
(#10893293)
After editing a zap in Firefox, and I click on the save button at the bottom of the page, something nothing happens. It's like it only gets focus with my first click. So I have to click a second time for the save button to go and save the zap.
Sorry for this type of report that's probably difficult to reproduce.
Re:Why Zapier Doesn't Use Gevent Yet
by miseg on Wednesday January 13, 2016 @04:50AM
(#10893268)
If I create a zap in a folder, I'd normally expect to be brought back to that folder, not to see the home screen again where folders are listed. I want to be able to visually see my new zap listed.
Re:Why Zapier Doesn't Use Gevent Yet
by robgolding on Wednesday January 13, 2016 @06:16AM
(#10893542)
We haven't evaluated RQ specifically, since RabbitMQ is a core part of our infrastructure and we're really happy with it. We have experienced the pain of managing Celery, and considered alternatives a few times though! A post about our experiences there is on the list to write! :)
Re:Why Zapier Doesn't Use Gevent Yet
by sandGorgon on Wednesday January 13, 2016 @10:16AM
(#10894864)
Ahh... For being on Rabbitmq, I suppose there are few other options. Without going into a blog post, why not Redis? Other than fairly sophisticated message routing... Any other reason?
Re:Why Zapier Doesn't Use Gevent Yet
by agopaul on Wednesday January 13, 2016 @08:01AM
(#10893941)
I think he was referring to global variables inside the 3rd party libraries rather than their codebase. If a library is not thread-safe they cannot use it.
Crossbar.io would help though, but is not even necessary.
They added folders which is fantastic.
But editing a Zap is slow in the UI. I can't open a Zap into a new tab by Ctrl+Clicking the Zap title. Lots of Zap management is checking one zap's setup, and adding that configuration in turn to another zap.
For example, if the source trigger is Sendowl. Add a filter to the zap, like product name. The name of the filter is longer than the drop-down list. When coming back to edit the zap, I can't see what trigger is selected.
Sorry for this type of report that's probably difficult to reproduce.
what have been your experiences/protips ? And what's your opinion on something like RQ
Um. What? Why do you have global variables for your celery tasks? They should be idempotent and small.
Also, even your http requests should be globally stateless as that is the nature of http.