Re:RPC over AMQP
by nitrogen on Friday June 13, 2014 @12:50AM
(#7887491)
The real issue is that most of the time is spent marshalling all the data into YAML
Using MessagePack would speed that up significantly, too. I've done a comparison of MessagePack vs. JSON vs. YAML vs. Ruby .marshal in the past[0], and I'm currently building a system using AMQP+MessagePack (as of now the only part that's public is [1]).
Basically, YAML is great for configuration files because of its flexible syntax, but terrible for communication because of its flexible syntax.
Re:RPC over AMQP
by hydrogen18 on Friday June 13, 2014 @09:20AM
(#7888640)
You're absolutely right. The performance and security implications of YAML make it an awful choice.
Unfortunately, upgrading the server component is trivial but upgrading all the clients is next to impossible. The logical next step after this project is to implement a saner set of endpoints that breaks up all the amalgamated functions. At some point in the future, there will come a day in which we'll simply ignore the requests from older clients.
Not only is it faster than the serializations you mentioned, messages generally have a much smaller memory footprint, and it is therefore very friendly for use in contexts where bandwidth is likely to be at a premium (e.g. mobile carrier data limits). I use it exclusively in mobile apps I develop.
Re:RPC over AMQP
by Cieplak on Thursday June 12, 2014 @11:43PM
(#7887350)
AMQP is so nice from an engineer's perspective. If shit goes south somewhere in the system, I get a pagerduty alerting me that a deferred queue has exceeded some depth and I can start working to resolve the issue (if a message fails to be processed it is sent to the deferred queue). Once the issue is resolved, all the messages are still waiting in the deferred queue and will get processed as soon as they are requeued, and I don't have to replay nginx logs. Anything that can be asynchronous, make asynchronous and save yourself the headache of dealing with 500s when a component of your distributed system fails.
Re:RPC over AMQP
by illumen on Friday June 13, 2014 @02:53AM
(#7887682)
So to speed up a request, the author wants to introduce a whole message queue, and rewrite it in another language. This seems like a big maintenance burden to me.
Could the request be changed to use a faster message format instead of YAML?
If YAML can't be replaced, could the YAML marshalling be sped up?
Re:RPC over AMQP
by jbert on Friday June 13, 2014 @03:42AM
(#7887780)
I'm a little curious why in this case one wouldn't choose to do "RPC over HTTP".
Assuming the following points from the post are correct:
a) the perf problem is YAML marshalling
b) that can be solved via a go rewrite
then perhaps an easy way to solve "I want to call the old code" is to access it over HTTP, using a cheaper marshalling method than YAML.
I guess the tradeoff is whether you'd prefer an AMQP broker in there mediating the requests or an HTTP load balancer (or http client pool code in the go app). And whether you see other uses for that broker in your setup.
Re:RPC over AMQP
by cakoose on Friday June 13, 2014 @01:11AM
(#7887523)
You can make it so that finding a free slot is always O(1). Keep another array that is a stack of available slot indexes. When you need a free slot, pop off an index and use that slot. When you're done with a slot, push its index on the stack.
Re:RPC over AMQP
by noname123 on Friday June 13, 2014 @02:59AM
(#7887696)
Why not use zmq (http://zeromq.org/bindings:ruby)? And for data serialization use Google Protobuf's port ruby-protobuf; and avoid introducing a central message broker as another point of failure?
Re:RPC over AMQP
by hydrogen18 on Friday June 13, 2014 @09:23AM
(#7888657)
I've used zmq (or is it 0mq) in the past.
My complaint with 0mq is that you wind up needing some sort of service discovery service from what I can tell. In order to get everything working, each client needs to know about at least one servant. Ideally, each client needs to about all servants and use a load balancing algorithm across them. I'd need to keep this configuration at all clients synchronized. I'm aware there are tools to do this, but ultimately I am not the person in charge of operating this.
Using MessagePack would speed that up significantly, too. I've done a comparison of MessagePack vs. JSON vs. YAML vs. Ruby .marshal in the past[0], and I'm currently building a system using AMQP+MessagePack (as of now the only part that's public is [1]).
Basically, YAML is great for configuration files because of its flexible syntax, but terrible for communication because of its flexible syntax.
[0] http://blog.mikebourgeous.com/2012/10/12/optimizing-message-... (missing numbers for MessagePack, but mentions 2x speed up beyond best other option)
[1] https://github.com/deseretbook/classy_hash (includes a benchmark script that shows how ridiculously fast MessagePack is compared to YAML)
Unfortunately, upgrading the server component is trivial but upgrading all the clients is next to impossible. The logical next step after this project is to implement a saner set of endpoints that breaks up all the amalgamated functions. At some point in the future, there will come a day in which we'll simply ignore the requests from older clients.
Could the request be changed to use a faster message format instead of YAML?
If YAML can't be replaced, could the YAML marshalling be sped up?
For sure... for example, by rewriting it in another language?
Assuming the following points from the post are correct:
a) the perf problem is YAML marshalling b) that can be solved via a go rewrite
then perhaps an easy way to solve "I want to call the old code" is to access it over HTTP, using a cheaper marshalling method than YAML.
I guess the tradeoff is whether you'd prefer an AMQP broker in there mediating the requests or an HTTP load balancer (or http client pool code in the go app). And whether you see other uses for that broker in your setup.
The documentation describes how it works here: http://docs.spring.io/spring-amqp/reference/htmlsingle/#remo...
My complaint with 0mq is that you wind up needing some sort of service discovery service from what I can tell. In order to get everything working, each client needs to know about at least one servant. Ideally, each client needs to about all servants and use a load balancing algorithm across them. I'd need to keep this configuration at all clients synchronized. I'm aware there are tools to do this, but ultimately I am not the person in charge of operating this.
The project isn't complete but is functional enough to be used in production.