News for hackers. Stuff that matters.
Re:Microservices and the First Law of Distributed Objects by superuser2 on Wednesday August 13, 2014 @01:19PM (#8173982)
Docker is a promising building block for solving logging and service management and that's something I've been playing with for a while. You stream your logs to your container's stdout/stderr and can pick them up from the Docker REST API. You can also hit the Docker REST APIs across your hosts to discover the IP addresses and ports of the containers running various images. None of this is "solved" yet but it's getting there.

Serialization... depends, but you can get pretty far with JSON over HTTP.

So I guess you do need your microservices to be uniform in some respects, but the consistency of "every microservice is a Docker container exposing a REST API that speaks JSON and streams logs to stdout" would seem to be enough.

It's easy to build that in several languages... if Rails makes sense, use Rails. If Sinatra makes sense, use Sinatra. If Java makes sense, use Java.

You could run into trouble if you start using languages that only one or two of your staff members know, but it does allow for some flexibility (i.e. just because certain parts of the application are enterprisey doesn't mean everything has to be.)

Re:Microservices and the First Law of Distributed Objects by AdrianRossouw on Wednesday August 13, 2014 @01:46PM (#8174077)
Docker is currently switching to their own microservices library called libchan - https://github.com/docker/libchan

It is msgpack over SPDY+TLS. Every language that will want to talk to docker will need a libchan implementation eventually.

disclaimer: I'm the co-founder of the project to build the node.js version: https://github.com/graftjs/jschan

My actual goal is to pull this into the browser though, and figure out what we can do with that : https://github.com/GraftJS/graft

Until that is ready though, we have already built out our own microservices 'toolset' that does the [1] docker based deployment,

and a small library that handles the [2] REST API, and that handles the [3] logging to logstash/kibana

1. http://longshoreman.io

2. https://github.com/wayfin/waif

3. https://github.com/wayfin/dockstash

Re:Microservices and the First Law of Distributed Objects by PaulHoule on Wednesday August 13, 2014 @02:10PM (#8174221)
Hell yeah, latency matters and msgpack + SPDY beats the heck out of JSON/http.
Re:Microservices and the First Law of Distributed Objects by PaulHoule on Wednesday August 13, 2014 @02:09PM (#8174219)
I definitely like the direction that Docker is going.

JSON over HTTP sucks. I mean, it's OK if user experience doesn't matter, but if user experience does matter, then latency matters, and if latency matters you find pretty quickly that serialization/deserialization overhead is a big deal -- particularly in a microservices environment where an external request is going to mean a large number of microservice requests happen, which means data could be serialized/deserialized many times. If you're happy being in the middle of the pack, go on with JSON, but if you are playing to win you need something faster (preferably with a framework that means development as easy or easier than JSON)

With multiple languages you don't just have the problem of training but you have other complexity-multiplying problems.

For instance, one of the reasons why languages like Haskell are always a bridesmaid and never a bride is that they lack a complete "standard library". You need a urlencode() and decode function if you write web apps. If you write your own you may be more concerned with making it tail recursive than making it correct. I remember how the urldecode in Perl was busted for a decade and never got fixed, and that's a mainstream language. As flawed as PHP is, it took off because it had "good enough" implementations of everything web developers need in the stdlib.

In a microservices system there tend to be functions similar to urlencode that need to be replicated across the system, and if these are implemented over and over again in different languages it is inevitable that some or all implementations will be buggy. If you can centralize these things in one library everybody depends upon, however, life is easy and happy.

Re:Microservices and the First Law of Distributed Objects by dragonwriter on Wednesday August 13, 2014 @04:47PM (#8174933)
> For instance, one of the reasons why languages like Haskell are always a bridesmaid and never a bride is that they lack a complete "standard library". You need a urlencode() and decode function if you write web apps.

Okay:

  cabal install urlencoded
> If you write your own you may be more concerned with making it tail recursive than making it correct

You know, if there is one criticism I've never heard -- or expected to hear -- of Haskell programmers in general, its that they are overly concerned with optimizations over correctness.

Re:Microservices and the First Law of Distributed Objects by superuser2 on Wednesday August 13, 2014 @02:54PM (#8174443)
You don't need to use the same language to use the same serialization library, however. If the latency of HTTP/JSON is truly too high, there's Protocol Buffers, Thrift, and a host of others in that space.

HTTP/JSON is nice, however, for the wealth of tooling that already exists and the relative ease of standardizing API design with REST.

Re:Microservices and the First Law of Distributed Objects by PaulHoule on Wednesday August 13, 2014 @03:15PM (#8174536)
+1 for Thrift.

If you like the REST architecture it works just fine with the message body being Thrift or whatever. I think of the ElasticSearch API which lets you use Thrift or XML or JSON.

The interesting distinction between binary serialization formats is if the schema is separated from the data.

For instance, if both sides of the system know you are encoding a 24 bit color value you can send 3 bytes and that is it; coding and decoding can be very quick and even possibly done on a "zero copy" basis.

If you are using something like JSON you not only have the waste involved with converting "255" to an #FF byte, but you also have to embed the schema in the sense of "here is an array of three integers (which happen to be bytes)" or "here is key 'red' and value R,..."

Thus, JSON is not "schema-free", it is "schema embedded in the data" and this inevitably bulks up the data and slows down encoding and decoding. Yes, general-purpose compression eats some of the storage/network encoding overhead, but you'll get it even tighter if you eliminate that fat before you put it through the compressor.

Now, separating the schema from the data means you need to make both sides aware of it, which is why you need some strategy for handling this in a systematic way rather than hoping things will work out OK without a plan.

Re:Microservices and the First Law of Distributed Objects by kalkal on Wednesday August 13, 2014 @02:37PM (#8174355)
> With multiple languages you don't just have the problem of training but you have other complexity-multiplying problems.

> For instance, one of the reasons why languages like Haskell are always a bridesmaid and never a bride is that they lack a complete "standard library". You need a urlencode() and decode function if you write web apps. If you write your own you may be more concerned with making it tail recursive than making it correct. I remember how the urldecode in Perl was busted for a decade and never got fixed, and that's a mainstream language. As flawed as PHP is, it took off because it had "good enough" implementations of everything web developers need in the std lib.

I don't understand what you're saying here - are you arguing that we shouldn't use languages that don't have web functionality built into the std lib? What about web frameworks, e.g. Java + Spring?

Re:Microservices and the First Law of Distributed Objects by kyllo on Thursday August 14, 2014 @01:10AM (#8176187)
Haskell's libraries and packages are actually very well fleshed out at this point. That's not the reason for poor adoption. It really is just that it's academic and mathy and difficult to understand.
Re:Microservices and the First Law of Distributed Objects by PaulHoule on Wednesday August 13, 2014 @03:17PM (#8174546)
In Java you can find implementations of that stuff and will probably use it. If there is anything wrong it is that there are too many different implementations out there.

In the case of Haskell you ~might~ find implementation of that stuff but you'll probably think you're too smart to have to reuse somebody else's code and also be too smart to have to deal with the corner cases.

Re:Microservices and the First Law of Distributed Objects by dragonwriter on Wednesday August 13, 2014 @04:50PM (#8174953)
> In the case of Haskell you ~might~ find implementation of that stuff but you'll probably think you're too smart to have to reuse somebody else's code and also be too smart to have to deal with the corner cases.

What is there besides extreme language bigotry to suggest that Haskell programmers tend to think they are too smart to do either of these things? Haskell programmers use other people's code all the time. And Haskell programmers handle corner cases as much as any other programmers.

Re:Microservices and the First Law of Distributed Objects by sanderjd on Wednesday August 13, 2014 @04:36PM (#8174894)
Your criticism of Haskell here just doesn't have the ring of truth to me. Haskell has lots of good, modern frameworks and libraries for doing web-related things, which aren't difficult to use or any more corner-case-y than their corollaries in other languages.

I get the vibe that there is some underlying point you're attempting to make and that the Haskell-specific stuff is secondary. Maybe your point is that using a technology stack with unproven maturity in a given domain (in this case web application development) is riskier and likely more time-consuming than using the more common stacks for that domain. If so, then I agree. (But I'm also very appreciative of the early-movers who put in the time and effort to make immature ecosystems around nice technologies more mature; somebody has to do it!)

Re:Microservices and the First Law of Distributed Objects by codygman on Thursday August 14, 2014 @12:16AM (#8176091)
This is a silly comment, being that Haskell is very library and reuse oriented.
Re:Microservices and the First Law of Distributed Objects by angersock on Wednesday August 13, 2014 @05:43PM (#8175150)
I'm curious--what sort of workload are you looking at where JSON/HTTP is insufficient?

We do a lot of realtime signal streaming, and haven't had many problems. We'll be switching to a different format (or embedded format) to better handle some numerical pickiness on our end, but the fundamental transport is fine.

Re:Microservices and the First Law of Distributed Objects by XorNot on Thursday August 14, 2014 @04:29AM (#8176561)
I think 'streaming' is the operative difference there. If you're opening a connection and sending large amounts of JSON 1-way then header/encoding overhead is basically not a problem. But a 2-way RPC protocol is basically making a separate HTTP request each time. For log file messages you're going to be sending more header more object definitions and headers than data.

I'm curious how you're formatting your JSON too - some formats are going to be way lighter then others.

Re:Microservices and the First Law of Distributed Objects by angersock on Thursday August 14, 2014 @11:37AM (#8177965)
Ah, good point on the RPC stuff!

That suggests pretty much mandatory use of keep-alive headers in infrastructure services, no?

Re:Microservices and the First Law of Distributed Objects by jburwell on Wednesday August 13, 2014 @04:31PM (#8174873)
There are few issues with this approach. First, many security regulations require deterministic capture of audit-oriented log events (e.g. login failure/success, invalid access attempts, etc). Typically, you are required to provide a local spool to handle circumstances where a remote logging service is unavailable/overloaded. Simply logging to stdout is insufficient and hoping it gets picked by a process over HTTP is woefully inadequate in these circumstances. Second, as mentioned previously, HTTP headers will be larger than most logging events. Therefore, this approach likely doubles/triples (admittedly back of the napkin estimates) the data being transferred. rsyslog and its ilk learned this lesson long ago -- using a lightweight TCP protocol to minimize transfer overhead. Personally, I favor a locally spooling rsyslog instance in each container configured to push to a central logging service.

Besides the issues of implementation, log data needs to be analyzed. Standardizing on a logging library and associated configuration provides important consistency to use a common set of log analytics in alerting systems not to mention keeping system admins performing forensics operations sane. Finally, in addition to logging, there is service instrumentation which requires consistency to yield useful visualization and alerting for operations teams.

While Docker may allow permit a vast polyglot infrastructure, carefully choosing stacks deployed will have a tremendous impact on operational robustness. The greater the consistency across containers reduces the effort required to deploy and operate new services.

Re:Microservices and the First Law of Distributed Objects by superuser2 on Thursday August 14, 2014 @02:51AM (#8176349)
What you're attacking is not how Docker logging works.

The Docker daemon collects stdout/stderr from the beginning of each container's life, always, and stores it locally. You can then ask the Docker API for all the log events from a particular container ID at any time.

You would have one system that polls the Docker APIs on all your hosts and dumps the logs into the management system of your choice at some interval.

Log events are not going to HTTP live, you are not making an HTTP request per entry, and the event will be captured whether or not the network is available at the moment.

Re:Microservices and the First Law of Distributed Objects by bcoates on Wednesday August 13, 2014 @02:03PM (#8174186)
Why not SNMP? It seems to be the protocol everyone loves to hate but there's a ton of ecosystem around it and it directly solves the status/discovery case