News for hackers. Stuff that matters.
Re:Goodbye Microservices: From 100s of problem children to 1... by NickBusey on Tuesday July 10, 2018 @01:38PM (#17500385)
Right, but the thing that makes Linux actually useful isn't really the kernel is it? I would say what makes it useful is all the various small, targeted programs (some might call them microservices) it lets you interact with to solve real world problems.

If Linux tried to be an entire computing system all in one code base, (sed, vim, grep, top, etc., etc.) what do you think that would look like code base/maintainability wise? Sounds like a nightmare to me.

Re:Goodbye Microservices: From 100s of problem children to 1... by joshvm on Tuesday July 10, 2018 @03:15PM (#17501225)
One example I'm familiar with that sounds like microservices is the Robot Operating System (ROS). At its heart it's just a framework for pub/sub over IP, it just happens to be targeted towards robotics. A ROS system comprises of 'nodes' for each logical operation e.g. image acquisition -> camera calibration -> analysis -> output.

The system is defined by a graph of these nodes, since you can pipe messages wherever they're needed; all nodes can be many-many. Each node is a self-contained application which communicates to a master node via tcp/ip (like most pub/sub systems a master node is requried to tell nodes where to send messages). So you can do cool stuff like have lots of seprate networked computers all talking to each other (fairly) easily.

It works pretty well and once you've got a particular node stable - e.g. the node that acquires images - you don't need to touch it. If you need to refactor or bugfix, you only edit that code. If you need to test new things, you can just drop them into an existing system because there's separation between the code (e.g. you just tell the system what your new node will publish/subscribe and it'll do the rest).

There is definitely a feeling of duct tape and glue, since you're often using nodes made by lots of different people, some of which are maintainted, others aren't, different naming conventions, etc. However, I think that's just because ROS is designed to be as generic as possible, rather than a side effect of it running like a microservice.

Re:Goodbye Microservices: From 100s of problem children to 1... by blt on Tuesday July 10, 2018 @09:07PM (#17503628)
IMO, the main benefit of ROS is the message definitions. Having exactly one definition of "point cloud" that everybody accepts means everyone's code will be compatible. That isn't normally the case in library ecosystems. If ROS was replaced by <robotics_types.hpp> I think we'd get 90% of the benefit.
Re:Goodbye Microservices: From 100s of problem children to 1... by chrisco255 on Tuesday July 10, 2018 @04:00PM (#17501549)
There's actually no reason you can't architect microservices like this. You can put RabbitMQ or some AMQP service as a comm layer between services. But then you have to architect your system to be event-driven. It's not a bad approach.
Re:Goodbye Microservices: From 100s of problem children to 1... by thermodynthrway on Tuesday July 10, 2018 @05:09PM (#17502194)
I've worked on a system that used Kafka in-between. It makes consistency issues a ton worse because everything is async. At least with HTTP you can do synchronous calls
Re:Goodbye Microservices: From 100s of problem children to 1... by bsder on Tuesday July 10, 2018 @06:44PM (#17502856)
Repeat after me: there is no such thing as a synchronous network.

I get where you are coming from. But you will go async sooner or later if you need any reasonable error recovery or reliability.

The only question is how much pain you will suffer before you do so.

Re:Goodbye Microservices: From 100s of problem children to 1... by thermodynthrway on Tuesday July 10, 2018 @07:24PM (#17503088)
This is just not true. Google uses almost entirely immediately consistent databases and gRPC internally. Eventual consistency is hardly ever required for REST type calls even at massive scale.

gRPC has no queuing and the connection is held open until the call returns. All of Google's cloud databases are immediately consistent for most operations

Re:Goodbye Microservices: From 100s of problem children to 1... by jacques_chester on Tuesday July 10, 2018 @07:53PM (#17503268)
My understanding is that Spanner is optimistically immediately consistent, with very clever logic for deciding on whether to bounce the attempted transaction (TrueTime).

But strictly speaking, even inside a single multicore CPU, there is no such thing as immediate consistency. The universe doesn't allow you to update information in two places simultaneously. You can only propagate at the speed of light.

Oh, and the concept of "simultaneous" is suspect too.

Our hardware cousins have striven mightily and mostly successfully for decades to create the illusion that code runs in a Newtonian universe. But it is very much a relativistic one.

Re:Goodbye Microservices: From 100s of problem children to 1... by rcxdude on Wednesday July 11, 2018 @06:42AM (#17505855)
ROS is not a particularly good implementation of this concept though. And it tends to encourage extremely fine splitting of jobs, which basically just turns your robotics problem into a distributed robotics problem, making it much harder in the process (as well as less efficient).
Re:Goodbye Microservices: From 100s of problem children to 1... by lmm on Tuesday July 10, 2018 @01:48PM (#17500487)
> If Linux tried to be an entire computing system all in one code base, (sed, vim, grep, top, etc., etc.) what do you think that would look like code base/maintainability wise?

You mean like BSD?

Re:Goodbye Microservices: From 100s of problem children to 1... by boramalper on Tuesday July 10, 2018 @01:57PM (#17500567)
Same with BSD; the userspace programs are not part of the kernel, and developed separately by the same entity.

Busybox is a slightly better example (even though it’s also a userspace program).

Re:Goodbye Microservices: From 100s of problem children to 1... by jcelerier on Tuesday July 10, 2018 @03:34PM (#17501356)
> Same with BSD; the userspace programs are not part of the kernel, and developed separately by the same entity.

they're part of the same repo and built at the same time than the kernel. Run a big "make universe" here : https://github.com/freebsd/freebsd and see for yourself. That they are different binaries does not matter a lot, it's just a question of user interface. See for instance busybox where all the userspace is in a single binary.

Re:Goodbye Microservices: From 100s of problem children to 1... by zerd on Tuesday July 17, 2018 @03:42PM (#17552572)
I would argue that the fact that they're in the same repo does not make it a monolith at all. All of google and facebook is in a single repo and built at the same time, but it doesn't make them monoliths.
Re:Goodbye Microservices: From 100s of problem children to 1... by apeace on Tuesday July 10, 2018 @01:59PM (#17500586)
My understanding is that OpenBSD is constructed this way, and I have also heard that their code is well-organized and easy to follow.

There is the "base" system which is the OS itself and common packages (all the ones you mentioned), then there is the "ports" repo which contains many open-source applications with patches to make them work with OpenBSD.

Here is a Github mirror of their repos: https://github.com/openbsd

I think OpenBSD has reaped many of the same advantages described by Segment with their monorepo approach, such as easily being able to add pledge[1] to many ports relatively quickly.

https://www.slideshare.net/GiovanniBechis/pledge-in-openbsd

Re:Goodbye Microservices: From 100s of problem children to 1... by fastball on Tuesday July 10, 2018 @02:07PM (#17500668)
"packages" and "ports" are both collections of "microservices" in your analogy, so you can't really assert the whole thing is a monolith.
Re:Goodbye Microservices: From 100s of problem children to 1... by lmm on Wednesday July 11, 2018 @06:55AM (#17505922)
The whole thing is released and deployed together, which is the usual distinction between microservices and not. I don't think anyone's advocating not using modules and folders to structure your code in your repository (certainly I hope not), but a single deployment artifact makes life a lot easier.
Re:Goodbye Microservices: From 100s of problem children to 1... by blablabla123 on Tuesday July 10, 2018 @01:58PM (#17500578)
But there is a big difference. These small targeted programs are invoked in user land, usually by the user. Microservices get invoked directly by the user when debugging is going on. Otherwise they are expected to automagically talk to each other and depending on the abstraction even discovery each other automatically.

Also I can pipe these tools together from the same terminal session, like

  tail -f foo | grep something | awk ...
You don't have that in general with Microservices. Unix tools are Lego, Microservices aren't. They are Domino at best.

Probably one could come up with an abstraction to do Lego with Microservices but we're not there yet.

Re:Goodbye Microservices: From 100s of problem children to 1... by paxys on Tuesday July 10, 2018 @02:29PM (#17500880)
Look at the package dependency tree of an average linux program. They are absolutely examples of "microservices" talking to each other.
Re:Goodbye Microservices: From 100s of problem children to 1... by digi_owl on Tuesday July 10, 2018 @03:09PM (#17501176)
And they can be a right mess to wrangle because upstream gets into some artistic frenzy, and shouts down anyone that worries about the ensuing breakages as luddites and haters.
Re:Goodbye Microservices: From 100s of problem children to 1... by wsy on Tuesday July 10, 2018 @04:55PM (#17502068)
And still nobody would argue to pack everything together into one executable because of these issues. As everything in software engineering, it is about finding the right trade-off.
Re:Goodbye Microservices: From 100s of problem children to 1... by digi_owl on Tuesday July 10, 2018 @05:17PM (#17502266)
Oh, but why do that when we have containers...
Re:Goodbye Microservices: From 100s of problem children to 1... by lmm on Wednesday July 11, 2018 @06:53AM (#17505910)
The big problems with microservices come from distributed transactions being difficult and clocks getting out of sync. Multiple services on a single machine don't have that problem.
Re:Goodbye Microservices: From 100s of problem children to 1... by jacques_chester on Tuesday July 10, 2018 @07:55PM (#17503284)
Re:Goodbye Microservices: From 100s of problem children to 1... by thermodynthrway on Tuesday July 10, 2018 @01:53PM (#17500536)
With web apps the main concern is data consistency between relations. On the OS level you have these same concerns with memory and disk, and there's database-like systems in the kernel and drivers to handle it. Essentially all these utilities are running within the same "database" which is disk and memory management handled by the kernel. Usually microservices have their own databases, which is where consistency hell begins
Re:Goodbye Microservices: From 100s of problem children to 1... by kolpa on Tuesday July 10, 2018 @02:01PM (#17500605)
> Usually microservices have their own databases

That's news to me, and seems insane.

Unless you mean "their own database tables", not "database servers". But that's just the same as having multiple directories and files in a Unix filesystem.

Re:Goodbye Microservices: From 100s of problem children to 1... by jsight on Tuesday July 10, 2018 @02:26PM (#17500851)
To be blunt, its news to a lot of people, but it also isn't wrong. Microservices really shouldn't share a database, and if they do then they aren't "microservices".
Re:Goodbye Microservices: From 100s of problem children to 1... by copperx on Tuesday July 10, 2018 @02:57PM (#17501082)
What is that called, then? Having small programs, each responding to a different part of the API, all storing data in a single shared DB?
Re:Goodbye Microservices: From 100s of problem children to 1... by ljw1001 on Tuesday July 10, 2018 @03:26PM (#17501296)
(a) a prototype (b) a set of applications that share a database.

You would have to have an oddly disconnected schema if modifications to the program don't result in programs accessing parts of the database that other programs are already accessing. If this isn't a problem it means you're using your database as nature intended and letting it provide a language-neutral, shared repository with transactional and consistency guarantees.

so maybe not microservices, but fine nonetheless.

EDIT: two more comments:

- this is exactly what relational databases were designed for. If people can't do this with their micro-services, maybe their choice of database is the issue.

- "micro-service" as the original post suggests, is not synonymous with good. "monolith" is only synonymous with bad because it got run-over by the hype-train. If you have something that works well, be happy. Most people don't.

Re:Goodbye Microservices: From 100s of problem children to 1... by jsight on Wednesday July 11, 2018 @10:27AM (#17507176)
I think you are exactly right. Microservice architectures are definitely not automatically good, and there is nothing wrong with a well architected "monolith".
Re:Goodbye Microservices: From 100s of problem children to 1... by matrix on Tuesday July 10, 2018 @03:28PM (#17501310)
It's called: a distributed monolith
Re:Goodbye Microservices: From 100s of problem children to 1... by Beldur on Tuesday July 10, 2018 @04:54PM (#17502061)
Distributed Monolith
Re:Goodbye Microservices: From 100s of problem children to 1... by erik_seaberg on Tuesday July 10, 2018 @02:26PM (#17500857)
If you're going to go microservices, you want service A to use service B's public API (MQ or RPC or whatever), not to quietly depend on the schema B happened to choose. And sharing a database server instance turns overloads into cascading failures, unless the stack is very good at enforcing resource limits on noisy neighbors.
Re:Goodbye Microservices: From 100s of problem children to 1... by VBprogrammer on Tuesday July 10, 2018 @02:22PM (#17500813)
That architecture is commonly known as a distributed monolith. If you put two services on the same DB you can guarantee that someone will be joining onto a table they shouldn't have before the week is out.
Re:Goodbye Microservices: From 100s of problem children to 1... by ljw1001 on Tuesday July 10, 2018 @03:42PM (#17501414)
This is another way of saying that micro-services are for easy problems, where you can avoid having many processes share the same complex database.
Re:Goodbye Microservices: From 100s of problem children to 1... by wsy on Tuesday July 10, 2018 @05:02PM (#17502120)
It's exactly the other way around:

If you can afford all your components sharing the same database without creating a big dependency hell, then your problem is _too small_ for microservices.

If your problem is so large that you have to split it up to manage its complexity, start considering microservices (it might still not be the right option for you).

Re:Goodbye Microservices: From 100s of problem children to 1... by ljw1001 on Tuesday July 10, 2018 @05:12PM (#17502224)
like all those simple airline reservation and banking systems?
Re:Goodbye Microservices: From 100s of problem children to 1... by wsy on Tuesday July 10, 2018 @05:37PM (#17502414)
Yes, they do fine as examples. I have worked both on decades old booking systems and banking software. Both were dependency hells. Refactoring was impossible. Everybody was super-careful even with tiny changes because the risk to break something was just too high.

If it could be avoided, these systems were not touched anymore. Instead, other applications where attached to the front and sides.

Re:Goodbye Microservices: From 100s of problem children to 1... by collyw on Wednesday July 11, 2018 @03:27AM (#17505071)
Isn't that a feature of old systems in general?
Re:Goodbye Microservices: From 100s of problem children to 1... by wsy on Wednesday July 11, 2018 @08:34AM (#17506340)
I don't think so. For example, the Linux kernel is old, but still quite well maintainable.

So I would say: it applies to systems where proper modularization was neglected. In the anecdotical cases I referred to, one major element of this deficiency was a complex database, shared across the whole system.

Re:Goodbye Microservices: From 100s of problem children to 1... by WJW on Tuesday July 10, 2018 @03:52PM (#17501479)
Not to mention the endless fun that can be had when one service changes the database schema without telling the others!
Re:Goodbye Microservices: From 100s of problem children to 1... by y4mi on Tuesday July 10, 2018 @04:30PM (#17501861)
It's neither new nor insane... Putting everything on the same server becomes a scalability nightmare at some point.

This video is strangely relevant in this thread... https://www.youtube.com/watch?v=X0tjziAQfNQ

Re:Goodbye Microservices: From 100s of problem children to 1... by thermodynthrway on Tuesday July 10, 2018 @05:11PM (#17502214)
Nah, just use a database that can scale with your app and run a ton of instances. You'll be hard pressed to find any size app that doesn't fit in RAM these days.
Re:Goodbye Microservices: From 100s of problem children to 1... by kgwgk on Tuesday July 10, 2018 @02:33PM (#17500899)
Between “database servers” and “database tables” there is another level of granularity: “database users”.
Re:Goodbye Microservices: From 100s of problem children to 1... by VBprogrammer on Tuesday July 10, 2018 @02:44PM (#17500980)
While that's true. It also means you are only one grant away from sharing database tables. Maybe with good discipline you will be ok but all it take is for one dev to take that one shortcut.
Re:Goodbye Microservices: From 100s of problem children to 1... by user68858788 on Tuesday July 10, 2018 @03:59PM (#17501545)
Agreed. I've only seen proper user separation happen when legal gets involved. Usually it's an all-access free-for-all, and after all, why shouldn't it be? It's a lot easier to pull data and logs from adjacent services yourself rather than taking a ticket and hoping the team still exists for a response.
Re:Goodbye Microservices: From 100s of problem children to 1... by ahoka on Wednesday July 11, 2018 @07:59AM (#17506194)
If you have different database servers then you are only a "connection string" away from breaking your separation. Using a different login and schema is perfectly fine in a lot of architectures.
Re:Goodbye Microservices: From 100s of problem children to 1... by dvlsg on Tuesday July 10, 2018 @04:03PM (#17501577)
Separating by schema would work, too. At least, they would be separated from each other but still on the same database.
Re:Goodbye Microservices: From 100s of problem children to 1... by inetknght on Wednesday July 11, 2018 @09:52AM (#17506850)
> I would say what makes it useful is all the various small, targeted programs

You mean GNU coreutils?

Re:Goodbye Microservices: From 100s of problem children to 1... by mirceal on Tuesday July 10, 2018 @04:06PM (#17501607)
Linux is the kernel.