If this were Medium, I'd highlight the hell out of that.

That's so true, and so nicely, succinctly put - it ought to be the reply to end every argument about whether microservices are good or bad.

Meanwhile, the rest of our codebase (a monolith) crashed every few days for one reason or another. We had an on-call rotation not because that's what you're supposed to do, but because we actually needed it.

Now I'm not saying that microservices make sense for everyone. In general, I agree that they are used incorrectly. Microservices are hot and software developers, generally speaking, like to use hot technologies. Yes, moving to a microservice was costly. We had to re-write a lot of code, we had to set up our own servers, and we had to get permission from the guardians that be to do all of this. But, for our use case, and I assume there are other use cases too, the benefits of detaching ourselves from the company's monolithic codebase far outweighed the costs for doing so.

TL;DR No argument is the end to every conversation. Few things are so black and white.

Sooner or later you get a feel for which bits are becoming at least API stable and could run independently. That's when I split them out.

Do it too soon and you end up choosing the wrong boundaries and tying yourself up in knots, do it too late and your monolith can become a mess that's difficult to detach the pieces of.

I don't go too far with this, just avoid things like shared static state and other anti-patterns.

It does assume that you either start with devs familiar with OTP or you have generalist devs that can pick things up quickly.

Could you expand on how the architecture actually looked? What made it a monolith and what made it micro serviced?

You reap the scaling benefits way later, if ever.

When you need to scale, then you can easily throw your libraries behind an RPC framework and call it microservices, but there's no need to pay that cost until you actually face that problem.

That doesn't mean you can't scale up from a monolith (even one without clean interfaces) - every startup growth story is a testament otherwise - but it's never as as easy as strapping an RPC layer over your library.

This is a real problem - I recall that Sanjay Ghemawat et al was working on it when I left Google, though I dunno if the solution they came up with is public yet. It's unlikely to seriously affect you unless you're Google-scale, though, by which time you've probably divided everything up into services and aren't taking advice from the Internet anyway. For companies that are a few teams working on a single product, it's easy enough to send a company-wide e-mail saying "Rebuild & redeploy anything that depends upon library X", and if you're doing continuous deployment or deploy only as a single artifact, the problem never affects you anyway.

For starters the vast majority of code is developed in-house in non-software companies. The vast number of products are a single team working essentially in a silo, not "a few teams working on a single product".

When people are talking about small companies, it's misleading to think "smaller than Google". Smaller-than-Google is still an enormous quantity of development. Enterprisy practices make much sense in scaling software in companies that are smaller-than-smaller-than-Google. if you hear "small company", think multiple steps further from that, a smaller-than-smaller-than-smaller-than-smaller-than-smaller-than-Google company.

Sure, but if you do that in place it will still break stuff that assumes it works like the last version, and if you do a versioned API or the like you still can't force all teams to adopt the new version.

Does your CI system not automatically build dependent artifacts--

> It's unlikely to seriously affect you unless you're Google-scale, though

--okay, whew. ;)

Of course, there are other limitations this imposes, but it does make it very simple to deploy a new library to all code which uses it.

You can deploy a new version of the dll and applications can pick it up when they restart. Linux will apply security patches this way.

Microservices are only a step ahead.

That said, in many cases the cost of a full restart can be accepted.

It helps cognitive load because such apps can be reasoned about without reading code elsewhere.

Except splitting into microservices is an unnecessarily complex design choice. That's almost always worse, and the cognitive load comes in when you now need to figure out how to get this stuff right. The scaling benefits also require that you get it right, small flaws in your system become massive issues.

It can be as simple as a simple class, or maybe a larger class as a single-file service, or an entire namespace with a several classes, or a separate library easily referenced. All the "logic" split benefits without the ridiculous hassle of microservices.

However, exercising proper software discipline and using languages with good/existent module systems, like OCaml or Go, can lead to the same modular results without the fixed overhead. If you don't have a full-time ops person or team, you almost always have no business running microservices.

This too!

Relational databases are great, but I spent large parts of my life as a developer writing layers converting to/from SQL and later ORMs. There's a huge gain in just not translating data. I know Postgres (and others) deal with JSON, but I can't escape the feeling it's a bit shoe horned in there – basic SQL statements have strange new operators like ->> -> #>>.

Relational databases are great for, well, relational data with strong consistency requirements. The popularity of the original MyISAM tables without integrity checks baffled me at the time. Why spend time marshalling data in/out of table form when you don't gain the benefits of a RDBM?

Not doing data translation saves _a lot_ of time. Plain key-value stores are amazing, document stores like Elasticsearch are great, ultimately the choice comes down to requirements and time saving is often a very heavy argument, especially for small companies/startups.

I never found data translation/serialization to be a big pain (just rely on a framework/lib that does it for you). It's a bigger pain to hand-roll joins that would be a one-liner in SQL or deal with issues that arise from having your data (unnecessarily) reside in many systems.

> Things such as joins, transactions, and means of enforcing data integrity are useful

If the domain needs transactions I'm already speccing them regardless of general usefulness. Yes all that stuff is useful, but all abstractions have a cost which isn't free just because it's hidden in the DB.

For a problem that didn't need transactions, but for which they were useful, why would you automatically want to couple the solution with your storage layer? If you're looking for the ability to express business logic clearly without cluttering it with error handling, for instance, software transaction memory would probably be a better level to work at.

Considering we speak about daemon software (services exposing some API to readers and writers) that provides CRUD behavior to the user (end-user or other developer), isn't that nearly always the case to guarantee write access to concurrent writers without the risk of crippling your data?

Furthermore I am not sure how this relates to FP.

> It's much easier and safer to declare your constraints rather than trying to enforce them.

But that is a strong point of RDBMs implementing SQL. You have some kind of schema (think type) and use selected functions (select, update, delete, create, etc.) to transform the data.

> Furthermore I am not sure how this relates to FP.

I'm saying that FP is great for ensuring correctness.

> But that is a strong point of RDBMs implementing SQL.

Right. And if I didn't have other functional languages available that might be a bigger issue.

I'd definitely use a trusted DB for storing bank accounts. The consistency is pretty much the first requirement and the data maps perfectly to tables. And 7B checking accounts isn't that big, compared to some problems so it'd probably scale pretty well even worst-case.

But I probably wouldn't for an MMO. Or at least, it wouldn't be where I stored every little thing going on around them - just the events (xp and gold earned) that they'd freak out about if we lost. But even just a log-structured DB would work well for that.

If there's no contention for a resource (in the bank case - the value in the account) there's much less reason for a transaction. I want the system to make its best effort but I don't want to wait around for the message if there isn't anything I can do on a failure anyways.

I start with text files and for most purposes I do not bother with anything else.

Next is a key/value store. Simple.

Relational databases carry large overhead in translating data (as above) and also in design and maintenance of the structure and getting data into and out of them. I spent many years with them, like them a lot, but they are too much complication for most purposes.

Even with relational data RDBMS are only good if you are not certain of how you will be accessing the data. In most cases you are sure.

I am constantly stunned how people reach straight for MySQL or Postgres when flat text files with grep would work just as well and be much quicker to implement

Consider your process writing to the file and dying during write() - do you recover and repair the file after you reschedule?

All that said, doing this requires careful thought, and DB normalization when its discovered that there is a 1:1 relationship between rows in a table and a particular key/value table. So, its not something that should be taken to extreme, but I find it aids in quick development, as every time you discover you need to store another piece of data for some edge condition it doesn't require lots of DB normalization. Also, I wouldn't really consider making the "value" field a blob, rather a very limited int or string.

Most often I see TDD used as a couch for a really bad type system.

Can't think of any less functional languages than these.

If you need a template then think of scheme. It is the best example of a minimal functional language.

Pure functions help but the principles of generalisation are deeper than this.

Tests verify contracts offline. This will miss real issues.

Can you elaborate on that one? Sounds interesting to me.

When business rules are encoded in datatypes, it's easy to check that the encoding and the transformations are complete, and the logic can easily be mock-tested.

[1] http://degoes.net/articles/modern-fp

If it feels like translating into a foreign language, then it likely is that exotic or you are using a wrong language.

This can be more sophisticated, like "this function requires a sorted list" so lets make a sorted list type, or packaging things up into biz logic types (a cost type that contains an integer instead of just using integers), but you can catch a wide variety of errors with static analysis if you make your code and logic amenable.

We approached this first outlining the procedure and specifying the types involved, then outlining functions from each type to the next. You could essentially think of our types as tables, so our outline was f1: t1: -> t2, f2: t2 -> t3,...,fn: tn -> t_output (although not quite that linear.)

This let us split up work on the different functions across a team of 6 people. Specifying the input and output types was basically enough to make sure the functions were correct most of the time, and baking the interfaces into types enforced by the compiler made it easy to refactor & coordinate on changes when necessary. Feedback when we made an error was generally immediately available, because IntelliJ would highlight the function that produced an output value of the wrong type, or the compiler would catch it.

In contrast, if we had relied primarily on unit tests to check the functions, that would have made coordination more difficult, refactoring harder, and would have required us to either generate or acquire test data to feed through each function. But this architecture let us successfully build out most of the logic even while we had no access to real data & a different team was working on data ingestion.

Assuming you are talking about real types and having something like

  f1 :: t1 -> t2

  f2 :: t2 -> t3

  g :: t1 -> t3
  g = f2 . f1

Does f1 consume all input before f2 can run? Is it "streamed-through", like in `sh`, e.g.

  $ find /home/foobar | grep hs$ | xargs wc -l

I really like the concept and it works great, but for me it is simpler to apply to smaller constructs and I am still investigating how to apply it to more "business-logic".

g = f5 . f4 . f3 . f2 . f1

where f1 reads from s3 and f5 writes to a db. Then the implementation work mostly involved breaking these down further, e.g. f1 = h4 . h3 . h2 . h1, where only h1 is stateful and everything else is pure.

Spark is lazily evaluated, and in practice it will stream through many operations–f1 will generally not be done consuming the input by the time f2 starts, although sometimes we force it to for debugging purposes.

Lazy evaluation and the discarding of side effects make logging difficult, which is one of the downsides. There are various monitoring and debugging tools that help but it's still definitely harder than the single machine case.

But especially here, where we're talking about reducing complexity, it feels off to me. PostgreSQL and MySQL seem to me like incredibly complex packages. SQL, the language, is not easy to master either; most programmers I meet know mostly basics. On top of that, there's a long ongoing history of security malpractice.

When talking about reducing complexity, CouchDB and Redis are far easier alternatives, in my humble opinion, though they go slightly against 'use the tools developers know'.

SQL might take some getting used, but it is also not rocket science. It shouldn't take more than a week's study to master the basics. There is of course a lot of awful SQL code out there, exactly because most programmers don't even know the basics. You can do incredibly powerful things in it that would take 10x the code in an OO/procedural language. In my opinion dumping an ORM on top is also not the best way to leverage the strengths of an RDBM.

It is slightly ironic that you bring up security malpractice in the context of PostgreSQL, when in the next sentence you advocate Redis as a far easier alternative. As was recently in the news the Redis defaults were for a long time insecure (google for Fairware ransomware).

I agree. Unfortunately, the way I see people usually using them is pretty bad - you should not let ORM-generated stuff dictate your business model. Database is a database. A storage layer. Business objects will not map 1:1 to ORM objects. Approaches like "let's inherit from ORM class and add business-related methods", in my experience, lead to total disaster. One has to respect the boundary between storage layer and business model layer.

The problem I encountered in those projects is the mismatch between storage mental model and business mental model, which lead to explosion of crappy code (AKA technical debt). In particular:

1. the classes I need for business model may have initially mapped well to database tables, but over time they stop; business logic and model changes much faster than you'd like your DB schema to

2. since many things in AR can fire SQL queries, you have to keep in mind the workings of your database when doing almost every operation on your model; it's an abstraction leak

3. code shooting off SQL queries is randomly called from all over your codebase; it's harder to keep track of it and, if needed, optimize those queries

I like AR as a convenient API to get data from/to database, but given the point 1., I eventually learned to isolate AR layer as something below business model layer, so that the pattern is that business model is explicitly serialized and deserialized from database, instead of the database being coupled with the logic of your program.

Now I vaguely recall complaining about this before on HN and getting my ass handed back to me by someone who pointed out that these are all ORM n00b mistakes. I wish I could find that comment (pretty sure I noted the link down somewhere). Yeah, I admit - in those two projects I mentioned, we were all ORM noobs. So we've learned those lessons the hard way.

I don't disagree, in fact I'd go further and say that data and logic should not be coupled, but this is the Active record pattern which is far from the only way to use an ORM, most ORM's won't even support this pattern by default.

Key-value stores are "easy", but what I think isn't easy is to reduce your business domain to a simple key-value model without sacrificing promises and gaurantees offered by a good relational database system.

PostgreSQL and MySQL are very complex, but the complexity is entirely contained. Both are well-tested and reliable, so developers can deploy them without worrying much about them.

I would disagree about SQL being difficult to master. The basics are all most people need, and are not at all difficult to learn. The more advanced stuff (e.g. CROSS APPLY) is not necessarily standard across implementations, and can usually be replaced with application code.

>When talking about reducing complexity, CouchDB and Redis are far easier alternatives, in my humble opinion, though they go slightly against 'use the tools developers know'.

I can't say I'm that familiar with CouchDB, but Redis is entirely inappropriate for most SQL use cases. It's a key value store, and is not meant to do any sort of advanced queries.

Of course, if you regard sql as something you'd rather not "waste" time on, of course you're going to find those kind of subtle distinctions confusing - sort of like how people think css is difficult.

The more reasonably "complex" bits are the update visibility semantics, i.e. which transaction isolation levels mean what in various scenarios.

That's really complex, and it's truly somewhat unique to sql in that most alternatives simply don't bother trying to solve those problems at all - that can be a bad thing, but it is simpler.

A lot of other technologies drop complications like foreign keys which you won't miss when you start developing software but it gives guarantees you will miss sorely when you start seeing inconsistent data 6 months in.

Sometimes it is good to get a pickup truck ahead of time, but often a smaller less versatile car will suffice. But not quite a motorbike.

SQL is well worth its time to learn. It's a good DSL for relational data. Most programming languages used for regular code are not very convenient with relational data. As for its security issues, this is actually simple - one has to respect SQL as a real programming language with its own syntax and grammar, instead of resorting to idiocies like gluing strings together in an ad-hoc manner.

I'm going to use a car analogy here. Modern cars are incredibly complicated machines. They're also generally very reliable, thanks to about a century of development and engineering, and (relatively) inexpensive thanks to economies of scale.

If I want to transport myself to work on weekdays, and transport my girlfriend and maybe another friend on a weekend trip, and carry a bunch of groceries once a week, I can do all of that with a standard 5-person car. It isn't completely optimal for any of those tasks: it has more space than it needs for any of them, especially the weekday commuting. For my daily commute, I don't even reach highway speeds because I live close to work, so the engine is seriously overkill.

So I could have a custom-designed vehicle for each of these use-cases. Each vehicle would be a little less complex than my current car. But that's a lot to maintain, and would surely be far more expensive and less reliable, since each one is a one-off, requiring custom design and engineering, special parts, etc., and not benefiting from the economies of scale and engineering resources that a mass-market car gets. So instead, I just go buy a ready-made car and use it, and it works great and I'm happy.

Is PostgreSQL overkill for a lot of uses? Probably so. But it's designed to be used for all kinds of different tasks, and while it may not be quite as efficient for any of those tasks as some custom-designed solution, it's far more flexible, and it's readily available, plus it's benefitted from an enormous amount of engineering and debugging that a custom-designed solution would not. Things like CouchDB don't have nearly the number of users and amount of development, so while they may make sense for some tasks, the fact that PostgreSQL has more lines of code does not necessarily mean it's less reliable, in fact the opposite is likely true, just like an off-the-lot Honda or Toyota is likely much more reliable than some custom-designed car that someone built in their garage or some high-end limited-production exotic car like a Ferrari or Bentley.

The reason for using an off-the-shelf solution is because it's fast and easy and reliable. It doesn't matter if you're not making use of 80% of the features or capabilities. And software isn't like cars or engines; hard drive space is nearly free, and except for certain applications you're not likely to see a significant downside to just using a standard SQL database versus something more tailor-made. The main problem is cost (like with Oracle), but with PostgreSQL or MySQL this isn't an issue since they're free (and Free). It also helps that they use a standardized query language which makes them much more accessible.

Like most things in life there is no right answer, with tools like Terraform you can build a very complicated microservices system with not much effort, but only if someone on your team is experienced enough. If you're in a small team it's probably not worth the effort of learning the techniques and putting them in practice. We hate premature optimisation after all.

Shorter version: Cost/benefit.