News for hackers. Stuff that matters.
Re:Don't start with microservices – monoliths are your friend by Pamar on Thursday December 16, 2021 @05:25AM (#29576857)
Maybe it is just me, but I always understood that properly designed microservices have their own specific datastore, which is not shared with other microservices even if these all collaborate to the same process.

If this is actually (still) true, that means that "the way you organize your code" is a bit simplistic. Your example of an "http api that converts pdfs to ..." is surely a valid example of a microservice, but most business products have to handle much more "state" than those, and this will create further complications which go far beyond "how to organize your code" (and make monoliths more appealing).

Re:Don't start with microservices – monoliths are your friend by jhoelzel on Thursday December 16, 2021 @05:35AM (#29576913)
Well I dont believe that i need a seperate datastore, but yes i need to communicate my state much more.

The PDF Example for instance: I have to provide all assets that will be converted to startdust or at least links for them I have to define where the finished stardust would be send to (e.g http post)

but its function is nonetheless independent. the epepheral disk on your container is a datastore too. If you need what is stored actually for longer than the request.... that is another story.

Re:Don't start with microservices – monoliths are your friend by VBprogrammer on Thursday December 16, 2021 @05:35AM (#29576911)
IMHO Microservices done well should actually cut a whole vertical through your applications feature space. So not only should it be responsible completely for its own storage of data but it should be responsible for how that data is shown on the front end (or as close to that as you can realistically achieve). A microservice should ideally be reviews or left navigation not customer authentication or order processing.
Re:Don't start with microservices – monoliths are your friend by barrkel on Thursday December 16, 2021 @06:30AM (#29577191)
Single writer multiple readers, ideally with permissions to enforce, is a useful hybrid. Enforce going through the front door if you want side-effects. Reporting and ad-hoc data enrichment can be painful to materialize otherwise.

When you have multiple bits of code responsible for writing the same columns, maintaining global invariants becomes much harder.

I can still see rationale for exceptions to the rule, e.g. status fields for rows which logically represent some ongoing process or job, where a UI controller may set something to "processing" and the background job sets it to "processed" when it's complete. But there are ways to close the loop here. For example, implement the background job by having the job processing system invoke the same system which owns the field and sets it in response to the UI, like this:

    UI start  ---> Back End ---> status := processing
                   Back End ---> add job to job queue

    Job Queue ---> Back End does work
                   Back End ---> status := processed
Re:Don't start with microservices – monoliths are your friend by Pamar on Thursday December 16, 2021 @07:09AM (#29577383)
Caveat: I am really not qualified to discuss the nuances (because I have never used microservices so the little I know is based on reading a bit on those here and on other online forums).

"Single writer multiple readers", yes, this is what I would probably use, but yet again, wasn't the "promise" of Microservices being able to work in total isolation?

If I have one table (e.g. "Customer") which is written by one specific microservice and read by a dozen or more... what happens when I decide that I have to change the schema because the current representation of "Zip Code" is not adequate anymore because, I dunno, we started dealing with UK customers now?

Lo and behold, I have to change code in 13 Microservices - the one actually writing to it, and the 12 more that only need to get the data to show or print or convert to JSON or whatever... ¯\_(ツ)_/¯

Re:Don't start with microservices – monoliths are your friend by barrkel on Thursday December 16, 2021 @08:45AM (#29577990)
Database schemas are generally isomorphic with RPC data structures - usually a subset one way or the other. If you had a service which handed out Zip Codes and the structure of Zip Codes changed (maybe you used numbers and now you realize you need to use strings, which I infer from your UK example) you'd have an isomorphic problem: updating 12 consumers and 1 producer for the new type in the RPC message.

Having an RPC to migrate means you can put code on the read path to support backward compatibility while migrating. But you can do the same thing with the database - support both fields until all readers have ported to the new field - and since big database migrations are expensive, in practice large systems do migrations that way anyhow. Introduce the new field, start writing to it, back-fill it, port readers across individually (there may be many read locations even within a single service), drop the old field.

If a system evolved from a monolith, one way to partition services is to provide views to limit set of visible columns. That's another point at which you can put shims for compatibility while porting schemas.

Re:Don't start with microservices – monoliths are your friend by HelloNurse on Thursday December 16, 2021 @09:38AM (#29578533)
First you publish the next version of your service with the new data, then the disparate teams in charge of 12 clients (not you) update the rest of the application, then the old version is retired and every team has coordinated properly. Microservices allow lean coordination, basically just asking every team when they think they'll be ready without anyone messing with someone else's code.
Re:Don't start with microservices – monoliths are your friend by jhoelzel on Thursday December 16, 2021 @07:41AM (#29577561)
actually you would only have to change the one reader and the one writer.

The services themselves do not differ, but are fanned out. E.G. so 100 workers watch your pdf queue to generate pdfs.

If your zip code is really used by many services, the question is how you want to communicate those objects and there are really a lot of choices out there. GRPC / Soap / Shared model package / and the list goes one.

On the other hand i have obsverved that people push full blown models around just to end up reading one field of it in their monolith. I believe the PHP frameworks of the past where the biggest offenders like that.

When i was working with sugarcrm for instance, it was not uncommon to pass huuuuge objects between functions because you needed 1 field. It's PHP so by default, function arguments are passed by value, so you can already see where the fun begins.

Re:Don't start with microservices – monoliths are your friend by rvense on Thursday December 16, 2021 @10:43AM (#29579313)
Microservices mean that different parts of your code are now communicating via (e.g.) HTTP instead of function calls.

It changes the situation regarding deployment, but it does not miraculously absolve you from changing code when data formats change. How could it?

Re:Don't start with microservices – monoliths are your friend by Pamar on Friday December 17, 2021 @02:51AM (#29589156)
I know, thanks.

My point is precisely that: if you have to handle large quantity of state (e.g.: travel agency handling diverse item bookings to sell as a complete holiday packages - note that this includes having conflicts on inventory, like "cruise cabin categories" or "hotel rooms") microservices add latency by "replacing function calls with RPC", and gain you... an unspecified advantage in terms of... deployment? The possibility to have hundreds of developers working on the system in parallel?

I have always worked on medium-size monoliths during most of my career, and "ah, if we had 137 developers all working on this everything would be magically solved, but alas, we have a monolith" was a sentence I uttered (or heard) exactly 0 times so far.

Re:Don't start with microservices – monoliths are your friend by jhoelzel on Thursday December 16, 2021 @07:08AM (#29577377)
Thank you for this as this is exactly how its implemented in a recent project of mine.

I didn't want to put unneeded information in the first post but here i have some space ;)

What I ended up doing was:

  - two sqs queues, one for processing one for failures after 3 tries

  - the go backend reads from its queue and the message contains a json object with all the information needed for pdf generation

  - the backend posts the file back to the "internal api" of the application, which saves it in s3 and also sets processing to true which means the user can go and download it.
This results in:

  - my pdf generated the way i want it (sorry i dont actually make stardust)

  - the queues are only reachable through the assosiated iam roles

  - if a job failes it will be added to the failed queue, which sends me an email

  - if i recieve an email by the failed queue, 90% of the time i have found another edge case 

  - since the message already contains everything that i need to build the pdf, my edge case has just defined itself

  - for the postback i supply an api token that is only valid for that one file
All this means my client has total flexibility in deployment.
Re:Don't start with microservices – monoliths are your friend by pibi on Thursday December 16, 2021 @05:31AM (#29576885)
Well, it's all about the data responsibility: who is the owner of the data, how others can access the data. Once you have defined these, you see that you can "share the access" with other microservices (for example read only mode on a view), as long as the ownership and the access rules are preserved.
Re:Don't start with microservices – monoliths are your friend by Pamar on Thursday December 16, 2021 @07:17AM (#29577424)
Yes, a view would be exactly how I would address the problem I described in my other answer above ("What happens if I need to change zip code representation in a data source that is read by multiple microservices?").

But this also means that we are now back into "YesSQL" territory, and specifically that we have to use a RDBMS which allows us to create Views. Goodbye NoSQL, goodbye Key+Valus datastore. (Or maybe you will just create an extra "newZipCode" and mantain it in parallel with "ZipCode" allowing every other consumer to adapt at their leisure...?).

So it is another step back to "more traditional ways" to design a system... or a recipe for a disaster as soon as you start dealing with significant quantities of "state".

Re:Don't start with microservices – monoliths are your friend by preseinger on Thursday December 16, 2021 @05:48AM (#29576986)
Yes, services should definitely encapsulate their state layer.