But the 15 lines of Golang are not just 15 lines of Golang in production. You need:
- auth? Who can talk to your service? Perhaps ip whitelisting?
- monitoring? How do you know if you service is up and running? If it's down, you need alerts as well. What if there is a memory problem (because code is not optimal)?
- how do you deploy the service? Plain ansible or perhaps k8s? Just scp? Depending on your solution, how do you implement rollbacks?
- what about security regarding outdated packages the Go app is using? You need to monitor it as well.
And so on. The moment you need to store data that somehow needs to be in sycn with the monolith's data, everything gets more complicated.
Production stuff is not just about lines of code.
So in the k8s world:
- auth: service meshes, network policies, ...
- monitoring: tons of tooling there to streamline that
- deploy: this at scale is trickier than you'd think, many seem to assume k8s on it's own here is the magic dust they need. But GitOps with ArgoCD + helm has worked pretty well at scale in my experience.
- Security is a CI problem, and you have that with every single language, not just Go. See Log4j.
Kubernetes is my bread & butter, but I do realise this has way too much overhead for small applications. However, once you reach a certain scale, it solves many of the really really hard problems by streamlining how you look at applications from an infrastructure and deployment side of things. But yes - you need dedicated people who understand k8s and know what the hell they're doing - and that's in my experience a challenge on it's own.
Let's also dispel a myth that k8s is only suitable for microservices. I have clients that are running completely separate monolith applications on k8s, but enough of those that managing them 'the old way' became very challenging, and moving these to k8s in the end simplified thing. But getting there was a very painful process.
(And yes, if you need extra argument to sell k8s, remind them about bin-packing more applications onto less hw)
1. auth? probably an internal service, so don't expose it to the outside network.
2. monitoring? if the service is being used anywhere at all, the client will throw some sort of exception if its unreachable.
memory problem? it should take <1 day to ensure the code for such a small service does not leak memory. if it does have memory leaks anyways, just basic cpu/mem usage monitoring on your hosts will expose it. then ssh in, run `top` voila now you know which service is responsible.
3. deployment? if its a go service, literally a bash script to scp over the binary and an upstart daemon to monitor/restart the binary.
rollback? ok, checkout previous version on git, recompile, redeploy. maybe the whole process is wrapped in a bash script or assisted by a CI/CD build job.
4. security? well ok, PDFs can be vulnerable to parser attacks. so lock down the permissions and network rules on the service.
Overall this setup would work perfectly fine in a small/medium company and take 5-10x less time than doing everything the FAANG way. i don't think we should jump to calling these best practices without understanding the context in which the service lives.
> if it does have memory leaks anyways, just basic cpu/mem usage monitoring on your hosts
Who keeps on monitoring like this? How frequently would you do it? In a startup there are somewhere in the range of 5 microservice of that scale per programmer and daily monitoring of each service by doing top is not feasible.
> 3. deployment? if its a go service, literally a bash script to scp over the binary and an upstart daemon to monitor/restart the binary.
Your solution literally is more complex than simple jenkins or ansible script for build then kubectl rollout restart yet is lot more fragile. Anyways the point stands that you need to have a way for deployment
For example, kubectl rollout assumes that your service is already packaged as a container, you are already running a k8s cluster and the team knows how to use it. In that context, maybe your method is a lot better. But in another context where k8s is not adopted and the ops team is skilled at linux admin but not at k8s, my way might be better. There’s no one true way and there never will be. Technical decisions cannot be made in a vacuum.
Haha. You're a bit of an optimist, eh?
The point was never comparing it to the FAANG way. The point is: it's easier (at the beginning) to maintain ONE monolith (and all the production stuff related to it) than N microservices.
Auth: nobody knows IP address of our server anyway, don't bother with that. And for extra security we have secret port number.
Monitoring? Well, we have our clients for that. They'll call us if something happens.
Deploy? Copy .war file and restart tomcat. Rollback? What's rollback?
Outdated packages? We call those stable packages.
I wish I would work in a miracle world you describe.
Figuring out what was wrong with it was hard, because the logging was inconsistent, all over the place in where it logged and also insufficient.
The deployments and environments were inconsistent, the sysadmins on clients's side manually changed stuff in the .war archives, all the way up to library versions, which was horrendous from a reproducibility perspective.
The project was also severely out of date, not only package wise, but also because the new versions that had been developed actually weren't in prod.
After ripping out the system's guts and replacing it with something that worked, i became way more careful when running into amateur hour projects like that and about managing the risks and eventual breakdown of them. I suggest that you do the same.
Don't put yourself at risk, especially if some sort of a liability about the state of the system could land on you. What are you going to do when the system gets breached and a whole bunch of personal data gets leaked?
Yes, if there is mission critical software that affects millions of people, do all that stuff.
If your pdf -> stardust service is more a convenience then don't spend hundreds of thousands on infrastructure.
I find it curious how we went from doing the basics of software development that would minimize risks and be helpful to almost any project out there to this.
To clarify, i agree with the point that you'll need to prioritize different components based on what matters the most, but i don't think that you can't have a common standard set of tools and practices for all of them. Let me address all of the points with examples.
> Auth: nobody knows IP address of our server anyway, don't bother with that. And for extra security we have secret port number.
Port scanning means that none of your ports are secret.
JWT is trivial to implement in most languages. Even basic auth is better than nothing with HTTPS, you don't always need mTLS or the more complicated solutions, but you need something.
This should take a few days to a week to implement. Edit: probably a day or less if you have an easily reusable library for this.
> Monitoring? Well, we have our clients for that. They'll call us if something happens.
This is not viable if you have SLAs or just enjoy sleeping and not getting paged.
There are free monitoring solutions out there, such as Zabbix, Nagios, Prometheus & Grafana and others. Preconfigured OS templates also mean that you just need the monitoring appliance and an agent on the node you want to monitor in most cases.
This should take close to a week to implement. Edit: probably an hour or less if you already have a server up and just need to add a node.
> Deploy? Copy .war file and restart tomcat. Rollback? What's rollback?
This is an error prone way to do things, as the experience of Knight Capital showed: https://dougseven.com/2014/04/17/knightmare-a-devops-caution.... In addition, manual configuration changes lead to configuration drift and after a few years you'll have little idea about who changed what and when.
In contrast, setting up Ansible and versioning your config, as well as using containers for the actual software releases alongside fully automated CI cycles addresses all of those problems. In regards to rollbacks, if you have automated DB migrations, you might have to spend some time writing reverse migrations for all of the DDL changes.
This should take between one to two weeks to implement. Edit: probably a day or less once per project with small fixes here and there.
> Outdated packages? We call those stable packages.
Log4j might be stable, but it also leads to RCEs. This is not a good argument, at least for as long as the software packages that we use are beyond our ability to control or comprehend.
This should be a regular automated process, that alerts you about outdated and/or insecure packages, at least use npm audit or something, or proactive scanning like OpenVAS. This should take close to a week to implement.
All of the mentioned software can easily run on a single node with 2 CPU cores and about 8 GB of RAM. I know this, because i did all of the above in a project of mine (mentioned technologies might have been changed, though). Of course, doing that over N projects will probably increase the total time, especially if they have been badly written.
In my eyes that's a worthwhile investment, since when you finally have that across all of your apps, you can develop with better certainty that you won't have to stay late and deliver 2 or 3 hotfixes after your release, which goes hand in hand with test coverage and automated testing.
That is hardly hundreds of thousands of dollars/euros/whatever on infrastructure. And if the personnel costs are like that, i'd like to work over there, then.
The main point is, that one should never spend a "a few days to a week" to implement a feature that at best i useless and at worst is detrimental to the service stood up.
Implement auth, if it is needed, implement monitoring, CI, CD, dependency monitoring, testing, everything, if it is needed.
But don't implement it as dogmatic consequences of doing software development.
And regarding the spend: one week worth of work could be USD 8k. So just the initial implementation of your JWT based authentication system is 4% into the "hundreds of thousands of dollars". Then you need to factor in the extra complexity on maintenance and before you know it we do not talk about hundreds of thousands of dollars but millions...
The point you should have made is that dogmatic approaches usually produce a lot of waste, but the example you gave us exactly why teams end up that way. Otherwise people come up with bullshit hacks like you describe and the entire team pays for it.
Though these were not the cases I talked about.
I talked about implementing features that are not necessary for the product because of dogmatic reasons.
Sure, i'm not necessarily advocating for a full blown RBAC implementation or something like that, merely something so that when your API is accidentally exposed to the rest of the world, it's not used for no good (at least immediately).
> Implement auth, if it is needed, implement monitoring, CI, CD, dependency monitoring, testing, everything, if it is needed.
> But don't implement it as dogmatic consequences of doing software development.
Now this is a bit harder to talk about, since the views here will probably be polarized. I'd argue that if you're developing software for someone else, software that will be paid for (and in many cases even in pro-bono development), most of that is needed, unless you just don't care about the risks that you (or someone else) might have to deal with otherwise.
If there's an API, i want it to at the very least have basicauth in front of it, because of the reasons mentioned above.
If there is a server running somewhere, i want to be alerted when something goes wrong with it, see its current and historic resource usage and get all of the other benefits running a few apt/yum commands and editing a config file would get me, as opposed to discovering that some bottleneck in the system is slowing down everything else because the memory utilization is routinely hitting the limits because someone left a bad JVM GC config in there somewhere.
If something's being built, i want it to be done by a server in a reasonably reproducible and automated manner, with the tasks are described in code that's versioned, so i'm not stuck in some hellscape where i'm told: "Okay, person X built this app around 2017 on their laptop, so you should be able to do that too. What do you mean, some-random-lib.jar is not in the classpath? I don't know, just get this working, okay? Instructions? why would you need those?"
Furthermore, if there is code, i want to be sure that it will work after i change it, rather than introducing a new feature and seeing years of legacy cruft crumble before my eyes, and to take blame for all of it. Manual testing will never be sufficient and integration tests aren't exactly easy in many circumstances, such as when the app doesn't even have an API but just a server side rendered web interface, which would mean that you need something like Selenium for the tests, the technical complexity of which would just make them even more half baked than unit tests would be.
Plus, if i ever stumble upon a codebase that lacks decent comments or even design docs/issue management, i will want to know what i'm looking at and just reading the code will never be enough to understand the context behind everything but if there are at least tests in place, then things will be slightly less miserable.
I'm tired of hating the work that i have to do because of the neglect of others, so i want to do better. If not for those who will come after me, then at least for myself in a year or so.
Do i do all of that for every single personal project of mine? Not necessarily, i cherrypick whatever i feel is appropriate (e.g. server monitoring and web monitoring for everything, tests for things with "business logic", CI/CD for everything that runs on a server not a local script etc.), but the beauty is that once you have at least the basics going in one of your projects, it's pretty easy to carry them over to others, oftentimes even to different stacks.
Of course, one can also talk about enterprise projects vs startups, not just personal projects, but a lot of it all depends on the environment you're in.
As for the money, i think that's pretty nice that you're paid decently over there! Here in Latvia i got about 1700 euros last month (net). So that's about 425 euros a week, or more like 850 if you take taxes and other expenses for the employer into account. That's a far cry from 100k. So that is also situational.
However, i just had a look at https://www.algas.lv/en, one site that aggregates the local salary information, based on a variety of stats. As someone with a Master's degree and ~5 years of experience and whose job description states that i'm supposed to primarily work in Java (even though i do full stack and DevOps now), i input those stats and looked at what information they have so far.
The average net salary monthly figures, at least according to the site, are:
- Lithuania: 1 858 EUR/month ( 464 EUR/week )
- Latvia: 1 871 EUR/month ( 467 EUR/week )
- Estonia: 2 233 EUR/month ( 558 EUR/week )
- Finland: 3 767 EUR/month ( 941 EUR/week )
A brief litmus test makes me think that the values are vaguely believable.For example, if we have a look at some other sources for Lithuania, such as https://www.glassdoor.com/Salaries/vilnius-software-develope..., we get a figure of 2 458 EUR/month --> 614 EUR/week.
Or, if we look at other sources for Estonia, such as https://www.payscale.com/research/EE/Job=Software_Developer/..., we get a figure of 28 000 EUR/year --> 2 333 EUR/month --> 583 EUR/week.
So there's definitely a bit of variance, but it's still in the same ballpark. Of course, there are better companies out there, but the reality for many is that they're not rewarded for their work with all that much money.
I actually wrote about how that has impacted my ability to make savings on my blog: https://blog.kronis.dev/articles/on-finances-and-savings
Again, I advocate developing in a timely manner, and not do over engineering (neither under engineering).
I made the above response with that in mind.
> And regarding the spend: one week worth of work could be USD 8k.
The original claim was that one week's worth could be 8000 USD, or let's say roughly 7094 EUR. That comes out to 28376 EUR per month.
Last month i made around 1700 EUR, so it's possible to calculate approximately how much my work cost to my employer. Let's do that with a calculator here: https://kalkulatori.lv/en/algas-kalkulators
After inputting the data that's relevant to me, i got the following:
Gross salary 2654.51 EUR
Social tax 278.72 EUR
Personal income tax from income till 1667 EUR 277.66 EUR
Personal income tax (987.51 EUR), from part ... 227.13 EUR
Social tax, employer's part 626.20 EUR
Business risk fee 0.36 EUR
Total employer's expenses 3281.07 EUR
It should be apparent that 28376 EUR is a far cry from 3281 EUR, which is how much my work cost to my employer.Thus, per week, 7094 EUR is also a far cry from 820 EUR, which is how much my work cost to my employer.
Also, 820 is actually pretty close to my initial guess of 850 EUR.
Of course, it's possible to argue that either i'm underpaid individually, or that many of my countrymen in Latvia are underpaid in general (on which i elaborated in an adjacent comment https://news.ycombinator.com/item?id=29595158), but then the question becomes... so what?
Does that mean that if you're in a well paid country like US, then you cannot afford proper development practices due to all of the payroll expenses that would cause? While that may well be, to me that sounds weird and plain backwards - if that were really true, then US would outsource even more to countries like mine and these outsourced systems would work amazingly well, since you can supposedly afford a team of developers here for what would buy you a single developer over there. And yet, most systems are still insecure, buggy and slow.
Maybe someone else is pocketing a lot of the money they receive in these countries, and is simply charging high consulting rates? The prevalence of WITCH companies here is telling, but that's a discussion for another time.
Our internal software plattform is getting to a point so it can answer most of these things - auth via the central OIDC providers, basic monitoring via annotations of the job's services, deployments via the orchestration and some infrastructure around it, including optional checks and automated rollbacks and automated vulnerability scanning on build-servers and for the running systems. It wouldn't be 15 lines of go, more like 15 lines, plus about 100-200 lines of terraform and/or yaml to get everything configured, and a ticket do register the service in the platform. It's pretty nice and our solution consultants like it very much.
The thing is - this took a team about a year to build and it'll take another half a year to get everything we currently want to do right. And it takes a non-trivial time to maintain and support all of this. This kind of infrastructure only makes business sense, because we have enough developers and consultants moving a lot faster with this.
Back when we were a lot smaller, it made a lot more sense to just push a single java monolith on VMs with chef or ansible, because that was a lot easier and quicker to get working correctly (for one thing).
It's also a common trope when I'm dealing with k8s decriers - yes, you might have one application that you can easily deploy, but suddenly there are 15 other medium-weight applications that solve different important problems and you want them all ;)
P.S. Recently a common thing in my own architectures is separate keycloak deployment that all services either know how to use, or have it handled at separate request router (service mesh or ingress or loadbalancer)
The task of the Microservice is to convert the pdf to stardust and to return it to its sender. so no auth. Furthermore its most likely only reachable through the local network, or at least should be if you want some stranger not to be able to also make stardust from pdfs.
Monitoring: are you trying to say that its a lot esaier to pick up one logfile thant lets say 15? because they should be aggregated somewhere anyway no?
Deployment: Depending on anything you listed how do i do anything? Of course if have to define it but if you want a fancy example: k8s argocd canary deployments done. I literally set it up once.
security? Really? Please dont get this wrong but this feels to me like whataboutism but well here i go:
i implement security just the same way as i would in the monorepo. The thing/person/entity just has to look into more repositories ;)
It comes down do one sentence i think: State is not shared, state is communicated.
Monitoring: it's easier to look at a stack trace, including some other team's external library, than a HTTP error code 500.
Deployment is certainly easier when you're just shipping code and a build. You don't have to faff around with the previous instance running, maybe having some active connections/transactions/whatever, needing to launch a new one. Maybe it's not hard overall, but less fun.
Auth can be a very broad term in the case of services, especially in the cloud. IAM could only allow access from certain other machines, etc...
If this is actually (still) true, that means that "the way you organize your code" is a bit simplistic. Your example of an "http api that converts pdfs to ..." is surely a valid example of a microservice, but most business products have to handle much more "state" than those, and this will create further complications which go far beyond "how to organize your code" (and make monoliths more appealing).
The PDF Example for instance: I have to provide all assets that will be converted to startdust or at least links for them I have to define where the finished stardust would be send to (e.g http post)
but its function is nonetheless independent. the epepheral disk on your container is a datastore too. If you need what is stored actually for longer than the request.... that is another story.
When you have multiple bits of code responsible for writing the same columns, maintaining global invariants becomes much harder.
I can still see rationale for exceptions to the rule, e.g. status fields for rows which logically represent some ongoing process or job, where a UI controller may set something to "processing" and the background job sets it to "processed" when it's complete. But there are ways to close the loop here. For example, implement the background job by having the job processing system invoke the same system which owns the field and sets it in response to the UI, like this:
UI start ---> Back End ---> status := processing
Back End ---> add job to job queue
Job Queue ---> Back End does work
Back End ---> status := processed"Single writer multiple readers", yes, this is what I would probably use, but yet again, wasn't the "promise" of Microservices being able to work in total isolation?
If I have one table (e.g. "Customer") which is written by one specific microservice and read by a dozen or more... what happens when I decide that I have to change the schema because the current representation of "Zip Code" is not adequate anymore because, I dunno, we started dealing with UK customers now?
Lo and behold, I have to change code in 13 Microservices - the one actually writing to it, and the 12 more that only need to get the data to show or print or convert to JSON or whatever... ¯\_(ツ)_/¯
Having an RPC to migrate means you can put code on the read path to support backward compatibility while migrating. But you can do the same thing with the database - support both fields until all readers have ported to the new field - and since big database migrations are expensive, in practice large systems do migrations that way anyhow. Introduce the new field, start writing to it, back-fill it, port readers across individually (there may be many read locations even within a single service), drop the old field.
If a system evolved from a monolith, one way to partition services is to provide views to limit set of visible columns. That's another point at which you can put shims for compatibility while porting schemas.
The services themselves do not differ, but are fanned out. E.G. so 100 workers watch your pdf queue to generate pdfs.
If your zip code is really used by many services, the question is how you want to communicate those objects and there are really a lot of choices out there. GRPC / Soap / Shared model package / and the list goes one.
On the other hand i have obsverved that people push full blown models around just to end up reading one field of it in their monolith. I believe the PHP frameworks of the past where the biggest offenders like that.
When i was working with sugarcrm for instance, it was not uncommon to pass huuuuge objects between functions because you needed 1 field. It's PHP so by default, function arguments are passed by value, so you can already see where the fun begins.
It changes the situation regarding deployment, but it does not miraculously absolve you from changing code when data formats change. How could it?
My point is precisely that: if you have to handle large quantity of state (e.g.: travel agency handling diverse item bookings to sell as a complete holiday packages - note that this includes having conflicts on inventory, like "cruise cabin categories" or "hotel rooms") microservices add latency by "replacing function calls with RPC", and gain you... an unspecified advantage in terms of... deployment? The possibility to have hundreds of developers working on the system in parallel?
I have always worked on medium-size monoliths during most of my career, and "ah, if we had 137 developers all working on this everything would be magically solved, but alas, we have a monolith" was a sentence I uttered (or heard) exactly 0 times so far.
I didn't want to put unneeded information in the first post but here i have some space ;)
What I ended up doing was:
- two sqs queues, one for processing one for failures after 3 tries
- the go backend reads from its queue and the message contains a json object with all the information needed for pdf generation
- the backend posts the file back to the "internal api" of the application, which saves it in s3 and also sets processing to true which means the user can go and download it.
This results in: - my pdf generated the way i want it (sorry i dont actually make stardust)
- the queues are only reachable through the assosiated iam roles
- if a job failes it will be added to the failed queue, which sends me an email
- if i recieve an email by the failed queue, 90% of the time i have found another edge case
- since the message already contains everything that i need to build the pdf, my edge case has just defined itself
- for the postback i supply an api token that is only valid for that one file
All this means my client has total flexibility in deployment.But this also means that we are now back into "YesSQL" territory, and specifically that we have to use a RDBMS which allows us to create Views. Goodbye NoSQL, goodbye Key+Valus datastore. (Or maybe you will just create an extra "newZipCode" and mantain it in parallel with "ZipCode" allowing every other consumer to adapt at their leisure...?).
So it is another step back to "more traditional ways" to design a system... or a recipe for a disaster as soon as you start dealing with significant quantities of "state".
I don't think this is true.
I think — at least as far as I've observed — microservices in practice means replacing various functions calls with slow and error-prone network requests.
This is a big and meaningful difference.
Actually it's been going on for years, and it's always the same argument. People think they're thought leaders for saying "Start with monoliths and only move to microservices if you absolutely need to!"
It's a pretty obvious conclusion to anyone who has worked in both environments, or have had to migrate from one to the other, so it's not particularly insightful. And yet here were are, 5+ years later saying the same thing over and over again.
It should be obvious, but apparently it's not. So many architecture astronauts drinking the kool-aid and making a mess. Premature microservices can easily kill a product.
It’s also about how you deploy your code. If you have 1000 micro services do you have 1000 deployment pipelines? If so how do you manage those pipelines? If not, you sacrifice independent deployment of each micro service.
Also deployment is part of code organistaion no? i like to point out the fact that i explicitly mentioned you do not need to decide, you can use both at the same time but i would like to try to answer your point anyway:
Maybe I am just spoiled with go and github, but those deployment pipelines can be directly integrated into your repository. The same way I can counter argue that your build ci and deployment phase will take significantly more time too and if you change a variable from uppercase to lowercase you will wait for minutes too.
I come from a c# background a long time ago and this has been true with it for eons: https://xkcd.com/303/
Another thing that I have noticed is that its easily scriptable too. What I end up doing is to provide a makefile into my repository too. This way i can define the build process amongst other things eaisly.
In the end: We have a predefined spec that will create the desired state on a host of our chosing.
Ansible really does not care if you deploy 1 or 1000 "services" for instance! Tools Like ArgoCD will also deploy stuff when you commit to your master.
There is tooling for everyone available, but what happnes quite often is that the people in charge of the company expected the developers to be all knowing entities that need to define everything end to end. The Kubernetes Space is vast because we are still defining and communication its future.
But recently? I am trying to think of something that would not be completly automateable in 2021
How? Can you provide any examples?
That way, i can attach as much functionality as i want without bloating the main app and processing scales with demand.
The application dev does not need acces to production whatever and the devops person does not need to know how the application works.
But then again these to people also need to communicate ;)
maybe we are getting caught up in sematics because its christmas, but "monorepo/monolith/microservices/etc" is -just- the way you organize your code.
Developing a montolith for years but now you have written a 15 line golang http api that converts pdfs to stardust and put it into on a dedicted server in your office? welp thats a microservice.
Did you write a 150 repo application that can not be deployed seperatly anyway? welp thats a monolith.
You can also build a microservice ecosystem without kubernetes on your local network. We have done it for years with virtual machines. Software defined networking just makes things more elegant.
So dont stop using microservices because its "hard" or start writing monoliths because its "easy", because none of that is true in the long run.
What is true is that you have a group of people trying to code for a common goal. The way you reach that goal together defines how you organize your code.