This is the age old problem of simple problems being harder to solve at scale. Running a blog on Digital Ocean is very different to running a million dollar application on AWS.
[1] https://medium.com/@Joachim8675309/devops-concepts-pets-vs-c...
I'd rather not engage in pointless unprovable arguments, if you don't mind.
> The problem is that you are treating your logs as pets when the truth is the logs are also cattle. Virtually all debug logs will pass through their life cycle without being read, so indexing them is just a flagrant waste of energy.
You shouldn't have debugging enabled on production systems unless you have an interim process that filters out debug messages before they get indexed (and thus you can toggle which logs get indexed there rather than reconfiguring / redeploying all of your application nodes).
Also nobody is suggesting logs should be treated as "pets". You still want to purge out older logs however the problem is you cannot always replicate reported errors so if you don't have those log messages captured then you're sod out of luck.
Don't get me wrong, there is a certain allure to the traditional method of systems administration - I've been on both sides of the fence - but central logging services have so many other benefits such as security (tamper proof logs, users don't require SSH), ease of use, persistent logging, etc. The only real downside is cost but that quickly becomes absorbed in your pricing plan when customers start asking for SLAs.
By the time you want centralized logs, you are probably already spending much more than the logging will cost on infrastructure.
What you're doing here is constructing a straw man argument while agreeing with the same point I was making.
> > ship log events out of the box and into something searchable, indexable, and can derive metrics
> That's just a way to spend a ton of money. There's really not a reason to ship or index logs...
That's what I said. ;)
> One of the ways to make it seem a reasonable suggestion is the broader topic of not logging into machines at all, ever, and why that may be a good thing.
Except none of that was being discussed. Only log access.
> They're explicitly talking about not logging into machines for log access.
Why can't you just admit that you didn't read the thread properly rather than insulting all of our intelligence with these piss poor mental gymnastics where you redefine the context of what people had very clearly written.
The conversation you're replying to, however, did. I know this for a fact because I was involved in that conversation and I made that distinction myself ;)
I guarantee that you haven't, as the number of people that can make that claim in relation to my background is vanishingly small and I know almost all of them by name.
I centralize logs. Not only does it make more sense for administration at scale, it's invaluable for security reasons and assists in compliance by providing a controllable guaranteed audit trail.
You may have a valid argument about cost here for some applications, but it's unwise to make arrogant claims you cannot back up.
Not only is that vanishingly unlikely to be true with this crowd it’s also not at all the kind of rhetoric that they’ll listen to either.
Your storage/indexing money spend scales with your company. If you have < 10 hosts you can probably handle logs with a cost that's a rounding error with something like loki+grafana.
If you have a bigger environment and more people, the cost will be more significant, but you'll also likely have more revenue to justify it.
Proper Information Security policy is one very good reason. If a box is compromised, you don't want the logs being tampered with.
You ship the logs to a central, dedicated server so that a clear audit trail can be established.
On the point, what do you do if the machine is not there anymore? Your model fails immediately. Also, some processes operate on an assumption that the "machine" will stand-up -> do-something()- > die(), which your model also doesn't support. Not to be messing with you, but I don't think there is the "right" model, but whatever works for us. I do like your model for immediate accessibility of the logs.
Anyone who needs access could remotely run a command that would return the results of whatever permitted query they submitted.
> ship log events out of the box and into something searchable, indexable, and can derive metrics
That's just a way to spend a ton of money. There's really not a reason to ship or index logs. Just leave them where they are produced and instead of shipping them to a central thing that inevitably becomes an expensive bottleneck, instead develop ways to access them in place (other than SSH) and frameworks for large-scale distributed searching where you push the predicate out to the machines where the logs exist.