Enterprise scale, enterprise pricing. Generally prices line up to the number of heads an org can save compared to doing things themselves with staff.

But if you're going to do it yourself, at least use structued logging rather than thinking of logs wholly as unstructured text.

> all I want is the production log in a .txt.

I feel the same way, but if you’re working for a reasonably sized organization, you likely don’t. Once you have that, you’ll probably do quite a few of the following:

- write “one off” scripts to grep and wc -l those logs

- make those scripts “multiple off” by running them from cron jobs

- getting tired of having to go look them up each day, mail the reports to you/put them on Slack/etc.

- getting tired of having to compare multiple reports, add the ability to summarize multiple reports in a single view.

- getting tired of daily checking that there’s nothing interesting again today, add outlier detection.

- on request from your superiors, put the results on a web page for them to look at

- on request from your superiors add charts to that web page

- after an outage that meant losing data that your superiors use to report to their superiors, spent weeks to make the contraption more robust an to add alerting

- etc.

It likely won’t grow as complex as NewRelic, but your bosses may start wondering whether it’s worthwhile to maintain that in-house, and be dependent on you working there.

Summary: it’s not fun having to use a wheel that has a zillion features and a thousand page manual that also is incomplete and changes regularly, but re-inventing the wheel is fun, but not always the right choice.

At our company, we use logrotate to put all logs into an S3 bucket by hostname (with a PutObject only key of course). We then have a service running in EC2 that does a cached get of those logs (based on hostname and timestamp range) then runs ripgrep over it and returns the lines that match.

It works well and requires very little maintenance, and it costs a tiny fraction of something more full-featured like Papertrail, NewRelic, etc.

What’s “cached” referring to here, in the case of the get?

We keep recently fetched files on the EC2 instance's EBS volume for a few days (or until they are evicted to make space), as this is faster than re-reading them from S3 each time someone does a search, and it's common to do a number of searches over the same time period when debugging an issue.

How does this work if the dev's want to do live debugging etc.

We don't handle that case in our S3 upload system. We only upload upon rotation, which adds a latency of around a day (less for very noisy log files). For live debugging, we either replicate the problem in local environment (if possible), or the dev logs into the production server and runs `tail` on the log file.

I generally feel like bad software will directly integrate APM APIs in their codebase, instead of outputting meaningful logging that APMs can consume.

And APM SaaS companies want this, because the engineers aren't bright enough to think, "Hey, what if we need to switch APM companies?"

So your B-tier engineers do it without questioning things, and your B-tier managers will complain to A-tier engineers that it's more work to protect yourself from vendor lock-in.

And now your org is paying 5-6 figures a month because some idiots couldn't append to a text file or add rows to a table using SQLite. And no one cares because it's not their money, and you don't care because no one else cares.

And what happens when your system scales out to 2 hosts? Now you have 2 txt files - no big deal just query them both. Now what about when you scale to 16 hosts? 100 hosts? What about when your system is composed of dozens of services all managed by different teams with different logging philosophies? And hosts are created and destroyed dynamically, where should the txt files go?

Tools like New Relic are not designed for small products that can run on a single host with logs in a txt file. They provide value when systems scale.

Firedoors in buildings are built to standards. There's standard lighting directig occupants towards a fire exit. A quota of fire exits to seats / floorspace / etc exists. Stair cases need to be of a certain width. Quite a bit more. Buildings across a country are built to these standards, and when they're not tragic things can happen.

This isn't to say all buildings are identical. Or save the same purpose. Or exist in the same climate. Or even have occupants that speak the same language. But do share a core philosophy that despite individual developers' or architects' preferences for wood or concrete, in the case of a fire a building's occupants should be provided safety.

Logs should be more like fire exits and less like fragile works of art. As I can find my way to a fire exit in hopefully any building, I should be able to deal with several .txt files.

There are cheap services and free DIY tools that make it easier to aggregate and grep text logs. Bigger, growing or more complex operations would like more and would rather pay for something tested that has more features. New Relic is one of those. You don’t have to be a good customer for every service provider if enough meet your needs.

I agree and use the simplest structured logging https://www.npmjs.com/package/livelogging

I feel the same way with DataDog. All I need is a tool to monitor logs. First I tried Signoz and the docker gave me an error on clickhouse. Then I tried Loki but the docker agent never sent the logs and I received no labels although I received a test one I sent. I tried azure log analytics and lost the plot after creating about 15 services. Paying a premium to DataDog just for log analytics seems to be an overkill

SigNoz has pretty broad compatibility with whatever docker environment, if you're interested in trying it out you may want to join the community slack and get it working: https://signoz.io/slack

Please don't take this wrong but if you want to adopt a new observability solution you'll need to invest some time in it. Signoz and Loki work well. If you can't set them up in a day it's time to look elsewhere but you really need to have a serious crack at them first.

Edit: typo

Your suggestion is valid and I did spend about 2 days in total across all 3. the least amount of time being the azure log analytics. I spent 10 minutes setting up the datadog agent and paying for that convenience.

The vicious cycle I see so many teams in is, the costs have gotten so high with SaaS tools, and the management of those costs so poor, that the solution is telling all teams "Don't for gods sake add monitoring to any more services, we can't afford to monitor anything more"

Not only is this a bad way to decide what gets monitored, it's also going to bite you during an outage.

Might be interesting to check out SigNoz (https://signoz.io)

Has all the three key signals like Metrics, Traces & logs in a single pane and it is much more cost effective compared to Datadog/NewRelic. And if you want, you can just host it your self as it is open source (https://github.com/signoz/signoz)

(PS: I am one of the maintainers)