News for hackers. Stuff that matters.
Re:Monitoring Is a Pain by wspeirs on Tuesday June 27, 2023 @11:01AM (#36493867)
>Cost, which still isn't solved yet.

I'd argue it _is_ solved... store your logs in S3. At ~$0.02/GB you can store a _lot_ of logs for like $20. The problem is that _most_ solutions (Honeycomb included) are SaaS-based solutions, and so they have to charge a margin on top of whatever provider is charging them.

You just need a tool (like log-store.com) that can search through logs when they're stored in S3!

Re:Monitoring Is a Pain by phillipcarter on Tuesday June 27, 2023 @11:06AM (#36493941)
At the small volume of data where you could "just put stuff in S3 and search it later", the cost of tracing with a SaaS vendor is also very small. Far smaller than the developer time needed to set up something else.
Re:Monitoring Is a Pain by wspeirs on Tuesday June 27, 2023 @11:32AM (#36494333)
>At the small volume of data where you could "just put stuff in S3 and search it later"

I was arguing exactly the opposite... leverage S3's near-infinite storage so you can store whatever you'd like. Searching through it _can_ be fast with the right tool.

Re:Monitoring Is a Pain by OldManRyan on Tuesday June 27, 2023 @11:14AM (#36494058)
Depends on how much you value not being vendor locked in. I'll spend a week of dev time in order to save a month or two in migration pain later down the line when my vendor jacks up cost. See Datadog as an example.
Re:Monitoring Is a Pain by phillipcarter on Tuesday June 27, 2023 @11:23AM (#36494197)
You can have your cake and eat it too with OpenTelemetry (see my original post). I don't see how that has to do with the cost of tracing being a factor in inhibiting its adoption though? I agree that vendor neutral instrumentation is worth it.
Re:Monitoring Is a Pain by OldManRyan on Tuesday June 27, 2023 @11:32AM (#36494329)
> ...the cost of tracing with a SaaS vendor is also very small. Far smaller than the developer time needed to set up something else.

I'm mainly responding to this point regarding developer time outweighing the cost of just going with SaaS vendor that provides that capability. I don't think we disagree with each other though

Re:Monitoring Is a Pain by chimeracoder on Tuesday June 27, 2023 @12:43PM (#36495334)
> You just need a tool (like log-store.com) that can search through logs when they're stored in S3!

As someone who has run logging systems professionally, "just" is doing a lot of work in that sentence.

Re:Monitoring Is a Pain by roflyear on Tuesday June 27, 2023 @02:39PM (#36496965)
Depends on what you're logging. If you're logging everything it can add up.
Re:Monitoring Is a Pain by poulsbohemian on Tuesday June 27, 2023 @03:22PM (#36497481)
I spent a lot of years traveling around the world fixing problem for major companies... logs aren't enough. Not only do you have the issue of auditors who are scared you might put things in logs that shouldn't be there, but there are security / access problems and the very basic problem of "did we log the thing we now need?" Not to say that logs can't be helpful, but they generally aren't enough on their own.