Re:Ask HN: How do you handle logging?
by jasonrojas on Thursday August 29, 2019 @01:29AM
(#20826908)
“ELK stack is free only in bits but not in engineering time.” — best thing I’ve read all week. Thank you.
This is so true!
After 3-4 attempts of 7 days over the last 1 year. I finally had it up and running happliy.
Re:Ask HN: How do you handle logging?
by AlchemistCamp on Thursday August 29, 2019 @11:18AM
(#20830258)
How long would it take you to do it again on another project?
Having walked this experience myself a few times. Borrowing from the GP: Repeats take an additional 3-4 attempts of 7 days over 1 year.
That would still take 3 days.
Sentry is not for logging really. It's designed for errors/exceptions. I believe you should use rollbar/sentry/airbrake regardless of whether you use centralised logging. Or even before it.
+1 for sentry. You can actually self host it for free too!
Another +1 for sentry. I used it (basically accidentally) for a year because it's part of the cookiecutter-django framework[1]. I don't think there was a single issue with it the whole time we used it.
+1. Sentry is amazing. We self host although they stopped shipping updates for over a year until recently. I'd pay for saas if it was up to me, tho.
> it's more likely for your network to be down than for your disk to fail
For most people, the network being down means they can't reach the disk.
Buffering unsent logs via local disk or RAM is critical due to network flakiness for sure, but not logging over the network as well is a bad idea 100% of the time.
> For most people, the network being down means they can't reach the disk.
If you're talking about cloud solutions, that's what instance stores are for. Logging to local disk and then forwarding is still the best answer. And there's still a lot of world out there that's running on actual hardware.
Re:Ask HN: How do you handle logging?
by scarface74 on Wednesday August 28, 2019 @09:50PM
(#20825967)
He specifically mentioned mobile apps. When developing mobile apps, you almost always operate in a “semi connected” state where ideally you can function without network access and rely on syncing.
He's actually working in the backend, not in the mobile app itself.
Re:Ask HN: How do you handle logging?
by linsomniac on Wednesday August 28, 2019 @06:31PM
(#20824737)
I set up Sentry a few years ago, based on a PyCon hallway-track talk I saw. We had a few false starts with it, but have integrated it with a couple of our newer platforms and have liked it.
It takes a kind of "ticket" approach to messages, it'll deduplicate and combine similar errors, and you go into a dashboard and see "We got ten thousand of this error, let me track it down, fix it, ack it and see if we keep getting it."
Re:Ask HN: How do you handle logging?
by MuffinFlavored on Wednesday August 28, 2019 @11:58PM
(#20826558)
It blows my mind that there isn't an easily Googable example of how to pipe JSON logs on disk to logstash. It's 2019. The software should be able to come online, check what logs it has already indexed, and then `tail -f` files matching a pattern. I really thought this was a more common use case but it's obvious things went the "directly log to socket" route you mentioned.
filebeat is one tool that does that, watching logs and piping to logstash.
I believe you can skip logstash now and have filebeat go directly to your elastic search indexer.
This simplified the ELK stack setup a whole bunch.
We do a similar thing. Additionally, we use logrotate to move these files into S3, which later get ETLed into Parquet so we can scoop it for long term analytics.
One should keep in mind that ELK is quite expensive once you reach the point where you need to pay for it.
How so? Honest question as I'm curious where the hidden costs might be.
From my limited experience the ElasticSearch requires tons of excessive RAM, storage and CPU. The bloat rate is about 1.5X to 2X your data size.
With cloud options available for both Elasticsearch and Splunk these days difficulty of setup / ease of use may be better evaluated from the client perspective. I’ve setup both ES and Splunk in the past few years and it’s not terribly different at lower end scale (< 100 GB logs / mo). But currently Splunk is not as good as ELK for metrics and vice verse and with the recent SignalFX acquisition that may change in a couple years but it definitely isn’t now. Also, there’s tons of options for streaming logs to ES besides Logstash including Filebeat which is at least written in Go (Splunk’s forwarders are probably in C given I swear they’ve been mostly the same since the early 2000s).
Elastic cloud is not a great option for beginners, at least not at this point. For instance there is no purge of old indexes out of the box, a lot of configuration options are not available, performance issues are inscrutable.
I think it needs at least of few more months or years to be a no-brainer to choose.
Re:Ask HN: How do you handle logging?
by TrickyRick on Thursday August 29, 2019 @09:39AM
(#20829368)
https://sematext.com/cloud/ Sematext Cloud has been working well for me in a relatively small startup (Couple of thousands active users). Cheap solution and easy to get started.
> network to be down than for your disk to fail
If network is down then how your users are going to reach your app?
Re:Ask HN: How do you handle logging?
by GauntletWizard on Wednesday August 28, 2019 @08:18PM
(#20825459)
Netsplits happen. User actions can be queued on a server that has just lost connectivity. Your logging server can be down or broken itself. A file on a unix system isn't going anywhere fast.
In Python, use the RotatingFileHandler to avoid running out of space.
2) Incrementally forward your log files to a server using something like fluentd that can pre-aggregate and/or filter messages.
Big advantage of logging to disk: if logging server is unreachable, forwarder can resume once it's up again. If you log directly over network, if things fail the very log messages you need to troubleshoot the failure are potentially gone.
3) Visualize. Create alerts.
I've evaluated a bunch of logging solutions. Splunk is the best, and affordable at low data volumes (they have a pricing calculator, you can check for yourself). It's medium hard to setup.
Sumo Logic is the easiest to set up, and at low data volumes, prices are similar to Splunk. You can get something working within an hour or less.
ELK stack is free only in bits but not in engineering time.
I've not actually tried Sentry.io but I saw it at PyCon and it looks pretty impressive. If you only care about tracking errors/events and not about general-purpose logging functionality per se, I would take a serious look at it.