News for hackers. Stuff that matters.
Re:A simple way to get more value from metrics by bitcharmer on Saturday May 30, 2020 @10:14AM (#23361319)
You'd be surprised how many serious tech shops have close to zero performance metrics collected and utilised.

I've done this in fintech a few times already and the best stack that worked from my experience was telegraf + influxdb + grafana. There are many things you can get wrong with metrics collection (what you collect, how, units, how the data is stored, aggregated and eventually presented) and I learned about most of them the hard way.

However, when done right and covering all layers of your software execution stack this can be a game changer, both in terms of capacity planing/picking low hanging perf fruit and day to day operations.

Highly recommend giving it a try as the tools are free, mature and cover a wide spectrum of platforms and services.

Re:A simple way to get more value from metrics by chucky_z on Saturday May 30, 2020 @07:00PM (#23365594)
I'd like to followup on this and say InfluxDB is fantastic -- with the caveat that you know exactly what you want, and use it with that in mind. It's great at storing some metrics with high volume, but I've found the OSS version falls over hard in some circumstances (high cardinality in tags absolutely murders performance, making it nearly unusable).

I'm in the middle of doing a test-run of vmagent + victoria-metrics as a mostly-replacement (see: not replacing the cases where it's known exactly what's wanted, and used explicitly for that), and victoria-metrics fully support telegraf pushes which is a big bonus.

Also they're apparently going to be dropping InfluxQL entirely, for Flux, which is weird to me? The docs don't outright state this, but InfluxDB 2.0 has only Flux examples in the docs I can find. It's a better language, however the learning curve is not trivial.

Re:A simple way to get more value from metrics by sciurus on Saturday May 30, 2020 @08:41PM (#23366185)
Thankfully they still support influxql via transpiling it to flux. They've talked about supporting promql eventually as well.

Tickscript is gone in 2.0, though. I'm not in love with Flux, but I certainly won't miss tickscript.

Re:A simple way to get more value from metrics by chucky_z on Sunday May 31, 2020 @01:31PM (#23371035)
Oh, interesting. What's the core difference between Tickscript and Flux? Was Tickscript what Kapacitor used for InfluxDB 1.7?
Re:A simple way to get more value from metrics by sciurus on Monday June 01, 2020 @12:58PM (#23380697)
> Was Tickscript what Kapacitor used for InfluxDB 1.7?

Yes. The learning curve was steep, and debugging was near-impossible. Flux is rough too, but it's already getting better documentation and tooling support than Tickscript had.

Re:A simple way to get more value from metrics by benraskin92 on Saturday May 30, 2020 @10:37AM (#23361478)
Might also want to give Prometheus a try as it's extremely simple to setup, and it is very well supported in the open source community.
Re:A simple way to get more value from metrics by contravariant on Saturday May 30, 2020 @10:18PM (#23366631)
Note that this isn't an either/or. You can use telegraf to push data to prometheus, or pull data from prometheus compliant http endpoints. You can also use grafana to show data from both InfluxDB and Prometheus.

This might be obvious to some but I thought it worth mentioning.

Re:A simple way to get more value from metrics by bitcharmer on Saturday May 30, 2020 @11:46AM (#23361931)
Prometheus was another option but millisecond-precise timestamps are a deal breaker in my field.
Re:A simple way to get more value from metrics by benraskin92 on Saturday May 30, 2020 @12:55PM (#23362546)
That makes sense -- have you taken a look at M3DB (https://www.m3db.io/)?
Re:A simple way to get more value from metrics by amenod on Saturday May 30, 2020 @12:00PM (#23362029)
Curious, would microseconds suffice? Or are we talking about higher precision still?
Re:A simple way to get more value from metrics by simcop2387 on Saturday May 30, 2020 @12:15PM (#23362162)
Very likely they need nanosecond precision, https://www.thetradenews.com/fintech-firms-reduce-trading-ti...
Re:A simple way to get more value from metrics by bitcharmer on Saturday May 30, 2020 @05:30PM (#23364934)
Just like the sibling comment indicated, we (HFT, algo trading) need nanosecond precision for some metrics.
Re:A simple way to get more value from metrics by chrisweekly on Saturday May 30, 2020 @10:30AM (#23361431)
Agreed; same exp (w many years of web perf optimization as ui arch and/or perf strategy consulting gigs. Even after all these years, simply using WebPageTest to capture test runs and share the results is often a transformational experience.
Re:A simple way to get more value from metrics by duckmysick on Saturday May 30, 2020 @06:45PM (#23365507)
> There are many things you can get wrong with metrics collection

Can you share the important things that are overlooked?