News for hackers. Stuff that matters.
Re:InfluxDB 1.0 GA Released: A Retrospective and What’s Next by scrollaway on Thursday September 08, 2016 @11:09AM (#12453697)
Influx is absolutely amazing. We're using it along with grafana to store and display our desktop and web apps' analytics (it completely replaced GA), store and display HTTP health analytics (piping custom uwsgi request logger into UDP input), and do continuous analysis of Hearthstone games.

It's incredibly fast and the grafana/influx/telegraf stack is really cool to play with. Highly recommended.

Re:InfluxDB 1.0 GA Released: A Retrospective and What’s Next by chrissnell on Thursday September 08, 2016 @01:37PM (#12455186)
Want to hear more about your HTTP analytics. Particularly interested in collecting selenium/sitespeed.io-style page timing metrics. There are a million commercial solutions for this but so few open source options.
Re:InfluxDB 1.0 GA Released: A Retrospective and What’s Next by scrollaway on Thursday September 08, 2016 @01:57PM (#12455399)
Glad you asked! I'm pretty proud of this hack :)

uWSGI has a very flexible logging system. You can create a log with just requests, and completely customize the format of the line.

On top of that, uWSGI supports creating multiple log targets, and supports logging directly into a UDP socket.

If you make it match InfluxDB's line format (https://docs.influxdata.com/influxdb/v1.0/write_protocols/li...), and you set up InfluxDB to accept connections from a matching UDP socket, then uWSGI can essentially log every request directly into influx.

Like this for example:

    logformat = uwsgirequest,host="%(host)",status=%(status) msecs=%(msecs),size=%(size) %(epoch)000000000
    logger = file:/var/logs/uwsgi/uwsgi.log
    req-logger = file:/var/logs/uwsgi/requests.log
    req-logger = socket:127.0.0.1:8083
You can also add 'path="%(var.PATH_INFO)"' to that but there is a potential data injection vulnerability if you don't re-parse the output.

With that, you can create nice graphs with Grafana to analyze your loading times, error rates, page sizes etc:

https://i.imgur.com/fzbXgUd.png

You can also set up Kapacitor to identify when your page load times are abnormally increasing. We had the idea after a database connection leak increased our average loading times by 300%.

Do note that this doesn't batch inputs, so 1 request = 1 connection to influx. But it's pretty easy to put a middleware agent in between those to do validation and batching.

The nice thing about all this is it's application level timing metrics. There's not a lot of ways to get that, except with application middleware which can't always catch everything you want.

Re:InfluxDB 1.0 GA Released: A Retrospective and What’s Next by chrissnell on Thursday September 08, 2016 @03:03PM (#12456205)
Sweet. Since it's UDP, it's connectionless so any penalty for not batching data points is likely minor. We're not a Python/uWSGI shop, but it looks like I might be able to emulate the Influx format as you have done using this nginx logging module:

https://github.com/vkholodkov/nginx-udplog-module

Re:InfluxDB 1.0 GA Released: A Retrospective and What’s Next by boduh on Friday September 09, 2016 @07:17AM (#12461082)
I am new to InfluxDB but I am interested in using it for some desktop app analytics. Are there any resources that you would recommend I start with? Maybe some from your own experience? How hard was for you to start using it for desktop app analytics?
Re:InfluxDB 1.0 GA Released: A Retrospective and What’s Next by scrollaway on Friday September 09, 2016 @12:45PM (#12463888)
Logging datapoints into influx is pretty straightforward. For example, logging every time your app is started, adding a tag for the platform, the region, logging datapoints whenever some important settings are toggled, etc.

Once you have this data, set up grafana and build some graphs to query it. Grafana has a graphical query builder, and internally it's essentially a sql-like. There's good tutorials in the influx/grafana docs for all that.