News for hackers. Stuff that matters.
Re:Show HN: Visualizing disk IO activity using log-scale ban... by apankrat on Wednesday April 11, 2018 @12:07PM (#16812537)
Author here.

We had a need for a tool to graph IO activity at arbitrary sampling rates, so there was finally a solid excuse to do some datavis work :)

The thing I wanted to show is not the program itself, but the idea of factoring values and displaying factorization to cover several magnitude levels in one go. I suspect that this has been done before, but I couldn't find anything similar when I was researching this beforehand.

Re:Show HN: Visualizing disk IO activity using log-scale ban... by arnioxux on Wednesday April 11, 2018 @05:49PM (#16816071)
This visualization is really cool and innovative! It's useful for seeing the changes in the stream of values even if they are small relative to absolute magnitude (e.g., 1000001 -> 1000005 -> 1000003 etc).

The tradition way to solve this is to rescale the y axis (for example google stock price chart ranges from 1000-1200). But this doesn't work for graphing IO because it spikes up/down too often in a huge range.

Your visualization plots the base-1024 digits so whenever the IO is sustained at a certain range, the top few digits will be held constant and the next digit will show the variation within that scale.

Unfortunately I don't think this solves the spiking problem completely. It's mentally hard to parse since two adjacent values are only comparable whenever their top digits match.

For example it's only meaningful to compare (1gb+2mb+3kb) with (1gb+2mb+5kb) on the kb graph. If it spikes down to 0gb+0mb+0kb, you need to switch to reading the gb graph (since comparing the kb value to one before is pretty meaningless unless for some crazy reason you actually care about the mod 1024 value of two very different numbers).

To fix this, you should color code the "runs" of values that are meaningful to compare! In other words, kb values have the same color only if their gb/mb values match. And mb values have the same color if their gb value matches.

Re:Show HN: Visualizing disk IO activity using log-scale ban... by jdmichal on Thursday April 12, 2018 @10:21AM (#16821182)
I like the idea of using a color signal to show a changed value, with the change cascading to lower "digits". So, for instance, if the GB change, then so do MB, KB, and B. If GB and MB remains the same, then only KB and B will change.

Hard part would be how to do this over a long span of columns without the whole thing looking like a unicorn puked on the chart...

Re:Show HN: Visualizing disk IO activity using log-scale ban... by LukeShu on Wednesday April 11, 2018 @12:25PM (#16812720)
I've never seen something similar before either, and it is a very useful visualization.

I'm thinking about how to do something similar in my monitoring. How would you extend it to allow zooming on the X-axis? With a continuous log scale, or a simple linear scale, it's easy to know how to compress the data horizontally: take the min/max/avg of the datapoints that go in to one horizontal pixel. I'm not sure how to apply that to your discrete log scale; it seems to work on the assumption that points along the X-axis are discrete.

Edit: I said Y-axis, I meant X-axis. I'm embarrassed.

Re:Show HN: Visualizing disk IO activity using log-scale ban... by apankrat on Wednesday April 11, 2018 @12:29PM (#16812765)
> How would you extend it to allow zooming on the Y-axis?

Well, the whole point is to NOT need to do any Y-scaling. What would be the case for zooming in?

Re:Show HN: Visualizing disk IO activity using log-scale ban... by LukeShu on Wednesday April 11, 2018 @12:44PM (#16812933)
I'm sorry, I meant X-axis. I'm embarrassed.
Re:Show HN: Visualizing disk IO activity using log-scale ban... by apankrat on Wednesday April 11, 2018 @01:00PM (#16813144)
Ha. Good question, actually.

The simplest option would obviously be to just show a conventional candlestick when zoomed out. There might be an option to do something interesting with rendering the stick when min/max have the same higher-order factors... but it might also end up looking like a royal mess.

Re:Show HN: Visualizing disk IO activity using log-scale ban... by arunc on Wednesday April 11, 2018 @12:17PM (#16812632)
Nice work. Do you plan to open source it?
Re:Show HN: Visualizing disk IO activity using log-scale ban... by apankrat on Wednesday April 11, 2018 @12:23PM (#16812699)
I see no point in that. It's trivial to do in several hours from scratch.
Re:Show HN: Visualizing disk IO activity using log-scale ban... by arunc on Wednesday April 11, 2018 @12:28PM (#16812749)
Even if you think your work is trivial, it may not be so for others. :) Also, gives an opportunity for people to extend the current feature set. Entirely your choice though.
Re:Show HN: Visualizing disk IO activity using log-scale ban... by IMTDb on Wednesday April 11, 2018 @07:58PM (#16816897)
And a great way for anyone wanting to start some windows programming to see an example of a nice simple program with a GUI that interacts with WIN32 apis
Re:Show HN: Visualizing disk IO activity using log-scale ban... by defgeneric on Wednesday April 11, 2018 @03:59PM (#16814906)
I came to the comments to look for a link to the source. I'm a systems programmer but don't have much knowledge of how you guys do things in Win32. A ~70kb executable with nice graphs was enticing and I was really interested to see the source. But as it is, I can't afford to download closed freeware that can't be audited just for some nice graphs.
Re:Show HN: Visualizing disk IO activity using log-scale ban... by 72deluxe on Wednesday April 11, 2018 @05:53PM (#16816100)
Not meaning to be rude, but how do you run Windows itself if you cannot audit it? How do you fetch updates from Microsoft if you cannot audit them? What about other applications, eg. anything from Adobe, Microsoft (for Office etc.) ?

Seems to be an odd complaint to make, to me. You could run ProcessMonitor to see what the application is doing.

Or read the excellent Windows Internals books if you've got a few weeks spare (and for Windows 10 it is missing Volume 2 at the moment - given that Windows 10 is a constantly moving target the poor book will be out of date as soon as it is written, let alone published).

Do you audit apps on Android / iOS / macOS too? I find my Android device dials home all the time and flings data out at an astounding rate, which is disappointing.

How do you use your satellite/cable/TV these days? How do you audit your entertainment systems? Mine came with the GPL on a bit of paper in the box given that it's got no end of Open Source software used inside it but I was unable to audit it - how did you manage it?

Do you read the source code of every Open Source application before running it? Have you combed over the entire Linux kernel before booting it? What do you think to Peter H. Anvin's bootloader code given that you use it every single time Linux boots? How long did auditing that take?

At what point do you start to trust an executable or code?

Re:Show HN: Visualizing disk IO activity using log-scale ban... by theclaw on Wednesday April 11, 2018 @07:47PM (#16816849)
I would assume that someone else would be one doing the auditing in this case. Some companies have paranoid IT.
Re:Show HN: Visualizing disk IO activity using log-scale ban... by 72deluxe on Thursday April 12, 2018 @04:36AM (#16819018)
That's a lot of code to read before using something!
Re:Show HN: Visualizing disk IO activity using log-scale ban... by blattimwind on Wednesday April 11, 2018 @04:54PM (#16815522)
Windows as a desktop operating system ships everything you need to write GUI apps with a plain C API, which allows to build fairly small tools. I'd guess this probably just uses good ol' fashioned GDI.
Re:Show HN: Visualizing disk IO activity using log-scale ban... by apankrat on Wednesday April 11, 2018 @06:05PM (#16816171)
Yep, Win32 API + GDI + UPX.
Re:Show HN: Visualizing disk IO activity using log-scale ban... by boromi on Thursday April 12, 2018 @04:14PM (#16824525)
by UPX do you mean https://github.com/upx/upx
Re:Show HN: Visualizing disk IO activity using log-scale ban... by eps on Wednesday April 11, 2018 @05:22PM (#16815820)
Just stick it into a VM like the rest of us.
Re:Show HN: Visualizing disk IO activity using log-scale ban... by boromi on Wednesday April 11, 2018 @12:43PM (#16812915)
How's the GUI setup?
Re:Show HN: Visualizing disk IO activity using log-scale ban... by apankrat on Wednesday April 11, 2018 @01:07PM (#16813206)
Basic window with stuff drawn on it with GDI and GDI+ in a double-buffered way.

Edit: Scratch the GDI+ bit, it's just GDI.

Re:Show HN: Visualizing disk IO activity using log-scale ban... by dracodoc on Wednesday April 11, 2018 @02:13PM (#16813834)
Maybe you can use slightly different color/background color for the bands? That way it's more obvious that the 3 bands are separated instead of one continuous band.
Re:Show HN: Visualizing disk IO activity using log-scale ban... by apankrat on Wednesday April 11, 2018 @03:32PM (#16814660)
It started this way, actually. But in the day-to-day use it ends up being an embellishment, because there would typically be a string of read/writes that fall on the first pixel of every band so they'll highlight the boundaries, for free.
Re:Show HN: Visualizing disk IO activity using log-scale ban... by dracodoc on Thursday April 12, 2018 @09:34AM (#16820725)
Different color could be a little bit distracting for minimal design, maybe there could be gaps between bands. The bands are independent with each other, a 999M 999K 999B band look like a continuous band but it should be 3 separate bars.

This is just pure nick picking, your current design probably suit your need already.

Re:Show HN: Visualizing disk IO activity using log-scale ban... by tlarkworthy on Wednesday April 11, 2018 @07:03PM (#16816576)
Re:Show HN: Visualizing disk IO activity using log-scale ban... by gg3 on Wednesday April 11, 2018 @12:19PM (#16812658)
How are you reading disk I/O rates? Is there a logs from Windows on this?
Re:Show HN: Visualizing disk IO activity using log-scale ban... by apankrat on Wednesday April 11, 2018 @12:22PM (#16812683)
IOCTL_DISK_PERFORMANCE

https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...

This relies on IO performance counters enabled for the target drive, but that's typically the case. If they are off, they can be enabled with "diskperf.exe".

Re:Show HN: Visualizing disk IO activity using log-scale ban... by ComodoHacker on Wednesday April 11, 2018 @03:04PM (#16814376)
>a tool to graph IO activity at arbitrary sampling rates

Why sampling rate is fixed then?

Re:Show HN: Visualizing disk IO activity using log-scale ban... by apankrat on Wednesday April 11, 2018 @03:13PM (#16814472)
Because public build is not the same as the one we use internally.
Re:Show HN: Visualizing disk IO activity using log-scale ban... by kiki_jiki on Wednesday April 11, 2018 @11:21PM (#16817726)
A Rainmeter version of this would be very cool.