News for hackers. Stuff that matters.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by philsnow on Wednesday April 08, 2015 @12:32PM (#9341940)
I got down to

     you should be able to install PCP from binary packages made available by the PCP development team on:

    ftp.pcp.io
and that threw up a red flag in my brain. Then I noticed that techblog.netflix.com doesn't redirect to https (and indeed can't serve https) (I use https://www.eff.org/HTTPS-EVERYWHERE and did not get content over https).

The directions _I_ saw for building from source looked pretty innocuous, but you might see a different set of directions if you're being MITMed. Observe an appropriate amount of caution.

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by mspier on Wednesday April 08, 2015 @12:40PM (#9342008)
Happy to discuss your concerns. PCP 3.10 should be available on Ubuntu's official repo pretty soon too.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by harshreality on Wednesday April 08, 2015 @04:12PM (#9343572)
His concerns seem plain to me. Unauthenticated channels for software distribution or software installation instructions are bad.

The techblog isn't using SSL, and the git pull url for PCP is using the git protocol which is also unauthenticated, rather than the authenticated https transport (ssh is only an option when user accounts make sense).

Someone's at a conference and follows the link over public wifi. They get the same page but with "here's how to get PCP: ftp evil.io or git clone git://git.evil.io/pcp" Even if the webpage were ssl-enabled so that an attacker can't rewrite the pcp.io links, an attacker or evil network operator could MITM git.pcp.io or ftp.pcp.io. (FTP?!)

Being in Ubuntu's repo doesn't make it safe if Ubuntu's maintainers have no (semi-)trustworthy way of getting the code.

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by justizin on Wednesday April 08, 2015 @06:17PM (#9344464)
Ubuntu's maintainers can check the MD5SUM file on ftp.pcp.io:

  ftp://ftp.pcp.io/projects/pcp/download/MD5SUM
The project seems to be hosted by Red Hat these days.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by willglynn on Wednesday April 08, 2015 @08:08PM (#9345016)
FTP is just as unauthenticated as everything else above, so having MD5SUMs available over FTP doesn't really change the situation.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by mrmondo on Thursday April 09, 2015 @06:13AM (#9346608)
Any chance you could provide standard Debian builds rather than Ubuntu specific?
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by nathan_scott on Thursday April 09, 2015 @09:53PM (#9352157)
To get the sources, you can use non-ftp/non-git access methods via the github mirror - https://github.com/performancecopilot/pcp
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by CWuestefeld on Wednesday April 08, 2015 @02:01PM (#9342610)
It's enormously frustrating to me that the Windows platform is so far behind.

I've been researching for a rebuild of our ecommerce site with the idea of a modern microservice architecture. Team skillset and some other considerations dictate a .Net environment.

Netflix and other companies have created a really rich platform, between logging and monitoring technologies like this; message queueing; deployment; and so on. In the Windows environment there's precious little to compare to.

One bright spot is the news story also on HN today [1] about MS announcing Docker-related Hyper-V technology - but in the next version of Windows server.

It might be that to satisfy those .Net compatibility wishes, we just go to Mono and do everything else in Linux.

[1] https://news.ycombinator.com/item?id=9342369

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by AloisMayr on Thursday April 09, 2015 @07:46AM (#9346870)
Want to try https://ruxit.com for monitoring your .NET ecommerce shop on Windows? It also monitors your host and real user interactions with your site.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by dkhenry on Wednesday April 08, 2015 @08:44PM (#9345144)
So I haven't looked into all the metrics that they are getting off of PCP but from an initial glance it doesn't look like they are getting a lot more information then you get off net-snmp.

In this one case the windows platform isn't going to hold you back too much. On top of that anything the .net stack on windows gives you access to WMI which can give you really simple really deep data from your application.

Don't buy into too much hype on what some of the big guys are doing on monitoring, most of their challenges seem to be around the scale they are monitoring at not with the depth of metrics.

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by brendangregg on Wednesday April 08, 2015 @09:41PM (#9345348)
For the initial release, you are right that you'll find most of these metrics in net-snmp. It's the selection that's more interesting, which is guided by the USE method: identifying utilization, saturation, and error metrics for key resources. We're missing some metrics (so do all monitoring tools), and we'll be extending Vector to include them (as can other people, as it's open source).

One challenge is that there are some metrics that the kernel doesn't make available, so we need to either use tracers to fetch them, patch the kernel, or use custom kernel modules. In recent weeks, I've been using (or trying to use): ftrace, perf_events, SystemTap, eBPF, and LTTng. I've also used others, but not in the last few weeks. I'm really looking forward to getting these advanced tools into Vector.

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by tonyarkles on Wednesday April 08, 2015 @08:06PM (#9345013)
> Team skillset and some other considerations dictate a .Net environment.

Not to be a dick but... that doesn't sound like a Windows limitation.

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by voltagex_ on Wednesday April 08, 2015 @07:32PM (#9344885)
I suspect that you could extend PCP (the monitoring component of this dashboard) to support Windows' performance counters. The profiling capabilities are certainly there.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by nathan_scott on Thursday April 09, 2015 @10:01PM (#9352185)
PCP can export many hundreds of Windows-specific counters (which it accesses via the PDH APIs). The Windows build does need some love though, as the OP mentioned - its out of date and needs a regular contributor to bring it up to date with the latest PCP sources.

Then Vector should Just Work (tm) for Windows servers too.

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by yareally on Wednesday April 08, 2015 @09:07PM (#9345218)
> Netflix and other companies have created a really rich platform, between logging and monitoring technologies like this; message queueing; deployment; and so on. In the Windows environment there's precious little to compare to.

Have you looked at any of the following for the various gripes you mentioned? I came from a primarily Linux background until recently, but I know .net and Windows have good tooling that extends beyond my modest list below. Looking to Microsoft to provide every option is probably not the best way to go about it, not sure if that was mostly what you were looking at so far.

https://www.rabbitmq.com/install-windows.html (company I work for uses it for messaging between their services)

http://nlog-project.org/ (we also use that as a base framework for our logging)

http://getakka.net/ (never used it on .net but can't do without it in the jvm)

https://github.com/dotnet/Orleans (sort of like Akka, but with additional features)

https://github.com/MSOpenTech/Redis (it's a bit behind the nix version, but still well maintained)

https://msdn.microsoft.com/en-us/data/gg577609.aspx (Netflix uses the Java port of that, fyi)

https://technet.microsoft.com/en-us/sysinternals/bb896653.as...

http://en.wikipedia.org/wiki/Resource_Monitor

https://www.jetbrains.com/dotmemory/

https://www.jetbrains.com/profiler/

http://www.appveyor.com/ (great for CI and easy to set up)

https://www.cygwin.com/ (when you miss a nix shell and tools)

http://en.wikipedia.org/wiki/Windows_PowerShell (for everything cygwin can't do easily)

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by wantab on Thursday April 09, 2015 @07:00AM (#9346722)
The web was built, and runs on, *nix/BSD. Windows is an outlier. Windows can't even get the slashes going the right way. There's a reason 80% of internet traffic does not use Windows.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by CWuestefeld on Thursday April 09, 2015 @10:26AM (#9347730)
I'm not sure how that was supposed to be helpful.

I also don't think it's strictly true. For sure the underlying networking came from Unix. But as for the web itself, once we got to real dynamic content, it seems to me that Microsoft were the ones that got things moving.

While the Unix world was mired in the awful world of CGI, Microsoft gave us high-performance ISAPI, and then Cold Fusion (also on the Windows platform) and Microsoft's ASP made programming a little more sane. While the Unix world tried to deal with JSP (which IMHO wasn't a very good solution), the Microsoft platform seems to have been the innovators for several years, until Ruby on Rails and then node.js and stuff started coming out.

Today, the Apache server powers far more sites than any other, it's true. But IIS shares the 2nd-place spot [1]. When you say "There's a reason 80% of internet traffic does not use Windows", that's pretty much true for the servers, but far off the mark when counting clients. And the reason for that is that Microsoft's strength hasn't historically been radical advances, but in figuring out ways to take the bleeding edge tech that doesn't really work quite right yet, and packaging it into commodity software that may not be as sexy as envisioned by those with the original ideas, but actually useful to the average guy.

[1] http://www.zdnet.com/article/web-servers-microsoft-iis-and-n...

EDIT - missing word "world" in 3rd para

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by wantab on Friday April 10, 2015 @04:29PM (#9356928)
Your examples presume there was something better than CGI at the time and the other products are better than something else. For one, I wouldn't be caught dead using any of the products you mentioned.

You claim IIS by using a ZDNet article from two years ago but the reality is IIS is number three behind Nginx and Apache. Still, being a distant #2 is nothing to brag about.

Now you're trying to claim clients are what powers the web but that's not the topic. What an amateur uses does not define what the professional uses. And to claim not wanting to be on the bleeding edge of things is no excuse for falling behind. Firefox and Chrome knocked IE off its perch years ago by being on the bleeding edge.

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by SEJeff on Thursday April 09, 2015 @10:19AM (#9347676)
Likely more than that seeing as how the majority of the load balancers and many of the routers / switches out there run some form of BSD (netscalar) or Linux (Arista), etc.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by fapjacks on Wednesday April 08, 2015 @04:20PM (#9343642)
I just tried installing this on a test VPS. PCP went well, but whenever I tried to input a "hostname" in Vector's UI to get stats, it just kept telling me it couldn't connect and to check hostname, regardless of what I put in the box. PCP was available and of course ports open and whatnot. I'm not sure what the problem was, but that was a bit frustrating. It looks like a slick product otherwise, especially for a first version! Thanks for releasing projects like this! I'll be trying again in the future, for sure.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by mspier on Wednesday April 08, 2015 @06:04PM (#9344377)
Can you open an issue on GitHub and post more details? Any errors on the JavaScript console?
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by samstave on Wednesday April 08, 2015 @05:27PM (#9344117)
After using Stackdriver for the last 18 months, I would never go back to rolling my own monitoring infra if I could avoid it. I had nagios, cacti, munin, graphite all running and had two ops guys pretty much 80% of their time managing it.

Stackdriver with pagerduty and I have 250,000 custom metrics being published and hundreds and hundreds of graphs on dozens of dashboards.

Although, I am looking at SignalFX to give even better version of this, but I manage nearly 1,000 machines with only a staff of four.

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by makeitsuckless on Thursday April 09, 2015 @08:07AM (#9346943)
Never heard of it, so checked out Stackdriver. Seemed very interesting, especially since it's geared at AWS (which we use).

Until I noticed the big red flag on the front page "Stackdriver is now part of Google."

Surely Google is going to kill this product and use the knowledge to offer something similar for Google's cloud services.

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by samstave on Thursday April 09, 2015 @11:41AM (#9348328)
We were concerned when.Google bought them and we had many meetings with them about this, I do not believe that will happen, but if that is still an issue for you, use signalFX.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by rubiquity on Wednesday April 08, 2015 @01:15PM (#9342297)
Am I wrong in saying this is a web interface that wraps PCP? So you can't really compare it to inspeqtor or collectd since those actually do the metrics collection.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by mspier on Wednesday April 08, 2015 @01:20PM (#9342331)
True. But that's the first release of Vector. We expect to make our custom PCP agents public soon.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by rubiquity on Wednesday April 08, 2015 @01:23PM (#9342361)
Sorry I should clarify I was making that statement for my understanding, not to degrade the project!
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by debaserab2 on Wednesday April 08, 2015 @12:23PM (#9341874)
What does this have that CloudWatch enhanced metrics doesn't? From the screenshots, the metrics look pretty similar. Not a slight at all against this project (it looks awesome), I'm just curious if your infrastructure is already AWS based what would cause you to choose a non-CloudWatch option.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by toomuchtodo on Wednesday April 08, 2015 @12:25PM (#9341891)
CloudWatch charges you per instance if you want 1 minute metrics instead of the standard (free) 5 minute metrics. Any tool that collects the data for you gets you out of that $3.50/instance/month detailed monitoring charge [1].

[1] http://aws.amazon.com/cloudwatch/pricing/

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by brendangregg on Wednesday April 08, 2015 @12:37PM (#9341981)
It's one second metrics, and has been designed to allow us to add other metric sources and visualizations (using, say, ftrace, perf_events, SystemTap, etc), although those are not in the initial release.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by thieving_magpie on Wednesday April 08, 2015 @03:08PM (#9343086)
It looks like it has some metrics that aren't present in cloudwatch without a lot of scripting, such as Memory utilization. I'm new to cloud watch though, so someone please correct me if that is wrong.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by Thaxll on Wednesday April 08, 2015 @02:27PM (#9342778)
Not sure what the difference is between that and Collect + Graphite / InfluxDB.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by nathan_scott on Thursday April 09, 2015 @10:05PM (#9352199)
There are many, many differences - there's some books about PCP that might help clarify PCP design points, see: http://pcp.io/documentation.html
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by Florin_Andrei on Wednesday April 08, 2015 @07:58PM (#9344978)
Make that Sensu + Graphite and you get fault detection and alerts too.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by easytiger on Thursday April 09, 2015 @06:44AM (#9346683)
Yea, but i guess they connect directly to the daemon on each host so no intermediate db. Which sounds like a terrible idea
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by capkutay on Wednesday April 08, 2015 @12:45PM (#9342049)
The charts look like they were built with nvd3. Can anyone confirm/deny?
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by mspier on Wednesday April 08, 2015 @12:49PM (#9342083)
That's right. Any suggestion of better reusable charts?
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by forrestthewoods on Wednesday April 08, 2015 @01:32PM (#9342414)
Honestly? Just write your own SVG code. If you inspect the elements the output data is super simple and easy to understand. NVD3 just wraps D3.js which just wraps utilities that output relatively basic data. Well, d3js is a data-binding system that's way too god damned complicated if all you want to do is make some simple charts which is how it's used 99% of the time.

I've spent more time trying to manipulate chart libraries into doing almost the same thing but just different enough to cause pain and suffering. Output your own path data and it's a million times easier.

For reference here's what I made: http://forrestthewoods.com/unbalanced-design-of-super-smash-...

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by capkutay on Wednesday April 08, 2015 @01:20PM (#9342332)
IMO nvd3 is one of the better high-level charting libraries in terms of number of charts, out-of-the box styles and interactivity.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by SEJeff on Thursday April 09, 2015 @10:21AM (#9347696)
It isn't d3 based (uses canvas instead of svg), but flot is fantastic as well:

http://flotcharts.org

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by tdicola on Wednesday April 08, 2015 @01:52PM (#9342541)
Wow this looks great, thanks for releasing it! Nice to see even 'simple' stuff like this that can help people who aren't running at Netflix scale is still released.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by vizzah on Wednesday April 08, 2015 @04:15PM (#9343600)
Could anyone who tried this monitoring tool compare it to Munin?
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by victorhooi on Wednesday April 08, 2015 @04:31PM (#9343734)
This is a slight tangent...but does anybody know what UI toolkit Netflix is using for this?

Or if it's in-house, any info on whether they have, or might release it?

I see bootstrap-submenu.css mentioned, but not Bootstrap itself:

https://github.com/Netflix/vector/tree/master/app/css

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by mspier on Wednesday April 08, 2015 @06:36PM (#9344575)
It's bootstrap (see bower.json dependencies), but with our own layer on top of that.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by xbryanx on Wednesday April 08, 2015 @01:26PM (#9342377)
I'm curious how this compares to Zabbix's agent and server. Does PCP give you finer grained details, or is it possibly more lightweight?
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by nathan_scott on Thursday April 09, 2015 @11:23PM (#9352418)
PCP is much finer-grained than Zabbix in terms of the metrics it makes available (esp. from the Linux kernel); not sure on Zabbix costs but PCP is quite light on all resources (mem, cpu, net) and very robust.

I've worked on production systems where everything else was failing (hardware, kernel, applications) but PCP kept chugging along, recording and telling the sad story to anyone that would listen.

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by strunz on Wednesday April 08, 2015 @12:03PM (#9341711)
Anyone care to compare this to something like collectd? https://collectd.org/
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by mspier on Wednesday April 08, 2015 @12:19PM (#9341854)
Goal is a bit different. Vector doesn't collect and persist metrics. We needed something that had as little overhead as possible so it could be deployed to all our hosts and simplify the process of analyzing those metrics.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by toomuchtodo on Wednesday April 08, 2015 @12:24PM (#9341882)
If its not collecting and persisting metrics, is it more of a glorified htop?
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by brendangregg on Wednesday April 08, 2015 @12:34PM (#9341956)
Not really. htop doesn't visualize historical trends from numerous sources at the same time. Vector also allows us to interface with other metric sources, beyond /proc.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by cpwatson on Wednesday April 08, 2015 @12:28PM (#9341917)
doesn't htop require you to go onto the box? We haven't released our custom pcp modules yet which allow more complex visualizations such as flamegraph generation from perf event sampling.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by lscotte on Wednesday April 08, 2015 @01:34PM (#9342430)
Yes, exactly - htop is a curses application that shows instantaneous samples of a few system metrics, but it mostly lists processes - this is nothing like Vector at all...
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by davidu on Wednesday April 08, 2015 @01:06PM (#9342231)
Wait, really?

Why not? Storage is cheap. Do you use something else to get historical visibility into metrics?

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by brendangregg on Wednesday April 08, 2015 @03:25PM (#9343225)
Yes, Atlas, which is also open source: http://techblog.netflix.com/2014/12/introducing-atlas-netfli... . Atlas monitors cloud-wide, and stores historical metrics at a one minute granularity.

Vector is for per-instance custom drilldowns. I gave a talk last year where I showed how they both fit together: http://www.brendangregg.com/blog/2014-09-27/from-clouds-to-r...

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by davidu on Wednesday April 08, 2015 @03:40PM (#9343331)
Got it... and thank you Brendan!
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by zantana on Wednesday April 08, 2015 @03:24PM (#9343207)
I think the reason is they have a bunch of ephemeral hosts which they don't want to put in a central web interface or collect statistics on. They are only interested in getting some insight as to what went wrong with this host so they can fix it in the future.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by toomuchtodo on Wednesday April 08, 2015 @02:01PM (#9342609)
Grafana, Graphite, StatsD, Skyline.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by fche on Thursday April 09, 2015 @08:58PM (#9352007)
pcp is happy to collect & persist metrics; vector just doesn't happen to render them. pcp offers other ways to interact with live or archive-saved data, including other webapps.
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by falcolas on Wednesday April 08, 2015 @10:12PM (#9345456)
Neither does collectd, it just gathers and transmits the data to an endpoint you specify. It's also quite lightweight (once you address the memory leaks, at least).
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by oimaz on Wednesday April 08, 2015 @01:19PM (#9342328)
Where can I find deb packages for pcp version 3.10 or higher?
Re:Introducing Vector: Netflix's On-Host Performance Monitor... by jmedefind on Wednesday April 08, 2015 @02:20PM (#9342738)
The git repo has a MakePkg script that will generate deb packages for you.

I found it really easy to use.

Re:Introducing Vector: Netflix's On-Host Performance Monitor... by lscotte on Wednesday April 08, 2015 @01:42PM (#9342477)
See http://pcp.io/docs/installation.html#deb or direct link at ftp://ftp.pcp.io/projects/pcp/download/deb/ - but I only see packages for i386 arch, so you may have to compile from source if on amd64 or other architecture.