News for hackers. Stuff that matters.
Re:Bhyve – BSD Hypervisor by SEJeff on Friday August 29, 2014 @04:58PM (#8244477)
Facebook uses Linux "where it is faster than FreeBSD, which is basically everywhere" per a recent whitepaper or presentation I saw. What I found interesting, is that Facebook runs FreeBSD in production at all.

It just goes to show how solid the BSDs really are. I use the hell out of keepalived (OSS vrrp daemon) + conntrackd on Linux, but strongly prefer carpd and pf. It moves a lot slower, but it caters to a different audience.

Re:Bhyve – BSD Hypervisor by allegory on Friday August 29, 2014 @05:27PM (#8244596)
An anecdote.

When the company I was working for at the time's AC failed in the machine room and melted three racks of kit, all 7 of our FreeBSD machines at the time were back online from scratch on hot spare kit in under two hours.

Our Windows guys were still working on restoring the master Active Directory pair and DFS cluster TWO DAYS LATER even with a DR strategy in place.

Quality engineering, documentation and automation ability shits on everything else IMHO and that's where FreeBSD lands every time. Performance isn't a major thing for me.

Linux on the other hand is heading in the direction of windows. I'll probably get flamed by the systemd proponents here but it took me three hours to work out how to debug this command:

   timedatectl set-ntp yes
Which returned:

   Failed to issue message call
Re:Bhyve – BSD Hypervisor by vezzy-fnord on Friday August 29, 2014 @06:15PM (#8244748)
Linux on the other hand is heading in the direction of windows.

The reasons for this, in part, are the different mentalities in the Linux and BSD communities. The Linux community used to be quite adamant about how "Linux is not Windows" and sticking to their paradigms, but more recently they've been absolutely obsessed with pipe dreams of conquering the desktop, and have been completely undermining their technical workflow in the process. Instead of "Linux is not Windows", it seems companies and communities alike have decided to pander to a crowd that has no respect for computing and Unix culture, who are used to closed systems that hold your hand, and quality is suffering in the process.

The BSD people on the other hand, have stayed true to their heritage, and show no signs of compromise. BSD people also tend to be very familiar with Linux as well, whereas the opposite is much less common.

Re:Bhyve – BSD Hypervisor by lozf on Saturday August 30, 2014 @02:14PM (#8247114)
Reminds me of the old saying that <em>"Linux is for people who hate Windows, and *BSD is for those who love Unix."</em>
Re:Bhyve – BSD Hypervisor by nisa on Friday August 29, 2014 @07:47PM (#8245049)
> Failed to issue message call

Reminds me of some mad attempts to debug some problem in Unity/GNOME... I gave up pretty fast after stranding on orphaned freedesktop.org pages.. I distinctly remember ranting on reddit about the problems and I've got a few .odt presentations from GNOME developers that not so really explained some concepts.. it's a different world. I can't judge if it's better this way for the Desktop but I've found it impossible to debug without sacrificing enormous time and energy about things I'd never wanted to deal with.

Stark contrast has been xCAT by IBM.. it's used to manage clusters of 10000 machines and it's only small readable Perl and Shellscripts.. totally awkward at the beginning but once you get a grasp it's a flexible power tool.

Re:Bhyve – BSD Hypervisor by anon4 on Saturday August 30, 2014 @05:20AM (#8246001)
A friend of mine calls it "desktrap loonix".

The sad part is it's still better than windows.

Re:Bhyve – BSD Hypervisor by allegory on Saturday August 30, 2014 @09:21AM (#8246374)
Despite punching windows in the face a while back, I don't agree that GNOME is anywhere near windows. Not even going back to windows 2000.
Re:Bhyve – BSD Hypervisor by anon4 on Saturday August 30, 2014 @06:32PM (#8247773)
>GNOME

Oh dear Eris, don't get me wrong, I don't claim GNOME in particular is a software fit for any purpose.

My point was more like that the base system is still really good and systemd even with its failings is better than dealing with Windows. As for the desktop, there's enough choice that you should be able to find something that's maximally convenient while minimally retarded, with a small amount of customization. That something will most surely not be GNOME.

That's kind of what I mean - it's gotten worse in some ways, but you can still pick your poison and it's not as bad as having to deal with whatever retardation Microsoft come up with.

Re:Bhyve – BSD Hypervisor by incision on Friday August 29, 2014 @06:09PM (#8244730)
>'Our Windows guys were still working on restoring the master Active Directory pair and DFS cluster TWO DAYS LATER even with a DR strategy in place.'

I'd be overjoyed if I never had to admin Windows again, but it's a bit unfair to frame that as an OS comparison.

The most basic of direct from TechNet 'best practices' would reduce the impact of a failure of that sort to minutes.

That said, that sort of failure not at all uncommon. Further, I expect such disasters would happen far less often if an existing, poorly implemented Microsoft environment could be remediated into something better as directly as anything unix-like.

Re:Bhyve – BSD Hypervisor by allegory on Friday August 29, 2014 @07:10PM (#8244948)
I think it's completely fair. Windows is completely non-deterministic due to the insane amount of coupling inside it and the sheer snowball of crap that has stuck to it over the years.

In this case, they had tested DR strategy etc in place. Unfortunately when it came to doing it for real (onto the same backup hardware the DR plan was tested against), TSHTF and Windows threw an obscure COM error loading the AD catalog. That required contacting MS 1st line who didn't know what it was so had to escalate it to the AD guys. One obscure registry key change and an ACL change (related to ESE) and it was back.

All best practices followed, yet complexity, poor design and secret knowledge crept in and shot the whole process.

Stuff like that scares the shit out of me having been on the end of it way too many times now.

Many a night have I spent up at 2AM trying to work out why the hell something odd has gone on inside Windows and taken something out in production. Not once in 20 years has a proper Unix (Solaris/FreeBSD) box woken me up or shafted me for hours.

Re:Bhyve – BSD Hypervisor by shawnreilly on Saturday August 30, 2014 @12:22AM (#8245641)
The scenario described would indicate that there were delta's between the testing environment (where the DR strategy was tested), and the production environment. It was probably related to OS updates/changes being applied over time, resulting in a configuration that changes. I've always found it good practice to build and maintain a staging environment that mimics the production environment in all aspects. When configuration changes (aka security patch) are needed, they are tested and validated on the staging environment before they are deployed on the production environment. This gives you an opportunity to test and validate the results in a non-production (aka no rules, no SLA's) environment. Part of this involves validating that procedures such as DR will continue to work as expected on the new configuration, before it gets rolled out to production. From my experience, this methodology minimizes scenarios of unexpected behavior in the production environment (aka downtime). I would recommend this methodology (or anything similar) regardless of the OS/distribution you're using.
Re:Bhyve – BSD Hypervisor by allegory on Saturday August 30, 2014 @04:55AM (#8245969)
That's exactly what was done. The guys we had were shit hot ex Microsoft admins.

The DR environment was the staging environment for the patches. Periodically the production kit would be block copied back to the DR environment and sysprepped.

Every step to prevent differences between the two clustered environments were taken.

Re:Bhyve – BSD Hypervisor by Rapzid on Friday August 29, 2014 @10:48PM (#8245436)
You're just not trying hard enough. There was a Xen hypervisor bug for years that could, when the stars aligned and the moon was full, jump the clock ahead 50 minutes. Nobody knew what was causing it until during a bug session the right guy was looking at the right bit of code and noticed an obscure problem in some in-line assembly.

Every system has bugs.

Re:Bhyve – BSD Hypervisor by allegory on Saturday August 30, 2014 @05:00AM (#8245976)
Every system has bugs, but as the other reply said, quality engineering and most importantly transparency determine the impact.

Never been a fan of Xen. It doesn't strike me as quality software. Then again no virtualization solution has to me, yet. I'd rather just deploy all the services to the base machine and use MAC to isolate them.

Re:Bhyve – BSD Hypervisor by vezzy-fnord on Friday August 29, 2014 @11:04PM (#8245481)
This isn't about bugs, it's about system transparency.
Re:Bhyve – BSD Hypervisor by bdunbar on Saturday August 30, 2014 @10:50AM (#8246574)
"Our Windows guys were still working on restoring the master Active Directory pair "

I was a Solaris admin at a previous employer. We ran annual DR exercises, and saw similar issues with the Windows kit. Even with months to prepare for the exercise, it took the entire exercise, sometimes longer, to bring 'Windows stuff' back to life.

What we (the unix team) did was to write into _our_ DR plan a few paragraphs on how to be up and running without DNS, or Active Directory.

Re:Bhyve – BSD Hypervisor by allegory on Saturday August 30, 2014 @03:58PM (#8247409)
I see DHCP, DNS and AD (in that order), at least in a windows environment as step one of anything.

Solaris was nice and easy to get back apart from one machine I dealt with: the starfire E10k. The SSP failed on one that we had and the entire system uses crypto key exchange so only the original SSP can bring the system up. Fortunately we had an E15k being installed at the time so it was an emergency migration job.

I miss big iron. I had an old maxed out 1000E and a disk tray full of 4.3Gb SCSI disks at home for a bit (until I got the electricity bill!)

Re:Bhyve – BSD Hypervisor by bdunbar on Monday September 01, 2014 @12:30AM (#8251553)
"I see DHCP, DNS and AD (in that order), at least in a windows environment as step one of anything."

It gets _messy_ but I think it's worthwhile to write out in biz continuity plan how to function without DHCP, DNS, and AD.

Take my case: we sometimes had the tier 1 and 2 services (the unix-based ones at least) up _days_ before DNS and AD were available. In the real world the business can't wait for DNS and AD to have access to their tier one and two apps: customers are waiting for product.

Note that a lot of work went into getting the Solaris stuff to this point: in particular using ZFS and zones made the process or restoration a breeze. By the time of our fourth annual DR exercise, getting stuff up and running was essentially waiting for it to restore from tape, adjust for things like 'DNS y/n' and done.

Re:Bhyve – BSD Hypervisor by harshreality on Friday August 29, 2014 @08:29PM (#8245147)

  Failed to issue message call
What was that, a dbus configuration problem? Are you quoting it correctly because if you really ran into a failure message that's not googleable that's scary.
Re:Bhyve – BSD Hypervisor by pantalaimon on Saturday August 30, 2014 @10:09AM (#8246478)
Re:Bhyve – BSD Hypervisor by allegory on Saturday August 30, 2014 @05:02AM (#8245981)
No idea. I never solved the problem. It just worked again randomly after a period of time (a couple of hours).

That was enough to scare me away from CentOS 7 and systemd to be honest.

Re:Bhyve – BSD Hypervisor by anon4 on Saturday August 30, 2014 @07:54AM (#8246208)
On the whole I don't think systemd is that much more complicated to understand than a collection of bash scripts. It's a domain-specific declarative language for dependency-based execution. At least, once you understand it, it has less gotchas than bash scripting.

It's the rest of the utilities that they're trying to couple to it that are bullshit. Starting with the journal - asking for the last few lines of the journal can take half a minute for some reason. Fortunately you can ignore those parts and just use the smart init system, i.e. you can ignore the cron replacement and run plain cron, you can ignore the nptd replacement and run plain ntpd, etc.

Re:Bhyve – BSD Hypervisor by jedberg on Friday August 29, 2014 @06:49PM (#8244868)
reddit originally ran on FreeBSD. They switched to Linux before I got there for ease of administration. I wanted to switch it back to BSD, but never got around to it as it wasn't very high priority. I tried to switch again when we moved to Amazon, but sadly it wasn't an option so I went to Ubuntu instead.

I've always thought that BSD is superior from an admin perspective, as in it's easier to do it right and keep it solid.

I may be biased though as my first job was IT for a company that had BSD committers so everything we did was on BSD and I learned how to admin from arguably some of the best BSD admins in the world.

Re:Bhyve – BSD Hypervisor by lifeisstillgood on Saturday August 30, 2014 @02:27AM (#8245838)
I would be interested in interviewing (for the worlds smallest blog audience) some of "the worlds best BSD admins" - is there any chance of an introduction ?
Re:Bhyve – BSD Hypervisor by justincormack on Friday August 29, 2014 @05:22PM (#8244575)
They have a job ad out [1] saying

"Facebook is seeking a Linux Kernel Software Engineer to join our Kernel team, with a primary focus on the networking subsystem. Our goal over the next few years is for the Linux kernel network stack to rival or exceed that of FreeBSD."

[1] https://www.facebook.com/careers/department?req=a0IA000000Cz...

Re:Bhyve – BSD Hypervisor by contingencies on Friday August 29, 2014 @08:47PM (#8245189)
That's very interesting, but isn't very convincing. Is there some kind of study on the supposed differences? I like BSD, but I've always felt from admin experience that the Linux networking stack is far more capable in terms of features.
Re:Bhyve – BSD Hypervisor by justincormack on Saturday August 30, 2014 @05:02AM (#8245982)
Well obviously it is a brief statement. Linux generally does have enormous numbers of features, but they are no use if they do not apply to you. My guesses about Facebook is that network stack issues would be around ipv6, performance or large scale (eg lost of routes).
Re:Bhyve – BSD Hypervisor by UNIXgod on Friday August 29, 2014 @11:01PM (#8245478)
BSD is the implementation to the networking stack.
Re:Bhyve – BSD Hypervisor by contingencies on Friday August 29, 2014 @11:41PM (#8245568)
From the API design / pedigree side, sure. But that's a given, and a tangent to the curiosity I am trying to have sated :)
Re:Bhyve – BSD Hypervisor by ams6110 on Friday August 29, 2014 @06:58PM (#8244903)
BSDs do change stuff too. OpenBSD recently replaced sendmail with OpenSMTPD, and apache with nginx. FreeBSD 10 has some big changes[1], depending on what you're doing with it.

Still, in general I agree. I run OpenBSD as my main desktop system because it works and mostly stays out of my way.

[1] http://www.freebsd.org/releases/10.0R/announce.html

Re:Bhyve – BSD Hypervisor by Forbo on Friday August 29, 2014 @07:08PM (#8244941)
Just a heads-up, OpenBSD is actually removing nginx from the base install in favor of httpd.

http://undeadly.org/cgi?action=article&sid=20140827065755

Edit: Adding in the link to the actual commit: http://marc.info/?l=openbsd-cvs&m=140908174910713&w=2

Re:Bhyve – BSD Hypervisor by ams6110 on Friday August 29, 2014 @07:22PM (#8244982)
Wow that was pretty fast, by OpenBSD standards. nginx was just added to base a release or two ago. Thanks.
Re:Bhyve – BSD Hypervisor by talideon on Friday August 29, 2014 @07:53PM (#8245066)
It's understandable when you know that httpd was spun out of relayd: they basically had almost all the work done and figured they'd might as well do the last few bits needed ot make it a standalone HTTP daemon.
Re:Bhyve – BSD Hypervisor by cjg_ on Friday August 29, 2014 @07:09PM (#8244947)
Actually, OpenBSD just replaced nginx with their own httpd.
Re:Bhyve – BSD Hypervisor by Nux on Saturday August 30, 2014 @07:27AM (#8246166)
What httpd would that be?

I thought they replaced their custom Apache with (also custom) Nginx.

Re:Bhyve – BSD Hypervisor by clarry on Saturday August 30, 2014 @10:31AM (#8246527)
http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.sbin/httpd/...

For some rationale, see this comment (by Reyk): http://undeadly.org/cgi?action=article&sid=20140827065755&pi...

Nginx is considerably larger than the old Apache was. Commits that took it to a new version were pretty big. I didn't see a whole lot of OpenBSD developers hardening the code. Simple and small seems to be the OpenBSD way. I like it.

Re:Bhyve – BSD Hypervisor by Bluerise on Saturday August 30, 2014 @08:13AM (#8246236)
They wrote their own. A completely new implementation.
Re:Bhyve – BSD Hypervisor by sturadnidge on Friday August 29, 2014 @05:19PM (#8244561)
Be interested to see that paper / presentation if you can track it down, especially given https://www.facebook.com/careers/department?req=a0IA000000Cz...
Re:Bhyve – BSD Hypervisor by krakensden on Friday August 29, 2014 @05:13PM (#8244535)
whatsapp was a FreeBSD shop, and they weren't purchased that long ago.
Re:Bhyve – BSD Hypervisor by X-Istence on Friday August 29, 2014 @11:45PM (#8245575)
WhatsApp is still a FreeBSD shop AFAIK.
Re:Bhyve – BSD Hypervisor by Keyframe on Friday August 29, 2014 @05:27PM (#8244593)
Define faster though.
Re:Bhyve – BSD Hypervisor by SEJeff on Friday August 29, 2014 @05:40PM (#8244642)
There are 2 common things where "faster" makes sense.

1. Throughput

2. Latency

Both are relevant for webapps like facebook.