News for hackers. Stuff that matters.
Re:CoreOS: Boot on Bare Metal with PXE by hosay123 on Wednesday September 11, 2013 @03:09PM (#6369254)
Good luck finding a network infrastructure and PXE server able to boot a few hundred machines simultaneously during a power event. Yes in theory PXE boot sounds great. In reality it's a pointless SPOF.

Also most likely makes it harder to reuse most of the trusted boot infrastructure that already exists for Linux. So we can assume at least in the initial release Mallory can race with the real PXE server assuming a network that hasn't been partitioned with a crazy complex config (i.e. basically all of them).

Re:CoreOS: Boot on Bare Metal with PXE by fintler on Wednesday September 11, 2013 @03:59PM (#6369608)
We commonly reboot entire clusters at once (around 10,000 servers in larger clusters -- each running a full Linux OS) over PXE without a problem. We have a configuration management machine that creates an image, then we push that down to a small cluster of TFTP servers that serve it out. The strain on NFS (we keep parts of the OS in RAM, and load other parts on demand over NFS) after we kexec from the PXE kernel into the production kernel causes more problems than the initial TFTP traffic (but it usually works fine as well). Btw, after booting, we use PanFS (DirectFlow) or Lustre for computing stuff, not NFS.

Although it's not what we use, here's a program that does a similar type of management: http://warewulf.lbl.gov/trac If you take the time to combine Warewulf with something like Puppet or Chef, you'll have a nice system for managing 100s of thousands of machines (I could easily see this scaling to over a million servers if you have the cash to build something like that).

If you're wondering about dynamic libraries in an environment like this, take a look at https://github.com/hpc/Spindle

And yes, I still get giddy when I type one command to reboot 10,000 servers.

Re:CoreOS: Boot on Bare Metal with PXE by ajdecon on Thursday September 12, 2013 @02:57AM (#6372482)
Since you mention kexec and TFTP+NFS, are you currently using Perceus? Or is there another system out there with that combo?
Re:CoreOS: Boot on Bare Metal with PXE by fintler on Sunday September 15, 2013 @03:44PM (#6390051)
We're using a modified Perceus tied into cfengine.
Re:CoreOS: Boot on Bare Metal with PXE by devicenull on Wednesday September 11, 2013 @03:50PM (#6369555)
So, a 'PXE server' is nothing more then a DHCP server and some method of fetching files (historically, TFTP, but this is not a requirement).

ISC DHCPD is pretty bulletproof. Many large carriers/ISPs use it, so scaling this end should be pretty straightforward. You'd probably want to statically configure leases for all your servers though.

Pretty much all PXE roms built into NICs only support TFTP. I would use TFTP to load a iPXE, then use iPXE to load the operating system over something else (probably HTTP). Scaling HTTP is well understood (this is about the easiest HTTP scaling you can do, downloading a few static files).

Nothing about this is hard to scale, nor does it require any special network setup.

Re:CoreOS: Boot on Bare Metal with PXE by zdw on Wednesday September 11, 2013 @08:26PM (#6371155)
iPXE is awesome.

My standard install is to embed a script into iPXE that chainloads a config file over TFTP (this works properly with non-ISC DHCPD servers, like OpenBSD's fork and dnsmasq), then loads various OS's and diagnostic tools over HTTP or NFS.

So much easier than slinging around optical media or hard disks for installation/diagnostic/booting diskless clients.

Re:CoreOS: Boot on Bare Metal with PXE by bretpiatt on Wednesday September 11, 2013 @03:22PM (#6369350)
I'm unsure why PXE cannot be a reliable network service just as DHCP or DNS. It can be clustered and made reliable as any other system can. You can also load balance and distribute out the PXE traffic in the event of a full refresh of all nodes just fine.
Re:CoreOS: Boot on Bare Metal with PXE by stephengillie on Wednesday September 11, 2013 @03:38PM (#6369464)
I don't get this either. PXE is a few special DNS settings & a TFTP server & a very small boot file. The concept has been around since before 2001, yet it's treated like some kind of weird voodoo.

Using PXE in your environment means you have to feck around with DNSmasq, or manually configure DNS, or use enterprise-level bloatware.

Re:CoreOS: Boot on Bare Metal with PXE by baruch on Thursday September 12, 2013 @02:30AM (#6372401)
TFTP is a UDP based file transfer protocol that uses a stop-and-wait and a simple retry loop, if you use only that you risk hitting congestion very fast and having a hard time to fix it.

The suggestions above to use iPXE and switch to TCP transfer methods such as HTTP make a lot of sense in that aspect.

Re:CoreOS: Boot on Bare Metal with PXE by ook on Wednesday September 11, 2013 @06:11PM (#6370485)
Use iPxe (http://ipxe.org/) to boot from a http image server.
Re:CoreOS: Boot on Bare Metal with PXE by stephengillie on Wednesday September 11, 2013 @03:34PM (#6369433)
Only if your datacenter or server room is underpowered. A properly specced location can handle all servers at full load.
Re:CoreOS: Boot on Bare Metal with PXE by jlgaddis on Wednesday September 11, 2013 @07:21PM (#6370866)
> In reality it's a pointless SPOF.

Like pretty much anything else, it's not a problem when implemented correctly.

Even if you have hundreds or thousands of machines, it's not that difficult to configure your network switches (assuming you don't buy your network gear at Office Depot) to distribute the DHCP and TFTP across multiple servers -- if you even need to, that is; DHCP and TFTP are two of the most lightweight networking protocols in use.

Re:CoreOS: Boot on Bare Metal with PXE by pkj on Thursday September 12, 2013 @12:48AM (#6372110)
If you are ok with CentOS/RHEL, there is a scalable way to boot multiple machines simulataneously using ........ bittorrent !

http://www.rocksclusters.org/rocks-doc/papers/two-pager/pape...

Rocks is used at production cluster installations with thousands of servers.

http://www.rocksclusters.org/rocks-register/index.php?sortby...