News for hackers. Stuff that matters.
Re:OnMetal: The Right Way To Scale by ggreer on Thursday June 19, 2014 @01:58PM (#7917379)
This sounds pretty cool, but I have a few suggestions:

1. I want to know how much these things cost. Even a line like, "Under $x/hour." would be useful.

2. The blog post says that the memory and compute types have no disk. This confused me at first, but then I saw on http://www.rackspace.com/cloud/servers/onmetal/ that they have 32GB boot drives. You might want to add that to the comparison table in the blog post.

3. Has any thought been given to letting users provision OnMetal servers in specific cabinets? Sometimes users want to have servers next to each other for better network connectivity. Other times, they want them in different cabinets for increased reliability.

Lastly, kudos to Rackspace for building this. Most people don't realize how virtualization can hurt performance. Even without noisy neighbors, a modern Xeon still takes >500 cycles to switch between VM and host.

Re:OnMetal: The Right Way To Scale by jayofdoom on Thursday June 19, 2014 @02:07PM (#7917430)
1) I don't think prices have been announced yet.

2) They do have 32GB SATADOMs, but these are intended for boot media and configuration only. They are not intended for anything more than boot media. Think something like an SD Card in a solid-state machine.

3) Yes, I want this. No, we don't do it currently.

FWIW, we didn't build all this. Ironic existed long before we came around, we added a deploy driver and set it all up and exposed it for customers. We'll have a blog up shortly with exactly how we're running this in production.

The coolest part of all this is something nobody has mentioned here yet -- this is all on Open Compute hardware. When I managed servers, I always wanted access to the cool hardware and scale that companies like Facebook and Google had. Hopefully with OnMetal, we're going to give our customers the cool hardware I always wanted.

Edit: As a note, I'm a Racker and I work on the team that built this.

Re:OnMetal: The Right Way To Scale by jayofdoom on Thursday June 19, 2014 @03:06PM (#7917742)
Re:OnMetal: The Right Way To Scale by ggreer on Thursday June 19, 2014 @02:41PM (#7917621)
> Think something like an SD Card in a solid-state machine.

Ah, that's still way better than no disk. I was initially worried about having to deal with PXE. I wonder if logging will cause trouble for the SATADOMs. I'm guessing they don't do much in the way of wear-leveling.

I didn't recognize your nick at first, but we worked in the same office. I was part of the Cloudkick acquisition. I left Rackspace in August of 2012 to start Floobits. Small world. :)

Re:OnMetal: The Right Way To Scale by jayofdoom on Thursday June 19, 2014 @03:07PM (#7917746)
I would say if you're running applications in the cloud, you should ship your logs and forgo logging them on disk. Or setup an awesome ES+Logstash cluster using IO nodes :).

You should've left off the last paragraph; I assumed you knew who I was :).

Re:OnMetal: The Right Way To Scale by ggreer on Thursday June 19, 2014 @03:26PM (#7917844)
I agree with you about log shipping, but customers are bound to misconfigure their servers and log to the SATADOM.
Re:OnMetal: The Right Way To Scale by Serow225 on Thursday June 19, 2014 @09:35PM (#7919140)
Kudos, very cool. Maybe instead of 'no disk' something like 'SSD for boot media, no local bulk storage' would be clearer, as I was a little confused by this as other posters below were. It's also a little interesting that the 'High CPU' option has the fewest cores of the three options... Any comments on that?
Re:OnMetal: The Right Way To Scale by vehementi on Thursday June 19, 2014 @10:41PM (#7919309)
It's low on other resources so it's probably lowest cost per CPU.
Re:OnMetal: The Right Way To Scale by russell_h on Friday June 20, 2014 @01:01PM (#7921580)
That's basically correct, we were trying to balance a few factors:

  - Total cost
  - Cost per core
  - Clock speed (this actually matters a lot for, say, a Python webapp)
We chose to run only a single CPU in this box to keep the total cost of the box low. This slightly increase the cost per core (switch ports, the chassis, rack space, etc being basically fixed overhead per node), but by selling in smaller increments we think it we can keep the total cost lower for most users.

We'll be gathering feedback on the resource balance, and iterating as we go.

Re:OnMetal: The Right Way To Scale by _Adam on Thursday June 19, 2014 @02:20PM (#7917502)
Cool! Can you talk about some unexpected challenge you faced getting this rolled out?
Re:OnMetal: The Right Way To Scale by jayofdoom on Thursday June 19, 2014 @03:25PM (#7917840)
Honestly? Using 512GB of ram is pretty hard. I learned during some of this that you can actually get tmpfs to be CPU limited if you try to write to it too quickly.
Re:OnMetal: The Right Way To Scale by mbell on Thursday June 19, 2014 @09:08PM (#7919085)
> Honestly? Using 512GB of ram is pretty hard.

It's extremely easy. Insert 512GB of data into postgres, there, done.

Re:OnMetal: The Right Way To Scale by Serow225 on Thursday June 19, 2014 @09:32PM (#7919132)
Not if you are in Oil and Gas :)
Re:OnMetal: The Right Way To Scale by servercobra on Thursday June 19, 2014 @03:34PM (#7917891)
Working with IPMI can be tough. We nailed the issues eventually, but there were a few frustrating days of images not booting properly with no explanation. Until we found the right combination of settings, we had issues with flakey BMCs that would stop responding for no apparent reason.
Re:OnMetal: The Right Way To Scale by fleitz on Thursday June 19, 2014 @02:04PM (#7917416)
Yup, this is what I like about Docker (I'm sure there's some overhead as well), application isolation with out paying for it seriously in terms of performance.

That said you can pull a lot of performance out of a modern 2U server, 24 cores and 256 GB of RAM can be had pretty cheaply these days, pair it with 8 SSDs and you've got a real workhorse.

Even a traditional MySQL/Postgresql DB can maintain upwards of 140,000 tps on hardware like that.

Re:OnMetal: The Right Way To Scale by x0x0 on Thursday June 19, 2014 @02:54PM (#7917693)
And you can probably rack that for order $11k
Re:OnMetal: The Right Way To Scale by fleitz on Thursday June 19, 2014 @03:23PM (#7917829)
Or rent it from EC2 for a month.