This is a cool thread.

So please spec out a couple servers for us. How would you get a bunch of memory (32GB, 64GB even) into one of them and still have it be a good web/app server?

My current favorite is the Altus 600 from Penguin Computing. Cheap chassis with dual sockets and 16 RAM slots. More slots means cheaper chips. 2GB chips for that board are about 300 bucks. Throw two 2.2GHz Opterons and 2x 300GB drives in there. Add an SATA controller card, 4 SATA/CF adapters and CF cards (in the CD-ROM bay), and you've got a nice machine: 1U, dual processors, 32GB of RAM, 600GB of platter, 32GB of flash for about USD$7500.

Expensive, yeah. 5 grand of that is RAM. You could get away with 4GB of RAM and still have a fucking fast web/app server as long as your active dataset fits on the flash.

$215: http://www.newegg.com/Product/Product.asp?Item=N82E168201343...

Brings the memory cost down to around $3440.

Wrong speed but yeah, that's the ballpark. I used Crucial to price the RAM.

The problem is that not many people benchmark the write access time for flash. It's pretty bad, often 50ms+. There is a product I know of, the Zeus IOPS, which gets extremely fast write speeds to flash. Unfortunately it costs about $250/GB. For that price, you can get a RAM based solution, which is even faster. Ones like http://www.hyperossystems.co.uk/ can come with a battery backup and a hard drive that it will dump to and restore from if the power goes out. Price is about $220/GB. The prices sound high, but if you're trying to scale a database, it can potentially solve an otherwise intractable problem without any software changes.

Does SATA / CF have any advantage over a pack of USB drives?

I am not sure what out-of-the-box solution can you use to read/write packs of flash memory in parallel, but I'm pretty sure there exists something (you could do it on the application level as well).

Edit: USB RAID-0 array benchmarks here:

http://arstechnica.com/reviews/hardware/flash.ars/8

http://www.bigbruin.com/reviews05/thumbraid_5

The advantage is that you're not going through the USB bus. The spec says 480Mbit but in real life it's much much lower. Does your rackmount server have quality USB2.0 (heh) or the cheapest 1.1 crap the manufacturer can find (likely)?

On the other hand I don't have hard numbers, just some bonnie tests on random flash sticks I had lying around. Check out the newer USB sticks that are "Readyboost Ready", such as Corsair. They might be quick enough.

It's too bad that the transfer rate is so low. It should be pretty simple to build a very fast device which spreads the blocks across a large number of flash cards. I suppose you could do a raid array of addonics adapters, but it's unclear how that would perform (and it might be a mess dealing with a large number of them).

For many database applications, I don't think the transfer rate matters that much.

Regardless, the DDR-ram solid state solutions have faster transfer rates than a 4 drive raid0. The only drawback is that they cost about $100 per GB.

Oh, and the onboard battery will only let you unplug your machine for 8hrs. After that, the state is lost.