For example each of these "pods" only has 4GB of RAM. If I was doing lots of random I/O on 67TB of drives, I would want a heck of a lot more ram for caching efficiencies.

That's a strange choice, HTTPS would incur quite a bit of overhead for something that is essentially a (large) drive at the end of a network cable used internally only. Why the encryption ?

quote from the article:

"A Backblaze Storage Pod isn’t a complete building block until it boots and is on the network. The pods boot 64-bit Debian 4 Linux and the JFS file system, and they are self-contained appliances, where all access to and from the pods is through HTTPS. Below is a layer cake diagram.

Starting at the bottom, there are 45 hard drives exposed through the SATA controllers. We then use the fdisk tool on Linux to create one partition per drive. On top of that, we cluster 15 hard drives into a single RAID6 volume with two parity drives (out of the 15). The RAID6 is created with the mdadm utility. On top of that is the JFS file system, and the only access we then allow to this totally self-contained storage building block is through HTTPS running custom Backblaze application layer logic in Apache Tomcat 5.5."

That's an odd choice for a storage server protocol stack.

To help prevent a network compromise from resulting in storage management compromise? Just because something is internal doesn't mean it's safe. Once a host/network segment is compromised, you don't want it to be easy to jump to the next.

Otherwise, you've built M&M security. Hard candy shell on the outside, soft gooey chocolate insides. Mmm.

Of course these days there are well-understood solutions to those problems, compared to when S3 first started, but it's still not stuff you can pull off the shelf. (Although with the Cassandra distributed database, the metadata problem is close to that point now. </plug>)

Hardware is cheap, power and developers aren't.