Many of our customers do use S3 and we make use of S3 extensively ourselves. However, Drupal often expects to operate on a POSIX compatible filesystem. Drupal 7 does support PHP file streams which can be configured to use S3, but not every Drupal module follows the best practices. Plus, we support every flavor of Drupal under the sun (including custom code).

All of our enterprise customers receive a highly available setup running on multiple nodes--thus, we have the need for a persistent filesystem attached to multiple EC2 instances. We utilize GlusterFS to ensure all of our clients have the filesystem capabilities their apps may need.

Basically, they need EBS without the single-instance (writable) mount limit, if they're using it to sync Drupal installations (php files, temp folders, etc.) across multiple web servers. S3 isn't really intended for that kind of workload (it would be possible to use something like s3fs, but I don't think that will give you a POSIX-compatible file system which is required for Drupal, and it's probably too slow as well (lots of small files, etc.))