News for hackers. Stuff that matters.

A comment on Quantcast File System on S3.

Re:Quantcast File System on S3 by lumens on Tuesday November 29, 2016 @01:44PM (#13065195)
For some context:

Quantcast moved off Hadoop in favor of a home-rolled solution (QFS) because of Hadoop wasn't scaling well to our needs (40PB+ data processed/day).

This release of QFS (https://github.com/quantcast/qfs) includes support for S3, among other improvements.

[ Parent | on HN ]
Re:Quantcast File System on S3 by bertan on Tuesday November 29, 2016 @02:56PM (#13065768)
I wonder how it compares/relates to https://minio.io

Can it run on minio, for example?

[ Parent | on HN ]
Re:Quantcast File System on S3 by mck35 on Tuesday November 29, 2016 @03:41PM (#13066077)
One difference I see is that QFS is block-based and each file is composed of multiple S3 objects. Hence, there is no file size limit.
[ Parent | on HN ]