Understanding Google’s File System (2020)

Re:Understanding Google’s File System (2020) by nosefrog on Thursday September 01, 2022 @05:32AM (#32674053)

Colossus, the successor to GFS, is one of the coolest technologies at Google. Almost everything we have internally is built on top of it.

[ Parent | on HN ]

Re:Understanding Google’s File System (2020) by wuschel on Thursday September 01, 2022 @05:52AM (#32674198)

Could you point me towards some materials re Colossus?

[ Parent | on HN ]

Re:Understanding Google’s File System (2020) by atombender on Thursday September 01, 2022 @05:58AM (#32674241)

https://cloud.google.com/blog/products/storage-data-transfer...

[ Parent | on HN ]

Re:Understanding Google’s File System (2020) by hppb on Thursday September 01, 2022 @07:17AM (#32674796)

This talk by Denis Serenyi is good.

http://www.pdsw.org/pdsw-discs17/slides/PDSW-DISCS-Google-Ke...

[ Parent | on HN ]

Re:Understanding Google’s File System (2020) by JamesSwift on Thursday September 01, 2022 @09:18AM (#32675840)

What specifically makes it so cool in your opinion?

[ Parent | on HN ]

Re:Understanding Google’s File System (2020) by dekhn on Thursday September 01, 2022 @10:40AM (#32676942)

after a couple decades of breaking every cluster filesystem I've used at scale, Colossus was the first one that managed to handle my loads effortlessly. It required far less attention from SREs when abusive users sent it a lot of load. It also made using reed solomon codes easy.

[ Parent | on HN ]

Re:Understanding Google’s File System (2020) by the-rc on Thursday September 01, 2022 @03:14PM (#32681560)

The truly problematic Colossus clusters usually fell into one of two buckets: a) those with shrinking machine counts (but always increasing traffic) because the hardware was ancient and repairs/upgrades were not viable and b) those where most of the quota was held by a single project, with massive jobs. One cluster at IDI was mostly Image Search's and it wasn't fun at all.

[ Parent | on HN ]