Compute Engine machine types with up to 96 vCPUs and 624GB of memory

Re:Compute Engine machine types with up to 96 vCPUs and 624G... by malux85 on Friday October 06, 2017 @07:35AM (#15415940)

scientific computing and simulation. I have had need for machines like this - for a project that was very compute heavy but embarrassingly parallel, but also was very "chatty" - I.e updated the data structures a lot during compute

I can't be too specific about it, but if involved creating a very large tree structure and updating, pruning and transversing the tree a lot

If the algorithm is updating and reading a large data structure a lot it's only practical from a speed point of view to hold the whole structure in RAM

[ Parent | on HN ]

Re:Compute Engine machine types with up to 96 vCPUs and 624G... by neutronicus on Friday October 06, 2017 @09:29AM (#15416602)

Is this really a better solution than just getting hours on NERSC (or some other government supercomputer)?

[ Parent | on HN ]

Re:Compute Engine machine types with up to 96 vCPUs and 624G... by tallanvor on Friday October 06, 2017 @12:03PM (#15417859)

Private companies want to do simulations as well, and with this type of solution, you can pretty much run them on demand rather than having to wait in line.

[ Parent | on HN ]

Re:Compute Engine machine types with up to 96 vCPUs and 624G... by neutronicus on Friday October 06, 2017 @01:23PM (#15418503)

...I guess.

My advisor's company straddles the public / private divide, but we've definitely done some simulations for private clients on NERSC, and I assume we weren't misusing hours allocated for some other purpose.

[ Parent | on HN ]

Re:Compute Engine machine types with up to 96 vCPUs and 624G... by mahdix on Friday October 06, 2017 @07:56AM (#15416046)

That's interesting. What tools do you use to run your computations in parallel on a cluster? Hadoop?

[ Parent | on HN ]

Re:Compute Engine machine types with up to 96 vCPUs and 624G... by malux85 on Friday October 06, 2017 @07:59AM (#15416060)

For this type of problem - it must be done on a single node. The overhead of network communication would have killed the latency requirement - that's why we need a huge machine like this.

The code was written in C

(We also maintain a Hadoop and Cassandra cluster, and I use Spark for distributed computation - but those are different projects)

[ Parent | on HN ]

Re:Compute Engine machine types with up to 96 vCPUs and 624G... by make3 on Friday October 06, 2017 @07:46AM (#15415994)

how does the problem fair on gpus

[ Parent | on HN ]

Re:Compute Engine machine types with up to 96 vCPUs and 624G... by hyperbovine on Friday October 06, 2017 @07:54AM (#15416031)

If it requires synchronizing across workers, poorly.

[ Parent | on HN ]

Re:Compute Engine machine types with up to 96 vCPUs and 624G... by make3 on Saturday October 07, 2017 @05:55PM (#15425903)

depends how often you need to synchronize. look at deep learning, you need to sync the weights as often as possible

[ Parent | on HN ]

Re:Compute Engine machine types with up to 96 vCPUs and 624G... by malux85 on Friday October 06, 2017 @07:55AM (#15416037)

This is correct

[ Parent | on HN ]