Where did you see that?

Also how do two masters students with no experience in NLP get $50-500k in compute credits? How do I get that deal?

It was in the article last night, they deleted it for some reason after my comment.

One of the authors has a peer reviewed NLP publication [1], the other has several publications in computer vision. I don’t know how they got research credits from Google.

[1] https://arxiv.org/abs/1905.13153

Well that is pretty dubious. That completely changes the metric of how much money and how difficult this costs. (And I definitely believe you because there's several other comments with that same number). Could be a typo? But I feel like that's something you'd say. Then again, these are masters students.

It's $50k per training run. Their main contribution has been finding the optimal hyperparameters, not described in the OpenAI paper. Obviously you need more than one training run to to that.

They didn't, their advisors -- Stefanie Tellex and George Konidaris -- did.

And the answer to that part is, they asked: https://www.tensorflow.org/tfrc

Interesting. I'm doing ML in a niche area so I'm wondering if I can get funding (for my PhD). Obviously there's politics involved, but honestly I just want to try out methods and experiment (typical grad student I guess). So if they have a funding route I'm stoked. Thanks for the info.

In most programs, if you get accepted into the Ph.D. program, you're funded. Obviously, you should check first. Different programs treat these things differently with TA requirements, but most Ph.D. students are funded by an "RA" (research assistant, aka, doing your actual eventual-thesis research...).

For more info, my colleague Mor wrote a very useful doc about the Ph.D. process in CS: https://www.cs.cmu.edu/~harchol/gradschooltalk.pdf

Oh I'm funded. Definitely was told not to go unless I'm fully funded. But $500k for computing resources is bigger than a lot of small term contracts (I'm mostly on research and usually don't have to TA unless there's a hiccup in funding schedules. Gov money...). I definitely see that amount for long term, but this seems like a pretty short project.

Got it. (1) Yes, 500k is a lot, but (2) the TFRC program is making fairly expensive TPUs available at shockingly good prices for researchers willing to do some tire-kicking and feedback-providing. So 500k of TPU time is easier to come by than, say, 500k of generic compute.