Colab is effectively JupyterLab/JupyterHub with Google's own add-ins (like integration with Google Drive). JupyterHub is a huge PITA to manage, and Colab also offers a degree of free compute and limited access to a dated, but still free K80 GPU.

From the tweet thread, it seems like there was some misunderstanding over where the files are being stored and being executed. This is a pretty common issue with Google Drive. I.e. if someone shares a file with me, and I copy it to a folder, it's just a pointer to the original file. Only after clicking "Add to My Drive" does it count against my storage allocation, and only then is it a distinct copy.

My guess is that the researchers expected each user to be able to run the game in their own personal free Colab environment, not be running it against the university's compute and storage budget.

Google Colab now comes with free T4 GPUs.

Occasionally. You randomly get a K80 or T4. It's annoying.

As an "SRE engineer", I don't agree. Certainly in an university/R&D environment, you cannot compete with what 'the cloud' is able to offer within the reach of a few mouse clicks. It would be absolutely moronic to set up an entire infra yourself with GPU's, ... specifically tailored to for 1 research project to then throw it away after a few months, or see the scope change every 2 weeks.

One of the reasons such AI tech is able to move this fast because the cloud has become a commodity. People start to expect this kind of flexibility. It's impossible to self-host every possible thing some guy might want to try out.

Once you have a stable application - sure you might want to invest in own hosting for this, but even then it's an expensive up-front investment for something that might not go anywhere. I know a few places where they have a bunch of maxed-out nVidia DGX systems catching dust for exactly this reason.

At 10k per day it is absolutely worth it to hire someone to take care of a bunch of cheap servers.

That's what, $2.5MM/yr?

Servers with GPU's are not going to be cheap :)

Servers are a CapEx purchase, and can be depreciated or re-purposed.

10k/day makes sense for a short project, but if this is going to be an on-going thing -- like 2-3 years -- then absolutely buy that hardware and have an internal team run it.

It doesn't seem to be as simple as downloading a binary.

We (those who read HN) could probably download the code and run it at home, but I think the authors want non-technical users to play. To do that, they need an accessible Python runtime, so they're hosting the game in a Colab notebook. The download in question is referring to downloading the weights of the neural net into the VM running the notebook.

If only redistributing Python apps wasn't so difficult.

The game's GitHub page[1] states that you would need a "beefy" GPU ~12 GB and CUDA to play the game locally.

I think that's why the author was serving the game through Colab since the majority of users probably don't have a 12GB GPU.

[1]https://github.com/AIDungeon/AIDungeon/

Ooof, yes, now I see where the $10K/day is coming from...

As I have demonstrated, I've really not much of a clue when it comes to AI, but do users really need 12Gb GPU RAM, 100% of the time? Maybe it's possible to use one GPU for multiple users?

Google Colab gives each user a dedicated Nvidia Tesla K80 GPU for 12 hours for free, which is super cool and presumably why the project is on Colab. But as each user spins up their own Colab instance it pulls down the 6GB of GPT-2 model weights, incurring 30-40 cents of data egress charges against the GCP Storage Bucket that the data is stored in.

60k users yesterday * 6GB each -> 360TB of data egress!

Normally, a scenario like this wouldn't involve bandwidth costs because GCP -> GCP same-region bandwidth is free, but Colab is technically not part of GCP, so the bandwidth charge is being assessed as egress to the public internet, which is pricy for that much data. Though it's probably still a lot cheaper than paying for the GPU-hours for that many users.

This is sort of a killer example for the layperson of what ML can do, so hopefully Google will recognize this and comp most/all the data egress since it's a drop in the bucket for them, but every person that uses it can still go "Wow, Google's services allows for some amazing stuff."

Oh, it does. NOT.FOR.FREE, though; that's the entire reason it exists at all.

In other words, yes, this is an amazing amount of raw power - with a corresponding price tag.

But you could set up your own file-serving node and then data transfer to end-user Colab instances would be free, right? ("Free" — costing only as much your CDN costs for 360TB/day, which is still quite pricey, I think, but not as much as $10K/day. I.e. Google wouldn't charge you for data transfer here.)

Yup. I think I saw yesterday that they were looking to move the model to BitTorrent.

The $10K/day was actually coming from the large egress fees they were getting for transferring the models and agents from Google Cloud Storage to the Colab notebooks. I think if you were to serve the game as a web app you definitely wouldn't need one instance of a 12 GB GPU for each user. But the thing about Colab is that you need a Google account to use it, and you run your own notebook, independent from the author's account.

$10k/day is just for file transfer, not for GPU.

Ok, ok, let me see if I now got this right: The AI needs a VM (because users rarely have 12GB GPUs at home), and this VM then downloads about 6GB from GitHub each time a user opens up a new session instead of sourcing that from a locally cached copy?

EDIT: Almost, the VM runs on Colab, which only works with Google, and Google's charging for the upload to Colab? ...more like collaborator, amirite? dodges rotten eggs

Yeah, now you almost got it right! Colab is a service provided by Google.

The funny thing is the high egress fees Google charges for transferring data between two of its services (GCS -> Colab).

Funny, cheap and cheerful, little hidden costs in the fine-print. Oh my Google, how could we ever be mad at you? opens a can of laughter

I played it on a cpu. I downloaded it to remove the profanity filter. It's several seconds for each answer, so it's not ideal, but workable.

A lot of HN users couldn't run it and have a reasonable experience because they don't own computers with GPUs capable of generating a response in a timely manner.

Could something like PyInstaller[1] work?

[1]https://www.pyinstaller.org

If agree in theory, but I'd like to disagree with some assumptions. AI devs don't necessarily have experience with efficient network infrastructure. (And ops people with AI development)

You can't just "buy servers for the same price". Where are you going to put them? How are you going to power them? How is the bandwidth for them provisioned? At some scale these are non-trivial questions - you can't just buy a rack, stick it next to your desk and plug into an extension cord.

The system deployment is something you need to spend time on as well. Bare metal provisioning and deployment of GPU libraries to make things run smoothly takes time.

And finally when the hype dies down in a week or two, what are you going to do with that infrastructure?

Cloud services are not trivial either. But they do have some advantages.

Possibly the author of the parent comment was puzzled why does an AI lab not have a specialist who provisions the bare metal. I would think that if the hype died in a week, an AI lab would have other projects which require similar equipment to run.

I agree with your post in general as intuition tell me (without further details about ops situation) that cloud is a competitive fit in this scenario.

At this "scale" a single 3Gbps guaranteed pipe is ~330€/mo on OVH. You could simply buy ten of them just to be sure and still be paying 1/3 a month of what that would cost you for a day in the cloud. That includes all the maintenance and power for the equipment, you simply have to know how to use the OS.

My understanding is that each user runs a separate Colab instance with its own hardware, it's not the kind of thing you could replicate easily with one server and a couple GPUs.

I see, thanks. That's no longer the case now, is it? I'm presently downloading the torrent, and it seems like it will then run completely local?

Well, I guess that was the cost of getting it to go viral. The torrent seems quite active now.

> What am I missing?

There are a couple of things. First off, academic funding for things like AI-capable clusters is a whole thing, let alone the support staff for them. This is difficult for a host of reasons, some good and some bad.

Secondly, this isn't about hosting files really, it's about providing a suitable environment to run the model in. You can expect a AI research lab to have figured out how to do training in a reasonable way, but what they have here is more of an inference in production problem. That's really outside the expected expertise of a research lab. In the old days you just wouldn't be able to access this simply. Today there are turn-key(ish) solutions with built in scaling; they used one - it scaled and now they are wincing at the bill.

This isn't crazy, as the google->google egress fees are not obvious.

> What am I missing?

They overbudgeted the "infrastructure" section of their grants and have money to burn; and nobody's interested in setting up another datacenter.

Politics is what you are likely missing.