News for hackers. Stuff that matters.

atylerrice

Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool on Tuesday February 14, 2023 @10:03PM (#34799522)
You can also use int8 quantization which comes for free with huggingface and isn’t a huge cost to accuracy. Which cuts the memory usage almost in half again. (almost in half because some layers might stay 32 bit if they make a tiny part of the model)
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool on Tuesday February 14, 2023 @10:01PM (#34799505)
This is incorrect. Thought would be interesting. Pruning usually removes some parameters from the model (setting certain ones to 0 for instance). The smaller models all have less layers and therefore weren’t pruned but were smaller to begin with.
Re:ChatGPT Hardware a Look at 8x Nvidia A100 Powering the Tool on Tuesday February 14, 2023 @09:59PM (#34799479)
Well the batch size also helps with making training more stable. So a larger batch size computes the gradient across more examples so it ends up being more stable than only using a batch size of day one. A batch size of one is only fixing the error on one example rather than many.

a larger Batch size can help keep the gpu fed but usually that’s not a problem with these larger models.

I don’t usually comment, but aws is actually rather expensive and i’ve hit this annoying quota problem as well. Been back and forth and still hasn’t been raised. I can also recommend coreweave banana.dev and pipeline.ai as great alternatives. This service looks awesome good luck with the launch!
I’ve seen this a lot. I think sometimes in my experience it comes down to what is work?

As a developer I would classify a lot of the meetings I’ve had at past companies where it takes up 40% of my time as not really working. Sometimes these meetings push over the hours for the week but weren’t really productive in any meaningful way.

Re:Transformers in Medical Computer Vision on Wednesday March 17, 2021 @01:43PM (#26494516)
I totally agree but I’m excited for some recent initiatives to fix this. https://www.researchallofus.org/ This research project is open even to at home scientists although I think some of the genomic data may reasonably not be as open as other parts.
I built Hatchedwith.com mostly because I wanted to mess around with language models. I can’t seem to find any customers even though I got two from a fake landing page but realistically it’s not the product I advertised in the beginning I’m going to lower the pricing to extremely cheap and call it done unless google starts picking up on the tens of thousands of pages.