For that 70B parameter model: an A100?
Source: have a $200 board in my computer right now with two full-length PCIe slots.
Meh. If you want 16GB of VRAM for several hundred dollars, can't you just pull a brand new 30-series off the shelf and have ten times more computing power than those old pascal cards? You'll even have more VRAM if you go for the 3080 or 3090. Admittedly, the 3090 is closer to $700 or so, but it should still make a P100 very sad in comparison.
Another curious contender is the decommissioned Nvidia CMP series GPUs from miners. For example, the Nvidia CMP 170HX basically uses the same Nvidia A100 PCB with its features downsized or disabled (8 GB VRAM, halved shaders, etc). But interestingly, it seems to preserve the full 1500 GB/s memory bandwidth, making it potentially an interesting card for running memory-bound simulations.
[1] Prices are so low exactly because most people don't. SXM-to-PCIe adapters also exist which cost $100-$200 which is nearly as much as you have saved. It should be trivial to reverse-engineer and make an free and open source version...
...But it doesn't seem to be cheap. Not really worth it over a 4090 for the same price.
I'm not sure what exact LORA/quantization settings would be ideal, but check out https://github.com/OpenAccess-AI-Collective/axolotl#config
The main bottleneck to doing distributed training like this is the communication between nodes.
Consider the trivial example of training a network to distinguish between sample A and sample B. Give it a hundred As in a row and it just learns "everything is A". Give it a hundred Bs in a row and it relearns "no, everything is B". To train it to distinguish, you must alternate As and Bs (and not too regularly, either!)
Some inference frameworks came up with schemes for just this, and it was horrifically slow.
This is the part that raised my eyebrows.
Finetuning 70B is not just hard, its literally impossible without renting a very expensive cloud instance or buying a PC the price of a house, no matter how long you are willing to wait. I would absolutely contribute to a "llama training horde"