Re:Cost Per 1M tokens Of Self Hosting Llama-3
by Melatonic on Friday June 14, 2024 @07:10PM
(#40686034)
Read the other comment and also immediately thought of Cloudflare tunnel instead - is there a reason you chose that? Wondering if I should do the same with my old Titan XP (probably slower than your 3070 but it does have 12gb of vram)
Re:Cost Per 1M tokens Of Self Hosting Llama-3
by czhu12 on Saturday June 15, 2024 @10:23AM
(#40690007)
No reason other than that cloudflare is quite a nice reverse proxy and I already use it for managing my DNS's. Thus far, I've not noticed any issues with it, other than when my home wifi goes down.
Have a 13B running on a 3070 with 16 gpu layers and the rest running off CPU.
Performs okay, but way cheaper than renting a GPU on the cloud.