News for hackers. Stuff that matters.
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by sneak on Friday June 14, 2024 @03:34PM (#40684197)
Look into Nebula (or Tailscale if you trust third parties). I have all my workstations and servers on a mesh network that appears as a single /24 that is end to end encrypted, mutually authenticated and works through/behind NAT. I can spawn a vhost on any server that reverse proxies an API to any port on any machine.

It’s been an absolute gamechanger.

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by logtrees on Friday June 14, 2024 @04:03PM (#40684518)
Whooooaaa that is mind-blowing. Thanks for sharing. <3
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by elorant on Friday June 14, 2024 @04:27PM (#40684773)
Is there any resource that goes into more detail about how to setup all this?
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by sneak on Friday June 14, 2024 @04:37PM (#40684853)
https://github.com/slackhq/nebula

the docs are good. when creating the initial CA make absolutely sure you set the CA expiration to 10-30 years, the default is 1 which means your whole setup explodes in a year without warning.

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by aborsy on Friday June 14, 2024 @05:02PM (#40685103)
Why do you have to trust a third party?

It’s end to end encrypted, and with tail lock enabled, nodes can not be added without user’s permission.

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by sneak on Saturday June 15, 2024 @02:09AM (#40687835)
The idea of “user’s permission” is determined by tailscale and/or the oidc provider. I don’t know anything about “tail lock”, perhaps it is a new mitigation for this issue?

I didn’t start with tailscale because the only way you could log into it was with Google or GitHub or something. I don’t trust Microsoft or Google with auth for my internal network. I thought about running Headscale but Nebula was faster/easier for me.

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by aborsy on Saturday June 15, 2024 @04:50AM (#40688429)
Yes, Microsoft and Google will not be able to authenticate to your network if you enable tail lock. A node in your network has to sign.
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by hammyhavoc on Saturday June 15, 2024 @10:54AM (#40690184)
Well, one example, depending on your threat model—their privacy policy states that they retain info and comply with subpoenas.

There's also potential for malicious updates to compromise a network (as there is with most software unless you're auditing the source for each update).

E2EE is only as meaningful as where the keys reside, and how easily those keys are abused.

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by aborsy on Sunday June 16, 2024 @04:15AM (#40695516)
That’s interesting!

The metadata is generally public information, I don’t care about that.

The malicious updates and key abuse are more concerning. It’s true for all software, and probably better done with OS, like on iOS.

The VPN could steal the keys, but that’s a lawsuit!

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by hammyhavoc on Sunday June 16, 2024 @01:01PM (#40698493)
Are the keys not already kept on their own infra?
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by aborsy on Monday June 17, 2024 @11:16AM (#40706461)
No, private keys don’t leave user’s devices. This is the case in all such products.

But with a malicious update, they could ship them to their infra, targeting some users. The product then becomes malware!

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by 1oooqooq on Friday June 14, 2024 @05:26PM (#40685302)
why either of these over plain wireguard if you're not provisioning accounts?
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by sneak on Friday June 14, 2024 @05:40PM (#40685398)
Wireguard doesn’t do nat punching and is not mesh, it’s p2p only.

totally different use case.

Re:Cost Per 1M tokens Of Self Hosting Llama-3 by 1oooqooq on Friday June 14, 2024 @07:36PM (#40686189)
true. i do may punch with a pnp client on the server side.
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by thot_experiment on Saturday June 15, 2024 @04:42AM (#40688395)
I feel like wire guard definitely does nat punching unless I misunderstand you, I've been doing this sort of thing to have my phone and desktop on the same "LAN" all the time so I can moonlight in from anywhere (among other things) and they're definitely natted.
Re:Cost Per 1M tokens Of Self Hosting Llama-3 by drio on Friday June 14, 2024 @07:41PM (#40686228)
@sneak, can you comment on your experience with nebula vs Tailscale?