News for hackers. Stuff that matters.
Re:DeepSeek-R1 by simonw on Monday January 20, 2025 @02:20PM (#42772077)
I got a quantized Llama 70B model working, using most of my 64GB of RAM but it's usable:

    ollama run hf.co/unsloth/DeepSeek-R1-Distill-Llama-70B-GGUF:Q3_K_M
That's a 34GB download. I'm accessing it via https://github.com/open-webui/open-webui which I ran like this:

    uvx --python 3.11 open-webui serve
I have TailScale on my laptop and phone so I can run experiments directly from my phone while leaving my laptop plugged in at home.
Re:DeepSeek-R1 by jt_b on Monday January 20, 2025 @03:22PM (#42772661)
Thanks for all your writing on these topics Simon! It has turned me from a bit of a naysayer to an optimist around this tooling, especially being able running stuff locally with access to tools. Have an M4 Pro mac mini arriving this week to have a similar self hosted setup over tailscale.
Re:DeepSeek-R1 by brewtide on Friday January 24, 2025 @07:11AM (#42812355)
Maybe related, but I've also got a tailscale instance running for the same use case (on an older it box but...) I've also installed open-webui attached to ollama. That way the interface I can deal with when on my phone is just a simple nice to use webpage. May want to look into this? Thus far it's worked very slick.
Re:DeepSeek-R1 by jt_b on Saturday January 25, 2025 @11:57PM (#42827804)
Nice, thanks for the suggestion. I got it set up just before leaving town for a few days, so have been doing a little tinkering with it. I was hoping to have a setup with LM Studio, where my laptop could use the API Server from the mini over the TS network. Unfortunately doesn't seem to be the case, so I'll set up a configuration like you mentioned to just have a global client from any device on the netowrk.

It's very cool to be able to have access to such a high horsepower machine from anywhere though. Next step is figuring out the networking interface to be able access the host GPU/ollama API from pods running in a Colima VM/k3s cluster setup.

Re:DeepSeek-R1 by matt_daemon on Monday January 20, 2025 @04:41PM (#42773390)
> Here's what I got - the joke itself is rubbish but the "thinking" section is fascinating

Very funny. How does the 70B model do in comparison?

Re:DeepSeek-R1 by pmarreck on Monday January 20, 2025 @07:40PM (#42774956)
Time to put this 128GB M4 Mac to work...

    bash> ollama run hf.co/unsloth/DeepSeek-R1-Distill-Llama-70B-GGUF:Q5_K_M
(49GB!)

(It wouldn't accept the Q_8 because it was sharded.)

Will report back lol

What app do you use on your phone to access your home LLM servers over tailscale?

Re:DeepSeek-R1 by liamwire on Monday January 20, 2025 @09:37PM (#42775781)
Not to speak for him, but I’d suspect Simon would be using a terminal emulator. He often mentions preferring to use his own command line tool, named LLM, as it allows one to log everything to an SQLite database for later exploration.
Re:DeepSeek-R1 by pmarreck on Monday January 20, 2025 @09:47PM (#42775884)
Hah, I probably have a similar tool just called "ask" which logs to a JSON log instead of a sqlite db
Re:DeepSeek-R1 by pmarreck on Monday January 20, 2025 @09:48PM (#42775897)

    bash> ollama run hf.co/unsloth/DeepSeek-R1-Distill-Llama-70B-GGUF:Q5_K_M
    pulling manifest
    ...
    success
    >>> Tell me a "what's the difference between" joke about techbros and another funny comparison of your choosing
    <think>
    Okay, so I need to come up with a "What's the difference between" joke comparing techbros and another group. Hmm. Let me think.

    First, what defines a techbro? They're usually associated with startups, tech gadgets, maybe working in Silicon Valley, into things like crypto, NFTs, and they often have a certain lingo or way of
    dressing—like hoodies, jeans, maybe beanies. They might be seen as overly optimistic about their ideas changing the world but sometimes disconnected from practical realities.

    Now, I need another group to compare them with. Maybe something else in tech but different enough to create humor. Let's think: developers? No, too similar. Maybe hipsters? Or perhaps venture capitalists?
    Wait, what about "developers who actually ship code"? That could work because it plays on the idea that while techbros talk a lot about their ideas, developers are more focused on delivering actual
    products.

    So the joke structure is usually two lines: one for each group, highlighting a key difference. It needs to be punchy and capture the essence of both groups in a humorous way.

    Let me draft it:

    "What's the difference between a techbro and a developer who actually ships code?"

    "Techbros are still explaining their MVP to you three years later, while developers just sent you a pull request titled 'it works now'."

    Wait, does that make sense? Let me break it down. A Minimum Viable Product (MVP) is something techbros often talk about but might take forever to actually build or ship. On the other hand, developers who
    are focused on shipping code would just send a pull request indicating they've fixed something and it's done now.

    I think that captures the difference between all talk and actual action. It's relatable for people in the tech industry who know the dynamics between idea people and implementers.

    Is there another angle? Maybe comparing them to something else, like entrepreneurs or investors, but I think the developer angle is solid because it directly contrasts talking about ideas versus executing
    them.

    So the final answer would be that joke. It highlights the techbro's tendency to over-explain without delivery, while developers focus on getting things done.
    </think>

    **What's the difference between a techbro and a developer who actually ships code?**

    *Techbros are still explaining their MVP to you three years later, while developers just sent you a pull request titled "it works now."*

    This joke humorously contrasts the tendency of techbros to focus on ideas and discussions with the action-oriented approach of developers who deliver tangible results.
Re:DeepSeek-R1 by thih9 on Tuesday January 21, 2025 @11:00AM (#42781587)
IMO passable joke, not unlike modern content; if I saw it in the wild I’d laugh, feel called out and scroll down without thinking that AI may have written this.
Re:DeepSeek-R1 by Mo3 on Tuesday January 21, 2025 @05:43PM (#42786058)
> Let's think: developers? No, too similar

Ouch

Re:DeepSeek-R1 by m463 on Wednesday January 22, 2025 @09:24PM (#42799947)
Funny, too much musing about the joke but then actually shipping it. must be from an AI, sort of an unfunny valley.
Re:DeepSeek-R1 by darthrupert on Thursday January 23, 2025 @01:09AM (#42801171)
The part between the "think" tags is (I guess) what ChatGPT-o1 hides under vague ux updates like "thinking about the problem". Perhaps the UI over ollama should hide them too.

Personally, I like to see the thought processes.

Re:DeepSeek-R1 by dielll on Monday January 20, 2025 @11:10PM (#42776533)
On Android use Termux.

It's a terminal emulator

Re:DeepSeek-R1 by simonw on Monday January 20, 2025 @11:24PM (#42776601)
I'm running the Open WebUI Python web server app and accessing it on my phone via Mobile Safari.
Re:DeepSeek-R1 by Hasnep on Monday January 20, 2025 @11:43PM (#42776710)
I use the GPTMobile app on Android to access ollama over Tailscale.

https://github.com/chocydar/GPTMobile

Re:DeepSeek-R1 by sebws on Tuesday January 21, 2025 @04:00AM (#42777895)
The tailscale app on iOS can turn on a VPN on the phone to connect into your tailnet. Once this is enabled, you can then access the tailnet IPs from anything, e.g., in your browser to hit a web ui. You can then take it further with local DNS and a reverse proxy to access your home server services at a URL.
Re:DeepSeek-R1 by jondwillis on Monday January 20, 2025 @10:32PM (#42776250)
I set up a cloudflare tunnel recently so I could also use my local LLMs remotely! It’s nice.
Re:DeepSeek-R1 by regularfry on Tuesday January 21, 2025 @08:18AM (#42779772)
I've previously had a 70B Q2 wedged into a 4090. Not much space for context but it runs...
Re:DeepSeek-R1 by unstatusthequo on Tuesday January 28, 2025 @06:06PM (#42859337)
Yes, I did the same but with Docker serving open-webui. It's definitely not fast. Took 2 minutes to answer "what is the highest peak in California," but it eventually go there. OpenAI o1 took 6 seconds.

Pasting the logic for anyone interested:

<details type="reasoning" done="true" duration="102"> <summary>Thought for 102 seconds</summary> > Okay, so I need to figure out what the tallest peak in California is. Hmm, I remember that the highest mountain in the contiguous United States is Mount Whitney, but wait, isn't that in California? Let me think. Yes, I'm pretty sure it's in California. But maybe there are other tall peaks too. > > Wait, no, Mount Whitney is definitely the tallest. It's over 14,000 feet. But sometimes people might confuse it with other mountains like Shasta or something else. Oh, wait, isn't Mount Shasta also a volcano? I think that's in California too but it's not as tall as Whitney. > > So, putting it together, Mount Whitney is the tallest peak in California at 14,505 feet. It's located in the Sierra Nevada range. People hike there a lot because it's such a high point. But I should double-check to make sure I'm not mixing up anything else. Yeah, that seems right. > I wonder if there are any taller peaks elsewhere. Oh, Denali in Alaska is higher, but that's not in California. So within the state, Whitney is the highest. Got it. > So the answer is Mount Whitney at 14,505 feet above sea level. It's a popular spot for hikers and climbers. Makes sense because it's such an iconic peak in the Sierra Nevada mountains. > I think I've got all that right. No other peaks in California are taller than Whitney. So yeah, Mount Whitney is the tallest. > Just to make sure, sometimes there are different names or maybe alternate names for peaks, but Whitney is pretty straightforward. Yeah, it's definitely the one. > Okay, so I'm confident now. The answer is Mount Whitney. > *Final Answer* > The tallest peak in California is Mount Whitney, with an elevation of 14,505 feet above sea level. > > \boxed{Mount Whitney} </details> To determine the tallest peak in California, we consider notable mountains such as Mount Whitney and Mount Shasta. Mount Whitney is recognized as the highest peak in the contiguous United States, standing at 14,505 feet. It is located in the Sierra Nevada mountain range and is a popular destination for hikers. Mount Shasta, although also a significant peak, does not surpass Whitney's elevation. Therefore, the tallest peak in California is confirmed to be Mount Whitney.

*Final Answer* The tallest peak in California is \boxed{Mount Whitney}.