It was about time to build a new desktop anyways (roughly 4 to 6 years before the old one goes to frolic at the server farm in the basement) and $2,000 will easily buy a machine that can run the quantized 65b models right now. So I spent slightly more than I normally do on this latest box and it's happily spitting out 10+ tokens a second.
You're not going to beat GPT-4 yet, but you have direct control over where your info goes, what model you're running, compliance with work policies against using public AI, and relatively cheap fixed costs.
Not to mention, the local version works with no internet and isn't subject to provider outages (not entirely true - but you're the provider and can resolve).
Seems like an easy win for anyone who might be buying a desktop for graphic/gaming anyways.
It's also not licensed commercially, so I avoid some things with it (ex: I do a lot of personal learning/investigation, but it doesn't touch or write anything related to work or personal projects)
The open models are a little further behind, but it's interesting to see them spin off into niches where they have strengths based on tuning/training.
Based on the rate of progress in the open source world, it won't be more than a year before we have an open source model that is truly superior to GPT 3.5
The commercial & api based models are still more capable general purpose tools. But the current open tooling can do some nifty stuff, and the community around it is moving at a breakneck speed still.
In some areas, it's acceptably good. In some areas it's not. But it's getting better really fast.
Can you elaborate on what those capabilities are?
That said:
- For many uses, it doesn't matter. For many of the ways I use it, I don't care. For basic use (e.g. clean up an email for me), it's basically the same. For things like complex reasoning, algorithms, or foreign languages, the hosted service is critical.
- GPT3-grade models have more soul. OpenAI trained GPT3.5 and 4 to never do anything offensive, and that has a lot of negative side effects, well-documented in research. The way I'd describe it, though, is the difference between talking to a call center rep and your grandma (with mild Alzheimer's, perhaps). They both have their place.
- Different models are often helpful in workflows.
My experience is anecdotal. Please don't take it as more than one data point. If other people post their anecdotal experiences, you'll get the plural of "anecdote."
I'm absolutely disgusted by OpenAI for this "do no offense" approach. How can people so smart be so damn uneducated?
Then again, this industry has disgusted me for a long time so it's not really a surprise.
https://huggingface.co/ (finding models and downloading them)
https://github.com/ggerganov/llama.cpp (llama)
https://github.com/cmp-nct/ggllm.cpp (falcon)
For interactive work (art/chat/research/playing around), things like:
https://github.com/oobabooga/text-generation-webui/blob/main... (llama) (Also - they just added a decent chat server built into llama.cpp the project)
https://github.com/invoke-ai/InvokeAI (stable-diffusion)
Plus a bunch of hacked together scripts.
Some example models (I'm linking to quantized versions that someone else has made, but the tooling is in the above repos to create them from the published fp16 models)
https://huggingface.co/TheBloke/llama-65B-GGML
https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ
https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored...
etc. Hugging face has quite a number, although some require filling out forms for the base models for tuning/training.
Which won't run everything, but will run model in the GGML format such as https://huggingface.co/TheBloke/llama-65B-GGML
The steps are basically:
1. Download a model
2. Make sure you have the latest nvidia driver for your machine, along with the cuda toolkit. This will vary by OS but is fairly easy on most linux distros.
3. compile https://github.com/ggerganov/llama.cpp following their instructions (in particular, look for LLAMA_CUBLAS for enabling GPU support)
4. Run the model following their instructions. There are several flags that are important, but you can also just use their server example that was added a few days ago - it gives a fairly solid chat interface.
1) Go to https://gpt4all.io/index.html
2) Click the downloader for your OS
3) Run the installer
4) Run gpt4all, and wait for the obnoxiously slow startup time
... and that's it. On my machine, it works perfectly well -- about as fast as the web service version of GPT. I have a decent GPU, but I never checked if it's using it, since it's fast enough.
128gb ram ~400
reasonable processor/mobo/psu ~600
2Tb m2 drive ~94
In hindsight - I don't know that the second GPU was worth the spend. The c++ tooling is doing a very good job right now at spreading work between GPU vram and main ram and still being fast enough. Even ~4/5 tokens a second is fast enough to not feel like you're waiting.
I'd suggest skipping the second card and dropping the price quite a bit (~2100 vs ~2900) unless you want to tune/train models.
My experience is only a few systems will share load across GPUs. I didn't bother with dual GPUs for that reason.
4-5 tokens per second is slower than my system. I'm getting in the teens. I'm a little surprised since yours is newer, faster, and has way more RAM.
If I push everything into VRAM - I get 12.2 tokens on average running quant 4 llama 65b.
If I run a smaller model I get considerably faster generation. Ex: llama 7b runs at 52 tokens/sec, but it's small enough I don't need the second GPU.
Ex - here's my nvidia-smi output while 65b is running
Does that mean $1 trillion? If so, citation badly needed.
If it meant good grades without doing the work and it became socially acceptable it is ~150 million global tertiary students times 10 months of the year times whatever that is worth on a monthly basis. Maybe only $150B ARR right now at $100/mo (textbooks to read cost more than this, but you don’t need them anymore), so not $1T on students but still massive, and if you capture them in pre/early career it is easily $1T in ARR once your queue deepens.
So how do we value ChatGPT then? Should it be as valuable as Coursera, Udacity etc put together? Should it be that PLUS Harvard Deloitte and ServiceNow? Or since they have no major moat should it be Zero? It’s hype of this scale that can rationalize unprecedented figures.
https://www.cnbc.com/amp/2021/03/31/coursera-ipo-cour-begins...
Coursera is a discount retailer of tertiary education so I don’t see any relevant connection.
ChatGPT/OpenAI specifically is probably already overvalued unless they have some first-mover advantage that I can’t see.
Someone else has said it many times over but I agree that LLMs will disappear into the technology stack — a rising tide that lifts many market caps.
https://www.forbes.com/sites/susanadams/2021/01/28/this-12-b...
Huge percentage of the teenagers at her school are using it to do their assignments.
Later I her to read a few chapters of her book for summer reading. When I quizzed her it, she got it right.
Found out later she used ChatGPT to give her a summary of the first chapters. She did not enjoy having first weeks of summer without a phone.
School is having to change all assignments to be written in class on paper. No phones allowed.
How about accepting new realities and changing homework assignments instead?
But you cannot unironically think we can substitute fundamental skills like essay writing and critical thinking for a degree in 'prompt engineering'?
Personally, I don't think that. However, I also think it's logical and good for a child to select the most efficient tool available for a task.
I am not this girl's parent, and I'm not sure how I would have handled the situation if I was. However, I worry that simply taking away her phone may have been counterproductive. I would have erred toward some sort of open conversation about the purpose of the assignment and what she is hoping to learn.
Phone is her currency.
A warning on losing it will usually fix any behavior problems.
After one really bad episode of mouthing off to her mother, it sat in the blender for a week.
There was a firm understanding that it was being turned on if she didn’t correct herself.
She’s still a mouthy teenager, but she’s learned to keep it at a playful level and not getting out of hand.
Critical thinking is still a skill, but even there GPT-4 is great at giving suggestions and ideas when you're not sure what to do. I got into a long recurring argument with extended family members at an event last month. I pulled out my laptop and dropped in everyone's complaints into GPT and asked the best way to move forward. An argument that normally would have gone on for hours was over in 20 minutes, and we actually felt a sense of mutual understanding for once.
The world is changing very rapidly right now.
Suggestions for how to change homework assignments? It seems like a difficult problem, so reverting to hand writing things with no phones is probably the best option for now...in my opinion.
What does someone learn from doing this? What happens when they actually have to use their brain to do something hard?
At this point, why bother?
Of course! How to apply the calculator to problems, and how to discern if a problem is calculator-solveable. We've decided as a society it's better for students to use calculators as a tool in mathematics, why is chatGPT different in literature?
You have to understand the problem and how to use the tool.
If one's understanding of how to use the tool is limited to "access the tool and dump in the question, then blindly paste the response into my homework document", I'd say you are not learning much. It's obvious that every contemporary teenager can learn that in a few minutes. That's not giving us the capabilities to move forward as a society.
But we can? Students using ChatGPT to get homework answers is the whole thing we are talking about.
E.g., we could not use a calculator at all when learning arithmetic.
Once we learned arithmetic and were learning (say) algebra, we could use the calculator to do arithmetic, but not to compute the algebra for us.
While learning trigonometry, we could use the calculator for arithmetic, but not for computing sines and cosines.
While learning calculus, we could use the calculator for arithmetic and sines and cosines, but we could not use it to perform integration or differentiation for us.
And so on.
I don't think Google in 2024 is guaranteed to be the critical-thinking tool that 2004 Google might have been.
2. Consider making it non-graded (optionally providing feedback if desired that will not be counted in course grading.)
3. Require oral/in-person explanation of homework to paid peer assistants recruited from the previous cohort.
4. For writing, change to a writing workshop model where ChatGPT is a permitted tool that is incorporated into the workshop and students can learn how it can be helpful and how it can run astray. Help students to find and write in their own distinctive voice, even if they are assisted by ChatGPT.
They never should have moved away from this if they are worried about cheating.
But if students are using ChatGPT to summarize/prep for stuff at home, fine. No different from the Cliff Notes I used when I was that age.
Good for you. I know that kind of situation is not fun for a parent either.
AI tools exist, we should be encouraging our kids to learn how to exist in a world with them, not some nostalgic extinct world without.
Yes, but other than being conditioned to believe the experience of reading is superior, why are we striving for that experience?
Doing the square root of 147 by hand is a more educational experience than using a calculator, but I can understand the concept of a square root without having to factorize 147 (the same cannot be said of my pre-calculator ancestors).
It also saves me from having to remember the decimal representation of sqrt3.
With math, when you are first learning the very basics it helps to do a few problems by hand to get a feel of things. Sure if you have a degree in STEM have have deal with advance math you can probably get away with just reading the theory only.
We just built a very strong culture encouraging it because for a long time it was the most efficient way to do both things.
Now it’s not, but the tradition of it being important is hard to shake.
The closest analogue in my mind is how the classics in their original Greek were considered the only true way to learn for a very long time (even once most of the important knowledge had been translated to English very well).
Most of the things a kid does on a computer in 2023 are actively geared toward not using your brain in any way.
This is more analogous to a parent in 1985 taking away both the phone and the TV from their kid.
Eye of the beholder, I guess. I certainly see people struggle to use LLM's so I disagree with the implication that it's a zero-skill activity.
Honestly, the pupils of my generation had other sources for finding summaries of the books to read for school (e.g. on the web, or in form of other books that contain a summary of the book to read (the respective pages were copied and these copies were passed among classmates).
Thus: just the methods change over generations, how pupils behave is much more constant.
I don't know, if high school and undergrad students being off for the summer is enough to shrink your app maybe it's not all that?
But for actual non-fiction usage, you have to spend so much time triple-checking everything they say to ensure that it isn't simply complete nonsense. What's the point?
I'm convinced the people who say it's nothing but a BS machine have never tried to use it step by step for a project. Or they tried to use it for a project most humans couldn't do, and got upset when it was only 95% perfect.
Still, it's way better and more efficient than Google. Less than not being lazy and using my two braincells tbh.
My newest use is
Hello, i' m working on X, I use Y tech, my app do Z and I want to implement W. Can you provide a plan on how and where to start?
Personnally, a lot of bash, C, AWK at the moment (typescript + html/css until last april, now i'm back to the basics). The figure i gave in my post were more for that.
The last time i used it was yesterday, i wanted to hack something on an old game i used steam+proton for. I knew it was a weird Wineprefix, so i asked ChatGPT for it, i might have asked poorly, but after fiddling, i had the response (tbh i had to look how to get the game ID, so in the end i lost more time than not), then when it still didn't work, cause the path was shit, i entered all necessary context in ChaGPT4, and it couldn't find the easy "USER=steamuser" env variable to add before launching Wine. I stopped after 10 minutes, looked into a example Wine cfg file, understood the issue and fixed the problem myself.
I mean, it's probably good for really basic stuff, so it could have helped me when i was starting, but 80% of the stuff i code automatically without really thinking about it, and when i have to stop to think, ChatGPT isn't helping. Also tbh, VSCode is really, really good and fix my old, time-consuming task of "what's this argument again?"
The Unreal engine code is documented and publicly avaiable for OpenAI to ingest and it still gets the basics wrong.
I wasted hours trying to get it to explain to me what I didn't know, if it doesn't understand the internals of Unreal, I have no hope for it on bigger and better codebases.
It doesn't parse, it doesn't explain, it does not grok. It guesses at best and the blood sucking robot-horse is not telling the truth.
In my experience with coding (I've only done javascript and python myself) you have to tell it to explain and grok. It takes on the role you give it. Even just saying something like "you are a professional unreal developer specializing in C++, I am your apprentice writing code to (x). I want you to parse the following code in chunks, and tell me what might be wrong with it" before typing your prompt can help the output immensely. It starts to parse things because it's taken on the role of a teacher.
People love to hate on the idea of "prompt engineering" but it really is important how you prime the thing before asking it a question. The other thing I do is feed it the code slowly, and in logical steps. Feeding it 20 lines of code with a particular purpose / question you'll get a much better answer than feeding 200 lines of code with "what's wrong here?" You still need to know 90% of what's going on, and it becomes very good at helping out with that 10% you're missing. But for all I know it is just really bad at C++, that wouldn't surprise me. The things I'm using it for are definitely more simple.
Knowing that, it makes sense that your prompt should be as specific as possible if you want the results to be as specific as possible.
The best results I got was feeding it Lisp code that I wanted translated to C (to compile it). It took very little effort on my part because I described what each of the snippets did separately, and the expectation when combined and used together.
Through this, I learned that C doesn't have anything akin to the Lisp's (ATOM). ChatGPT stated clearly that its version of ATOM should only be expected to work in the code it was writing, but might not work as expected if copied out for another use of Lisp's (ATOM).
I asked it to give examples of where it wouldn't work, and it gave me an example of a code snippet that used (ATOM) that would not have worked correctly with the snippet that did work correctly with my original purpose.
Having said that, I myself learned that working with code function by function with ChatGPT, and being explicit about what you need, gives very good results. Focusing on too many things at one time can derail the whole session. One or two intermingling functions works great though.
In my testing prompts did not unlock an ability in GPT to grok the structure of code.
Empirical testing of LLM's is going to prove and map out it's weaknesses.
It is wise to infer from intution and examples what it can handle, leave the empirical map of it's capabilities to the academics, for the provable conclusions.
I wanted to create a web app, something I haven't done in a very long time. Just a simple throwaway back-of-the napkin app for personal use. I described what I wanted it to do, and asked what might be a good frontend/backend. It listed a few, I narrowed it down even more. Ended up deciding on flask/quasar.
After helping me setup VS Code with the proper extensions for fancy editing, and guiding me through the basic quasar/flask setup, it then was able to help me immensely creating a basic login page for the app. Then it easily integrated openAI api into it with all the proper quasar sliders for tokens/temperature/etc. Then it created a pretty good CSS template for the app as well, and a color scheme that I was able to describe as "something on adobe color that is professional and x and x (friendly, warm, whatever you want to put in)". Everything worked flawlessly with very little fuss, and I'd never used flask or quasar before in my life. You can also delve VERY deep into how to make the app more secure, as I did for fun one evening even though it's not going to be internet facing.
Another thing I did was go over some pfSense documentation with it. I had some clarifying questions about HAProxy, as well as setting up Acme Certificates with my specific DNS provider. It was extremely helpful with both. It also taught me about nitty gritty settings in the Unbound DNS resolver in a way that's much more informative than the documentation, and helped me set up some internal domains for pihole, xen orchestra, etc with certificates. Also helped me separate out my networks (IoT, Guest network, etc), and taught me about Avahi to access my hue lights through mDNS.These are things I always wanted to do, I just never felt like going down a google rabbit hole getting mostly the wrong answers.
Last example I'll give is it was able to help me set up docker-compose plex within portainer that then uses my nvidia GPU for acceleration. The only thing I had to change from the instructions it gave was to get updated nvidia driver #s and I grabbed the latest docker-compose file. I'd never used portainer in my life before, nor do I have experience with nvidia drivers within linux, and I feel like learning it was many times faster being able to ask a chatbot question vs trying to google everything. Granted I still had to RTFM for the basics, as everyone should always do.
I think perhaps my use cases are a bit more "basic" than many HN users. Like I said I'm not asking it to do problems most humans wouldn't be able to do, as I know it isn't quite there yet. But for things like XCP-ng, portainer, linux scripts, learning software you've never used before, or even just framing a problem I'm having in steps I hadn't thought of it's been invaluable to me. For me it's like documentation you can ask clarifying questions to. And almost none of the things I've asked it would work at all if it were wrong, I would know immediately.
A few weeks back I was looking into how white supremacy works cause I didn't get it at all. We both came to a nice insight (it's a lot like a business monopoly) https://chat.openai.com/share/930e257f-addd-4371-ac37-370261...
Exactly; search engines give you those blue links and short descriptions of the search results which are not enough for you to grasp what is the website about. I think what search engines need to do is tackle the complexity of going through the results of a search engine. Google page rank seemed like a silver bullet back in the day but the websites which are the most popular are not necessarily of the best quality. What we need is to lower the complexity for casual users when they deal with search results.
On the other hand ChatGPT is like an answer machine that can give you satisfactory answer on your fist try but if not, you need to talk with it, push it and explore what answers it gives you, just like you said. I think ChatGPT type search engine will be more suitable for people who are "lazy" or for the people who don't have time to "Google" and go through search results and look around the web for the helpful and useful information.
This is exactly what I don't want a search engine to do for me. Going through the list of results and evaluating them is an important part of my process, if what I'm trying to do is learn something new.
[0] https://blogs.bing.com/search/2022-08/Shopping-Searches-are-...
They don't all look the same. They all tend to go to different places. I find that it's reasonably easy to spot a great deal of garbage sites just from their domain name or url, and that weeds out a large chunk. Ignoring multiple results for the same site also weeds out a large chunk (I only need one of them).
The rest, I just click on and take a look at the page. It's pretty quick and easy to weed out most of the garbage ones with a quick skim.
The rest, I sample, read captions and boxes, skim paragraphs and such to determine if it's along the lines of what I want. That's pretty quick too.
For the most part, it's the same process that you use when researching in a library.
The reason that I want to do this myself rather than outsourcing it is because I'll inevitably learn something in the process that will shift my viewpoint to one that's more targeted or meaningful for the purpose I have in searching.
It doesn't matter how good the engine is at collating and summarizing results -- even if it's perfect, my understanding not only of what I'm looking to learn, but also discovery of important but serendipitous or unexpected knowledge, is lessened.
It's a bit like the difference between reading Cliff's (or Cole's) Notes about a book and reading the book.
At least they tell you where the text came from, so you know to skip it. It's worse when they just post an LLM response as their own.
"Code" is really a much much much smaller and much much much more structured output than "English words".
Presumably, the system was trained with a very small amount of "untrue code" in the sense of stuff that just absolutely could never work. And also presumably, it was trained with a lot of free-form text that was definitely wrong or false, and highly likely to have been originally created to be purposefully misleading, or at a minimum, fiction.
That the system outputs reliable code tells us nothing about its current ability to output highly reliable free form text.
Somewhat more general, but I've pretty much already decided that if I find people using it to talk to me without telling me, I won't be talking to them. Goes for businesses as well as personal things - don't gaslight me, or you will lose the option to do so.
I'm curious on if you feel human generated content does not contain falsehoods.
Its not, though. ChatGPT, the website is basically an (increasingly non-exclusive, given BingAI) frontend. What is claimed to be revolutionary is the underlying models (in the narrow view) and similar generative AI systems (in the broader view). As more applications are built with either OpenAIs own underlying models or those from the broader space, the ChatGPT website should be expected to represent a smaller share of the relevant universe of use.
only they can tell though.
No. Every tech company at the moment is scrambling to build LLMs into their product. That's where the real value is going to be.
Speaking for myself, I used to use the crappy chat interface but I now exclusively use tooling I've built up around their API.
So, n=1, I'm using OpenAI much more, despite using chat.openai.com less.
If it's the case that "there are new tools coming, but most of them aren't out yet" - and I believe it is[0] - then the overall userbase of ChatGPT doesn't matter to OpenAI, because soon enough the same models will come back with a vengeance, in a different, more streamlined form.
In fact, I feel that the major change will happen if and when Microsoft gets their Office 365 and Windows copilots working and properly released: they'll have instant penetration into every industry, including scientists, lawyers, doctors, and office workers.
--
[0] - It's been only few months. Between playing around, experimenting, then developing, testing and marketing a tool, there just hasn't been enough time to do all of that.
I’m more curious about their B2B operations which is likely what will trickle into more people’s lives. Their APIs seem to enable quite a few interesting possibilities with a low up front technical investment.
Too me the whole “generate me a bunch of text and display it as text” is a niche use case for a lot of people. Integrations for web search, document search, summarization, data extraction/transformation, and sentiment analysis are more useful and less likely to have hallucinations affect the end product.
Regarding their revenue I’m curious how the Azure hosted OpenAI services work for OpenAI. Billing is all through Microsoft and the documentation tries really hard to make it clear these are Azure services. I wonder if Microsoft just pays a licensing fee or if there is some revenue sharing going on.
For ChatGPT specifically, yes, because it keeps getting hyped so much, and I think B2C is going to be what ChatGPT ends up with. My company isn't known for its tech innovation and we're spinning up our own LLM based on our own data. It doesn't need to be super fast because I doubt we're going to go the chatbot route. It will likely be for content generation, so less horsepower is fine. No need to pay OpenAI for excess capacity.
They could go public now with an unreal valuation based on the hype of B2B usage that may never materialize. Revealing that you're mainly a cheat/study tool for students puts you in a box with Chegg and others, at a much lower valuation.
https://www.pewresearch.org/internet/wp-content/uploads/site...
> "Chollet thinks he knows what’s going on: summer vacation... a significant portion of students using ChatGPT to do their homework. It’s one of the most common uses for ChatGPT, according to Sam Gilbert, a data scientist and author."
https://finance.yahoo.com/news/chatgpt-suddenly-isn-t-boomin...