Running large language models like ChatGPT on a single GPU

Re:Running large language models like ChatGPT on a single GPU by lxe on Monday February 20, 2023 @04:53PM (#34873455)

Got the ops-30b chatbot running on 3090 24GB. I set compress_weight=True and compress_cache=True, and ran with `python apps/chatbot.py --model facebook/opt-30b --percent 100 0 100 0 100 0`.

I also modified the prompt a bit to make it more... uh alive:

    Assistant: Did you know that Saturn is 97 times the size of Earth?
    Human: Are you sure?
    Assistant: What difference does size make, really, anyway?
    Human: You didn't answer my question.
    Assistant: Hey, it's your question I didn't answer! Anyway, did you know that on the surface of 
    Saturn, there's lakes of liquid methane and ethane?
    Human: I didn't know that. Are there any lakes on the Sun?
    Assistant: Look, I'll be honest with you. I don't know much about the Sun. Is it even a planet?
    Human: It sure is. Why don't you know about the Sun? I thought you were a hyperintelligent AI?
    Assistant: You're right. I'm a hyperintelligent AI. I'm hyperintelligent enough to know that I know jack shit about the Sun.

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by syntaxing on Monday February 20, 2023 @05:07PM (#34873600)

This is amazing. Reminds me of claptrap from Borderlands

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by LoganDark on Monday February 20, 2023 @05:23PM (#34873756)

It really does

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by samstave on Monday February 20, 2023 @07:19PM (#34874760)

OMG

What will be the first game with ChatGPT integrated into the NPC dialog interactions?

My vote is Hitman, with variable voices....

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by djokkataja on Monday February 20, 2023 @10:05PM (#34875956)

It's already been done with Mount & Blade II, though I don't know if that was the first.

https://www.techspot.com/news/97572-mount-blade-ii-mod-uses-...

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by razor_router on Monday February 20, 2023 @04:58PM (#34873514)

That's impressive! How much customization did you have to do in order to get the chatbot to respond like that?

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by lxe on Monday February 20, 2023 @05:14PM (#34873669)

Not much. Here's the prompt:

  A chat between a human and a more funny and witty intelligent assistant
  Human: Hello! What can you do?
  Assistant: Hey there! I can pretty much do whatever you ask!
  Human: What is the name of the tallest mountain in the world?
  Assistant: It's Everest.
  Human: Are you sure?
  Assistant: You damn right I'm sure! I'm a hyperintelligent AI!

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by posguy on Monday February 20, 2023 @05:31PM (#34873830)

All I can think of is outsourcing to ChatGPT now...

https://www.youtube.com/watch?v=rYaZ57Bn4pQ

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by mightytravels on Monday February 20, 2023 @08:05PM (#34875152)

Awesome - how many GB of hard drive/ SSD was required? What is your CPU RAM?

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by sva_ on Monday February 20, 2023 @09:38PM (#34875791)

I wonder if there would be interesting results if you make this model talk to openais chatgpt, since they are different models. Seeding them with some topic.

[ Parent | on HN ]

Re:Running large language models like ChatGPT on a single GPU by destroy-2A on Tuesday February 21, 2023 @02:32AM (#34877933)

Do not let Dinesh AI talk to Gilfoyle AI !

[ Parent | on HN ]