I've run hundreds of millions (150m so far in a couple of weeks of non-continuous running as I tweaked things) of tokens through my 2x 3090 with a 13b llama2 model I fine tuned on tasks like: summary, knowledge graph generation, writing using the knowledge graph, grammar, spelling, and transcription correction, etc.
This type of stuff is going to be done at scale with a modest budget if you have the skills to tune more efficient and faster models to your use cases.
If I didn't have to worry about my grandparents getting scammed because they were having 99.99% of it effectively filtered or warned about at one layer or another before it actually became a problem...can you imagine how much you could lower that type of fraud/abuse?
Exactly! Apple for instance already does this with some ML tasks that only run when your device is plugged in, I think it's a great compromise.
Some thermostats prioritize low-usage times, too.
I find there to be a giant gap in learning about this stuff between material that boils down to "use magic words and system prompts to improve results from one of the big models" and "how do LLMs work from first principles".
I still haven't found a great resource that covers this middle ground, which seems to me to be where a lot of the power of these approaches is going to reside.
Read papers, build institution, experiment.
That last part may be the most important.
My sense is that I have a much better grasp of the foundational material here, having read in depth books and papers about that, but still can't quite wrap my head around the question of how people are actually "operationalizing" this into useful software.
But to your point about experimentation, it might just be the kind of thing where there is no path to enlightenment besides working on a project and running into and overcoming all the hurdles along the way.
> Design your tasks to be repeatable and small steps, call the OpenAI API and log all requests/responses. > Filter out any bad responses and take a representative sample of the data you have collected from OpenAI,and train a Mistral or Llama2 model with the request/response pairs.
> Measure the quality of your model vs OpenAI for the same inputs, and then swap out the model in your workflow once happy with the results.
> This is how I made this model: Design your tasks to be repeatable and small steps, call the OpenAI API and log all requests/responses. > Filter out any bad responses and take a representative sample of the data you have collected from OpenAI,and train a Mistral or Llama2 model with the request/response pairs.
Just one example.
I've gone through a few of these "new kinds of software becoming useful" transition periods - most notably applications moving to the web, and then native smart phone applications - and in none of those transitions was there a dearth of resources on how to spin up on doing useful things due to this "moat" concern.
Nobody was protecting their iphone app dev moat by not publishing books and training courses on Objective-C and XCode...
Filter out any bad responses and take a representative sample of the data you have collected from OpenAI,and train a Mistral or Llama2 model with the request/response pairs.
Measure the quality of your model vs OpenAI for the same inputs, and then swap out the model in your workflow once happy with the results.
Getting a consistently good result for one shape of input may not indicate that same performance for another shape of input for example.
Anyway, nothing to say you can't use both, or have a fallback system.
Some of these open source models can even be run on your local machine. It’d be very inexpensive to run thousands of pages through it.
A cheap Mac mini with apple’s neural cores is good enough that it roleplays smut with a human at human speed. We’re going to see a rapid increase in throughput to price. We’ve already got small LLMs that run on mobile phones.
Paying additional 0.2 cents per request, if it can significantly improve your success rate, is not really that much and some people use LLMs for even simpler parsing tasks to save time on development efforts.
One can also imagine using GPT as part of a pipeline. Maybe one can extract most emails using regexps or other similar measures, but also use simple (computationally) heuristics to collect paragraphs of text which are likely to contain emails (which the regexps did not manage to extract), and then use chatGPT (or cheaper LLMs) to extract only from the most valuable paragraphs. Probably usefull (for the spammers) to also be able to categorize the paragraph, so programmers get programming-spam and chefs get chef-spam.
https://blogs.nvidia.com/blog/2023/10/20/eureka-robotics-res...
(not advocating for collecting email, just responding to all the comments that LLMs aren’t appropriate because of speed/cost)
That's why it's awesome to be able to locally run open-source LLaMA! Not to mention that by running it locally OpenAI is not aware of your shenanigans.
I'm sure some people would prefer to basically block email by default but that's not most of us.
Cost of LLM compute will continue to fall, models will take less CPU time to run, and the hardware will become more powerful.
'My email address is my username@"the company no one ever got fired for buying from".com'
Write me a regex for that.
GPT on the other hand:
I agree that the LLM is more versatile and retract what I wrote earlier saying that all patterns could be accounted for. I should have said most easy to figure out patterns.
Maybe not a big third-party commercial one, a smaller, purpose tuned one that the scraper runs on their own, OTOH...
Regarding the examples provided, there is nothing there that a simple regex couldn't achieve, so I don't really see the benefit of introducing a LLM into the flow, besides making it slower and more costly. john [at] company [dot] com was never a safe obfuscation in the first place, and ~99% of text obfuscations are known (because they have to be read by a human ultimately, and conventions are a thing).