News for hackers. Stuff that matters.
Re:My AI costs went from $100 to less than $1/day: Fine-tuni... by tylervigen on Thursday January 18, 2024 @06:39PM (#39049546)
Ethics and restrictive terms aside, it doesn't seem like GPT 4 was necessary for what the poster did. How much worse or more difficult would it have been to use Mixtral or 3.5 to generate the first 100 good prompt-response pairs and then manually tweak them as the poster did?
Re:My AI costs went from $100 to less than $1/day: Fine-tuni... by thrwayaistartup on Thursday January 18, 2024 @06:41PM (#39049576)
Or just... write 100 good prompt-repsonse pairs yourself.

2024 will be the year of synthetic data. 2025 will be the year of "you know you can use your own brain and type out 100 datapoints faster and cheaper than generating and filtering assloads of synthetic data, right?"

Maybe we can even skip 2024 :)

Re:My AI costs went from $100 to less than $1/day: Fine-tuni... by sp332 on Thursday January 18, 2024 @06:50PM (#39049664)
Databricks had their employees write up 15,000 of them. https://www.databricks.com/blog/2023/04/12/dolly-first-open-...
Re:My AI costs went from $100 to less than $1/day: Fine-tuni... by icyfox on Thursday January 18, 2024 @07:39PM (#39050118)
Favorite part of this piece:

> We were initially skeptical whether we would get to 10,000 results. But with nightly leaderboard gamification, we managed to break 15,000 results within a week. Out of fear of eating into our productivity, we closed the contest.

I've hosted a few of these corporate data labeling events. If sufficiently gamified / there's a good enough UX, they can be surprisingly engaging. It helps a lot if you have a large employee base though. Distributing results over 5000 employees is exponentially easier than even 50 - in practicality, even larger than the orders of magnitude.

Re:My AI costs went from $100 to less than $1/day: Fine-tuni... by code_runner on Thursday January 18, 2024 @07:08PM (#39049848)
I’ve worked at plenty of places where we did a ton of labeling by hand.

People concerned with data quality from LLMs should really see the inconsistencies we came up with!

Re:My AI costs went from $100 to less than $1/day: Fine-tuni... by ThrowawayTestr on Thursday January 18, 2024 @07:43PM (#39050157)
Anybody have this downloaded and can paste a few examples?
Re:My AI costs went from $100 to less than $1/day: Fine-tuni... by sp332 on Thursday January 18, 2024 @08:02PM (#39050320)
Re:My AI costs went from $100 to less than $1/day: Fine-tuni... by schreiaj on Thursday January 18, 2024 @08:07PM (#39050358)
Yes and no, for text type stuff? Yes you're right. But I think in the vision space synthetic data will remain useful for a lot of things. I'm currently working on building a pipeline for personal projects to go from CAD models of environment to segmented training data. So far it looks almost as useful as real world data at a fraction of the cost of manual labeling.
Re:My AI costs went from $100 to less than $1/day: Fine-tuni... by cjbprime on Thursday January 18, 2024 @06:53PM (#39049704)
Using 3.5 seems like it would have been okay (technically but not legally given OpenAI ToS), but Mixtral is already somewhere around that level, so it's not achieving a goal of improving towards GPT-4 level responses without having to write those yourself, which seems to have been the goal OP wanted to get to.

I've heard that training e.g. Mixtral on Mixtral's own outputs is a really bad idea, don't know full details.

Re:My AI costs went from $100 to less than $1/day: Fine-tuni... by gumballindie on Thursday January 18, 2024 @06:53PM (#39049706)
I am sorry what ethics? Openai is built on stolen content and your worry is people using openai’s output is the unethical issue? Whoa.
Re:My AI costs went from $100 to less than $1/day: Fine-tuni... by atleta on Thursday January 18, 2024 @08:57PM (#39050707)
Well, it seems that initially started with GPT4 but his costs were becoming high so he had to do something and had to do it quickly. Technically he could have written a few hundred responses himself while the site was still using GPT4 using the prompts from the users but that could have been slow (expensive)/boring, etc.