I don't think so.

Cross discipline skills are more valuable in the real world, outside of hyper specialization seen in the California tech scene. Even there, it's coming to an end.

Exactly what I was thinking. If this model will be used for coding, does it need to be fed the entire (mostly pirated) collection of works from Anna's Archive? Probably not. Maybe some, primarily computer-science oriented.

The main corpus would be a collection of thoroughly reviewed, clean, logical, well-documented, and secure code in whatever languages this thing will need to be an expert in.

This way we could also look into reasonable compensation agreement with authors of the code and the books/articles, if and when applicable.

Has no one tried that?

Unless your task happens to be completely contained in the set of your "curated works", that is, trivial and computer-sciency, the resulting model is likely to have for example absolutely no concept of common sense, society, the natural world, what humans consider artistic or of good taste. It will probrably make, say, a good and correct compiler, but fail spetacularly on anything else more complex that a toy crud.

Are you saying that LLMs have common sense, good taste, or can determine what is artistic?

Human lawyers, coders, accountants and designers are a lot more than their jobs.

They are people with common knowledge, a lawyer has to understand carpentry to understand how to deal with how a client's roof fell off, a coder has to understand transportation for writing a travel app, an accountant has to understand medicine to to accounting for a pharmacy, a designer how to understand cooking to design a toaster. Not expert knowledge, but enough to understand that toasters are hot and need electricity.

A LLM for people working these jobs also need to have common knowledge, that's the whole point of using LLMs, you can prompt them with intrinsically ambiguous and incomplete natural language and they are supposed to fill in the gaps. And the more common knowledge they have, the better they are at it. So why settle for less? If it is an optimization problem, we already have the solution, it is the "mixture of experts", which is better at optimizing then having an "lawyer LLM", "coder LLM", etc... because it is better adapted to the internal structure of the neural network behind it.

[delayed]

It actually goes the opposite way. Before we had translation models, summarization models, computer vision -- now one big llm does it all and better