So, like movies or software
Where is the torrent with a runnable copy of paypal, or amazon?
Even without that, if you want downloadable and runnable software platforms, look to public Git repositories. Some of the people who have no financial motivation will release what they do alongside installation procedures and quality of life scripts and architecture documentation.
Most of the open source platforms doesn't publish this documentation and doesn't make installation easy to keep a sizeable moat and protect the platform they have developed, hence this is why we have a division between "Free Software" and "Open Source".
In short, "The bulk of really valuable commercial code" is self contained, but not open source, or if open source it's not Free Software and made deployable for other parties. Otherwise it loses monetary value in the eyes of the people who develop that for the monies.
Otherwise we have have Elasticsearch incident, where they pivot and move to "Source Available" model to protect their castle.
You guess, and so perhaps do other things as well, with poor acuity.
The incalculable value of open source software has approximately no bearing on this assertion.
Yes I love linux and bsd too I'm not defacing your religion. I'm actually quite Stallmanesque in making my own life harder by only using ooen source software as much as possible and being super fun a family gatherings talking about it.
You sure nailed that one.
https://news.ycombinator.com/item?id=34525936
Looking down that page, there are some valid magnet links.
(This probably says more about how hard it is to build C++ than anything else)
May not take hours, but determined engineer should be able to figure it out.
Just getting the code to compile, link, baking in assets, etc. For a single architecture is a much more reasonable goal.
I'd imagine it'd be on the scale of 1-3months for an engineer to get working full time, but large error bars around this figure
This was done (if I remember right) when governments and big customers had access to the Windows source-code.
That's probably true, but I wouldn't be surprised if something like windows doesn't have a README file. And it does have build instructions they may well be in some wiki separated from the source code.
Video games build systems are a thing onto themselves even taking the c++ issues out of the equation.
Hell even comparatively simole microservices in modern CRUD apps have resorted to docker to do away with pains of rebuilding stuff.
I was trying to come to grasp with how much resource there is concentrated in one of these models. Somehow I come to the conclusion that it cost more than buying a jet airliner to train one of these models. And it is about the same order of money as commissioning and building a skyscraper in Manhattan. Is that correct approximately?
That's a $30Mil if you want to train at that scale. Also IIRC it took 23 days to train the biggest model. Someone else can do the power consumption cost calculations.
In any case the cost per run is going to be lower than 30m
Edit: todays pricing looks like about 20% higher, still. How are these prices so different.
If an A100 costs $15k and is useful for 3 years, that’s $5k/year, $425/mo. 2048 A100’s cost $870k for a month.
A100 cards consume 250w each, with datacenter overheads we will call it 1000 kilowatts for all 2048 cards. 23 days is 552 hours, or 552,000 kilowatt hours total.
Most dataceneters are between 7 and 10 cents per kilowatt hour for electricity. Some are below 4. At 10 cents, that's $53,000 in electricity costs, which is nothing next to $30 million in capital costs.
You mean in terms of money. I think this is exactly the problem that we have in CS, nobody really cares about CO2.
The manufacturing process, however, is totally decentralized, and NVIDIA mostly manufactures in China where coal is cheap.
https://www.cnbc.com/2022/09/01/nvidia-says-us-government-al...
All consumer SKUs that I know of are manufactured in China. By volume this is certainly the majority of manufacturing.
I believe capex <> opex is more 1:1 nowadays, so something feels off here...
A100 costs $2/h, so it is $2M to train biggest model. Easily kikstart crowdfundable project.
One way you can distill the first three is to use AWS/Azure/GCP costs. But then you are still missing a major factor which is the humans that worked on it, and the human may very well exceed the hardware cost.
At least $10 million/yr just for the talent.
You are right! Wow. Thank you for correcting me.
> GPT-3 probably cost under $5 million,
Is that one training run or includes all the fiddling to find the right hyperparameters? Or there aren't many of those in these training or they are not that sensitive?
See last page of their most recent available audited financials. https://rct.doj.ca.gov/Verification/Web/Download.aspx?saveas...
It’s like having an overfit equation to a sample of data points, instead of the simpler actual line they fall near.
They end up being black boxes, we have almost no idea how they work inside, and we have no idea how overtrained they are when something simpler could do the same thing.
They are wasteful. If LLaMa 13B is as powerful as previous 65B models, that's a significant amount of unnecessary paramaters lost/pruned in just this iterative upgrade alone. How small can they go? The fewest parameters that get the job done 99% as well is the way to go.
There is also the difference between the rules and use of language being directly compressed into the model, vs all the information known to humans compressed into the model. A smaller model that ingests relevant information on the fly (more like Bing, that supplements itself with search), may be less wasteful and perform better.
The current models being released are chosen because "they work" not because they are least fit and most performant optimized.
Sure, the binary probably depends on a lot of internal proprietary infrastructure, but also most of that infrastructure is easy to write a mock implementation of, as long as you are happy for it to be in-ram, not multi-homed and don't need it to scale to billions of users.
Most of the binaries have a standalone mode for running on a developers PC with few/no dependencies anyway.
And anything more complex than that would probably have dependencies on so many fat client libs, so much infrastructure, and so many external services, that you'll need months-years to even make sense of them, let alone mock them up.
The real issues are (again) in dependencies and complex tooling. You can have beautiful code and then in the middle of it, an ML inference call that assumes a crazy ML model and set of hardware to run it on.
Even just decent quality code keeps side effects isolated to the application fringe as much as possible.
Google has no incentives to allow an arbitrary component run standalone. Quite the contrary.
What they do get in return for the coupling is that they can evolve the common libraries and code patterns across the board (there are even automated code refactoring tools that help you do massive code changes, automating code review sessions across hundreds and hundreds of teams, with all changes tested against all reverse dependencies etc etc). All this allows for a level of internal code quality that is hard to see elsewhere.
Unless you really care a lot about that one requirement you seem to care about. In that case, yeah, you'd choose a different tradeoff.
The infrastructure in these companies is a huge amount of scaffolding that's non-trivial to replicate.
I think you have to consider that some things are systems, and it is the assembly of their components that imparts the true quality.
But these ML models are like neutron stars of computation density
The crazy thing about these models is that the compute power going into them is at least somewhat reversible.
How Well Can DeepMind's AI Learn Physics? https://www.youtube.com/watch?v=2Bw5f4vYL98 https://arxiv.org/abs/2002.09405 https://sites.google.com/corp/view/learning-to-simulate/home
Discovering Symbolic Models from Deep Learning (Physics) https://www.youtube.com/watch?v=HKJB0Bjo6tQ
Scientific Machine Learning: Physics-Informed Neural Networks with Craig Gin https://www.youtube.com/watch?v=RTPo6KgpvBA
Steve Brunton's channel is even more mind blowing than Two Minute Papers, https://www.youtube.com/@Eigensteve
Not only can we bank computation, speed up physical simulations by 100x but I also saw some work on being able to design outcomes in GoL (game of life).
There was a paper on using a NN to build or predict arbitrary patters in GoL, but I can't find it right now.
https://btm.qva.mybluehost.me/building-arbitrary-life-patter...
The large datasets involved let us usefully (for some value of useful) bank lots of compute, but it's not obvious to me that it's done particularly efficiently compared to other things you might precompute.
For converged model training, training is often quite inefficient because the weight updates decay to zero and most epochs are having a very small individual effect. I think for e.g. stable diffusion, they dont train to anywhere near convergence so weight updates have a bigger average effect. Not sure if that applies to llms
Fractal Image Compression, https://en.wikipedia.org/wiki/Michael_Barnsley https://www.abebooks.com/servlet/BookDetailsPL?bi=3131987970...
From Wikipedia:
> The ruling of the court was written by Justice Sandra Day O'Connor. It examined the purpose of copyright and explained the standard of copyrightability as based on originality.
> The case centered on two well-established principles in United States copyright law: that facts are not copyrightable, and that compilations of facts can be.
> "There is an undeniable tension between these two propositions", O'Connor wrote in her opinion. "Many compilations consist of nothing but raw data—i.e. wholly factual information not accompanied by any original expression. On what basis may one claim a copyright upon such work? Common sense tells us that 100 uncopyrightable facts do not magically change their status when gathered together in one place. … The key to resolving the tension lies in understanding why facts are not copyrightable: The ″Sine qua non of copyright is originality."
> ...
> The standard for creativity is extremely low. It need not be novel; it need only possess a "spark" or "minimal degree" of creativity to be protected by copyright.
> In regard to collections of facts, O'Connor wrote that copyright can apply only to the creative aspects of collection: the creative choice of what data to include or exclude, the order and style in which the information is presented, etc.—not to the information itself. If Feist were to take the directory and rearrange it, it would destroy the copyright owned in the data. "Notwithstanding a valid copyright, a subsequent compiler remains free to use the facts contained in another's publication to aid in preparing a competing work, so long as the competing work does not feature the same selection and arrangement", she wrote.
> The court held that Rural's directory was nothing more than an alphabetic list of all subscribers to its service, which it was required to compile under law, and that no creative expression was involved. That Rural spent considerable time and money collecting the data was irrelevant to copyright law, and Rural's copyright claim was dismissed.
---
And so, my (I am not a lawyer) take on this is that the numbers of the model are not copyrightable. The selection of the source material is... kind of. This gets into a "a recipe is not copyrightable, yet a recipe book is"
The model may, however, be a trade secret. ( https://en.wikipedia.org/wiki/Trade_secret )
It's not automatic, would require some ML Engineering, but nothing is stopping you if you have the Pytorch graph and weights.
/s
Calculating things takes time and unrelated to output size. There are NP problems that simply outputs true or false yet requires more computational power than the universe can support
The bar to competition is far lower, as already evidenced by the plethora of AI products being put forward. Its a race to the bottom on pricing
There's an ass ton of hardware that might otherwise be idle.
I really like that expression.
This has not been the case for most commercial software for the past 20 years, during the cloud era. If you could steal a dump of random Facebook source code, it would be 99% useless because it’s so closely tied to the infrastructure. There’s almost nothing you could usefully run on your own PC or server VM.
But these ML models are like neutron stars of computation density. You can’t really peek inside to see what’s going on either. An unknown stolen model’s properties would need to be discovered by experimentation.