News for hackers. Stuff that matters.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by Erlich_Bachman on Monday January 18, 2021 @04:40AM (#25820000)
It's a shame that it has turned out to be necessary to externally re-make and re-train a model that has come out of company called `OPEN`AI. Wasn't one of the founding principles of it that all of the research would be available to the public? Isn't that the premise on which the initial funding was secured? Best of luck to Eleuther.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by wraptile on Monday January 18, 2021 @04:52AM (#25820080)
OpenAI turning out to be a total bait and switch. Especially true when your co-founder is actively calling you out on it[1]

Remember kids: if it's not a non-profit organization it is a _for_ profit one! It was silly to expect anything else:

> In 2019, OpenAI transitioned from non-profit to for-profit. The company distributed equity to its employees and partnered with Microsoft Corporation, who announced an investment package of US$1 billion into the company. OpenAI then announced its intention to commercially license its technologies, with Microsoft as its preferred partner [2]

1 - https://edition.cnn.com/2020/09/27/tech/elon-musk-tesla-bill...

2 - https://en.wikipedia.org/wiki/OpenAI

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by dvfjsdhgfv on Monday January 18, 2021 @06:21AM (#25820600)
It will be interesting to see the attitude of Microsoft towards this project in the light of their "Microsoft loves open source" propaganda.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by Closi on Monday January 18, 2021 @06:36AM (#25820709)
I don't know where people got this idea that Microsoft can't participate positively in Open Source, and do that sincerely, without open sourcing absolutely everything.

Of course they can - just because you contribute to open source, and do that because you also benefit from open source projects, doesn't mean you have to do absolutely everything under open source.

Especially considering OpenAI isn't even Microsoft's IP or codebase.

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by dvfjsdhgfv on Monday January 18, 2021 @08:02AM (#25821365)
> I don't know where people got this idea that Microsoft can't participate positively in Open Source, and do that sincerely, without open sourcing absolutely everything.

I'm not claiming that. Of course there is place for closed and open elements of their offerings. Let me clarify.

In the past, Microsoft was very aggressive about open source. When they realized this strategy of FUD brings little result, they changed their attitude 180 and decided to embrace it putting literal hearts everywhere.

Personally, I find it hypocritical. There is no love/hate, just business. They will use whatever strategy works to get their advantage. What I find strange is that people fell for it.

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by Closi on Monday January 18, 2021 @08:17AM (#25821481)
But why on this thread then, about GPT-3? It's not even their own company, IP or source to give away.

But even when Microsoft can't open source it because it's not theirs, we still have people posting in this thread that this is further evidence that Microsoft is hypocritical. It sounds a lot like a form of Confirmation Bias to me where any evidence is used as proof that Microsoft is 'anti-open-source'.

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by taf2 on Monday January 18, 2021 @08:52AM (#25821738)
I think it is because each model from OpenAi was public until Microsoft became an investor.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by taf2 on Monday January 18, 2021 @08:50AM (#25821724)
How about when Steve Ballmer said something along the lines of

“Linux is a cancer that attaches itself in an intellectual property sense to everything it touches”

Pretty sure that is hostile towards open source? Linux being one of the flagship projects of open source.

[edit] source https://www.zdnet.com/article/ex-windows-chief-heres-why-mic...

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by Zee2 on Tuesday January 19, 2021 @03:13AM (#25831076)
Disclaimer: Microsoft employee

In my experience, I work at a completely different company than the one that Ballmer ran. Nearly everyone I talk to speaks of the "Ballmer era" in a negative light, and confirms that Satya literally turned the entire company on its head.

Many things happen every day that would never have happened under Ballmer.

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by Closi on Monday January 18, 2021 @09:05AM (#25821879)
It’s hostile to the GPL licence which means anything licensed under GPL can’t be used in Microsoft’s proprietary products.

I would personally say Microsoft wasn’t necessarily driven by anti open-source hate necessarily, they were just very anti-competitor. Microsoft tried to compete with their biggest competitor? Colour me shocked.

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by Daho0n on Monday January 18, 2021 @06:53AM (#25820822)
I don't think this should be seen in the light of "open source everything" but more that many see Microsoft doing open source not as part of "being good" but part of their age old "embrace extend extinguish" policy.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by pessimizer on Monday January 18, 2021 @07:32AM (#25821121)
I don't know where people got the idea that companies can be "sincere." Sincerity is faithfully communicating your mental state to others. A company's mental state can change on a dime based on the decisionmaking of people who rely on the company for the degree of profit it generates. Any analog to sincerity that you think you see can probably be eliminated by firing one person after an exceptionally bad quarter (or an exceptionally good one.)
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by Closi on Monday January 18, 2021 @08:25AM (#25821529)
Sincere to me just means that you are being truthful, or not trying to be deceptive.

And I think companies can be sincere - because companies are really just groups of people and assets when you get down to the nuts and bolts of it.

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by eeZah7Ux on Monday January 18, 2021 @09:58AM (#25822376)
> companies can be sincere

"sincere", "honest", "hypocritical" usually refers to a long-term pattern. Being able to be sincere from time to time is besides the point.

> companies are really just groups of people

...with profit as their first priority.

For-profit companies "can be sincere" only as long as it's the most profitable strategy.

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by eeZah7Ux on Monday January 18, 2021 @09:18AM (#25821983)
Like many other companies, Microsoft loves unpaid labor.

Free Software is about giving freedom and security all the way to the end users - rather than SaaS providers.

If you remove this goal and only focus on open source as a development methodology you end up with something very similar to volunteering for free for some large corporation.

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by person_of_color on Monday January 18, 2021 @05:19AM (#25820238)
So OpenAI employees get Microsoft RSUs?
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by unixhero on Monday January 18, 2021 @05:30AM (#25820302)
What is an RSU?
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by agravier on Monday January 18, 2021 @05:38AM (#25820343)
It means restricted stock unit, and it's a kind of company stock unit that may be distributed to some "valued" employees. There is usually a vesting schedule, and you can't do whatever you want with it.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by ourcat on Monday January 18, 2021 @05:36AM (#25820338)
Restricted Stock Units
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by garmaine on Monday January 18, 2021 @06:25AM (#25820632)
Why would they? It's a separate company.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by mhuffman on Monday January 18, 2021 @05:16AM (#25820215)
But I was told GPT-3 was too powerful for mere mortal hands (unless you have an account!) and that it would be used for hate speech and to bring about skynet.

How will this project avoid those terrible outcomes?

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by ForHackernews on Monday January 18, 2021 @05:48AM (#25820399)
New research has revealed that intelligence is not a prerequisite for generating hate speech on social media platforms.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by visarga on Monday January 18, 2021 @06:16AM (#25820560)
By putting the cat back in the bag. Oh, it's too late ... useless to think about it - we can't de-invent an invention or stop people from replicating. Its like that time when NSA wanted to restrict crypto.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by dvfjsdhgfv on Monday January 18, 2021 @06:24AM (#25820617)
I don't know a single intelligent person who believed this argument, it simply doesn't hold up.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by JacobiX on Monday January 18, 2021 @04:53AM (#25820088)
It's a recurring theme in OpenAI research, they become more and more closed. For instance their latest model called DALL·E hit the headlines before the release of the paper. Needless to say, the model is not available and no online demo has been published so far.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by cbozeman on Monday January 18, 2021 @12:07PM (#25823841)
Because its winner-take-all in this research, not "winner-take-some".

Andrew Yang talked about this and why breaking up Big Tech won't work. No one wants to use the second best search engine. The second best search engine is Bing and I almost never go there.

Tech isn't like automobiles, where you might prefer a Honda over a Toyota, but ultimately they're interchangeable. A Camry isn't dramatically different and doesn't perform dramatically better than an Accord. Whoever builds the best AI "wins" and wins totally.

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by visarga on Monday January 18, 2021 @06:19AM (#25820584)
But they still released the CLIP model which is the complement of DALL-E and used in the DALL-E pipeline as a final filter. There are collabs with CLIP floating around and even a web demo.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by JacobiX on Monday January 18, 2021 @07:13AM (#25820957)
Thank you for this info, as you mentioned CLIP is used for re-ranking DALL-E outputs, by itself it is just an (image, text) pairs classification network.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by dave_sullivan on Monday January 18, 2021 @05:30AM (#25820305)
Really. OpenAI assembled some of the best minds from the deep learning community. The problem isn't that they are a for-profit SaaS, the problem is they lied.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by thelastwave on Monday January 18, 2021 @11:16AM (#25823202)
And ended up making an AI service that's really good at... lying.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by spiderfarmer on Monday January 18, 2021 @04:44AM (#25820026)
I don't really care if OpenAI offers commercial licenses as long as the underlying research is truly open. This way alternative options will become available eventually.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by querez on Monday January 18, 2021 @05:08AM (#25820178)
Arguably openAI is one of the most closed industry AI labs (among those that are still participating in the research community), on par only with deep mind (though deepmind at least publishes way more). Funnily enough, FAIR and Google Brain have a vastly better track record wrt. publishing not only papers but also code and models.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by visarga on Monday January 18, 2021 @06:09AM (#25820506)
It was probably bait and switch to hire top researchers and get initial funding. Now that OpenAI is a household name, they don't have to pretend anymore.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by b3kart on Monday January 18, 2021 @06:16AM (#25820559)
I buy the former, researchers might be happier knowing their work potentially benefits all of humanity, not just a bunch of investors. But wouldn’t it be more difficult to get funding as a non-profit?
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by littlestymaar on Monday January 18, 2021 @07:06AM (#25820902)
It's just never going to be difficult to get funding when you have Elon Musk and Sam Altman as founders (and even more so when founders put one billion of their own money into it).
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by b3kart on Monday January 18, 2021 @07:25AM (#25821064)
Sure, but that's OpenAI's particular set of circumstances. Generally speaking I struggle to see investors preferring a nebulous non-profit over a for-profit with a clear path to market.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by littlestymaar on Monday January 18, 2021 @07:31AM (#25821113)
Sure, but we're explicitly talking about OpenAI here.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by b3kart on Monday January 18, 2021 @07:42AM (#25821207)
Of course. It's just that the comment I've been responding to suggested OpenAI going the "open"/non-profit route was to 1) get top researchers and 2) get investment. I was arguing that this doesn't seem to (generally) be a good way to get investment, but I agree with you in that in their case investment just wasn't a consideration at all.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by Tenoke on Monday January 18, 2021 @12:18PM (#25823995)
The research is open to the public. Here's the gpt3 paper https://arxiv.org/abs/2005.14165

Also gpt2 models and code at least were publicly released and so has a lot of their work.

And yes, they realized they can achieve more by turning for profit and partnering with Microsoft. So true, they are not fully 'open' but pretending they don't release things to the public and making the constant 'more like closedai aimirite' comments is getting old.

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by Sambdala on Monday January 18, 2021 @06:17AM (#25820571)
Wild-Ass Guess (Ass-Guess) incoming:

OpenAI was built to influence the eventual value chain of AI in directions that would give the funding parties more confidence that their AI bets would pay off.

This value chain basically being one revolving around AI as substituting predictions and human judgement in a business process, much like cloud can be (oversimply) modeled as moving Capex to Opex in IT procurement.

They saw that, like any primarily B2B sector, the value chain was necessarily going to be vertically stratified. The output of the AI value chain is as an input to another value chain, it's not a standalone consumer-facing proposition.

The point of OpenAI is to invest/incubate a Microsoft or Intel, not a Compaq or Sun.

They wanted to spend a comparatively small amount of money to get a feel for a likely vision of the long-term AI value chain, and weaponize selective openness to: 1) establish moats, 2) Encourage commodification of complementary layers which add value to, or create an ecosystem around, 'their' layer(s), and 3) Get insider insight into who their true substitutes are by subsidizing companies to use their APIs

As AI is a technology that largely provides benefit by modifying business processes, rather than by improving existing technology behind the scenes, your blue ocean strategy will largely involve replacing substitutes instead of displacing direct competitors, so points 2 and 3 are most important when deciding where to funnel the largest slice of the funding pie.

_Side Note: Becoming an Apple (end-to-end vertical integration) is much harder to predict ahead of time, relies on the 'taste' and curation of key individuals giving them much of the economic leverage, and is more likely to derail along the way._

They went non-profit to for-profit after they confirmed the hypothesis that they can create generalizeable base models that others can add business logic and constraints to and generate "magic" without having to share the underlying model.

In turn, a future AI SaaS provider can specialize in tuning the "base+1" model, then selling that value-add service to the companies who are actually incorporating AI into their business processes.

It turned out, a key advantage at the base layer is just brute force and money, and further outcomes have shown there doesn't seem to be an inherent ceiling to this; you can just spend more money to get a model which is unilaterally better than the last one.

There is likely so much more pricing power here than cloud.

In cloud, your substitute (for the category) is buying and managing commodity hardware. This introduces a large-ish baseline cost, but then can give you more favorable unit costs if your compute load is somewhat predictable in the long term.

More importantly, projects like OpenStack and Kubernetes have been desperately doing everything to commodotize the base layer of cloud, largely to minimize switching costs and/or move the competition over profits up to a higher layer. You also have category buyers like Facebook, BackBlaze, and Netflix investing heavily into areas aimed at minimizing the economic power of cloud as a category, so they have leverage to protect their own margins.

It's possible the key "layer battle" will be between the hardware (Nvidia/TPUs) and base model (OpenAI) layers.

It's very likely hardware will win this for as long as they're the bottleneck. If value creation is a direct function of how much hardware is being utilized for how long, and the value creation is linear-ish as the amount of total hardware scales, the hardware layer just needs to let a bidding war happen, and they'll be capturing much of the economic profit for as long as that continues to be the case.

However, the hardware appears (I'm no expert though) to be something that is easier to design and manufacture, it's mostly a capacity problem at this point, so over time this likely gets commoditized (still highly profitable, but with less pricing power) to a level where the economic leverage goes to the Base model layer, and then the base layer becomes the oligopsony buyer, and the high fixed investment the hardware layer made then becomes a problem.

The 'Base+1' layer will have a large boom of startups and incumbent entrants, and much of the attention and excitement in the press will be equal parts gushing and mining schaudenfreude about that layer, but they'll be wholly dependent on their access to base models, who will slowly (and deliberately) look more and more boring apart from the occasional handwringing over their monopoly power over our economy and society.

There will be exceptions to this who are able to leverage proprietary data and who are large enough to build their own base models in-house based on that data, and those are likely to be valuable for their internal AI services preventing an 'OpenAI' from having as much leverage over them and being much better matched to their process needs, but they will not be as generalized as the models coming from the arms race of companies who see that as their primary competitive advantage. Facebook and Twitter are two obvious ones in this category, and they will primarily consume their own models, rather than expose them as model-as-a-service directly.

The biggest question to me is whether there's a feedback loop here which leads to one clear winning base layer company (probably the world's most well-funded startup to date due to the inherent upfront costs and potential long-term income), or if multiple large, incumbent tech companies see this as an existential enough question that they more or less keep pace with each other, and we have a long-term stable oligopoly of mostly interchangeable base layers, like we do in cloud at the moment.

Things get more complex when you look to other large investment efforts such as in China, but this feels like a plausible scenario for the SV-focused upcoming AI wars.

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by visarga on Monday January 18, 2021 @06:32AM (#25820674)
Apparently you don't need to be a large company to train GPT-3. EleutherAI is using free GPU from CoreWeave, the largest North American GPU miner, who agreed to this deal to get the final model open sourced and have their name on it. They are also looking at offering it as an API.
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by Sambdala on Monday January 18, 2021 @06:52AM (#25820809)
I think it's great they're doing this, but GPT-3 is the bellwether not the end state.

Open models will function a lot like Open Source does today, where there are hobby projects, charitable projects, and companies making bad strategic decisions (Sun open sourcing Java), but the bulk of Open AI (open research and models, not the company) will be funded and released strategically by large companies trying to maintain market power.

I'm thinking of models that will take $100 million to $1 billion to create, or even more.

We spend billions on chip fabs because we can project out long term profitability of a huge upfront investment that gives you ongoing high-margin capacity. The current (admittedly early and noisy) data we have about AI models looks very similar IMO.

The other parallel is that the initial computing revolution allowed a large scale shift of business activities from requiring teams of people doing manual activities, coordinated by a supervisor towards having those functions live inside a spreadsheet, word processor, or email.

This replaces a team of people with (outdated) specializations with fewer people accomplishing the same admin/clerical work by letting the computer do what it's good at doing.

I think a similar shift will happen with AI (and other technologies) where work done by humans in cost centers is retooled to allow fewer people to do a better job at less cost. Think compliance, customer support, business intelligence, HR, etc.

If that ends up being the case, donating a few million dollars worth of GPU time doesn't change the larger trends, and likely ends up being useful cover as to why we shouldn't be worried about what the large companies are up to in AI because we have access to crowdsourced and donated models.

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by ccostes on Monday January 18, 2021 @10:23AM (#25822609)
I think calling this a "wild-ass guess" undersells it a bit (either that or we have very different definitions of a WAG).Very well though-through and compelling case.

My biggest question is whether composable models are indeed the general case, which you say they confirmed as evidenced by the shift away from non-profit. It's certainly true for some domains, but I wonder if it's universal enough to enable the ecosystem you describe.

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by jariel on Monday January 18, 2021 @11:26AM (#25823318)
This is neat, but almost no startups of any kind, even mid size corps, have such complicated and intricate plans.

More likely: OpenAI was a legit premise, they started to run out of money, MS wanted to license and it wasn't going to work otherwise, so they just took the temperature with their initial sponsors and staff and went commercial.

And that's it.