> Dataset Sampling prop. Epochs Disk size
> CommonCrawl 67.0% 1.10 3.3 TB
> C4 15.0% 1.06 783 GB
> Github 4.5% 0.64 328 GB
> Wikipedia 4.5% 2.45 83 GB
> Books 4.5% 2.23 85 GB
> ArXiv 2.5% 1.06 92 GB
> StackExchange 2.0% 1.03 78 GB
At no point do they use information from Facebook customers
Facebook has been weaponized numerous times, why would this be any different? Why would they be given any benefit of the doubt?
But Facebook research really wants to build amazing tech to power new experiences. The company has attracted many smart and amazing developers who want to pioneer innovation and want to see their work thrive in the broader tech community.
I’m sure this will have the same issues as the other Large Language Models and there’s a lot of work to be done to figure out how to distill a reliably useful system from something trained on so much human language which is unreliable.
I’m heartened to see them talk about needing to research how to remove bias, toxicity, disinformation and hallucinations.
This is something we need to focus on for all LLMs so I’m happy to see more focus on that in the community.
What you helped work for was to build a giant spy tool for corporate and government actors. Your own friends, neighbors, and even family surely have some kind of shadow profile on facebook, sleazily tied to other services, packaged, and sold on equally sleazy and opaque data markets.
Personally, I would support the government just shutting down facebook altogether, by force if necessary. They have lost all trust with me, and any benefit to society has easily been drowned by their numerous public failures, up to and including genocide.
Like, once your company is involved in genocide, that should be it. Game over.
CommonCrawl 67.0% C4 15.0% Github 4.5% Wikipedia 4.5% Books 4.5% ArXiv 2.5% StackExchange 2.0%
So in other words, nothing here in this research has used people's social media posts, and there's no private data whatsoever because the sources used were all public.
You've provided a strong datapoint indicating that the training sources in question were ethically sourced* - that's good! I'm glad to hear that Facebook avoided _this specific way of acting unethically_. This does not affect in any way my judgement of the likelihood of Facebook acting in _other_ unethical ways in this area, such as using the collected data in unethical ways.
* I don't, personally, hold the opinion that "just because something was posted publicly means that it is ethically permissible to do _anything_ with that content", since it's unreasonable to believe that the average internet user is giving informed consent to uses like this - but I do concede that that's a more subjective perspective and is probably not supported by the letter of the law. Luckily, this isn't relevant to my argument.
As an aside - I clicked around in a few pages on CommonCrawl's site (https://commoncrawl.org/big-picture/what-we-do/, https://commoncrawl.org/big-picture/frequently-asked-questio..., https://commoncrawl.org/the-data/get-started/) and didn't see any statement to the effect that social media posts were excluded, and a Google search for "Does CommonCrawl index social media" didn't provide a definitive answer, either. You're under no obligation to prove it to me - and, as I detailed above, even if you did, it wouldn't affect my general distrust of Facebook.
What in facebooks past gives anyone reason to believe this model would be used for anything other than nefarious purposes?