IshanMi

I'm working on a long-term project to better understand Operating Systems, video game development, and Rust by building the simplest possible OS in Rust that boots directly into a game of Doom, which will also be re-written in Rust.

I'm giving myself 18 months- it's been super fun so far!

The author mentions that his father was in touch with a PE-backed chemical company a month before he passed, who re-engaged a month after- sounds like that might have been his succession plan?

Personally, I found looking at open source work to be much more instructive in learning about AI and how things like training data and such are done from the ground up. I suspect this is because training data is one of the bigger moats an AI company can have, as well as all the class action lawsuits surrounding training data.

One of the best open source datasets that are freely available is The Pile by EleutherAI [1]. It's a few years old now (~2020), but they did some really diligent work in putting together the dataset and documenting it. A more recent and even larger dataset would be the Falcon-RefinedWeb dataset [2].

[1]: https://arxiv.org/abs/2101.00027 [2]: https://arxiv.org/abs/2306.01116

You're right about Calhoun- I was actually referring to Dennis Muilenburg, the CEO who was fired in 2019 after the two 737 MAX crashes and the subsequent groundings. My comment wasn't clear enough- I'll edit it for clarity.

Personally, I think the issue isn't one of degrees or qualifications, but rather one of values.

The CEO prior to this one (Muilenburg) also had degrees in Aerospace Engineering, but chose to value profit maximization over things like security and a good engineering culture. Innocent people had to pay the price for it.

I remember someone telling me that they thought of luck as being as one of four kinds:

- blind luck (which is what people usually mean by "luck")

- stirring the pot (if you apply to 100 jobs you might get more "luck" than if you applied for just 10)

- domain expertise (the more knowledge you have, the more opportunities you can spot that others might miss)

- reputation (if you're the world's best painter then when people want a painting they'll immediately think of you)

It's also interesting to think that IBM released an 8-trillion parameter model back in the 1980s [0]. Granted it was an n-gram model so it's not exactly an apples-to-apples comparison with today's models, but still, quite crazy to think about.

[0]: https://aclanthology.org/J92-4003.pdf

Thank you for sharing- upon reflection I should have been looking for primary source evidence like this instead of reading too deeply into Wikipedia pages, as someone else pointed out. I will definitely be more careful in the future.

Could I ask how you would normally go about trying to find memos like this for other companies?

I think you're right and I clearly read too much into the Wikipedia pages, instead of looking for more primary source documents like those others have posted. I will be more careful in the future. Thank you for pointing that out!

I found some discrepancies when reading deeper into Mistral that felt a bit peculiar.

If you look at the Wikipedia page for Mistral AI [1] it lists the company as having 3 founders, one from DeepMind & 2 from Meta. This is consistent with what I thought- that it's an AI lab founded by engineers.

But if you go to Cedric O's Wikipedia page [2], it has Cedric O, a politician and former Secretary of State for the Digital economy, listed as a co-founder. The source cited is [3]. Furthermore, if you look at this post about Mistral's lobbying for the AI Act [4], Cedric O is said to be just an investor.

These discrepancies seemed a bit weird and suspicious to me; it felt like they might have been trying to downplay how close their ties were with the French government?

[1]: https://en.wikipedia.org/wiki/Mistral_AI [2]: https://en.wikipedia.org/wiki/Cédric_O [3]: https://sifted.eu/articles/brunch-with-cedric-o [4]: https://www.euractiv.com/section/artificial-intelligence/new...

I still remember discovering & using numba for the first time in university. At the time I didn't know any programming languages except for Python, and wasn't super great at Python either. We had to write and run molecular simulations for our numerical methods and simulations class, and people were writing their code in C/C++ and were blazing through it. I remember finding out about Numba through the High Performance Python O'Reilly book, and by just adding a single @jit decorator, my simulations were dramatically faster.

Amazing stuff, honestly felt like black magic at the time.

This paper analyzed ~14,000 users and I'm wondering if that's a big enough sample size considering there are 350M or so users on Twitter? Is there a statistics test you could use to tell?

Or is the number we should compare 14,000 to not the total number of users, but the subset of people who actually migrated, i.e. left Twitter entirely and joined another social network?

Are there any estimates for how many people in total that might be?

For companies like Twitch or Kick, how much do layoffs like these affect the actual streamers?

(Like are the layoffs usually accompanied by them reducing the streamer's revenue share, or things like that?)

Try the "historical papers" on this repo: https://github.com/aimerou/awesome-ai-papers And also you can find papers with their implementations in code here: http://paperswithcode.com

In case you're unsure which papers would be good to implement, here's a nice GitHub repo: https://github.com/aimerou/awesome-ai-papers

Try out the "historical papers"! :)

Focusing on Deep Learning specifically: - Most LLMs currently use the transformer architecture. You can learn about this visually (https://bbycroft.net/llm), or through this blog post (https://jalammar.github.io/illustrated-transformer/), or through any number of Andrej Karpathy's blog posts and materials. - To stay on top of papers that get published every week, I read a summary every Sunday: https://github.com/dair-ai/ML-Papers-of-the-Week - To learn more about the engineering side of it, you can join Discord servers such as EleutherAI's, or follow GitHub discussions of projects like llama.cpp

Personally I think the best way to develop per unit time is probably to try to re-implement some of the big papers in the field. There's a clear goal, there are clear signs of success, there are many implementations out there for you to check your work against and compare and learn from.

Good luck!

So if I'm understanding this correctly:

The SAM paper from this past April (that let you do zero-shot segmentation on any image, seemingly better than even OpenAI's CLIP) was using a ~600M parameter ViT model to generate image embeddings. And in order to make it less computationally expensive to generate those same embeddings, they replace that model with a smaller ViT encoder that was pre-trained using the masked auto-encoder back propagation method?

A lot of people are taught Excel early on in school. I wonder if this would result in more people learning or being introduced to Python at an early age?

This is what works for me so far:

- This repo has a list of important papers at the end of every week: https://github.com/dair-ai/ML-Papers-of-the-Week

- The best papers are usually discussed pretty quickly on relevant Discord servers I'm in (HuggingFace, EleutherAI, FastAI, etc).

- I follow discussions on a few GitHub repos and learn some implementation details that way. (llama.cpp, oobabooga, etc)

- Finally- there are quite a few YouTube channels that are good supplements (there was a good HN thread about this a while ago).

I'd also like to offer some encouragement- I remember feeling daunted at the beginning, thinking of how fast things were moving in AI, and how slow I'd be in comparison. But once I started reading papers, it became a lot easier to make connections and identify which papers were actually novel and exciting versus which ones were just variations of things that had been tried and tested before.

I promise that it gets easier and more enjoyable as soon as you get past the initial psychological barrier at the beginning- AI is really fun to use!

I think the 8 trillion parameters is accurate- Tangora is an N-gram model with a vocab size of 20,000 words and N = 3.

Parameters for an N-gram model = V^(N-1) * (V-1) Plugging in V=20,000 words and N = 3 for Tangora, you'd get 7.9996E12.

Most of the parameters are likely zero or close to it because many 3-grams are possible but not likely to occur. (However the aggregate probability of all 3-grams is substantial and thus they have to be included.)