sshumaker

There’s both medication (which takes a while to get dosage right but can be extremely helpful for some) and also techniques you can use to help compensate.

Are you serious? It’s multiple nodes to run a frontier model (a node is 8x GPUs), and they aren’t running on H100s. You are looking at 32+ GPUs.

It’s how LLMs work - they are effectively recursive at inference time, after each token is sampled, you feed it back in. You will end up with the same model state (not including noise) as if that had been the original input prompt.

It depends on how large the input prompt (previous context) is. Also, if you can keep cache on GPU with a LRU mechanism, for certain workloads it's very efficient.

You can also design an API optimized for batch workloads (say the same core prompt with different data for instruct-style reasoning) - that can result in large savings in those scenarios.

They are almost certainly doing this internally for their own chat products.

The simple version of this just involves saving off the KV cache in the attention layers, and restore it back instead of recomputing. It only requires small changes to inference and the attention layers.

The main challenge is being able to do this under scale, e.g. dump the weights out of GPU memory, persist them, and have a system to rapidly reload them as needed (or just regenerate).

This is a pretty standard technique if you're running the models yourself. e.g. ChatGPT almost certainly does this.

There's even work that is more sophisticated in this domain that allows 'template' style partial caching: https://arxiv.org/abs/2311.04934

Stealth startup | ML Engineer (edge inferencing) | Bay Area or LA (hybrid) | Full-time

We're a stealth startup building something unbelievably ambitious in the AI space that blends AI and gaming tech - co-founders are Andy Gavin [1] (co-founder of video game developer Naughty Dog) and myself (VP @ Microsoft, Credit Karma, previously Google and Naughty Dog). Venture-backed by top investors including First Round Capital and Battery. Get in on the ground floor and work directly alongside a living legend and a dream team of world-class talent.

We're looking for someone who has deep experience with CoreML and optimizing model inferencing for mobile usage (e.g. ANE on iOS).

If interested, reach out to me at: [email protected]

[1] Andy Gavin on the Making of Crash Bandicoot: https://www.youtube.com/watch?v=pSHj5UKSylk

I had the pleasure of working closely with Jeff Slutter early in my career. Was the first really fantastic engineer I worked with - he’s at Santa Monica Studios (God of War) these days. I think Matt Toschlog is there too now. Glad to see these folks working on an awesome AAA franchise.

Monarchies usually result in long-term thinking? I’m sure we haven’t read the same history books. Most monarchs throughout history have been very self-interested and their decisions have been focused on their own personal interest - the definition of short-term.

I’m not claiming democracy is any better - humans are notoriously bad at long-term thinking unfortunately.

Hamas gained a ton of support for their cause by the tragedy inflicted in the counterattack. I believe they celebrate the deaths of Palestinian innocents as much as they do the Israeli ones so they can extract propaganda wins. This has been part and parcel of their strategy for decades - it’s why they embed their military activity in soft targets like schools and hospitals.

The Israelis know this by now, so the fact that Israel was goaded into a ground war speaks as much to the political situation as anything else, but either way it’s tragic.

It seems like rather than adding a random amount to each sample (which lets them compute a mean by recreating the same audio and extracting out the differences), Safari could instead add randomness that is based on a key that rotates every hour. (Function of audio sample and key, so the noise would be the same in a given session, but useless for tracking an hour later).

Stealth startup | Lead mobile engineer, AI Engineer | Bay Area or LA (hybrid) | Full-time

We're a stealth startup building something unbelievably ambitious in the AI space that blends AI and gaming tech - co-founders are Andy Gavin [1] (co-founder of video game developer Naughty Dog) and myself (VP @ Microsoft, Credit Karma, previously Google and Naughty Dog). Venture-backed by top investors including First Round Capital and Battery. Get in on the ground floor and work directly alongside a living legend and a small team of world-class talent.

Lead Mobile Engineer - Swift and C/C++, CoreML a plus

AI engineer - Finetuning/Retraining LLMs (LLama/Mixtral/etc), MLOps a plus

If interested, reach out to me at: [email protected]

[1] Andy Gavin on the Making of Crash Bandicoot: https://www.youtube.com/watch?v=pSHj5UKSylk

You’re being downvoted because this is a hot take that isn’t supported by evidence.

I just tried exactly that with dalle-3 and it worked well.

More to the point, it’s pretty clear LLMs do form a model of the world, that’s exactly how they reason about things. There was some good experiments on this a while back - check out the Othello experiment.

https://thegradient.pub/othello/

You can also look at Bertopic which has this functionality as an open source library:

https://maartengr.github.io/BERTopic/index.html

Sometimes you can use a heuristic to estimate K, or use a variant that terminates at some distance threshold.

That said, something like hdbscan doesn’t suffer from this problem.

I have an eightsleep, which cools the bed down dramatically (circulates cold water). Makes a huge difference in my ability up sleep.

{Stealth AI Startup} | AI Audio (speech) Engineer | SF or LA hybrid, remote possible

Very early stage startup, venture backed from top VCs, working on one of the hardest and most ambitious problems of our generation in AI. A true "AI-first" startup, not just a thin CRUD app on top of GPT.

Co-founders are Naughty Dog co-founder Andy (https://en.wikipedia.org/wiki/Andy_Gavin) and myself (https://linkedin.com/in/sshumaker). This is the most exciting project of our careers (a tough feat given our backgrounds) - it feels like literal science fiction.

We're looking for hands-on experience (even as a dedicated hobbyist) building + training audio models for text-to-speech, speech-to-text, or other novel AI speech applications, and someone who is deeply innovative and passionate about the space. It's still very early so this would be an opportunity to join on the ground floor - with meaningful equity, plus competitive comp.

Please contact me - Scott Shumaker ([email protected]) - if you're interested and want to learn more.

I talked to Pavel about taking a role in my team earlier this year (I ended up leaving Microsoft myself since). He was passionate about making the developer experience excellent - code quality, clean APIs, etc. Thats a tall order for some parts of Microsoft with 30+-year old codebases. He mentioned he was interested in rockets so I hope he found a gig doing that.

I don’t know about [1]. I asked an example from the paper above to GPT-4: “[If you had to guess] how many thumbs did Lincoln’s maternal grandmother have?”

Response: There is no widely available historical information to suggest that Abraham Lincoln's maternal grandmother had an unusual number of thumbs. It would be reasonable to guess that she had the typical two thumbs, one on each hand, unless stated otherwise.

If you ask it properly it gets it right.

From a pure measurement standpoint, could Jupiter fit in the space between the earth and moon?

The average distance from the Earth to the Moon is about 238,855 miles (384,400 kilometers). Jupiter, the largest planet in our solar system, has a diameter of about 86,881 miles (139,822 kilometers).

So, if you were to somehow place Jupiter in between the Earth and the Moon, it would fit with a significant amount of room to spare. However, it's important to note that this is a purely theoretical situation and not something that could actually happen without cataclysmic consequences due to gravitational forces and other factors.

I’m fairly confident this is untrue. At Microsoft at least, it’s a big deal when there is a privacy issue, even a small localized one on a single product - and creates a small firestorm.

We’ll get engineers working long hours focused on it, consulting closely with our legal and trust teams. One of the first questions we ask legal when we suspect a privacy issue is “Is this a notifiable event?”

It’s not really about getting slapped by regulators - it’s the fact that much of Microsoft’s business is built by earning the trust of large companies and small ones. Many of them are in the EU of course, but we have strict compliance we apply broadly. It’s just not worth damaging our reputation (and hurting our business) for some shortcut somewhere, as trust takes a long time to build and is easily broken.

I had the same issue, but support kept disconnecting me and I had to start all over again going through the entire “try all of the brain dead stuff like resetting” 3 times, before I finally gave up and just ordered a new Nest and ate the cost.

> They will instead pay a lot for MSFTs cloud service offering, which of course comes with the crucial promise that their data is safe and secured and handled in a way that is compliant with all privacy laws. Which of course isn't true, but that doesn't matter, the promise is what matters.

In what way is this not true? Obviously there is no perfection here, only degrees of risk. But this is literally why people pick MSFT over others. They have by far the strongest culture around maintaining trust in the enterprise space.

This is really a lot less necessary with GPT-4. What required careful prompting in 3.5 often you can give it something slapdash in 4 and it can do a great job figuring out intent.

Just use the Azure hosted solution, which has all of Azure's stronger guarantees around compliance. I'm sure it will update with GPT-4 pricing shortly.

https://azure.microsoft.com/en-us/products/cognitive-service...

(disclaimer: I work for Microsoft but not on the Azure team)

Lots of people have way more than that due to retirement accounts (which sometimes need more liquidity or people want to get out of the market so they have more cash on hand). Or even if you have a down payment on a house in most of the coastal states. 250k in the bank is not much money these days.

That’s how PageRank effectively worked, and people created deep networks of pages voting for each other. To solve this problem for real you either need Sybil protection or to make it economically infeasible (impose a tax for making content available).

To some extent I have been wondering for a while if prioritizing Ads in Google Search is Google’s way of creating that economic barrier to spam content (for some meaning of spam) - you can take the fact that a brand is willing to spend money as some indication of “quality”.

In a leadership role. I would love this - particularly to help make me aware of when I should be holding more space for others. But the fact that the meeting data (audio/video) gets transmitted to a third party is a complete deal killer.

Revisionist history? The Jews who converted to avoid expulsion and (some of whom) were practicing their religion in secret would like a word.

https://www.chabad.org/library/article_cdo/aid/2430792/jewis...

It did use one time pads:

Two paper tape readers are mounted in front of the 700 Unit at the top. One is a 5-level tape reader, which is used to read the plaintext (when used offline). The other one is a 6-level reader which is used to read the keystream tape. The signals of both readers are 'mixed' in the 804 unit by means of modulo-2 addition (XOR).

…

For proper security it is important that the keystream tape contains a sequence of evenly spread truely random characters. Producing such a random keystream was a major challenge during WWII.

When the need for keystream tapes increased during the course of war, the manual production was replaced by electromechanical methods. The machine that was used for the production of Rockex key tapes was codenamed DONALD DUCK, possibly because it speaks gibberish [3]. It wasn't before the application of a noise source however, that truely random key streams were produced. In the UK, a noise generator with five flip-flops was developed at GCHQ just after WWII by former GPO-engineer Don Horwood, who had also worked on Colossus at Bletchley Park.

…

Furthermore, he allowed only two identical tapes and instructed them to be both destroyed immediately after use. This way the machine became a real One-Time Pad system.

From https://www.cryptomuseum.com/crypto/uk/rockex/index.htm