reliableturing

Here’s an actual hot take compared to the sentiment in this discussion: this will be the best selling iPhone ever.

Specs wise sure, I’d also love a bigger battery than it being thin*. But the iPhone has been an unbelievable fashion statement, and this insanely sexy iPhone will be the strongest yet.

I’m pretty sure when it comes out, people will actually hold it in their hands and the sentiment will turn. Not talking to you tech nerds, but for the other 99% of the world.

Location: Paris, France

Remote: hybrid ok

Willing to relocate: yes

Technologies: Python (advanced), C# (advanced), Java, PyTorch, SLURM, Docker, Git, Vim

CV: Contact me

Email: hn.brittle810 at passmail dot net

Senior ML Research Engineer (PhD) with 6+ years’ experience building scalable systems and turning research into production. Strong product mindset with scientific rigor, moving easily from high-level product ideas to technical detail.

Location: Paris, France

Remote: Yes/Hybrid

Willing to relocate: No

Technologies: Python, PyTorch, SLURM, C#, CosmosDB, Cursor, Git, Octopus, TeamCity, SQL, Java, HuggingFace, Azure, ArgoCD

Résumé/CV: arthurhemmer.com

Email: arthurhemmer monkeytail proton dot me

Based in Paris and about to finish my (industrial) PhD in neurosymbolic document understanding. Looking for a position such as Research Scientist/Engineer in AI in general, though most of my experience is in NLP and OCR. Have a bit of professional experience as I started my PhD a few years after having worked as a Data Scientist.

I’m not sure what this paper is supposed to prove and find it rather trivial.

> All of the LLMs knowledge comes from data. Therefore,… a larger more complete dataset is a solution for hallucination.

Not being able to include everything in the training data is the whole point of intelligence. This also holds for humans. If sufficiently intelligent it should be able to infer new knowledge, refuting the very first assumption at the core of the work.

I know, really looking forward to watching 27h of video in a day

I agree but I already planned to upgrade to a 16 Pro up from my 12. Biggest thing I’m looking forward to? Battery life.

In the announcement they said “big boost”, and looking at the comparison page I will go from 17h video playback to 27h. That’s not even including the battery degradation I built up in the last 4 years. I’m practically going to double my battery life.

They probably meant vLLM https://docs.vllm.ai/en/latest/

Interesting question indeed! There isn't much of a consensus on this as you can see from the other comments. Nonetheless, I spend much time thinking about this so I'd like to take a jab at it as well.

I think it partially has to do with the concepts of modality and grounding. A modality is a channel through which information can be conveyed. You probably learned early on about the 5 human senses: vision, hearing, taste, smell and touch. The grounding problem refers to the fact that any symbols (read: language) we use, usually refers to perceptions in one or more of these modalities. When I write "shit", you can probably imagine a representation of it in different modalities (don't imagine them all!).

Interestingly, large language models (such as ChatGPT) don't have any of these modalities. Instead, they work directly on the symbols we use to communicate meaning. It's quite surprising that it works so well. An analogy that helps understand this is that asking an LLM anything is much like asking a blind person what the sun looks like. Obviously they cannot express themselves in terms of vision, but they could say that it feels warm and maybe even light because it doesn't make any noise. It would be a good approximation and they would be referring to the same physical phenomenon, but that's all it is, an approximation. They could say its a large yellow/white-ish circle if they heard this from someone else before, but since the blind person cannot see, they have no 'grounded truth' to speak from. If the sun would suddenly turn red, they would probably repeat the same answer. My point being: you can express one modality in another, but it'll always be an approximation.

What's interesting is that the only 'modality' of these LLMs is language, which is the first of its kind so we don't know what to expect from this. In a sense, LLMs are simply experiments to the question "what would a person that could only 'perceive' text look like?". Turns out, they're a little awkward. Obviously there's much more to the story (reasoning, planning, agency, etc.) but I think its fundamental to your question why reading for humans and AIs (LLMs) is not the same: LLMs have such a limited and awkward modality that any understanding can only be an approximation of ours (albeit a pretty good one), hence learning from reading will be much different as well.

Hope this helps your understanding.

This is exactly part of this Google USM approach, although these pretrained models are significantly smaller than ChatGPT. They reference this paper [1] which contains more details on the pretrained text-only alignment with the speech model.

[1] https://arxiv.org/abs/2209.15329

As someone who tends to side with Chomsky in these debates, I think Norvig makes some interesting points. However, I would like to pick one of his criticisms to disagree on:

"In 1969 he [Chomsky] famously wrote:

     But it must be recognized that the notion of "probability of a sentence" is an entirely useless one, under any known interpretation of this term.

His main argument being that, under any interpretation known to him, the probability of a novel sentence must be zero, and since novel sentences are in fact generated all the time, there is a contradiction. The resolution of this contradiction is of course that it is not necessary to assign a probability of zero to a novel sentence; in fact, with current probabilistic models it is well-known how to assign a non-zero probability to novel occurrences, so this criticism is invalid, but was very influential for decades."

I think Norvig wrongly interprets Chomsky's "probability of a sentence is useless" as "the probability of a novel sentence must be zero". I agree that we've shown that it's possible to assign probabilities to sentences in certain contexts, but that doesn't mean that it can fully desribe a language and knowledge. This seems to me yet another case of 'the truth is somewhere in the middle' and would be weary of the false dichotomy that is put forward here. Yes we can assign probabilities to sentences and they can be useful, but it's not the whole story either.

Thanks for the write up! As someone starting a doctoral degree early next year, I greatly appreciate it