News for hackers. Stuff that matters.
Re:Meaning Machine – Visualize how LLMs break down and simul... by jdspiral on Tuesday April 22, 2025 @06:55PM (#43767059)
I built a tool called Meaning Machine to let you see how language models "read" your words.

It walks through the core stages — tokenization, POS tagging, dependency parsing, embeddings — and visualizes how meaning gets fragmented and simulated along the way.

Built with Streamlit, spaCy, BERT, and Plotly. It’s fast, interactive, and aimed at anyone curious about how LLMs turn your sentence into structured data.

Would love thoughts and feedback from the HN crowd — especially devs, linguists, or anyone working with or thinking about NLP systems.

GitHub: https://github.com/jdspiral/meaning-machine Live Demo: https://meaning-machine.streamlit.app

Re:Meaning Machine – Visualize how LLMs break down and simul... by macleginn on Wednesday April 23, 2025 @04:45AM (#43769915)
The presentation is nice! The main point, however, is a bit misleading. From the title, one would assume that we will see something about how LMs do all these things implicitly (as was famously shown for syntax in this paper: https://arxiv.org/pdf/2005.04511, for example), but instead the input is simply given to a bunch of pretrained task-specific models, which may not have much in common and definitely do not have very much in common with what today's LLMs are doing under the hood.
Re:Meaning Machine – Visualize how LLMs break down and simul... by toxik on Wednesday April 23, 2025 @04:52AM (#43769954)
You shouldn’t link directly to the pdf, here is the abs page

https://arxiv.org/abs/2005.04511

Re:Meaning Machine – Visualize how LLMs break down and simul... by selfhoster11 on Thursday April 24, 2025 @06:15AM (#43780978)
I''m getting an error message with Streamlit: You do not have access to this app or it does not exist
Re:Meaning Machine – Visualize how LLMs break down and simul... by jdspiral on Thursday April 24, 2025 @07:45AM (#43781570)
I moved the app, it’s now tokenizer-machine.streamlit.app.