The Bluesky Dictionary

Re:The Bluesky Dictionary by wantlotsofcurry on Wednesday August 06, 2025 @06:03PM (#44818386)

I'm very curious as to how this works in the backend. I realize it uses Bluesky's firehose to get the posts, but I'm more curious on how it's checking whether a post contains any of the available words. Any guesses?

[ Parent | on HN ]

Re:The Bluesky Dictionary by avibagla1 on Wednesday August 06, 2025 @06:39PM (#44818679)

Hey! this is my site - it's not all that complex, i'm just using a sqlite db with two tables - one for stats, the other for all the words that's just word | count | first use | last use | post.

I... did not expect this to be so popular

[ Parent | on HN ]

Re:The Bluesky Dictionary by gumboshoes on Wednesday August 06, 2025 @09:42PM (#44819744)

What is your source dictionary to compare to? Seems kind of small. Also, how are you handling inflected forms?

[ Parent | on HN ]

Re:The Bluesky Dictionary by avibagla1 on Wednesday August 06, 2025 @11:33PM (#44820311)

https://github.com/words/an-array-of-english-words

using this, a combo of "covered enough" for the bit and easy to use

also, since i'm tracking every word (technically a better name for this project would be The Bluesky Corpus) all inflected forms are different words, which aligns with my thinking

[ Parent | on HN ]

Re:The Bluesky Dictionary by blendo on Wednesday August 06, 2025 @10:59PM (#44820127)

What are the table sizes?

And what ingress bandwidth do you have?

[ Parent | on HN ]

Re:The Bluesky Dictionary by avibagla1 on Wednesday August 06, 2025 @11:41PM (#44820359)

DB is currently 58mb (damn lol)

Ingress is actually pretty manageable, ~900kbps

[ Parent | on HN ]

Re:The Bluesky Dictionary by f311a on Wednesday August 06, 2025 @06:15PM (#44818495)

You can probably fit all words under 10-15MB of memory, but memory optimisations are not even needed for 250k words...

Trie data structures are memory-efficient for storing such dictionaries (2-4x better than hashmaps). Although not as fast as hashmaps for retrieving items. You can hash the top 1k of the most common words and check the rest using a trie.

The most CPU-intensive task here is text tokenizing, but there are a ton of optimized options developed by orgs that work on LLMs.

[ Parent | on HN ]

Re:The Bluesky Dictionary by gpm on Wednesday August 06, 2025 @06:09PM (#44818443)

Probably just a big hashtable mapping word -> the number of times it's been seen, and another hashset of all the words it hasn't seen. When a post comes in you hash all the words in it and look them up in the hashtable, increment it, and if the old value was 0 remove it from the hash set.

250k words at a generous 100 bytes per word is only 25MB of memory...

[ Parent | on HN ]

Re:The Bluesky Dictionary by stwrzn on Wednesday August 06, 2025 @06:31PM (#44818621)

I very much hope that the backend uses one of the bluesky jetstream endpoints. When you only subscribe to new posts, it provides a stream of around 20mbit/s last time I checked, while the firehose was ~200mbit/s.

[ Parent | on HN ]

Re:The Bluesky Dictionary by avibagla1 on Wednesday August 06, 2025 @06:39PM (#44818683)

yes it does!

[ Parent | on HN ]

Re:The Bluesky Dictionary by bangaladore on Wednesday August 06, 2025 @06:09PM (#44818438)

Maybe I'm being naive, but with only ~275k words to check against, this doesn't seem like a particularly hard problem. Ingest post, split by words, check each word via some db, hashmap, etc... and update metadata.

[ Parent | on HN ]

Re:The Bluesky Dictionary by somebehemoth on Wednesday August 06, 2025 @10:58PM (#44820113)

I think the cool part is watching words go brrr.

[ Parent | on HN ]