News for hackers. Stuff that matters.
Re:Show HN: Word2vec Algorithm in ~100sloc with NumPy by eesmith on Thursday June 01, 2023 @08:01AM (#36149981)
I have a suggestion for improving clarity:

    word_bag: dict[str, int] = dict()  # Multiset

    for line in raw_corpus:
        words = line.split()
        
        for word in words:
            if word in word_bag:
                word_bag[word] += 1
            else:
                word_bag[word] = 1
Try:

    word_bag = collections.Counter()
    for line in raw_corpus:
        word_bag.update(line.split())

then, to filter:

    word_bag = {word: count for (word, count) in word_bag.items()
         if count >= min_frequency}
or, if you can avoid printing the filtered word bag:

    return set(word for (word, count) in word_bag.items()
        if count >= min_frequency)
Also, that:

    raw_corpus    : list[list[str]],
should be list[str]. It's your corpus which is a list[list[str]]. (And you might want to use "line_words" instead of "line"

Could you explain why lines are processed independently?

    for i, line in enumerate(corpus):
        for j, target_word in enumerate(line):
Does it make a difference that some lines have fewer words than others?

I would think that

  for positive_words in sliding_window(flatten(corpus), window_size):
using the sliding_window and flatting recipes at https://docs.python.org/3/library/itertools.html#itertools-r... would be more appropriate. Though using it directly would not handle the leading/trailing partial windows.

I liked seeing the 'η' - I haven't yet used non-ASCII in my code!

Re:Show HN: Word2vec Algorithm in ~100sloc with NumPy by extasia on Thursday June 01, 2023 @11:35AM (#36152650)
Hi eesmith, thanks for your insight! I've applied a lot of the improvements you mentioned, the collections.Counter one didn't even cross my mind- very neat:)

The lines are processed independently as they are separate sentences. One line may be from one source, and the next from a completely different source. The problem with simply concatenating them is that the target words on the boundaries between sentences can end up with context words from unrelated sentences.

>I liked seeing the 'η' - I haven't yet used non-ASCII in my code!

Then you may enjoy this april fool's python issue I made https://github.com/python/cpython/issues/103172

Yeah, I like using utf-8 chars for mathematical symbols, it makes my brain hurt a little less when mapping between the two! I also like using ŷ for predictions in ML contexts as that's the canonical symbol used in the literature.

Re:Show HN: Word2vec Algorithm in ~100sloc with NumPy by eesmith on Thursday June 01, 2023 @11:47AM (#36152819)
> The lines are processed independently as they are separate sentences.

Ahh! I didn't understand that about the corpus. Thanks!