News for hackers. Stuff that matters.
Re:Elegant n-gram generation in Python by e271828 on Sunday August 17, 2014 @08:50PM (#8191025)
This is an elegant solution for sequences, but it doesn't work for arbitrary iterables (which need not support slicing). While this generalization might not be needed for language ngrams, the general problem of taking n items at a time from an iterable pops up in various places.

Here's a generator that yields ngrams from an arbitrary iterable:

  from collections import deque
  from itertools import islice
  
  def ngram_generator(iterable, n):
      iterator = iter(iterable)
      d = deque(islice(iterator, n-1), maxlen=n)
      for item in iterator:
          d.append(item)
          yield tuple(d)

There are no comments on this comment yet.