News for hackers. Stuff that matters.
Re:KiloGrams: Large N-Grams for Malware Classification by woliveirajr on Friday August 02, 2019 @09:07AM (#20592633)
>Larger values of n are not tested due to computational burden or the fear of overfitting

I would say that many times larger n-grams are tested in the early stage, give poor result, and then are dropped in subsequent tests.

Re:KiloGrams: Large N-Grams for Malware Classification by EdwardRaff on Friday August 02, 2019 @11:28AM (#20593732)
Thats true in many cases! For our malware work and research, we've gotten a lot of consistent push-back that n-grams needed to be larger. A lot of older work (with much smaller datasets) said 15-20 grams were best, but that couldn't be tested again with a modern large corpus. A common source of that intuition was that x86 assembly is variable length, and a single instruction could be up to 15 bytes long.

The primary point of our paper is that now we can circle back and really test all these hypothesis about larger n-grams for malware! And even if you only want smaller 6-grams, we get a nice big speedup too :)