News for hackers. Stuff that matters.

A comment on KiloGrams: Large N-Grams for Malware Classification. Parent comment

Re:KiloGrams: Large N-Grams for Malware Classification by crazygringo on Friday August 02, 2019 @02:05PM (#20595173)
I've done a lot of work with n-grams in NLP and in my experience it's only useful up to 4-6 words at a time, unless you're trying to index proverbs.

The reason being that grammar is intensely hierarchical, so that mere "linear" processing that n-grams do stops being useful beyond things like compound words or short sayings like "don't mind if I do!"

[ Parent | on HN ]
Re:KiloGrams: Large N-Grams for Malware Classification by EdwardRaff on Friday August 02, 2019 @02:56PM (#20595596)
Oh, I totally agree with everything you've said! Some collaborators I'm chatting with have more niche NLP applications where larger n would be valuable though. I don't want to go into too much detail yet since it's their project idea, and not sure on their comfort level on blasting it out to the public yet :)
[ Parent | on HN ]