> Damn. That's terrible. We are never going to get anywhere with 20 executions a second

...

> Lucky for us there is a trick we can use to greatly improve that speed.

I just dusted off my fuzzpy project [1] this week and was bringing it up to date with recent CPython, LLVM/libFuzzer.

> Notice the speed: 373.4/sec, up from 20! And we've already found 100 unique crashes!

Most of my test cases run at < 10/sec. I feel like I just got lapped while I thought I was on a casual solo stroll. O.o

I intentionally left Py_InitializeEx() in the critical section for fear of contaminating global state, though. But I see that I should've at least experimented with it out.

[1] https://bitbucket.org/ebadf/fuzzpy

EDIT: sorry, I went a little off the deep end -- this post is in response to the article listed in the first paragraph "After finding 1800+ crashes in Python 3.5 using a method shown here ... "

I didn't get the point of the 1800+ crashes. Python isn't designed to be safe from corrupted byte code in the .pyc files.

Yolo! :P

    PyObject *arg = PyTuple_GET_ITEM(varnames, j); 

> ... can be null if varnames is an empty tuple (which it can be if the pyc file is malformed).

This might be a case of "program works as designed" if the dev team thinks that's an acceptable reaction to malformed pyc input. But, hey, it's worth reporting because I would probably be inclined to fix it if I were the BDFL.

I hope we can get this to run much faster than a few executions per second. Having a way to speed this up could greatly help eliminate errors in the Erlang VM.

The OTP team is usually pretty good at working with tooling for correctness around the VM, so I would think this isn't too hard to get going.

Wondering if using erlang:halt() rather than init:stop(0) would be better since halt(0) shuts down the VM instantly while init:stop(0) takes time to take down the entire supervision trees in the VM in proper order.

When I quickchecked maps, my approach was to use the distribution feature to run the maps code itself on another VM and trying for force it into a crashing state. This avoids having to restart the VM all the time.

AFL fuzz is severely limited if you need to boot up the VM, then run a program, then stop the VM again. It is rarely that part you want to test.

Perhaps we can strike middle-ground. Rip out the ETERM encoder/decoder to binary data and fuzz that. It should be a far simpler target.

Hmm, not bad at all Python 3.5 1800+ crashes, Erlang VM 9.

And I suspect Erlang's VM is more complex. Has a register VM machine. A M:N scheduler (M cpus, N processes). Fairly intricate memory allocation, hot code loading and such.

I think the author implies that 9 (unique) crashes are with limited number of executions due to slow speed. Besides, all 1800+ crashes might not be unique.

Here's my stats on python so far:

(normal afl) http://mindscope.pw/static/afl-python-slow.png

(aflfast) http://mindscope.pw/static/afl-python-fast.png

sorry for screenshots ;)

I hope he's able to work with the Erlang VM devs and get this running at a faster pace. I'd be curious how many more this is able to find when it's able to run at a faster pace.

You can also get QuickCheck for Erlang, although that stays at the Erlang fuzzing level, not AFL style branch fuzzing. Still, finds bugs!

Fuzzing is different from QuickChecking:

QuickCheck uses hand-crafted known properties and generators. Fuzzing don't. QuickCheck can be used to find negative tests, but this is the sole target of a fuzzer. There are some similarities in the approach, but it is not the same kind of tool.

QuickCheck is for testing Erlang code, not the Erlang VM itself.

Did you file bugs for python?

The marshal format used by pyc files is explicitly documented[1] to be unsafe, just a fast way to cache the compiled byte code to a .pyc file next to the .py source file. The Python VM is not a sandbox so there has been no need for an untrusted bytecode format.

[1] see big red warning box at https://docs.python.org/3/library/marshal.html

That's true, but looking at crashes, the 'exploitable' marks some of them as exploitable, there are also heap errors. Might be an easy entry point to do... something ;)

We did similar excercise a year ago with zpaq, and Matt and community was able to fix it so well that now you can fuzz it pretty much forever :)

Not yet, currently sitting there with around 2000 unique cases and wondering how to do it efficiently.