News for hackers. Stuff that matters.
Re:Rpmalloc – A lock-free memory allocator by CyberDildonics on Sunday March 19, 2017 @11:30AM (#13907662)
Looks like they are missing jemalloc in the benchmarks, which would be good to see.

I think the simplicity of a 1300 line allocator is very alluring. At that level it should avoid bloating small programs and is hopefully easy enough to understand if digging in to it.

Also it looks like it is split up into 3 files. consolidating it to a single header file C library would be very nice.

One oddity is the 16 bytes alignment. 64 bytes is the false sharing boundary in x64 so I'm surprised to see this in a concurrent allocator, but maybe I've misunderstood exactly what that means here.

Re:Rpmalloc – A lock-free memory allocator by maniccoder on Sunday March 19, 2017 @04:54PM (#13909566)
I'm going to add jemalloc, lockless and scalloc in the benchmark comparisons soon(ish).

Regarding the files, only rpmalloc.[h|c] is needed. The malloc.c file is just for overriding the malloc family of entry points, and provide automatic init/fini glue.

The 16 byte alignment is just a note that all returned blocks are naturally aligned to 16 byte boundaries due to the fact that all size buckets are a multiple of 16 bytes internally. It is needed for SSE instructions, which is why I mention it.

Re:Rpmalloc – A lock-free memory allocator by rurban on Monday March 20, 2017 @02:22AM (#13912165)
Fair would be a comparison single-threaded to ptmalloc2 (glibc). I assume the slowdown will be ~5%.

And the size granularity is split into 3 (small, medium, large) which is fine, but other's have finer granularities. So benchmarks with object systems would be nice, not just random data.

Re:Rpmalloc – A lock-free memory allocator by maniccoder on Monday March 20, 2017 @04:48AM (#13912639)
There already is a benchmark against glibc single threaded, compile the rpmalloc-benchmark repository linked in the readme and run the runall script. Compare rpmalloc 1 thread case against crt 1 thread (with crt being the C runtime, i.e glibc in this case).

Graphs and analysis for a Linux system will be added soon, but a spoiler:

rpmalloc 22.7m memory ops/CPU second, 26MiB bytes peak, 15% overhead

jemalloc 21.3m memory ops/CPU second, 30MiB bytes peak, 33% overhead

tcmalloc 20.4m memory ops/CPU second, 30MiB bytes peak, 32% overhead

crt 13.5m memory ops/CPU second, 24MiB bytes peak, 7% overhead

rpmalloc is faster and has less overhead than both jemalloc and tcmalloc, and a lot faster than glibc at the cost of extra memory overhead.

The reasoning behind the granularities is that 16 bytes is a reasonable granularity for smaller objects (below 2KiB in size). Medium sized requests (between 2KiB and 32KiB) have a 512 byte granularity, while large requests (up to 2MiB) have a 62KiB granularity.

If you have a suggestion for additional benchmark models I would be happy to add an implementation for it.

Re:Rpmalloc – A lock-free memory allocator by rurban on Monday March 20, 2017 @08:33AM (#13913531)
Ouch, so this looks like the final coffin into ptmalloc2, and esp. a shout to emacs, which still hasn't finished their portable dumper. But not much missing there, should be ready soon. (I got a not working branch on my GitHub) Hope glibc will adopt this soon.
Re:Rpmalloc – A lock-free memory allocator by olliej on Monday March 20, 2017 @12:59AM (#13911823)
Any plan to add the allocators used in modern browsers? bmalloc (webkit), which ever tcmalloc variant chrome currently uses, the current firefox allocator, I have no idea if edge has an allocator that is not the default windows one?
Re:Rpmalloc – A lock-free memory allocator by maniccoder on Monday March 20, 2017 @01:41AM (#13911998)
I will look into it
Re:Rpmalloc – A lock-free memory allocator by ahoka on Sunday March 19, 2017 @04:57PM (#13909587)
16 bytes is the required aligment for long double, which is the largest (at least on x86) required alignment. If you need more use memalign or something similar.
Re:Rpmalloc – A lock-free memory allocator by krenoten on Sunday March 19, 2017 @01:42PM (#13908520)
Indeed. jemalloc uses a lock-free radix tree at its core.
Re:Rpmalloc – A lock-free memory allocator by TwoBit on Sunday March 19, 2017 @01:36PM (#13908484)
jemalloc doesn't support non-Unix platforms (eg Windows). You may or may not be able to find a Windows port of any particular jemalloc version, but the developers ignore Windows because they don't use it in their server software. We gave up using jemalloc on Windows because it was such a crap shoot.
Re:Rpmalloc – A lock-free memory allocator by kbenson on Sunday March 19, 2017 @04:33PM (#13909455)
That sounds like a good basis for a "portable" version project. Other very targeted software (such as OpenSSH[1]) does this. I'm not sure what it takes to make jemalloc windows compatible, but even if the jemalloc developers don't want to support windows, it's possibly they might be more amenable to working with a project dedicated to portability in general.

Also, my understanding is that Rust uses jemalloc, and Rust runs on windows, so what does Rust do? Just use a known good ported jemalloc version?

1: https://www.openssh.com/portable.html

Re:Rpmalloc – A lock-free memory allocator by steveklabnik on Sunday March 19, 2017 @04:49PM (#13909547)
We don't use jemalloc on Windows, we use the system allocator, aka HeapAlloc.
Re:Rpmalloc – A lock-free memory allocator by CyberDildonics on Monday March 20, 2017 @12:48AM (#13911776)
There are already versions out there on github that have windows support built in, I've used it on windows. I also had problems with it however.
Re:Rpmalloc – A lock-free memory allocator by jasone on Monday March 20, 2017 @09:12PM (#13919004)
jemalloc does support Windows, and stability has improved dramatically in recent releases, primarily because we now have continuous integration coverage for several Windows configurations.