No problem. I've just checked and it looks like libcgroup is not maintained any more and fallen behind the cgroups development. This may work instead:

systemd-run --scope -p MemoryMax=4M rtorrent

You can also set other quotas - `man systemd.resource-control` for more info.