News for hackers. Stuff that matters.
Re:Unfork: The Inverse of Fork(2) by touisteur on Saturday May 29, 2021 @11:15AM (#27325763)
That's very interesting. Userfaultfd opened a lot of features in Linux.

When I saw 'unfork' I thought 'someone found out how to replace the current process with a child process that was a older snapshot'. Any idea on how to do this? Like 'OK I did something wrong, let me back up a bit, kill me, replace me with my child x and refather every other one of my children processes to child x and gooo'.

Re:Unfork: The Inverse of Fork(2) by remram on Saturday May 29, 2021 @01:26PM (#27326760)
Easier to do it the other way: when you fork, keep the parent as the snapshot and keep going in the child. That way you can back up by terminating and resuming in the parent. I think this is what timetravelpdb does for Python: https://github.com/TomOnTime/timetravelpdb
Re:Unfork: The Inverse of Fork(2) by touisteur on Saturday May 29, 2021 @02:14PM (#27327104)
Yeah, thanks it was my first idea, but then 1) I'd have to re-attach everything to the child process and 2) I'd have to go one step further for every snapshot, right ? If everything goes well and I want to continue, next snapshot I'll have to fork the child process, and so on, no? The idea is to keep a somewhat short (some minutes?) of snapshots.

Thanks for the timetravelpdb link!

Re:Unfork: The Inverse of Fork(2) by thomasballinger on Saturday May 29, 2021 @03:56PM (#27327824)
Here's another implementation of what's described above, it forks on each invocation of readline to provide undo for interactive interpreters. https://github.com/thomasballinger/rlundo
Re:Unfork: The Inverse of Fork(2) by touisteur on Saturday May 29, 2021 @05:04PM (#27328220)
OK I love this, and not just because of the calls to system(nc) :-)

Your code made me think, maybe I'm tying myself into knots... But my thing is making hundred thousands of checkpoints so I'd have to have as many forks as savespoints. No way to coalesce parents, maybe reparenting could work there...

Re:Unfork: The Inverse of Fork(2) by IanGabes on Saturday May 29, 2021 @01:19PM (#27326714)
You might be interested in CRIU: https://github.com/checkpoint-restore/criu
Re:Unfork: The Inverse of Fork(2) by touisteur on Sunday June 06, 2021 @04:48PM (#27416188)
I'm hoping somewhere deep in Google or Cloudflare someone is implementing all this in chained ebpf scriptw...
Re:Unfork: The Inverse of Fork(2) by cpufry on Thursday June 03, 2021 @11:07AM (#27382446)
this is neat. thanks.
Re:Unfork: The Inverse of Fork(2) by CJefferson on Saturday May 29, 2021 @12:52PM (#27326512)
I would really like that function. While fork is often a bad idea, it can be an easy way of adding parallelization and backtracking to old large serial C code bases.
Re:Unfork: The Inverse of Fork(2) by snarfy on Saturday May 29, 2021 @12:06PM (#27326135)
iirc emacs implements a generic 'unexec' function which serializes the current process state so it can be reloaded later. Seems like you could use it to make snapshots.
Re:Unfork: The Inverse of Fork(2) by touisteur on Saturday May 29, 2021 @12:22PM (#27326239)
Thanks for the emacs pointer. Interesting. It screams of continuations.

I know how to do snapshots of a process, what with criu. But if I just want a checkpoint to go back later, not a full serialization, and I don't want to save/restore a complete process, but take advantage of fork()'s CoW to save the least possible in a stopped process, then be able to come back. The rest of fork()' semantics are a problem, with threads, sockets, signals that are not passed down. An example of that approach is perf-fuzz where they add a new syscall to make fuzzing faster.

[0] https://github.com/sslab-gatech/perf-fuzz

Re:Unfork: The Inverse of Fork(2) by touisteur on Saturday May 29, 2021 @04:37PM (#27328074)
And now I realize I was very wrong about what unexec does/did. Wow.
Re:Unfork: The Inverse of Fork(2) by the8472 on Saturday May 29, 2021 @12:55PM (#27326530)
Does `rr` fit the bill?
Re:Unfork: The Inverse of Fork(2) by touisteur on Saturday May 29, 2021 @02:06PM (#27327057)
From memory rr has a substantive recording overhead, and is specially made for debugging, right? But yes it is very useful to analyse a past state and to understand a chain of events.

I should clarify my use case: I would use such a feature (go back to previous state) for a speculative execution tool. I'd execute the happy path all the time assuming no error occured, but if I found out later that something went wrong somewhere, I'd want to go back and start from there knowing what went wrong, and so on. With as little perf loss as possible. Not sure my explanation makes sense.

I know about dmtcp, criu, vm snapshots, but they all come with big overhead (I don't want to pay too much for the checkpoint).

The closest I found was @gamozolabs amazing work on snapshot fuzzing (pushing the limit of what's possible on x86_64 hardware, including using Intel PML - similar to userfaultfd but hw-accelerated...).

Re:Unfork: The Inverse of Fork(2) by Agingcoder on Saturday May 29, 2021 @04:32PM (#27328049)
Rr was initially designed to reproduce flaky tests I think. They then realized that they could modify it for reverse debugging.

The recording overhead is quite acceptable, about 50%.

Rr also has a 'chaos mode' which changes the thread scheduling, and which greatly facilitates finding the 'unhappy' path.

Re:Unfork: The Inverse of Fork(2) by touisteur on Saturday May 29, 2021 @05:06PM (#27328225)
OK thanks for the feedback on recording overhead, I'll have to try for the checkpoint/restore use case.
Re:Unfork: The Inverse of Fork(2) by Agingcoder on Sunday May 30, 2021 @12:21PM (#27334306)
You might be interested by pernosco, written by the same people. It's rather spectacular.

https://pernos.co

The recording overhead is the same (it leverages rr) but you can explore your bug to your heart's content.

Re:Unfork: The Inverse of Fork(2) by mattgreenrocks on Saturday May 29, 2021 @12:49PM (#27326486)
This is the first I've heard of userfaultfd! It is rather mind-blowing. Thank you for the comment.
Re:Unfork: The Inverse of Fork(2) by touisteur on Saturday May 29, 2021 @05:18PM (#27328285)
There are so many features in the Linux kernel it sometimes blows my mind. eventfd, signalfd, timerfd, memfd, pidfd. The whole fricking tc/qdisc featureset (OMG). netlink. io_uring. criu. SO_REUSEPORT. Teaming. Namespaces. veths. vsocks. Dpdk/netmap/af_packet. XDP ! Seccomp.

I mean look at that https://developers.redhat.com/blog/2018/10/22/introduction-t...

Amazing.

Re:Unfork: The Inverse of Fork(2) by agumonkey on Saturday May 29, 2021 @11:43AM (#27325967)
ProlOS
Re:Unfork: The Inverse of Fork(2) by touisteur on Saturday May 29, 2021 @11:52AM (#27326034)
OK I really tried but my Googlefu seems lacking. Any pointer, pretty please?
Re:Unfork: The Inverse of Fork(2) by agumonkey on Saturday May 29, 2021 @01:34PM (#27326826)
Oh it was mostly a pun blending prolog like backtracking at the os-process level.
Re:Unfork: The Inverse of Fork(2) by touisteur on Saturday May 29, 2021 @02:09PM (#27327071)
I figured. I went down the rabbit hole, prolo, and prolog and hoped it wasn't a pun :-) yes I think backtracking as a general OS mechanism. Since the kernel already knows which pages differ between both processes, and could 'just' plug back any socket, file, shm, pipe 'as is' when 'restoring'. All the state is know to the OS!

Maybe something with ebpf in twelve years...

Re:Unfork: The Inverse of Fork(2) by agumonkey on Saturday May 29, 2021 @02:12PM (#27327088)
and live exploratory debugging when your fs fails
Re:Unfork: The Inverse of Fork(2) by touisteur on Saturday May 29, 2021 @02:18PM (#27327119)
Well if I could synchronise that with lvm snapshots and go back... Once you start going the backtracking rabbit hole, there's lots of thing one can imagine.
Re:Unfork: The Inverse of Fork(2) by agumonkey on Saturday May 29, 2021 @04:29PM (#27328033)
time to make a pull-request
Re:Unfork: The Inverse of Fork(2) by touisteur on Saturday May 29, 2021 @04:38PM (#27328082)
Wish I had the chops. Time to find money to have someone do it you mean.
Re:Unfork: The Inverse of Fork(2) by agumonkey on Saturday May 29, 2021 @05:25PM (#27328341)
who sets up the kickstarter ? :p