News for hackers. Stuff that matters.
Re:Snappy-start: Tool for launching a Linux process from a s... by vidarh on Wednesday July 22, 2015 @10:33AM (#9929847)
Condor [1] has done this since sometime in the 90's. This [2] paper contains references to a paper about it from at least 97. An interesting part about the Condor approach is that it can proxy file and network access, so the file-descriptors etc. can remain open (of course this may not always be what you want).

[1] http://www.sysproject.info/condor.php

[2] http://research.cs.wisc.edu/htcondor/doc/condor-practice.pdf

Re:Snappy-start: Tool for launching a Linux process from a s... by Animats on Wednesday July 22, 2015 @02:07PM (#9931278)
This goes way back, to 1960s and 1970s transaction systems such as CICS and Univac's TIP. The idea is that you have a copy of an initialized transaction program in a ready to run state, and when a transaction comes in, it's attached to a copy of the program and executed.

This approach comes in two flavors - in one, each transaction gets a fresh copy of the program. In the other, the program is used over and over unless it aborts, in which case a new clean copy is loaded and used. This corresponds to CGI and FCGI under Apache.

Programs which run as transactions have to be prepared for some unusual events. Time, for example, suddenly jumps forward when a stored transaction program starts. File and network connections have to be connected to the right things. Stateful connections, as with databases, may not play well with this approach. This can work well when the program doesn't have that much live state when the transaction program is frozen. A program that has a lot of startup overhead related to getting external resources may not be a good candidate. Unfortunately, those are the ones for which this is the biggest win.

If the main program is program loading overhead, as in Python and Java, maybe the solution is to go hard-compiled. C, C++, Go, and Rust are hard-compiled, and you can even hard-compile Java with GCC. Python and Javascript don't have that option.

(Nightmare of the future - precompiled Javascript snapshot blobs loaded into browsers.)

Re:Snappy-start: Tool for launching a Linux process from a s... by falcolas on Wednesday July 22, 2015 @09:17AM (#9929404)
Fascinating. Imagine being able to (re)start Java, Ruby and Python processes nearly instantly... No more interpreter startup overhead, just right into the process. There's some interesting implications for deployment as well; snapshot a containerized process on one machine, reproduce the disk elsewhere, and restart the already partitioned process.

I would think you'd need rather robust error handling in the program being snapshotted: there would be several machine states which would change from run to run.

Re:Snappy-start: Tool for launching a Linux process from a s... by stcredzero on Wednesday July 22, 2015 @12:49PM (#9930753)
What Smalltalk did to enable all of the above you mention: Certain facilities like networking had to be quiesced just prior to pickling, and then specially revived on return from snapshot. There were also special hooks for return from snapshot. (IIRC, there is a Smalltalk method of that name.)

Snapshots becomes much more powerful when combined with a transactional log of all changes. This gets around certain problems with de-synchronization of source files with the image. Smalltalks that abandoned the change-log and returned to conventional source files suffered for it. (Enfin/ObjectStudio)

Re:Snappy-start: Tool for launching a Linux process from a s... by gnufied on Wednesday July 22, 2015 @09:51AM (#9929621)
Getting this error while running ./make.sh:

   Hello, world!
   Hello, world!
   * Running test: ./out/example_loader
   Traceback (most recent call last):
   File "tests/run_tests.py", line 57, in <module>
    Main()
   File "tests/run_tests.py", line 40, in Main
    RunTest(['./out/example_loader'], -1, use_elf_loader=False)
   File "tests/run_tests.py", line 34, in RunTest
    subprocess.check_call(['./out/restore'])
   File "/usr/lib64/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
   subprocess.CalledProcessError: Command '['./out/restore']' returned non-zero exit status -11
Interestingly the project has Github issues disabled.
Re:Snappy-start: Tool for launching a Linux process from a s... by gboudrias on Wednesday July 22, 2015 @10:39AM (#9929887)
> Interestingly the project has Github issues disabled.

This is pretty weird. Certainly goes against the spirit of free software in my book. It's not like they have to answer any issues.

(Oh right it's Google. Well that explains that.)

Re:Snappy-start: Tool for launching a Linux process from a s... by noja on Wednesday July 22, 2015 @10:58AM (#9929982)
Having Issues enabled sets the expectation that they are going to do something about your issue.
Re:Snappy-start: Tool for launching a Linux process from a s... by patrickaljord on Wednesday July 22, 2015 @10:55AM (#9929955)
Actually there is nothing in the definition of free software that requires a bug tracker, even by gnu.org and rms standards and they're pretty high.
Re:Snappy-start: Tool for launching a Linux process from a s... by teddyh on Wednesday July 22, 2015 @12:00PM (#9930397)
gboudrias said “spirit”, not “definition”.
Re:Snappy-start: Tool for launching a Linux process from a s... by peterwwillis on Wednesday July 22, 2015 @12:02PM (#9930415)
This is also sometimes called 'checkpointing', and has been a long-studied feature necessary for distributed processing systems. There's a lot of ways to skin this cat. Kernel modules, application libraries, userland tools, and features to support various kinds of applications (single-threaded, multi-threaded, preserving TCP connections, etc). Find the one that fits your use best.

Descriptions of how checkpointing works: https://en.wikipedia.org/wiki/Application_checkpointing http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.117... http://www.anchor.com.au/blog/2013/02/overview-of-checkpoint... https://lwn.net/Articles/478111/ https://www.usenix.org/legacy/event/usenix01/freenix01/full_...

Individual solutions: https://ckpt.wiki.kernel.org/index.php/Main_Page https://code.google.com/p/cryopid/ http://criu.org/Main_Page http://crd.lbl.gov/departments/computer-science/CLaSS/resear...

An almost-full list of solutions here: http://checkpointing.org/

Re:Snappy-start: Tool for launching a Linux process from a s... by endlessvoid94 on Wednesday July 22, 2015 @11:45AM (#9930287)
I've always wanted the ability to take a snapshot of a running process, say a ruby repl, that's thrown an unhandled exception, waiting for user input. Send that snapshot to a friend, who can debug the issue, send back, etc. Collaboration on a live system over the internet.

Yes, exactly like smalltalk. sigh.

Re:Snappy-start: Tool for launching a Linux process from a s... by ivank on Wednesday July 22, 2015 @08:58AM (#9929308)
I used http://criu.org/ a while ago to do this for Clojure REPLs
Re:Snappy-start: Tool for launching a Linux process from a s... by jkrippy on Wednesday July 22, 2015 @05:48PM (#9932559)
They're adding integration in with Docker too for Criu: http://criu.org/Docker

Here's a fun example of it in action: http://blog.kubernetes.io/2015/07/how-did-quake-demo-from-do...

Re:Snappy-start: Tool for launching a Linux process from a s... by gnufied on Wednesday July 22, 2015 @09:56AM (#9929649)
Looks really nice except having to compile custom Kernels I guess.
Re:Snappy-start: Tool for launching a Linux process from a s... by sdsk8 on Wednesday July 22, 2015 @10:32AM (#9929838)
but it is in user space, why the need for compile kernel?
Re:Snappy-start: Tool for launching a Linux process from a s... by x7f on Wednesday July 22, 2015 @12:04PM (#9930428)
Some distros (e.g Arch) don't have CONFIG_CHECKPOINT_RESTORE enabled by default.
Re:Snappy-start: Tool for launching a Linux process from a s... by mateuszf on Wednesday July 22, 2015 @10:39AM (#9929886)
Did it change the startup time significantly?
Re:Snappy-start: Tool for launching a Linux process from a s... by ivank on Wednesday July 22, 2015 @01:29PM (#9931018)
Yeah, it was much faster than starting up the normal way, at least for a small heap (213 ms to restore 128MB on an E5645). I didn't pursue it further because it required root (not sure if it still does) and would've required writing some kind of snapshot manager.
Re:Snappy-start: Tool for launching a Linux process from a s... by mwcampbell on Wednesday July 22, 2015 @08:54AM (#9929289)
Very cool. Would be even nicer if the snapshotting process created a self-contained ELF executable, rather than a snapshot file that needs a separate executable to start it back up. Seems like it should be feasible.

Since the README mentions kentonv, I wonder if this would be useful within Sandstorm.

Re:Snappy-start: Tool for launching a Linux process from a s... by kentonv on Thursday July 23, 2015 @05:13PM (#9938557)
Sandstorm is the motivating use case. :)
Re:Snappy-start: Tool for launching a Linux process from a s... by vezzy-fnord on Wednesday July 22, 2015 @01:18PM (#9930945)
This is a nice PoC tool, but as a checkpointing solution the idea of using ptrace(2) just reeks of being a huge hack.

If you want to do syscall replaying, check out the Approximate Replay-Trace Compiler (ARTC).

For a full checkpointing suite on modern Linux, see CRIU or DMTCP.

Re:Snappy-start: Tool for launching a Linux process from a s... by jbverschoor on Wednesday July 22, 2015 @09:04AM (#9929331)
Sounds like an ideal tool for debugging.

Trace while running, take a snapshot when there's something wrong

Re:Snappy-start: Tool for launching a Linux process from a s... by xyzzy123 on Wednesday July 22, 2015 @09:37AM (#9929523)
It's a cool hack, but only useful for fast startup just at this point. It can't do network sockets (for obvious reasons), or things like gettimeofday or even fds just now.

Reminds me of a Lisp image dump.

Re:Snappy-start: Tool for launching a Linux process from a s... by sanxiyn on Wednesday July 22, 2015 @10:08AM (#9929708)
If you want a debugging tool, try http://rr-project.org/
Re:Snappy-start: Tool for launching a Linux process from a s... by mateuszf on Wednesday July 22, 2015 @10:33AM (#9929846)
I'd imagine that it has some overhead. So running it on production all the time would not make much sense. Because sometimes restarting a process makes it very hard to reproduce the bug. But snasphost / restore would handle that case. The problem is - what to do with file / network descriptors?
Re:Snappy-start: Tool for launching a Linux process from a s... by stcredzero on Wednesday July 22, 2015 @12:44PM (#9930716)
Lots of Smalltalk production systems did exactly this. One could also log into running production systems and inspect live objects.
Re:Snappy-start: Tool for launching a Linux process from a s... by haddr on Wednesday July 22, 2015 @12:10PM (#9930479)
This is really great tool! I wonder if it could work with Java programs... Can't wait to try it...
Re:Snappy-start: Tool for launching a Linux process from a s... by borkabrak on Wednesday July 22, 2015 @01:04PM (#9930870)
I'm personally excited to see if I can get this to allow me to persist tmux sessions over a reboot.

Thoughts?

Re:Snappy-start: Tool for launching a Linux process from a s... by curiousjorge on Wednesday July 22, 2015 @12:31PM (#9930627)
how does this compare to digitalocean's loading from snapshot? It takes 30 seconds or more to load your snapshot. Is this any different? I'd love to not rely on DO because this is one of the key selling point of DO, saving a snapshot and loading it.
Re:Snappy-start: Tool for launching a Linux process from a s... by sciurus on Wednesday July 22, 2015 @12:42PM (#9930701)
This is about snapshots of a single program. What you are describing at DO is for an entire virtualized computer.
Re:Snappy-start: Tool for launching a Linux process from a s... by curiousjorge on Wednesday July 22, 2015 @12:47PM (#9930742)
if I wanted something like DO, what can I use?

if this works on a single program, what sort of use cases would this come in handy? Perhaps a java program that has a bit of startup time?

Re:Snappy-start: Tool for launching a Linux process from a s... by scintill76 on Wednesday July 22, 2015 @04:46PM (#9932195)
VirtualBox has snapshots. Here's a page with an image: http://ehikioya.com/moving-virtualbox-vm-snapshots/ The page is talking about cloning, but you can save/restore snapshots within the same VM too.