CRIU - User contributions [en]

Checkpoint/Restore

2015-08-05T06:25:57Z

Artem: /* Checkpoint */

This page describes the overall design of how Checkpoint and Restore work in CRIU.

== Checkpoint ==

The checkpoint procedure relies heavily on '''/proc''' file system (it's a general place where criu takes all the information it needs).
Which includes

* Files descriptors information (via '''/proc/$pid/fd''' and '''/proc/$pid/fdinfo''').
* Pipes parameters.
* Memory maps (via '''/proc/$pid/maps''' and '''/proc/$pid/map_files/''').
* etc.

The process dumper (lets call it a dumper further) does the following steps during checkpoint stage

=== Collect process tree and freeze it ===
The '''$pid''' of a process group leader is obtained from the command line (<code>--tree</code> option). By using this '''$pid''' the dumper walks though '''/proc/$pid/task/''' directory collecting threads and through the '''/proc/$pid/task/$tid/children''' to gathers children recursively. While walking tasks are stopped using the <code>ptrace</code>'s <code>PTRACE_SEIZE</code> command.

=== Collect tasks' resources and dump them ===
At this step CRIU reads all the information (it knows) about collected tasks and writes them to dump files. The resources are obtained via
# VMAs areas are parsed from '''/proc/$pid/smaps''' and mapped files are read from '''/proc/$pid/map_files''' links
# File descriptor numbers are read via '''/proc/$pid/fd'''
# Core parameters of a task (such as registers and friends) are being dumped via ptrace interface and parsing '''/proc/$pid/stat''' entry.

Then CRIU injects a [[parasite code]] into a task via ptrace interface. This is done in two steps -- at first we inject only a few bytes for ''mmap'' syscall at CS:IP the task has at moment of seizing. Then ptrace allow us to run an injected syscall and we allocate enough memory for a parasite code chunk we need for dumping. After that the parasite code is copied into new place inside dumpee address space and CS:IP set respectively to point to our parasite code.

From parsite context CRIU does more information such as
# Credentials
# Contents of memory

=== Cleanup ===

After everything dumped (such as memory pages, which can be written out only from inside dumpee address space) we use ptrace facility again and cure dumpee by dropping out all our parasite code and restoring original code. Then CRIU detaches from tasks and they continue to operate.

== Restore ==

The restore procedure (aka restorer) is done by CRIU morphing itself into the tasks it restores. On the top-level it consists of 4 steps

=== Resolve shared resources ===

At this step CRIU reads in image files and finds out which processes share which resources. Later shared resources are restored by some one process and all the others either inherit one on the 2nd stage (like session) or obtain in some other way. The latter is, for example, shared files which are sent with SCM_CREDS messages via unix sockets, or shared memory areas that are restoring via <code>memfd</code> file descriptor.

=== Fork the process tree ===

At this step CRIU calls fork() many times to re-created the processes needed to be restored. Note, that threads are not restored here, but on the 4th step.

=== Restore basic tasks resources ===

Here CRIU restores all resources but

# memory mappings exact location
# timers
# credentials
# threads

These for are delayed till the last stage for the reasons described further. On this stage CRIU opens files, prepares namespaces, maps (and fills with data) private memory areas, creates sockets, calls chdir() and chroot() and dome some more.

=== Switch to [[restorer context]], restore the rest and continue ===

The reason for restorer blob is simple. Since criu morphs into the target process, it will have to unmap all its memory and put back the target one. While doing so some code should exist in memory (the code doing the munmap and mmap). So we introduced the restorer blob -- the small piece of code, that doesn't intersect with criu mappings AND target mappings. At the end of stage 2 criu jumps into this blob and restores the memory maps.

At the same place we restore timers not to make them fire too early, here we restore credentials to let criu do priveledged operations (like fork-with-pid) and threads not to make them suffer from sudden memory layout change.

[[Category:Under the hood]]

External UNIX socket

2015-07-30T07:51:46Z

Artem: /* What to do with socketpair()-s? */

This explains the meaning of '''external socket is used''' error message, and the purpose of '''<code>--ext-unix-sk</code>''' option.

== Error meaning ==

When dumping a process sub-tree, criu checks that the resulting image is consistent and self-contained, meaning if an object A references another object B, and A goes into dump, then B should be dumped as well. For example, if there is a pipe, and criu dumps its one end (object A) because it belongs to a process it dumps, it must also dump the other end of the pipe (object B), meaning it should take a process owning B into the image as well. Same is true for unix sockets: if there is a socket (A) that is <code>connect()</code>ed to another socket (B), and criu dumps socket A (because it is opened by a process it dumps), it must also dump socket B and the task who owns it. Possibly socket B is dumped some time later in the dumping process, but it must be dumped.

This restriction is in place to make sure that upon restore applications will continue to work, rather than, say, receive SIGPIPE signals due to half-closed pipes or sockets.

With that said, "external socket is used" error means criu detected a unix socket connected to another socket which is not being dumped (because it belongs to a process not going into the image).

== Using the option ==

However, sometimes it is possible to dump and successfully restore only one end of a unix socket pair. One particular example is the datagram sockets with on-way connection (client to server) used e.g. by <code>logd</code>. Server opens a datagram socket and waits on it for messages to be written into a log file. Processes using logd also create datagram sockets and connect those to the server. These connections are thus uni-directional. In this case it is possible to dump a program with the client-side socket and on restore the socket needs to be reconnected back to the original server.

This is when <code>--ext-unix-sk</code> option should be used. By providing this option (for both dump and restore commands), user states "I know there may be uni-directional dgram unix connections, and I will make sure the server end will exist on restore".

For '''criu dump''', this option enables dumping ''datagram'' unix sockets with additional information about that other ("external") socket it is connected to.

For '''criu restore''', this option asks criu to re-connect such sockets back.

== Limitations ==
Named unix sockets with stream/seqpacket options can't be dumped\restored now, see plan below.

== What to do with stream/seqpacket? ==

Such sockets cannot be just dumped and restored as once we dump one end, the other one seen EOF on socket and may close one. The plan for this is to extend the --ext-unix-sk semantics to work like this

* On dump the <code>--ext-unix-sk $id</code> says that socket with $id is OK to be disconnected
* On restore the <code>--ext-unix-sk$id[=$path]</code> says that the socket $id should be reconnected back to the path it say on dump (or to the $path).

== What to do with socketpair()-s? ==
Unnamed unix sockets created with socketpair() systemcall can be dumped and restored.
For dump socketpair put inode of this socket in <code>--ext-unix-sk</code> option, for example:

<code>
criu dump -D images -o dump.log -v4 --ext-unix-sk=11890815 -t 16528
</code>

For restore socketpair the server should create new pair and call criu restore asking it to [[Inheriting FDs on restore|inherit]] one. Example of command:

<code>
criu restore -d -D images -o restore.log --pidfile restore.pid -v4 -x --inherit-fd fd[3]:socket:[11890815]
</code>

test source code also available for this feature and can be found in criu source tree under test/socketpairs.

== See also ==

* [[Advanced usage]]

[[Category:HOWTO]]
[[Category:Network]]

External UNIX socket

2015-07-30T07:21:04Z

Artem: /* Limitations */

This explains the meaning of '''external socket is used''' error message, and the purpose of '''<code>--ext-unix-sk</code>''' option.

== Error meaning ==

When dumping a process sub-tree, criu checks that the resulting image is consistent and self-contained, meaning if an object A references another object B, and A goes into dump, then B should be dumped as well. For example, if there is a pipe, and criu dumps its one end (object A) because it belongs to a process it dumps, it must also dump the other end of the pipe (object B), meaning it should take a process owning B into the image as well. Same is true for unix sockets: if there is a socket (A) that is <code>connect()</code>ed to another socket (B), and criu dumps socket A (because it is opened by a process it dumps), it must also dump socket B and the task who owns it. Possibly socket B is dumped some time later in the dumping process, but it must be dumped.

This restriction is in place to make sure that upon restore applications will continue to work, rather than, say, receive SIGPIPE signals due to half-closed pipes or sockets.

With that said, "external socket is used" error means criu detected a unix socket connected to another socket which is not being dumped (because it belongs to a process not going into the image).

== Using the option ==

However, sometimes it is possible to dump and successfully restore only one end of a unix socket pair. One particular example is the datagram sockets with on-way connection (client to server) used e.g. by <code>logd</code>. Server opens a datagram socket and waits on it for messages to be written into a log file. Processes using logd also create datagram sockets and connect those to the server. These connections are thus uni-directional. In this case it is possible to dump a program with the client-side socket and on restore the socket needs to be reconnected back to the original server.

This is when <code>--ext-unix-sk</code> option should be used. By providing this option (for both dump and restore commands), user states "I know there may be uni-directional dgram unix connections, and I will make sure the server end will exist on restore".

For '''criu dump''', this option enables dumping ''datagram'' unix sockets with additional information about that other ("external") socket it is connected to.

For '''criu restore''', this option asks criu to re-connect such sockets back.

== Limitations ==
Named unix sockets with stream/seqpacket options can't be dumped\restored now, see plan below.

== What to do with stream/seqpacket? ==

Such sockets cannot be just dumped and restored as once we dump one end, the other one seen EOF on socket and may close one. The plan for this is to extend the --ext-unix-sk semantics to work like this

* On dump the <code>--ext-unix-sk $id</code> says that socket with $id is OK to be disconnected
* On restore the <code>--ext-unix-sk$id[=$path]</code> says that the socket $id should be reconnected back to the path it say on dump (or to the $path).

== What to do with socketpair()-s? ==

To restore socketpair we need more help from the server -- the server should create new pair and call criu restore asking it to [[Inheriting FDs on restore|inherit]] one.

== See also ==

* [[Advanced usage]]

[[Category:HOWTO]]
[[Category:Network]]

External UNIX socket

2015-07-30T06:28:40Z

Artem: /* What to do with stream/seqpacket? */

This explains the meaning of '''external socket is used''' error message, and the purpose of '''<code>--ext-unix-sk</code>''' option.

== Error meaning ==

When dumping a process sub-tree, criu checks that the resulting image is consistent and self-contained, meaning if an object A references another object B, and A goes into dump, then B should be dumped as well. For example, if there is a pipe, and criu dumps its one end (object A) because it belongs to a process it dumps, it must also dump the other end of the pipe (object B), meaning it should take a process owning B into the image as well. Same is true for unix sockets: if there is a socket (A) that is <code>connect()</code>ed to another socket (B), and criu dumps socket A (because it is opened by a process it dumps), it must also dump socket B and the task who owns it. Possibly socket B is dumped some time later in the dumping process, but it must be dumped.

This restriction is in place to make sure that upon restore applications will continue to work, rather than, say, receive SIGPIPE signals due to half-closed pipes or sockets.

With that said, "external socket is used" error means criu detected a unix socket connected to another socket which is not being dumped (because it belongs to a process not going into the image).

== Using the option ==

However, sometimes it is possible to dump and successfully restore only one end of a unix socket pair. One particular example is the datagram sockets with on-way connection (client to server) used e.g. by <code>logd</code>. Server opens a datagram socket and waits on it for messages to be written into a log file. Processes using logd also create datagram sockets and connect those to the server. These connections are thus uni-directional. In this case it is possible to dump a program with the client-side socket and on restore the socket needs to be reconnected back to the original server.

This is when <code>--ext-unix-sk</code> option should be used. By providing this option (for both dump and restore commands), user states "I know there may be uni-directional dgram unix connections, and I will make sure the server end will exist on restore".

For '''criu dump''', this option enables dumping ''datagram'' unix sockets with additional information about that other ("external") socket it is connected to.

For '''criu restore''', this option asks criu to re-connect such sockets back.

== Limitations ==

Socket screated with socketpair() systemcall are not one-way connected and do not work with --ext-unix-sk

== What to do with stream/seqpacket? ==

Such sockets cannot be just dumped and restored as once we dump one end, the other one seen EOF on socket and may close one. The plan for this is to extend the --ext-unix-sk semantics to work like this

* On dump the <code>--ext-unix-sk $id</code> says that socket with $id is OK to be disconnected
* On restore the <code>--ext-unix-sk$id[=$path]</code> says that the socket $id should be reconnected back to the path it say on dump (or to the $path).

== What to do with socketpair()-s? ==

To restore socketpair we need more help from the server -- the server should create new pair and call criu restore asking it to [[Inheriting FDs on restore|inherit]] one.

== See also ==

* [[Advanced usage]]

[[Category:HOWTO]]
[[Category:Network]]

Installation

2015-07-21T09:40:17Z

Artem: /* Other deps */

<code>criu</code> is an utility to checkpoint/restore a process tree. This page describes how to manually build and install prerequisites and the tool itself.

== Installing from packages ==

Some distributions provide ready-to-use [[packages]]. If no, or the CRIU version you want is not yet there, you will need to get CRIU sources and compile it.

== Obtaining CRIU Source ==

You can download the source code as a release tarball or sync the [http://git.criu.org/?p=criu.git;a=summary git repository]. If you plan to modify CRIU sources the latter way is highly recommended.

=== Getting source tarball ===
: {{Latest release}}

=== Cloning git repository ===
git clone https://github.com/xemul/criu

== Dependencies ==

=== Compiler and C Library ===

CRIU is mostly written in C and the build system is based on Makefiles. Thus just install standard <code>gcc</code> and <code>make</code> packages (on Debian, <code>[https://packages.debian.org/build-essential build-essential]</code> will pull in both at once).

If you are cross compiling for ARM, use distribution packages or download prebuilt toolchains from Linaro.

<div class="toccolours mw-collapsible mw-collapsed" style="width:800px">
Downloading Linaro toolchains
<div class="mw-collapsible-content">
sudo apt-get install lib32stdc++6 lib32z1 # These are ia32 binaries
mkdir -p deps/`uname -m`-linux-gnu
cd deps
wget http://releases.linaro.org/14.09/components/toolchain/binaries/gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux.tar.xz
tar --strip=1 -C `uname -m`-linux-gnu -xf gcc-linaro-arm-linux-gnueabihf-4.9-2014.09_linux.tar.xz
wget http://releases.linaro.org/14.09/components/toolchain/binaries/gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux.tar.xz
tar --strip=1 -C `uname -m`-linux-gnu -xf gcc-linaro-aarch64-linux-gnu-4.9-2014.09_linux.tar.xz
cd ..
</div>
</div>

=== Protocol Buffers ===

CRIU uses the [https://developers.google.com/protocol-buffers/ Google Protocol Buffers] to read and write [[images]] and thus requires [https://github.com/protobuf-c/protobuf-c C language bindings]. The <code>protoc</code> tool is required at build time and the <code>libprotobuf-c.so</code> shared object is required at build and run time. [[CRIT]] also uses python language bindings for protocol buffers and requires the <code>descriptor.proto</code> file typically provided by a distribution's protobuf development package.

==== Distribution Packages ====
The easiest way is to install distribution packages.

* RPM package names
** <code>protobuf</code>
** <code>protobuf-c</code>
** <code>protobuf-c-devel</code>
** <code>protobuf-compiler</code>
** <code>protobuf-devel</code>
** <code>protobuf-python</code>
* Debian package names
** <code>libprotobuf-dev</code>
** <code>libprotobuf-c0-dev</code>
** <code>protobuf-c-compiler</code>
** <code>protobuf-compiler</code>
** <code>python-protobuf</code>

==== Building Protocol Buffers From Source ====
If you would like to build from source, you can use the following commands to obtain the source code repositories, configure, and build the code. On a Debian based system, you may have to install <code>autoconf curl g++ libtool</code> packages first.

<div class="toccolours mw-collapsible mw-collapsed" style="width:800px">
To build protobuf
<div class="mw-collapsible-content">
cd deps
git clone https://github.com/google/protobuf.git protobuf
cd protobuf
./autogen.sh
./configure --prefix=`pwd`/../`uname -m`-linux-gnu
make
make install
cd ../..
</div>
</div>

<div class="toccolours mw-collapsible mw-collapsed" style="width:800px">
To build protobuf-c
<div class="mw-collapsible-content">
cd deps
git clone https://github.com/protobuf-c/protobuf-c.git protobuf-c
cd protobuf-c
./autogen.sh
mkdir ../pbc-`uname -m`
cd ../pbc-`uname -m`
../protobuf-c/configure --prefix=`pwd`/../`uname -m`-linux-gnu \
PKG_CONFIG_PATH=`pwd`/../`uname -m`-linux-gnu/lib/pkgconfig
make
make install
cd ../..
</div>
</div>

<div class="toccolours mw-collapsible mw-collapsed" style="width:800px">
To cross-compile for ARM some more tricks will be required.
<div class="mw-collapsible-content">
For ARMv7

cd deps
mkdir -p pbc-arm
cd pbc-arm
../protobuf-c/configure --host=arm-linux-gnueabihf --prefix=`pwd`/../arm-linux-gnueabihf \
--disable-protoc PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make install PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
cd ../..

For ARM8

cd deps
mkdir -p pbc-aarch64
cd pbc-aarch64
../protobuf-c/configure --host=aarch64-linux-gnu --prefix=`pwd`/../aarch64-linux-gnu \
--disable-protoc PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
make install PATH=`pwd`/../`uname -m`-linux-gnu/bin:$PATH
cd ../..
</div>
</div>

=== Other deps ===

* <code>python-ipaddr</code> is used by CRIT to pretty-print ip.
* If <code>libbsd</code> available, CRIU will be compiled with setproctitle() support. It will allow to make process titles of service workers to be more verbose.
* The iproute2 tool version 3.5.0 or higher is needed for dumping network namespaces. The latest one can be cloned from [http://git.kernel.org/?p=linux/kernel/git/shemminger/iproute2.git;a=summary iproute2]. It should be compiled and a path to ip written in the environment variable <code>CR_IP_TOOL</code>.
* If you would like to use <code>make test</code> you should install <code>libaio-dev</code>.

== Linux Kernel ==

Linux kernel v3.11 or newer is required, with some specific options set. If your distribution does not provide needed kernel, you might want to compile one yourself.

=== Configuring the kernel ===

Most likely the first thing to enable is the <code>CONFIG_EXPERT=y</code> (General setup -> Configure standard kernel features (expert users)) option, which on x86_64 depends on the <code>CONFIG_EMBEDDED=y</code> (General setup -> Embedded system) one (welcome to Kconfig reverse chains hell).

The following options must be enabled for CRIU to work:

* ''General setup'' options
** <code>CONFIG_CHECKPOINT_RESTORE=y</code> (Checkpoint/restore support)
** <code>CONFIG_NAMESPACES=y</code> (Namespaces support)
** <code>CONFIG_UTS_NS=y</code> (Namespaces support -> UTS namespace)
** <code>CONFIG_IPC_NS=y</code> (Namespaces support -> IPC namespace)
** <code>CONFIG_PID_NS=y</code> (Namespaces support -> PID namespaces)
** <code>CONFIG_NET_NS=y</code> (Namespaces support -> Network namespace)
** <code>CONFIG_FHANDLE=y</code> (Open by fhandle syscalls)
** <code>CONFIG_EVENTFD=y</code> (Enable eventfd() system call)
** <code>CONFIG_EPOLL=y</code> (Enable eventpoll support)
* ''Networking support -> Networking options'' options for sock-diag subsystem
** <code>CONFIG_UNIX_DIAG=y</code> (Unix domain sockets -> UNIX: socket monitoring interface)
** <code>CONFIG_INET_DIAG=y</code> (TCP/IP networking -> INET: socket monitoring interface)
** <code>CONFIG_INET_UDP_DIAG=y</code> (TCP/IP networking -> INET: socket monitoring interface -> UDP: socket monitoring interface)
** <code>CONFIG_PACKET_DIAG=y</code> (Packet socket -> Packet: sockets monitoring interface)
** <code>CONFIG_NETLINK_DIAG=y</code> (Netlink socket -> Netlink: sockets monitoring interface)
* Other options
** <code>CONFIG_INOTIFY_USER=y</code> (File systems -> Inotify support for userspace)
** <code>CONFIG_IA32_EMULATION=y</code> (x86 only) (Executable file formats -> Emulations -> IA32 Emulation)

For some [[usage scenarios]] there is an ability to track memory changes and produce [[incremental dumps]]. Need to enable the <code>CONFIG_MEM_SOFT_DIRTY=y</code> (optional) (Processor type and features -> Track memory changes).

Note we also have our [[custom kernel]], which might contain some experimental CRIU related patches.

== Building CRIU From Source ==

=== Native Compilation ===
Simply run <code>make</code> in the CRIU source directory.

=== Compilation in Docker container ===

There's a ''docker-build'' target in Makefile which builds CRIU in Ubuntu Docker container. Just run <code>make docker-build</code> and that's it.

=== Non-standard compilation ===

<div class="toccolours mw-collapsible mw-collapsed" style="width:800px">
Building natively, but specifying built dependencies manually
<div class="mw-collapsible-content">
cd deps
rsync -a --exclude=.git --exclude=deps .. criu-`uname -m`
cd criu-`uname -m`
make \
USERCFLAGS="-I`pwd`/../`uname -m`-linux-gnu/include -L`pwd`/../`uname -m`-linux-gnu/lib" \
PATH="`pwd`/../`uname -m`-linux-gnu/bin:$PATH"
sudo LD_LIBRARY_PATH=`pwd`/../`uname -m`-linux-gnu/lib ./criu check
cd ../..
</div>
</div>

<div class="toccolours mw-collapsible mw-collapsed" style="width:800px">
Cross Compilation for ARM
<div class="mw-collapsible-content">
ARMv7
cd deps
rsync -a --exclude=.git --exclude=deps .. criu-arm
cd criu-arm
make \
ARCH=arm \
CROSS_COMPILE=`pwd`/../`uname -m`-linux-gnu/bin/arm-linux-gnueabihf- \
USERCFLAGS="-I`pwd`/../arm-linux-gnueabihf/include -L`pwd`/../arm-linux-gnueabihf/lib" \
PATH="`pwd`/../`uname -m`-linux-gnu/bin:$PATH"
cd ../..

ARMv8
cd deps
rsync -a --exclude=.git --exclude=deps .. criu-aarch64
cd criu-aarch64
make \
ARCH=aarch64 \
CROSS_COMPILE=`pwd`/../`uname -m`-linux-gnu/bin/aarch64-linux-gnu- \
USERCFLAGS="-I`pwd`/../aarch64-linux-gnu/include -L`pwd`/../aarch64-linux-gnu/lib" \
PATH="`pwd`/../`uname -m`-linux-gnu/bin:$PATH"
cd ../..
</div>
</div>

== Installation ==
CRIU works perfectly even when run from the sources directory (with the "./criu" command), but if you want to have in standard paths run <code>make install</code>.

You may need to install the following packages to generate docs in Debian-based OS's to avoid errors from install-man:
* <code>asciidoc</code>
* <code>xmlto</code>

== Checking That It Works ==

First thing to do is to run <code>criu check --ms</code>.

At the end it should say "Looks OK", if it doesn't the messages on the screen explain what functionality is missing. If you're using our custom kernel, then the <code>--ms</code> option should not be used, in this case CRIU would check for ''all'' the kernel features to work.

You can then try running the [[ZDTM Test Suite]] which sits in the <code>tests/zdtm/</code> directory.

== Further reading ==

Please see [[Usage]] and [[Advanced usage]], as well as [[:Category:HOWTO]].

[[Category:HOWTO]]