Difference between revisions of "Inheriting FDs on restore"

From CRIU
Jump to navigation Jump to search
(added to cli category)
 
(8 intermediate revisions by 3 users not shown)
Line 1: Line 1:
This HOWTO page describes how to use the --inherit-fd command line option.
+
This article describes why and how to use the <code>--external file[...]</code>, <code>--external tty[...]</code> and <code>--inherit-fd</code> command line options.
  
 
== Background ==
 
== Background ==
  
There are cases where a process's file descriptor cannot be restored
+
There are cases where a process' file descriptor cannot be restored
 
from the checkpoint images.  For example, a pipe file descriptor with one
 
from the checkpoint images.  For example, a pipe file descriptor with one
 
end in the checkpointed process and the other end in a separate process
 
end in the checkpointed process and the other end in a separate process
Line 16: Line 16:
 
In these cases, criu's caller should set up a new file descriptor to be
 
In these cases, criu's caller should set up a new file descriptor to be
 
inherited by the restored process and specify the file descriptor with
 
inherited by the restored process and specify the file descriptor with
the --inherit-fd command line option.
+
the <code>--inherit-fd</code> command line option.
  
== Warning ==
+
== Limitations ==
  
 
Please note that inherit fd support breaks applications that depend
 
Please note that inherit fd support breaks applications that depend
Line 26: Line 26:
 
at a different seek offset.
 
at a different seek offset.
  
You should consider inherit fd only for specific use cases that you know
+
You should consider using this feature only for specific use cases that you know
 
for sure won't break the application.  In other words, use it at your
 
for sure won't break the application.  In other words, use it at your
 
own risk.
 
own risk.
Line 32: Line 32:
 
== Argument Format ==
 
== Argument Format ==
  
The argument of --inherit-fd has the format fd[%d]:%s, where %d tells
+
The argument of <code>--inherit-fd</code> has the format <code>fd[%d]:%s</code>,
criu which of its own file descriptors to use for restoring the file
+
where <code>%d</code> tells criu which of its own file descriptors to use
identified by %s.
+
for restoring the file identified by <code>%s</code>. Note that the file descriptor
 +
number should be enclosed in literal square brackets (and as square brackets are
 +
handled specially by the command line shell, they might need to be properly escaped).
  
== Debugging Aid ==
+
Example: <code>criu restore ... --inherit-fd 'fd[7]:tmp/old' ...</code>
  
As a debugging aid, if the argument has the format debug[%d]:%s, criu
+
== Debugging aid ==
will just write out the string after colon to the file descriptor %d.
+
 
 +
As a debugging aid, if the argument has the format <code>debug[%d]:%s</code>, criu
 +
will just write out the string after colon to the file descriptor <code>%d</code>.
 
This can be used to leave a "restore marker" in the output stream of
 
This can be used to leave a "restore marker" in the output stream of
 
the process.
 
the process.
  
== Example 1 - Regular Files ==
+
== Examples ==
 +
 
 +
=== Regular files ===
  
Let's redirect the output of test.sh from [[Simple loop]]
+
Let's redirect the output of <code>test.sh</code> from the [[simple loop]] example
to /tmp/old, checkpoint it, and restore it to use /tmp/new as its
+
to <code>/tmp/old</code>, checkpoint it, and restore it to use <code>/tmp/new</code> as its
 
output file.
 
output file.
  
 
As you see below, we have used criu's file descriptor 7 (just as an
 
As you see below, we have used criu's file descriptor 7 (just as an
arbitrary descriptor) with the --inherit-fd option. Notice that the path
+
arbitrary descriptor) with the <code>--inherit-fd</code> option. Note that the file path
in the argument of --inherit-fd is relative to the root of the process
+
in the argument of <code>--inherit-fd</code> is relative to the root of the process
(i.e., tmp/old).
+
(i.e., <code>tmp/old</code>).
  
 
<pre>
 
<pre>
Line 62: Line 68:
  
 
This is an example of an application that does not depend on the state
 
This is an example of an application that does not depend on the state
of its output file, so --inherit-fd wouldn't break it.
+
of its output file, so <code>--inherit-fd</code> wouldn't break it.
  
 
If your application depends on a particular state, say the seek offset,
 
If your application depends on a particular state, say the seek offset,
Line 68: Line 74:
 
descriptor.
 
descriptor.
  
== Example 2 - External Unnamed Pipes ==
+
=== External unnamed pipes ===
  
 
To replace an external pipe that was connected to an external process
 
To replace an external pipe that was connected to an external process
Line 88: Line 94:
 
simplicity, we're having the parent checkpoint the child.
 
simplicity, we're having the parent checkpoint the child.
  
The source code for pipe.c can be found in criu source tree under
+
The source code for <code>pipe.c</code> can be found in criu source tree under
test/pipes.  Each output line is preceded by the process that has
+
<code>test/pipes</code>.  Each output line is preceded by the process that has
generated it.  Notice pipe:[472728] is replaced with pipe:[474324]
+
generated it.  Notice <code>pipe:[472728]</code> is replaced with <code>pipe:[474324]</code>
passed through criu's fd[3].
+
passed through criu's <code>fd[3]</code>.
  
 
<pre>
 
<pre>
Line 120: Line 126:
 
</pre>
 
</pre>
  
== Example 3 - External files, FIFO-s ==
+
=== External files, FIFOs ===
"criu dump" tries to resolve paths for each files, so the first example works only for files which can be resolved from dumped mount namespaces. If we have a file from another namespace, we call it as "external" and criu isn't able to restore it without external help. We need to enumerate all external files on dump and set inherit file descriptors on restore. This descriptors will be used only to open a file via /proc/self/fd, so it doesn't metter with which flags an inherited descriptor has been opened. The format of file id is file[mnt_id:inode].
+
"criu dump" tries to resolve paths for each files, so the first example works only for files which can be resolved from dumped mount namespaces. If we have a file from another namespace, we call it as "external" and criu isn't able to restore it without external help. We need to enumerate all external files on dump and set inherit file descriptors on restore. This descriptors will be used only to open a file via <code>/proc/self/fd</code>, so it doesn't matter with which flags an inherited descriptor has been opened. The format of file id is <code>file[''mnt_id'':''inode'']</code>.
  
 
<pre>
 
<pre>
Line 128: Line 134:
 
</pre>
 
</pre>
  
== Example 4 - External TTY-s ==
+
=== External TTYs ===
  
The format of tty id is tty[rdev:dev]. Why can we not use a pair of mnt_id and inode here?
+
The format of tty id is <code>tty[''rdev'':''dev'']</code>. Note that a pair of <code>''mnt_id'':''inode''</code> is not used here, as there can be two device files with the same <code>''rdev''</code> but different inode numbers.
  
 +
Here is an example of dumping and restoring an external TTY:
 
<pre>
 
<pre>
     $ ipython
+
     # setsid --ctty ipython
 
     In [1]: import os
 
     In [1]: import os
 
     In [2]: st = os.stat("/proc/self/fd/0")
 
     In [2]: st = os.stat("/proc/self/fd/0")
     In [3]: print "tty[%x:%x]" % (st.st_rdev, st.st_dev)
+
     In [3]: print("tty[%x:%x]" % (st.st_rdev, st.st_dev))
     tty:[8800:d]
+
     tty[8802:19]
   
+
    $ps -C sleep
+
     # criu dump -t `pgrep ipython` --external 'tty[8802:19]'
      PID TTY          TIME CMD
+
     # criu restore --inherit-fd 'fd[1]:tty[8802:19]'
    4109 ?        00:00:00 sleep
 
      
 
    $ ./criu dump --external 'tty[8800:d]' -D imgs -v4 -t 4109
 
     $ ./criu restore --inherit-fd 'fd[1]:tty[8800:d]' -D imgs -v4
 
 
</pre>
 
</pre>
  
Line 150: Line 153:
 
[[Category:HOWTO]]
 
[[Category:HOWTO]]
 
[[Category:API]]
 
[[Category:API]]
[[Category:CLI]]
 
 
[[Category:Files]]
 
[[Category:Files]]
 
[[Category:External]]
 
[[Category:External]]

Latest revision as of 12:19, 19 July 2020

This article describes why and how to use the --external file[...], --external tty[...] and --inherit-fd command line options.

Background[edit]

There are cases where a process' file descriptor cannot be restored from the checkpoint images. For example, a pipe file descriptor with one end in the checkpointed process and the other end in a separate process (that was not part of the checkpointed process tree) cannot be restored because after checkpoint the pipe will be broken.

There are also cases where the user wants to use a new file during restore instead of the original file at checkpoint time. For example, the user wants to change the log file of a process from /path/to/oldlog to /path/to/newlog.

In these cases, criu's caller should set up a new file descriptor to be inherited by the restored process and specify the file descriptor with the --inherit-fd command line option.

Limitations[edit]

Please note that inherit fd support breaks applications that depend on the state of the file descriptor being inherited. For example, an application that depends on the seek offset within a file at checkpoint time will fail after restore if the file is replaced with another file at a different seek offset.

You should consider using this feature only for specific use cases that you know for sure won't break the application. In other words, use it at your own risk.

Argument Format[edit]

The argument of --inherit-fd has the format fd[%d]:%s, where %d tells criu which of its own file descriptors to use for restoring the file identified by %s. Note that the file descriptor number should be enclosed in literal square brackets (and as square brackets are handled specially by the command line shell, they might need to be properly escaped).

Example: criu restore ... --inherit-fd 'fd[7]:tmp/old' ...

Debugging aid[edit]

As a debugging aid, if the argument has the format debug[%d]:%s, criu will just write out the string after colon to the file descriptor %d. This can be used to leave a "restore marker" in the output stream of the process.

Examples[edit]

Regular files[edit]

Let's redirect the output of test.sh from the simple loop example to /tmp/old, checkpoint it, and restore it to use /tmp/new as its output file.

As you see below, we have used criu's file descriptor 7 (just as an arbitrary descriptor) with the --inherit-fd option. Note that the file path in the argument of --inherit-fd is relative to the root of the process (i.e., tmp/old).

	$ ./test.sh > /tmp/old &
	<pid>
	$ sudo criu dump -j -t <pid>
	$ sudo criu restore -d -j --inherit-fd 'fd[7]:tmp/old' 7> /tmp/new

This is an example of an application that does not depend on the state of its output file, so --inherit-fd wouldn't break it.

If your application depends on a particular state, say the seek offset, you should seek to that offset before you let criu inherit the new descriptor.

External unnamed pipes[edit]

To replace an external pipe that was connected to an external process is more involved because unnamed pipes are typically created between a parent and a child process. Therefore, the external process has to invoke criu as its child and set up a new pipe between itself and criu to be inherited by the restored process.

To demonstrate how this is done, consider a parent process that spawns a child which will write messages to its parent through a pipe. When a couple of messages has been received by parent, it invokes criu to checkpoint the child. As a result of checkpoint, the child exits and the pipe will be broken. Then, the parent sets up a new pipe between itself and criu and invokes it to restore the child using the new pipe (instead of the old one).

Note that unlike restoring, checkpointing the child doesn't have to be done by the parent (i.e., can be done manually by the user). But for simplicity, we're having the parent checkpoint the child.

The source code for pipe.c can be found in criu source tree under test/pipes. Each output line is preceded by the process that has generated it. Notice pipe:[472728] is replaced with pipe:[474324] passed through criu's fd[3].

$ cc -Wall -o pipe pipe.c
$ sudo ./pipe
[child 26371] writing hello 1 to pipe:[472728] via fd 4
[parent 26370] read hello 1 from pipe:[472728]

[child 26371] writing hello 2 to pipe:[472728] via fd 4
[parent 26370] read hello 2 from pipe:[472728]

[dump 26372] criu dump -D /tmp/criu_img -o dump.log -v4 -j -t 26371 
[parent 26370] [child 26371] exited with status 9
[parent 26370] [dump 26372] exited with status 0
[parent 26370] creating a new pipe

[restore 26378] criu restore -d -D /tmp/criu_img -o restore.log --pidfile restore.pid -v4 -j --inherit-fd fd[3]:pipe:[472728]   
[parent 26370] [restore 26378] exited with status 0

[child 26371] writing hello 3 to pipe:[474324] via fd 4
[parent 26370] read hello 3 from pipe:[474324]

[child 26371] writing hello 4 to pipe:[474324] via fd 4
[parent 26370] read hello 4 from pipe:[474324]

[parent 26370] [child 26371] exited with status 0
$

External files, FIFOs[edit]

"criu dump" tries to resolve paths for each files, so the first example works only for files which can be resolved from dumped mount namespaces. If we have a file from another namespace, we call it as "external" and criu isn't able to restore it without external help. We need to enumerate all external files on dump and set inherit file descriptors on restore. This descriptors will be used only to open a file via /proc/self/fd, so it doesn't matter with which flags an inherited descriptor has been opened. The format of file id is file[mnt_id:inode].

    criu dump --external file[72:a3e7]
    criu restore --inherit-fd fd[4]:file[72:a3e7]

External TTYs[edit]

The format of tty id is tty[rdev:dev]. Note that a pair of mnt_id:inode is not used here, as there can be two device files with the same rdev but different inode numbers.

Here is an example of dumping and restoring an external TTY:

    # setsid --ctty ipython
    In [1]: import os
    In [2]: st = os.stat("/proc/self/fd/0")
    In [3]: print("tty[%x:%x]" % (st.st_rdev, st.st_dev))
    tty[8802:19]
 
    # criu dump -t `pgrep ipython` --external 'tty[8802:19]'
    # criu restore --inherit-fd 'fd[1]:tty[8802:19]'