Shared memory - Revision history

Sungjaecho97: /* Checkpoint */ twice 'the'

2018-01-10T08:02:26Z

Checkpoint: twice 'the'

Xemul at 12:51, 13 July 2017

2017-07-13T12:51:09Z

Kir: some rewording

2016-09-08T22:11:02Z

some rewording

Xemul: /* Checkpoint */

2016-09-08T18:49:12Z

Checkpoint

Xemul: /* Checkpoint */

2016-09-08T18:45:52Z

Checkpoint

Kir: rm non-anon memory

2016-09-08T18:44:38Z

rm non-anon memory

Kir: /* Dumping present pages */ rewording

2016-09-02T19:43:44Z

Dumping present pages: rewording

Kir: /* See also */ use a list

2016-09-02T19:39:56Z

Kir: /* Changes tracking */ slight rewording

2016-09-02T19:39:39Z

Changes tracking: slight rewording

Kir: /* Restore */ OK this is the way it works. Perhaps we need a picture here.

2016-09-02T19:37:15Z

Restore: OK this is the way it works. Perhaps we need a picture here.

@@ Line 25: / Line 25: @@
 Note that different processes can map different parts of a shared memory segment.
 In this case, CRIU first collects mapping offsets and lengths from all the processes
-to determine the the total segment size, then reads all the parts contents
+to determine the total segment size, then reads all the parts contents
 from the respective processes.

← Older revision		Revision as of 12:51, 13 July 2017
Line 90:		Line 90:
	[[Category:Memory]]		[[Category:Memory]]
	[[Category:Under the hood]]		[[Category:Under the hood]]
		+	[[Category:Editor help needed]]

@@ Line 1: / Line 1: @@
-This articles describes some intricacies of handling shared memory mappings,
+Every process has one or more memory mappings, i.e. regions of virtual memory it allows to use.
-i.e. anonymous (not file-based) mappings that are shared between a few processes.
+Some such mappings can be shared between a few processes, and they are called shared mappings.
 == Checkpoint ==
 During the checkpointing, CRIU needs to figure out all the shared mappings in order to dump them as such.
 It does so by calling <code>fstatat()</code> on each entry found in the <code>/proc/$PID/map_files/</code>,
-noting the ''device:inode'' pair of the structure returned by fstatat(). Now, if some processes have a
+noting the ''device:inode'' pair of the structure returned by <code>fstatat()</code>. Now, if some processes
-mapping with the same ''device:inode'' pair, this mapping is marked as shared between them and is dumped as such.
+have a mapping with the same ''device:inode'' pair, this mapping is marked as shared between these processes
-Dumping a mapping means writing an entry into proceess' mm.img file and storing its contents. For shared
+Note that <code>fstatat()</code> works because the kernel actually creates a hidden
-mapping the contents is stored into pagemap-shmem.img and pages.img pair of images (see [[Memory dumps]]).
+tmpfs file, not visible from any tmpfs mounts, but accessible via its
-It's important to note that the above mechanism works not just for the file-based mappings,
+Dumping a mapping means two things:
-but also for the anonymous ones. For an anonymous mapping, kernel actually creates a hidden
+* writing an entry into process' mm.img file;
-tmpfs file, and so <code>fstatat()</code> on the <code>/proc/$PID/map_files/</code> entry
+* storing the actual mapping data (contents).
-works the same way as for other files. The tmpfs file itself is not visible from any tmpfs
+For shared mappings, the contents is stored into a pair of image files: pagemap-shmem.img and pages.img.
-mounts, but can be opened via its <code>map_files</code> entry.
+For details, see [[Memory dumps]].
 == Restore ==

@@ Line 5: / Line 5: @@
 Every process has one or more so called mmapings -- regions of virtual memory which it's allowed to use. Some mappings can be shared between a few processes.
-During the checkpointing, CRIU need to figure out all the shared mappings in order to dump them as such.
+During the checkpointing, CRIU needs to figure out all the shared mappings in order to dump them as such.
-It does so by performing <code>fstatat()</code> for each entry in <code>/proc/$PID/map_files/</code>,
+It does so by calling <code>fstatat()</code> on each entry found in the <code>/proc/$PID/map_files/</code>,
-noting the ''device'' and ''inode'' fields of the structure returned by fstatat(). This information
+noting the ''device:inode'' pair of the structure returned by fstatat(). Now, if some processes have a
-is collected and sorted. Now, if any few processes have a mapping with same ''device'' and ''inode'',
+mapping with the same ''device:inode'' pair, this mapping is marked as shared between them and is dumped as such.
-this mapping is a shared one and should be dumped as such.
 It's important to note that the above mechanism works not just for the file-based mappings,

@@ Line 4: / Line 4: @@
 == Checkpoint ==
-Every process has one or more mmaped files. Some mappings can be shared between a few processes.
+Every process has one or more so called mmapings -- regions of virtual memory which it's allowed to use. Some mappings can be shared between a few processes.
 During the checkpointing, CRIU need to figure out all the shared mappings in order to dump them as such.

@@ Line 1: / Line 1: @@
-This articles describes some intricacies of handling shared memory mappings, i.e. mappings that are shared between a few processes.
+This articles describes some intricacies of handling shared memory mappings,
 == Checkpoint ==
-Every process has one or more mmaped files. Some mappings (for example, ones of shared libraries)
+Every process has one or more mmaped files. Some mappings can be shared between a few processes.
-are shared between a few processes. During the checkpointing, CRIU need to figure out
+During the checkpointing, CRIU need to figure out all the shared mappings in order to dump them as such.
 It does so by performing <code>fstatat()</code> for each entry in <code>/proc/$PID/map_files/</code>,
@@ Line 20: / Line 20: @@
 == Restore ==
-During the restore, CRIU already knows which mappings are shared, so they need to be restored as shared.
+During the restore, CRIU already knows which mappings are shared, so they need to be
-To restore file mappings, no tricks are needed, they are opened and mmaped with with a MAP_SHARED flag set.
+restored as such. Here is how it is done.
 Among all the processes sharing a mapping, the one with the lowest PID among the group

@@ Line 66: / Line 66: @@
 == Dumping present pages ==
-When dumping the contents of shared memory CRIU doesn't dump all the data. Instead, it determines which pages contain
+When dumping the contents of shared memory, CRIU does not dump all of the data. Instead, it determines which pages contain
-it and dumps only them. This is done similarly to how regular [[memory dumping and restoring]] works, i.e. by analyzing
+it, and only dumps those pages. This is done similarly to how regular [[memory dumping and restoring]] works, i.e. by looking
-the owners' pagemap entries for PRESENT or SWAPPED bits. But there's one feature of shmem dumps -- sometimes shmem
+for PRESENT or SWAPPED bits in owners' pagemap entries.
-page can exist in the kernel, but not mapped to any process. In this case criu detects one by calling mincore() on
-the shmem segment, which reports back the page in-memory status. And the mincore bitmap is AND-ed with the per-process
+There is one particular feature of shared memory dumps worth mentioning. Sometimes, a shared memory page
-ones.
+can exist in the kernel, but it is not mapped to any process. CRIU detects such pages by calling mincore()
 == See also ==

@@ Line 75: / Line 75: @@
 == See also ==
-[[Memory dumping and restoring]]
+* [[Memory dumping and restoring]]
+* [[Memory images deduplication]]
 [[Category:Memory]]
 [[Category:Under the hood]]

@@ Line 60: / Line 60: @@
 == Changes tracking ==
-For [[iterative migration]] it's very useful to track changes in memory. Until 2.5 changes were tracked for anonymous memory only, but now criu does this for shared memory as well. To do it criu scans all the shmem segment owners' pagemap (as it does for anon memory) and then AND-s the collected soft-dirty bits.
+For [[iterative migration]] it's very useful to track changes in memory. Until CRIU v2.5, changes were tracked for anonymous memory only, but now it is also shared memory can be tracked as well. To achieve it, CRIU scans all the shmem segment owners' pagemap (as it does for anonymous memory) and then ANDs the collected soft-dirty bits.
-The changes tracking made developers implement [[Memory images deduplication]] for shmem segments as well.
+The changes tracking caused developers to implement [[memory images deduplication]] for shmem segments as well.
 == Dumping present pages ==

@@ Line 20: / Line 20: @@
 == Restore ==
-Upon restore, CRIU already knows which mappings are shared, and the trick is to restore them as such.
+During the restore, CRIU already knows which mappings are shared, so they need to be restored as shared.
-For that, two different approaches are used, depending on the availability.
+To restore file mappings, no tricks are needed, they are opened and mmaped with with a MAP_SHARED flag set.
-The common part is, between the processes sharing a mapping, the one with the lowest PID
+Anonymous memory mappings, though, need some work to be restored as such. Here is how it is done.
-=== memfd ===
+Among all the processes sharing a mapping, the one with the lowest PID among the group
-Linux kernel v3.17 adds a [http://man7.org/linux/man-pages/man2/memfd_create.2.html memfd_create()]
+First, the creator need to obtain a file descriptor for the mapping. To achieve it, two different
-syscall. CRIU restore checks if it is available from the running kernel; it yes, it is used.
+approaches are used, depending on the availability.
-FIXME how
+In case [http://man7.org/linux/man-pages/man2/memfd_create.2.html memfd_create()]
-HOW: The memfd in question is created in the task with lowest PID (see [[postulates]]) among those having this shmem segment
+If <code>memfd_create()</code> is not available, the alternative approach is used.
-mapped, then criu waits for the others to get this file by opening the creator's /proc/pid/fd/ link.
+First, mmap() is called to create a mapping. Next, a file in <code>/proc/self/map_files/</code>
-Afterwards all the files just mmap() this descriptor into their address space.
+is opened to get a file descriptor for the mapping. The limitation of this method is,
-=== /proc/$PID/map_files/ ===
+Once the creator have the file descriptor, it mmap()s it and restores its content from
+the dump (using memcpy()). The creator then unmaps the the mapping (note the file
-This method is used if memfd is not available. The limitation is, /proc/$PID/map_files/ is not available
+descriptor is still available). Next, it calls futex(FUTEX_WAKE) to signal all the
-for users inside user namespaces (due to security concerns), so it's not possible to use it if there
+waiting processes that the mapping file descriptor is ready.
-FIXME how
+All the other processes that need this mapping wait on futex(FUTEX_WAIT). Once the
-HOW: The same technique as with memfd is used, with two exceptions. First is that creator calls mmap()
+Finally, all the processes (including the creator itself) call mmap() to create a
-not memfd_create() and creates the shared memory at once. Then it waits for the others to open its
+needed mapping (note that mmap() arguments such as length, offset and flags may
-/proc/pid/map_files/ link. After opening "the others" mmap() one to their address space just as if
+differ for different processes), and close() the mapping file descriptor as it is
-they would have done it with memfd descriptor.
+no longer needed.
 == Changes tracking ==