Filesystems pecularities - Revision history

Adrian: Added btrfs workaround with nocow

2021-02-17T11:23:29Z

Added btrfs workaround with nocow

Radostin: Improve page description

2018-05-12T00:54:50Z

Improve page description

Xemul: cat files

2015-02-26T15:46:31Z

cat files

Xemul: Add project homepage url

2014-08-21T16:11:08Z

Add project homepage url

Xemul: /* AUFS */

2014-08-21T15:41:56Z

AUFS

Xemul: Added AUFS problem

2014-08-21T15:41:21Z

Added AUFS problem

Xemul: /* NFS */

2014-08-21T15:32:06Z

NFS

Xemul: Added nfs problem

2014-08-21T15:25:43Z

Added nfs problem

Xemul: /* BTRFS */

2014-08-21T14:59:06Z

BTRFS

Xemul: Added btrfs problem

2014-08-21T14:57:49Z

Added btrfs problem

← Older revision		Revision as of 11:23, 17 February 2021
Line 10:		Line 10:

	In order to address this issue, CRIU performs path-to-device resolution in user-space by analysing the information obtained from the <code>/proc/$pid/mountinfo</code> files. The routine in question is <code>mount.c:phys_stat_resolve_dev()</code>.		In order to address this issue, CRIU performs path-to-device resolution in user-space by analysing the information obtained from the <code>/proc/$pid/mountinfo</code> files. The routine in question is <code>mount.c:phys_stat_resolve_dev()</code>.
		+
		+	=== BTRFS Workaround ===
		+
		+	One possible workaround to use BTRFS in combination with CRIU is to disable copy on write (COW). During the discussion in https://github.com/containers/podman/issues/9318 one workaround to use Podman's checkpoint/restore support was: <code>chattr +C /var/lib/containers</code>

	== NFS ==		== NFS ==

@@ Line 1: / Line 1: @@
 "All filesystems are equal, but some filesystems are more equal than others."
-This page describes how different different filesystems are and how this affects CRIU dump (and restore) process.
+This page describes how different filesystems affect the CRIU dump (and restore) process.
 == BTRFS ==

← Older revision		Revision as of 15:46, 26 February 2015
Line 32:		Line 32:

	[[Category:Under the hood]]		[[Category:Under the hood]]
		+	[[Category:Files]]

@@ Line 23: / Line 23: @@
 == AUFS ==
-AUFS is not (yet) upstream, but Docker guys use one, so does CRIU.
+[http://aufs.sourceforge.net AUFS] is not (yet) upstream, but Docker guys use one, so does CRIU.
-AUFS has a strange BUG -- when we <code>execv</code> some file on it, and then check for <code>/proc/$pid/maps</code> or <code>.../smaps</code> files or links in the <code>.../map_files/</code> directory, we would note, that those mappings, that correspond to the executed file are seen under "wrong" paths.
+This FS has a strange BUG -- when we <code>execv</code> some file on it, and then check for <code>/proc/$pid/maps</code> or <code>.../smaps</code> files or links in the <code>.../map_files/</code> directory, we would note, that those mappings, that correspond to the executed file are seen under "wrong" paths.
 How wrong are these paths? AUFS joins several subdirectories into one, all these subdirs are called ''branches''. And the file seen inside AUFS by one path "really" has some other one -- the path by the file is seen in the respective branch. So in proc in these strange cases we would see the path from branch, instead of the path, by which the file is seen in AUFS.

@@ Line 25: / Line 25: @@
 AUFS is not (yet) upstream, but Docker guys use one, so does CRIU.
-AUFS has a strange BUG -- when we <code>execv</code> some file on it, and then check for <code>/proc/$pid/maps</code> or <code>.../smaps</code> files or links in <code>.../map_files/</code> directory, we would note, that those mappings, that correspond to the executed file are seen under "wrong" paths.
+AUFS has a strange BUG -- when we <code>execv</code> some file on it, and then check for <code>/proc/$pid/maps</code> or <code>.../smaps</code> files or links in the <code>.../map_files/</code> directory, we would note, that those mappings, that correspond to the executed file are seen under "wrong" paths.
 How wrong are these paths? AUFS joins several subdirectories into one, all these subdirs are called ''branches''. And the file seen inside AUFS by one path "really" has some other one -- the path by the file is seen in the respective branch. So in proc in these strange cases we would see the path from branch, instead of the path, by which the file is seen in AUFS.

← Older revision		Revision as of 15:41, 21 August 2014
Line 22:		Line 22:

	== AUFS ==		== AUFS ==
		+
		+	AUFS is not (yet) upstream, but Docker guys use one, so does CRIU.
		+
		+	AUFS has a strange BUG -- when we <code>execv</code> some file on it, and then check for <code>/proc/$pid/maps</code> or <code>.../smaps</code> files or links in <code>.../map_files/</code> directory, we would note, that those mappings, that correspond to the executed file are seen under "wrong" paths.
		+
		+	How wrong are these paths? AUFS joins several subdirectories into one, all these subdirs are called ''branches''. And the file seen inside AUFS by one path "really" has some other one -- the path by the file is seen in the respective branch. So in proc in these strange cases we would see the path from branch, instead of the path, by which the file is seen in AUFS.
		+
		+	This is problematic, since CRIU needs to know the path by which the file was accessed inside AUFS in order to properly restore one. To fix this, CRIU checks that there's an AUFS mount in the game, reads the branches info from sysfs, and then, when meets an AUFS file in mappings, check for the path to belong to one of the branches and "fixes" one. This has sits in the <code>sysfs_parse.c:fixup_aufs_vma_fd</code>.

	[[Category:Under the hood]]		[[Category:Under the hood]]

@@ Line 15: / Line 15: @@
 In Linux files have an attribute called <code>st_nlink</code> -- the amount of names the file has. When a file is removed (which is done with the <code>unlink</code> system call) this counter is decremented and, if it hits zero, the file itself can be removed from disk. Not "is removed", but "can be removed", since the file can be held opened while someone unlinks one. In the latter case the physical removal of the file is delayed till the file is closed.
-NFS does special handing in case the nlink value is about to hit zero. The thing is -- if NFS client would send the last unlink request to the server, the latter would just go and kill the file physically, since it doesn't "know" that someone holds this file opened (this information is owned by the client). Instead, client marks the file as "to be deleted on close" and doesn't perform the last nlink decrement immediately. And only when the file is closed and the mentioned flag is seen, the last unlink is sent to the server. And one more thing -- to prevent naming collisions (in simple words -- <code>open()</code> by the old name shouldn't file this old file) NFS also renames the file, giving it a special name ".nfsXXX" where XXX is some unique identifier.
+NFS does special handing in case the nlink value is about to hit zero. The thing is -- if NFS client would send the last unlink request to the server, the latter would just go and kill the file physically, since it doesn't "know" that someone holds this file opened (this information is owned by the client). Instead, client marks the file as "to be deleted on close" and doesn't perform the last nlink decrement immediately. And only when the file is closed and the mentioned flag is seen, the last unlink is sent to the server. And one more thing -- to prevent naming collisions (in simple words -- <code>open()</code> by the old name shouldn't file this old file) NFS also renames the file, giving it a special name ".nfsXXX" where XXX is some unique identifier. This trick is called "NFS silly rename".
-How this affects CRIU? In the article "[[How hard is it to open a file]]?" it's said, that CRIU should be able to dump and restore files, that are unlinked, but opened. Briefly: if a file is such, CRIU cannot just save the file's path, as once dumped tasks are killed, the fill would stop existing. So CRIU takes these files into the images. But on NFS there's no such thing as "unlinked" file -- it prevents the nlink count from dropping to zero. For CRIU all NFS files look as alive ones.
+How does this affect CRIU? In the article "[[How hard is it to open a file]]?" it's said, that CRIU should be able to dump and restore files, that are unlinked, but opened. Briefly: if a file is such, CRIU cannot just save the file's path, as once dumped tasks are killed, the fill would stop existing. So CRIU takes these files into the images. But on NFS there's no such thing as "unlinked" file -- it prevents the nlink count from dropping to zero. For CRIU all NFS files look as alive ones.
-To handle this, CRIU checks that a file it dumps resides on NFS (this is simply by checking the <code>statfs</code>'s <code>fs_type</code> field). If the file is such CRIU then checks its name to be the ".nfsXXX" one. If both checks succeed the file is treated as "opened and unlinked" one.
+To handle this, CRIU checks that a file it dumps resides on NFS (this is simply by checking the <code>statfs</code>'s <code>fs_type</code> field). If the file is such CRIU then checks its name to be the silly-renamed one. If both checks succeed the file is treated as "opened and unlinked" one. The code in question is the <code>files-reg.c:nfs_silly_rename()</code>.
 == AUFS ==
 [[Category:Under the hood]]

@@ Line 7: / Line 7: @@
 When we <code>stat()</code> a file we can get on which device it resides by checking the <code>st_dev</code> value. However, kernel exposes the device value in some more places. In particular, the device is shown in the <code>/proc/$pid/[mounts|mountinfo]</code> files and in the <code>/proc/$pid/s?maps</code> ones. Moreover, the sock-diag subsystem recently added into the kernel, reveals the device (and inode) on which a unix socket is bound.
-The problem with btrfs is that it substitutes the real device number with a virtual one in the <code>stat()</code> system call. And once we get this value we cannot compare it to any other device number obtained from other sources, they will also differ (these virtual device numbers are unique).
+The problem with btrfs is that it substitutes the real device number with a virtual one in the <code>stat()</code> system call. And once we get this value we cannot compare it to any other device number obtained from other sources, they will always differ (these virtual device numbers are unique).
-In order to address this issue, CRIU performs path-to-device resolution in user-space by analysing the information obtained from the <code>/proc/$pid/mountinfo</code> files.
+In order to address this issue, CRIU performs path-to-device resolution in user-space by analysing the information obtained from the <code>/proc/$pid/mountinfo</code> files. The routine in question is <code>mount.c:phys_stat_resolve_dev()</code>.
 == NFS ==

← Older revision		Revision as of 14:57, 21 August 2014
Line 4:		Line 4:

	== BTRFS ==		== BTRFS ==
		+
		+	When we <code>stat()</code> a file we can get on which device it resides by checking the <code>st_dev</code> value. However, kernel exposes the device value in some more places. In particular, the device is shown in the <code>/proc/$pid/[mounts\|mountinfo]</code> files and in the <code>/proc/$pid/s?maps</code> ones. Moreover, the sock-diag subsystem recently added into the kernel, reveals the device (and inode) on which a unix socket is bound.
		+
		+	The problem with btrfs is that it substitutes the real device number with a virtual one in the <code>stat()</code> system call. And once we get this value we cannot compare it to any other device number obtained from other sources, they will also differ (these virtual device numbers are unique).
		+
		+	In order to address this issue, CRIU performs path-to-device resolution in user-space by analysing the information obtained from the <code>/proc/$pid/mountinfo</code> files.

	== NFS ==		== NFS ==