<link>http://people.skolelinux.org/pere/blog/</link>
<atom:link href="http://people.skolelinux.org/pere/blog/index.rss" rel="self" type="application/rss+xml" />
+ <item>
+ <title>Speeding up the Debian installer using eatmydata and dpkg-divert</title>
+ <link>http://people.skolelinux.org/pere/blog/Speeding_up_the_Debian_installer_using_eatmydata_and_dpkg_divert.html</link>
+ <guid isPermaLink="true">http://people.skolelinux.org/pere/blog/Speeding_up_the_Debian_installer_using_eatmydata_and_dpkg_divert.html</guid>
+ <pubDate>Tue, 16 Sep 2014 14:00:00 +0200</pubDate>
+ <description><p>The <a href="https://www.debian.org/">Debian</a> installer could be
+a lot quicker. When we install more than 2000 packages in
+<a href="http://www.skolelinux.org/">Skolelinux / Debian Edu</a> using
+tasksel in the installer, unpacking the binary packages take forever.
+A part of the slow I/O issue was discussed in
+<a href="https://bugs.debian.org/613428">bug #613428</a> about too
+much file system sync-ing done by dpkg, which is the package
+responsible for unpacking the binary packages. Other parts (like code
+executed by postinst scripts) might also sync to disk during
+installation. All this sync-ing to disk do not really make sense to
+me. If the machine crash half-way through, I start over, I do not try
+to salvage the half installed system. So the failure sync-ing is
+supposed to protect against, hardware or system crash, is not really
+relevant while the installer is running.</p>
+
+<p>A few days ago, I thought of a way to get rid of all the file
+system sync()-ing in a fairly non-intrusive way, without the need to
+change the code in several packages. The idea is not new, but I have
+not heard anyone propose the approach using dpkg-divert before. It
+depend on the small and clever package
+<a href="https://packages.qa.debian.org/eatmydata">eatmydata</a>, which
+uses LD_PRELOAD to replace the system functions for syncing data to
+disk with functions doing nothing, thus allowing programs to live
+dangerous while speeding up disk I/O significantly. Instead of
+modifying the implementation of dpkg, apt and tasksel (which are the
+packages responsible for selecting, fetching and installing packages),
+it occurred to me that we could just divert the programs away, replace
+them with a simple shell wrapper calling
+"eatmydata&nbsp;$program&nbsp;$@", to get the same effect.
+Yesterday I decided to test the idea, and wrapped up a simple
+implementation for the Debian Edu udeb.</p>
+
+<p>The effect was stunning. In my first test it reduced the running
+time of the pkgsel step (installing tasks) from 64 to less than 44
+minutes (20 minutes shaved off the installation) on an old Dell
+Latitude D505 machine. I am not quite sure what the optimised time
+would have been, as I messed up the testing a bit, causing the debconf
+priority to get low enough for two questions to pop up during
+installation. As soon as I saw the questions I moved the installation
+along, but do not know how long the question were holding up the
+installation. I did some more measurements using Debian Edu Jessie,
+and got these results. The time measured is the time stamp in
+/var/log/syslog between the "pkgsel: starting tasksel" and the
+"pkgsel: finishing up" lines, if you want to do the same measurement
+yourself. In Debian Edu, the tasksel dialog do not show up, and the
+timing thus do not depend on how quickly the user handle the tasksel
+dialog.</p>
+
+<p><table>
+
+<tr>
+<th>Machine/setup</th>
+<th>Original tasksel</th>
+<th>Optimised tasksel</th>
+<th>Reduction</th>
+</tr>
+
+<tr>
+<td>Latitude D505 Main+LTSP LXDE</td>
+<td>64 min (07:46-08:50)</td>
+<td><44 min (11:27-12:11)</td>
+<td>>20 min 18%</td>
+</tr>
+
+<tr>
+<td>Latitude D505 Roaming LXDE</td>
+<td>57 min (08:48-09:45)</td>
+<td>34 min (07:43-08:17)</td>
+<td>23 min 40%</td>
+</tr>
+
+<tr>
+<td>Latitude D505 Minimal</td>
+<td>22 min (10:37-10:59)</td>
+<td>11 min (11:16-11:27)</td>
+<td>11 min 50%</td>
+</tr>
+
+<tr>
+<td>Thinkpad X200 Minimal</td>
+<td>6 min (08:19-08:25)</td>
+<td>4 min (08:04-08:08)</td>
+<td>2 min 33%</td>
+</tr>
+
+<tr>
+<td>Thinkpad X200 Roaming KDE</td>
+<td>19 min (09:21-09:40)</td>
+<td>15 min (10:25-10:40)</td>
+<td>4 min 21%</td>
+</tr>
+
+</table></p>
+
+<p>The test is done using a netinst ISO on a USB stick, so some of the
+time is spent downloading packages. The connection to the Internet
+was 100Mbit/s during testing, so downloading should not be a
+significant factor in the measurement. Download typically took a few
+seconds to a few minutes, depending on the amount of packages being
+installed.</p>
+
+<p>The speedup is implemented by using two hooks in
+<a href="https://www.debian.org/devel/debian-installer/">Debian
+Installer</a>, the pre-pkgsel.d hook to set up the diverts, and the
+finish-install.d hook to remove the divert at the end of the
+installation. I picked the pre-pkgsel.d hook instead of the
+post-base-installer.d hook because I test using an ISO without the
+eatmydata package included, and the post-base-installer.d hook in
+Debian Edu can only operate on packages included in the ISO. The
+negative effect of this is that I am unable to activate this
+optimization for the kernel installation step in d-i. If the code is
+moved to the post-base-installer.d hook, the speedup would be larger
+for the entire installation.</p>
+
+<p>I've implemented this in the
+<a href="https://packages.qa.debian.org/debian-edu-install">debian-edu-install</a>
+git repository, and plan to provide the optimization as part of the
+Debian Edu installation. If you want to test this yourself, you can
+create two files in the installer (or in an udeb). One shell script
+need do go into /usr/lib/pre-pkgsel.d/, with content like this:</p>
+
+<p><blockquote><pre>
+#!/bin/sh
+set -e
+. /usr/share/debconf/confmodule
+info() {
+ logger -t my-pkgsel "info: $*"
+}
+error() {
+ logger -t my-pkgsel "error: $*"
+}
+override_install() {
+ apt-install eatmydata || true
+ if [ -x /target/usr/bin/eatmydata ] ; then
+ for bin in dpkg apt-get aptitude tasksel ; do
+ file=/usr/bin/$bin
+ # Test that the file exist and have not been diverted already.
+ if [ -f /target$file ] ; then
+ info "diverting $file using eatmydata"
+ printf "#!/bin/sh\neatmydata $bin.distrib \"\$@\"\n" \
+ > /target$file.edu
+ chmod 755 /target$file.edu
+ in-target dpkg-divert --package debian-edu-config \
+ --rename --quiet --add $file
+ ln -sf ./$bin.edu /target$file
+ else
+ error "unable to divert $file, as it is missing."
+ fi
+ done
+ else
+ error "unable to find /usr/bin/eatmydata after installing the eatmydata pacage"
+ fi
+}
+
+override_install
+</pre></blockquote></p>
+
+<p>To clean up, another shell script should go into
+/usr/lib/finish-install.d/ with code like this:
+
+<p><blockquote><pre>
+#! /bin/sh -e
+. /usr/share/debconf/confmodule
+error() {
+ logger -t my-finish-install "error: $@"
+}
+remove_install_override() {
+ for bin in dpkg apt-get aptitude tasksel ; do
+ file=/usr/bin/$bin
+ if [ -x /target$file.edu ] ; then
+ rm /target$file
+ in-target dpkg-divert --package debian-edu-config \
+ --rename --quiet --remove $file
+ rm /target$file.edu
+ else
+ error "Missing divert for $file."
+ fi
+ done
+ sync # Flush file buffers before continuing
+}
+
+remove_install_override
+</pre></blockquote></p>
+
+<p>In Debian Edu, I placed both code fragments in a separate script
+edu-eatmydata-install and call it from the pre-pkgsel.d and
+finish-install.d scripts.</p>
+
+<p>By now you might ask if this change should get into the normal
+Debian installer too? I suspect it should, but am not sure the
+current debian-installer coordinators find it useful enough. It also
+depend on the side effects of the change. I'm not aware of any, but I
+guess we will see if the change is safe after some more testing.
+Perhaps there is some package in Debian depending on sync() and
+fsync() having effect? Perhaps it should go into its own udeb, to
+allow those of us wanting to enable it to do so without affecting
+everyone.</p>
+</description>
+ </item>
+
<item>
<title>Good bye subkeys.pgp.net, welcome pool.sks-keyservers.net</title>
<link>http://people.skolelinux.org/pere/blog/Good_bye_subkeys_pgp_net__welcome_pool_sks_keyservers_net.html</link>
</description>
</item>
- <item>
- <title>Install hardware dependent packages using tasksel (Isenkram 0.7)</title>
- <link>http://people.skolelinux.org/pere/blog/Install_hardware_dependent_packages_using_tasksel__Isenkram_0_7_.html</link>
- <guid isPermaLink="true">http://people.skolelinux.org/pere/blog/Install_hardware_dependent_packages_using_tasksel__Isenkram_0_7_.html</guid>
- <pubDate>Wed, 23 Apr 2014 14:50:00 +0200</pubDate>
- <description><p>It would be nice if it was easier in Debian to get all the hardware
-related packages relevant for the computer installed automatically.
-So I implemented one, using
-<a href="http://packages.qa.debian.org/isenkram">my Isenkram
-package</a>. To use it, install the tasksel and isenkram packages and
-run tasksel as user root. You should be presented with a new option,
-"Hardware specific packages (autodetected by isenkram)". When you
-select it, tasksel will install the packages isenkram claim is fit for
-the current hardware, hot pluggable or not.<p>
-
-<p>The implementation is in two files, one is the tasksel menu entry
-description, and the other is the script used to extract the list of
-packages to install. The first part is in
-<tt>/usr/share/tasksel/descs/isenkram.desc</tt> and look like
-this:</p>
-
-<p><blockquote><pre>
-Task: isenkram
-Section: hardware
-Description: Hardware specific packages (autodetected by isenkram)
- Based on the detected hardware various hardware specific packages are
- proposed.
-Test-new-install: mark show
-Relevance: 8
-Packages: for-current-hardware
-</pre></blockquote></p>
-
-<p>The second part is in
-<tt>/usr/lib/tasksel/packages/for-current-hardware</tt> and look like
-this:</p>
-
-<p><blockquote><pre>
-#!/bin/sh
-#
-(
- isenkram-lookup
- isenkram-autoinstall-firmware -l
-) | sort -u
-</pre></blockquote></p>
-
-<p>All in all, a very short and simple implementation making it
-trivial to install the hardware dependent package we all may want to
-have installed on our machines. I've not been able to find a way to
-get tasksel to tell you exactly which packages it plan to install
-before doing the installation. So if you are curious or careful,
-check the output from the isenkram-* command line tools first.</p>
-
-<p>The information about which packages are handling which hardware is
-fetched either from the isenkram package itself in
-/usr/share/isenkram/, from git.debian.org or from the APT package
-database (using the Modaliases header). The APT package database
-parsing have caused a nasty resource leak in the isenkram daemon (bugs
-<a href="http://bugs.debian.org/719837">#719837</a> and
-<a href="http://bugs.debian.org/730704">#730704</a>). The cause is in
-the python-apt code (bug
-<a href="http://bugs.debian.org/745487">#745487</a>), but using a
-workaround I was able to get rid of the file descriptor leak and
-reduce the memory leak from ~30 MiB per hardware detection down to
-around 2 MiB per hardware detection. It should make the desktop
-daemon a lot more useful. The fix is in version 0.7 uploaded to
-unstable today.</p>
-
-<p>I believe the current way of mapping hardware to packages in
-Isenkram is is a good draft, but in the future I expect isenkram to
-use the AppStream data source for this. A proposal for getting proper
-AppStream support into Debian is floating around as
-<a href="https://wiki.debian.org/DEP-11">DEP-11</a>, and
-<a href="https://wiki.debian.org/SummerOfCode2014/Projects#SummerOfCode2014.2FProjects.2FAppStreamDEP11Implementation.AppStream.2FDEP-11_for_the_Debian_Archive">GSoC
-project</a> will take place this summer to improve the situation. I
-look forward to seeing the result, and welcome patches for isenkram to
-start using the information when it is ready.</p>
-
-<p>If you want your package to map to some specific hardware, either
-add a "Xb-Modaliases" header to your control file like I did in
-<a href="http://packages.qa.debian.org/pymissile">the pymissile
-package</a> or submit a bug report with the details to the isenkram
-package. See also
-<a href="http://people.skolelinux.org/pere/blog/tags/isenkram/">all my
-blog posts tagged isenkram</a> for details on the notation. I expect
-the information will be migrated to AppStream eventually, but for the
-moment I got no better place to store it.</p>
-</description>
- </item>
-
</channel>
</rss>
projects / homepage.git / blobdiff