+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
+ "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" dir="ltr">
+ <head>
+ <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
+ <title>Petter Reinholdtsen: Speeding up the Debian installer using eatmydata and dpkg-divert</title>
+ <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/style.css" />
+ <link rel="stylesheet" type="text/css" media="screen" href="http://people.skolelinux.org/pere/blog/vim.css" />
+
+
+ </head>
+ <body>
+ <div class="title">
+ <h1>
+ <a href="http://people.skolelinux.org/pere/blog/">Petter Reinholdtsen</a>
+
+ </h1>
+
+</div>
+
+
+ <div class="entry">
+ <div class="title">Speeding up the Debian installer using eatmydata and dpkg-divert</div>
+ <div class="date">16th September 2014</div>
+ <div class="body"><p>The <a href="https://www.debian.org/">Debian</a> installer could be
+a lot quicker. When we install more than 2000 packages in
+<a href="http://www.skolelinux.org/">Skolelinux / Debian Edu</a> using
+tasksel in the installer, unpacking the binary packages take forever.
+A part of the slow I/O issue was discussed in
+<a href="https://bugs.debian.org/613428">bug #613428</a> about too
+much file system sync-ing done by dpkg, which is the package
+responsible for unpacking the binary packages. Other parts (like code
+executed by postinst scripts) might also sync to disk during
+installation. All this sync-ing to disk do not really make sense to
+me. If the machine crash half-way through, I start over, I do not try
+to salvage the half installed system. So the failure sync-ing is
+supposed to protect against, hardware or system crash, is not really
+relevant while the installer is running.</p>
+
+<p>A few days ago, I thought of a way to get rid of all the file
+system sync()-ing in a fairly non-intrusive way, without the need to
+change the code in several packages. The idea is not new, but I have
+not heard anyone propose the approach using dpkg-divert before. It
+depend on the small and clever package
+<a href="https://packages.qa.debian.org/eatmydata">eatmydata</a>, which
+uses LD_PRELOAD to replace the system functions for syncing data to
+disk with functions doing nothing, thus allowing programs to live
+dangerous while speeding up disk I/O significantly. Instead of
+modifying the implementation of dpkg, apt and tasksel (which are the
+packages responsible for selecting, fetching and installing packages),
+it occurred to me that we could just divert the programs away, replace
+them with a simple shell wrapper calling
+"eatmydata $program $@", to get the same effect.
+Yesterday I decided to test the idea, and wrapped up a simple
+implementation for the Debian Edu udeb.</p>
+
+<p>The effect was stunning. In my first test it reduced the running
+time of the pkgsel step (installing tasks) from 64 to less than 44
+minutes (20 minutes shaved off the installation) on an old Dell
+Latitude D505 machine. I am not quite sure what the optimised time
+would have been, as I messed up the testing a bit, causing the debconf
+priority to get low enough for two questions to pop up during
+installation. As soon as I saw the questions I moved the installation
+along, but do not know how long the question were holding up the
+installation. I did some more measurements using Debian Edu Jessie,
+and got these results. The time measured is the time stamp in
+/var/log/syslog between the "pkgsel: starting tasksel" and the
+"pkgsel: finishing up" lines, if you want to do the same measurement
+yourself. In Debian Edu, the tasksel dialog do not show up, and the
+timing thus do not depend on how quickly the user handle the tasksel
+dialog.</p>
+
+<p><table>
+
+<tr>
+<th>Machine/setup</th>
+<th>Original tasksel</th>
+<th>Optimised tasksel</th>
+<th>Reduction</th>
+</tr>
+
+<tr>
+<td>Latitude D505 Main+LTSP LXDE</td>
+<td>64 min (07:46-08:50)</td>
+<td><44 min (11:27-12:11)</td>
+<td>>20 min 18%</td>
+</tr>
+
+<tr>
+<td>Latitude D505 Roaming LXDE</td>
+<td>57 min (08:48-09:45)</td>
+<td>34 min (07:43-08:17)</td>
+<td>23 min 40%</td>
+</tr>
+
+<tr>
+<td>Latitude D505 Minimal</td>
+<td>22 min (10:37-10:59)</td>
+<td>11 min (11:16-11:27)</td>
+<td>11 min 50%</td>
+</tr>
+
+<tr>
+<td>Thinkpad X200 Minimal</td>
+<td>6 min (08:19-08:25)</td>
+<td>4 min (08:04-08:08)</td>
+<td>2 min 33%</td>
+</tr>
+
+<tr>
+<td>Thinkpad X200 Roaming KDE</td>
+<td>19 min (09:21-09:40)</td>
+<td>15 min (10:25-10:40)</td>
+<td>4 min 21%</td>
+</tr>
+
+</table></p>
+
+<p>The test is done using a netinst ISO on a USB stick, so some of the
+time is spent downloading packages. The connection to the Internet
+was 100Mbit/s during testing, so downloading should not be a
+significant factor in the measurement. Download typically took a few
+seconds to a few minutes, depending on the amount of packages being
+installed.</p>
+
+<p>The speedup is implemented by using two hooks in
+<a href="https://www.debian.org/devel/debian-installer/">Debian
+Installer</a>, the pre-pkgsel.d hook to set up the diverts, and the
+finish-install.d hook to remove the divert at the end of the
+installation. I picked the pre-pkgsel.d hook instead of the
+post-base-installer.d hook because I test using an ISO without the
+eatmydata package included, and the post-base-installer.d hook in
+Debian Edu can only operate on packages included in the ISO. The
+negative effect of this is that I am unable to activate this
+optimization for the kernel installation step in d-i. If the code is
+moved to the post-base-installer.d hook, the speedup would be larger
+for the entire installation.</p>
+
+<p>I've implemented this in the
+<a href="https://packages.qa.debian.org/debian-edu-install">debian-edu-install</a>
+git repository, and plan to provide the optimization as part of the
+Debian Edu installation. If you want to test this yourself, you can
+create two files in the installer (or in an udeb). One shell script
+need do go into /usr/lib/pre-pkgsel.d/, with content like this:</p>
+
+<p><blockquote><pre>
+#!/bin/sh
+set -e
+. /usr/share/debconf/confmodule
+info() {
+ logger -t my-pkgsel "info: $*"
+}
+error() {
+ logger -t my-pkgsel "error: $*"
+}
+override_install() {
+ apt-install eatmydata || true
+ if [ -x /target/usr/bin/eatmydata ] ; then
+ for bin in dpkg apt-get aptitude tasksel ; do
+ file=/usr/bin/$bin
+ # Test that the file exist and have not been diverted already.
+ if [ -f /target$file ] ; then
+ info "diverting $file using eatmydata"
+ printf "#!/bin/sh\neatmydata $bin.distrib \"\$@\"\n" \
+ > /target$file.edu
+ chmod 755 /target$file.edu
+ in-target dpkg-divert --package debian-edu-config \
+ --rename --quiet --add $file
+ ln -sf ./$bin.edu /target$file
+ else
+ error "unable to divert $file, as it is missing."
+ fi
+ done
+ else
+ error "unable to find /usr/bin/eatmydata after installing the eatmydata pacage"
+ fi
+}
+
+override_install
+</pre></blockquote></p>
+
+<p>To clean up, another shell script should go into
+/usr/lib/finish-install.d/ with code like this:
+
+<p><blockquote><pre>
+#! /bin/sh -e
+. /usr/share/debconf/confmodule
+error() {
+ logger -t my-finish-install "error: $@"
+}
+remove_install_override() {
+ for bin in dpkg apt-get aptitude tasksel ; do
+ file=/usr/bin/$bin
+ if [ -x /target$file.edu ] ; then
+ rm /target$file
+ in-target dpkg-divert --package debian-edu-config \
+ --rename --quiet --remove $file
+ rm /target$file.edu
+ else
+ error "Missing divert for $file."
+ fi
+ done
+ sync # Flush file buffers before continuing
+}
+
+remove_install_override
+</pre></blockquote></p>
+
+<p>In Debian Edu, I placed both code fragments in a separate script
+edu-eatmydata-install and call it from the pre-pkgsel.d and
+finish-install.d scripts.</p>
+
+<p>By now you might ask if this change should get into the normal
+Debian installer too? I suspect it should, but am not sure the
+current debian-installer coordinators find it useful enough. It also
+depend on the side effects of the change. I'm not aware of any, but I
+guess we will see if the change is safe after some more testing.
+Perhaps there is some package in Debian depending on sync() and
+fsync() having effect? Perhaps it should go into its own udeb, to
+allow those of us wanting to enable it to do so without affecting
+everyone.</p>
+</div>
+
+ <div class="tags">Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>.</div>
+
+
+ </div>
+
+
+
+
+ <div id="sidebar">
+
+
+
+<h2>Archive</h2>
+<ul>
+
+<li>2014
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/01/">January (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/02/">February (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/03/">March (8)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/04/">April (7)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/05/">May (1)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/06/">June (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/07/">July (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/08/">August (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2014/09/">September (2)</a></li>
+
+</ul></li>
+
+<li>2013
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/01/">January (11)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/02/">February (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/03/">March (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/04/">April (6)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/05/">May (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/06/">June (10)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/07/">July (7)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/08/">August (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/09/">September (5)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/10/">October (7)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/11/">November (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2013/12/">December (3)</a></li>
+
+</ul></li>
+
+<li>2012
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/01/">January (7)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/02/">February (10)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/03/">March (17)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/04/">April (12)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/05/">May (12)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/06/">June (20)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/07/">July (17)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/08/">August (6)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/09/">September (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/10/">October (17)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/11/">November (10)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2012/12/">December (7)</a></li>
+
+</ul></li>
+
+<li>2011
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/01/">January (16)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/02/">February (6)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/03/">March (6)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/04/">April (7)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/05/">May (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/06/">June (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/07/">July (7)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/08/">August (6)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/09/">September (4)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/10/">October (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/11/">November (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2011/12/">December (1)</a></li>
+
+</ul></li>
+
+<li>2010
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/01/">January (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/02/">February (1)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/03/">March (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/04/">April (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/05/">May (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/06/">June (14)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/07/">July (12)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/08/">August (13)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/09/">September (7)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/10/">October (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/11/">November (13)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2010/12/">December (12)</a></li>
+
+</ul></li>
+
+<li>2009
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/01/">January (8)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/02/">February (8)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/03/">March (12)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/04/">April (10)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/05/">May (9)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/06/">June (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/07/">July (4)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/08/">August (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/09/">September (1)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/10/">October (2)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/11/">November (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2009/12/">December (3)</a></li>
+
+</ul></li>
+
+<li>2008
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2008/11/">November (5)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2008/12/">December (7)</a></li>
+
+</ul></li>
+
+</ul>
+
+
+
+<h2>Tags</h2>
+<ul>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (13)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/amiga">amiga (1)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/aros">aros (1)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/bankid">bankid (4)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/bitcoin">bitcoin (8)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/bootsystem">bootsystem (14)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/bsa">bsa (2)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/chrpath">chrpath (2)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/debian">debian (101)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu (149)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/digistan">digistan (10)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/dld">dld (15)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/docbook">docbook (12)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/drivstoffpriser">drivstoffpriser (4)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/english">english (253)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/fiksgatami">fiksgatami (21)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (12)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (13)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/freedombox">freedombox (8)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/frikanalen">frikanalen (11)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/intervju">intervju (41)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/isenkram">isenkram (9)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/kart">kart (18)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/ldap">ldap (9)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/lenker">lenker (8)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/ltsp">ltsp (1)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/mesh network">mesh network (8)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/multimedia">multimedia (29)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk (247)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug (162)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn (11)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/open311">open311 (2)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (48)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern (74)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/raid">raid (1)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/reactos">reactos (1)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/reprap">reprap (11)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/rfid">rfid (2)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/robot">robot (9)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/rss">rss (1)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/ruter">ruter (4)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/scraperwiki">scraperwiki (2)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/sikkerhet">sikkerhet (41)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/sitesummary">sitesummary (4)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/skepsis">skepsis (4)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/standard">standard (45)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/stavekontroll">stavekontroll (3)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (9)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (25)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin (1)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/valg">valg (8)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/video">video (43)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/vitenskap">vitenskap (4)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/web">web (33)</a></li>
+
+</ul>
+
+
+ </div>
+ <p style="text-align: right">
+ Created by <a href="http://steve.org.uk/Software/chronicle">Chronicle v4.6</a>
+</p>
+
+ </body>
+</html>
projects / homepage.git / commitdiff