blog/archive/2014/09/09.rss

   1 <?xml version="1.0" encoding="ISO-8859-1"?>
   2 <rss version='2.0' xmlns:lj='http://www.livejournal.org/rss/lj/1.0/'>
   3         <channel>
   4                 <title>Petter Reinholdtsen - Entries from September 2014</title>
   5                 <description>Entries from September 2014</description>
   6                 <link>http://people.skolelinux.org/pere/blog/</link>
   7
   8
   9         <item>
  10                 <title>Speeding up the Debian installer using eatmydata and dpkg-divert</title>
  11                 <link>http://people.skolelinux.org/pere/blog/Speeding_up_the_Debian_installer_using_eatmydata_and_dpkg_divert.html</link>
  12                 <guid isPermaLink="true">http://people.skolelinux.org/pere/blog/Speeding_up_the_Debian_installer_using_eatmydata_and_dpkg_divert.html</guid>
  13                 <pubDate>Tue, 16 Sep 2014 14:00:00 +0200</pubDate>
  14                 <description>&lt;p&gt;The &lt;a href=&quot;https://www.debian.org/&quot;&gt;Debian&lt;/a&gt; installer could be
  15 a lot quicker.  When we install more than 2000 packages in
  16 &lt;a href=&quot;http://www.skolelinux.org/&quot;&gt;Skolelinux / Debian Edu&lt;/a&gt; using
  17 tasksel in the installer, unpacking the binary packages take forever.
  18 A part of the slow I/O issue was discussed in
  19 &lt;a href=&quot;https://bugs.debian.org/613428&quot;&gt;bug #613428&lt;/a&gt; about too
  20 much file system sync-ing done by dpkg, which is the package
  21 responsible for unpacking the binary packages.  Other parts (like code
  22 executed by postinst scripts) might also sync to disk during
  23 installation.  All this sync-ing to disk do not really make sense to
  24 me.  If the machine crash half-way through, I start over, I do not try
  25 to salvage the half installed system.  So the failure sync-ing is
  26 supposed to protect against, hardware or system crash, is not really
  27 relevant while the installer is running.&lt;/p&gt;
  28
  29 &lt;p&gt;A few days ago, I thought of a way to get rid of all the file
  30 system sync()-ing in a fairly non-intrusive way, without the need to
  31 change the code in several packages.  The idea is not new, but I have
  32 not heard anyone propose the approach using dpkg-divert before.  It
  33 depend on the small and clever package
  34 &lt;a href=&quot;https://packages.qa.debian.org/eatmydata&quot;&gt;eatmydata&lt;/a&gt;, which
  35 uses LD_PRELOAD to replace the system functions for syncing data to
  36 disk with functions doing nothing, thus allowing programs to live
  37 dangerous while speeding up disk I/O significantly.  Instead of
  38 modifying the implementation of dpkg, apt and tasksel (which are the
  39 packages responsible for selecting, fetching and installing packages),
  40 it occurred to me that we could just divert the programs away, replace
  41 them with a simple shell wrapper calling
  42 &quot;eatmydata&amp;nbsp;$program&amp;nbsp;$@&quot;, to get the same effect.
  43 Yesterday I decided to test the idea, and wrapped up a simple
  44 implementation for the Debian Edu udeb.&lt;/p&gt;
  45
  46 &lt;p&gt;The effect was stunning.  In my first test it reduced the running
  47 time of the pkgsel step (installing tasks) from 64 to less than 44
  48 minutes (20 minutes shaved off the installation) on an old Dell
  49 Latitude D505 machine.  I am not quite sure what the optimised time
  50 would have been, as I messed up the testing a bit, causing the debconf
  51 priority to get low enough for two questions to pop up during
  52 installation.  As soon as I saw the questions I moved the installation
  53 along, but do not know how long the question were holding up the
  54 installation.  I did some more measurements using Debian Edu Jessie,
  55 and got these results.  The time measured is the time stamp in
  56 /var/log/syslog between the &quot;pkgsel: starting tasksel&quot; and the
  57 &quot;pkgsel: finishing up&quot; lines, if you want to do the same measurement
  58 yourself.  In Debian Edu, the tasksel dialog do not show up, and the
  59 timing thus do not depend on how quickly the user handle the tasksel
  60 dialog.&lt;/p&gt;
  61
  62 &lt;p&gt;&lt;table&gt;
  63
  64 &lt;tr&gt;
  65 &lt;th&gt;Machine/setup&lt;/th&gt;
  66 &lt;th&gt;Original tasksel&lt;/th&gt;
  67 &lt;th&gt;Optimised tasksel&lt;/th&gt;
  68 &lt;th&gt;Reduction&lt;/th&gt;
  69 &lt;/tr&gt;
  70
  71 &lt;tr&gt;
  72 &lt;td&gt;Latitude D505 Main+LTSP LXDE&lt;/td&gt;
  73 &lt;td&gt;64 min (07:46-08:50)&lt;/td&gt;
  74 &lt;td&gt;&lt;44 min (11:27-12:11)&lt;/td&gt;
  75 &lt;td&gt;&gt;20 min 18%&lt;/td&gt;
  76 &lt;/tr&gt;
  77
  78 &lt;tr&gt;
  79 &lt;td&gt;Latitude D505 Roaming LXDE&lt;/td&gt;
  80 &lt;td&gt;57 min (08:48-09:45)&lt;/td&gt;
  81 &lt;td&gt;34 min (07:43-08:17)&lt;/td&gt;
  82 &lt;td&gt;23 min 40%&lt;/td&gt;
  83 &lt;/tr&gt;
  84
  85 &lt;tr&gt;
  86 &lt;td&gt;Latitude D505 Minimal&lt;/td&gt;
  87 &lt;td&gt;22 min (10:37-10:59)&lt;/td&gt;
  88 &lt;td&gt;11 min (11:16-11:27)&lt;/td&gt;
  89 &lt;td&gt;11 min 50%&lt;/td&gt;
  90 &lt;/tr&gt;
  91
  92 &lt;tr&gt;
  93 &lt;td&gt;Thinkpad X200 Minimal&lt;/td&gt;
  94 &lt;td&gt;6 min (08:19-08:25)&lt;/td&gt;
  95 &lt;td&gt;4 min (08:04-08:08)&lt;/td&gt;
  96 &lt;td&gt;2 min 33%&lt;/td&gt;
  97 &lt;/tr&gt;
  98
  99 &lt;tr&gt;
 100 &lt;td&gt;Thinkpad X200 Roaming KDE&lt;/td&gt;
 101 &lt;td&gt;19 min (09:21-09:40)&lt;/td&gt;
 102 &lt;td&gt;15 min (10:25-10:40)&lt;/td&gt;
 103 &lt;td&gt;4 min 21%&lt;/td&gt;
 104 &lt;/tr&gt;
 105
 106 &lt;/table&gt;&lt;/p&gt;
 107
 108 &lt;p&gt;The test is done using a netinst ISO on a USB stick, so some of the
 109 time is spent downloading packages.  The connection to the Internet
 110 was 100Mbit/s during testing, so downloading should not be a
 111 significant factor in the measurement.  Download typically took a few
 112 seconds to a few minutes, depending on the amount of packages being
 113 installed.&lt;/p&gt;
 114
 115 &lt;p&gt;The speedup is implemented by using two hooks in
 116 &lt;a href=&quot;https://www.debian.org/devel/debian-installer/&quot;&gt;Debian
 117 Installer&lt;/a&gt;, the pre-pkgsel.d hook to set up the diverts, and the
 118 finish-install.d hook to remove the divert at the end of the
 119 installation.  I picked the pre-pkgsel.d hook instead of the
 120 post-base-installer.d hook because I test using an ISO without the
 121 eatmydata package included, and the post-base-installer.d hook in
 122 Debian Edu can only operate on packages included in the ISO.  The
 123 negative effect of this is that I am unable to activate this
 124 optimization for the kernel installation step in d-i.  If the code is
 125 moved to the post-base-installer.d hook, the speedup would be larger
 126 for the entire installation.&lt;/p&gt;
 127
 128 &lt;p&gt;I&#39;ve implemented this in the
 129 &lt;a href=&quot;https://packages.qa.debian.org/debian-edu-install&quot;&gt;debian-edu-install&lt;/a&gt;
 130 git repository, and plan to provide the optimization as part of the
 131 Debian Edu installation.  If you want to test this yourself, you can
 132 create two files in the installer (or in an udeb).  One shell script
 133 need do go into /usr/lib/pre-pkgsel.d/, with content like this:&lt;/p&gt;
 134
 135 &lt;p&gt;&lt;blockquote&gt;&lt;pre&gt;
 136 #!/bin/sh
 137 set -e
 138 . /usr/share/debconf/confmodule
 139 info() {
 140     logger -t my-pkgsel &quot;info: $*&quot;
 141 }
 142 error() {
 143     logger -t my-pkgsel &quot;error: $*&quot;
 144 }
 145 override_install() {
 146     apt-install eatmydata || true
 147     if [ -x /target/usr/bin/eatmydata ] ; then
 148         for bin in dpkg apt-get aptitude tasksel ; do
 149             file=/usr/bin/$bin
 150             # Test that the file exist and have not been diverted already.
 151             if [ -f /target$file ] ; then
 152                 info &quot;diverting $file using eatmydata&quot;
 153                 printf &quot;#!/bin/sh\neatmydata $bin.distrib \&quot;\$@\&quot;\n&quot; \
 154                     &gt; /target$file.edu
 155                 chmod 755 /target$file.edu
 156                 in-target dpkg-divert --package debian-edu-config \
 157                     --rename --quiet --add $file
 158                 ln -sf ./$bin.edu /target$file
 159             else
 160                 error &quot;unable to divert $file, as it is missing.&quot;
 161             fi
 162         done
 163     else
 164         error &quot;unable to find /usr/bin/eatmydata after installing the eatmydata pacage&quot;
 165     fi
 166 }
 167
 168 override_install
 169 &lt;/pre&gt;&lt;/blockquote&gt;&lt;/p&gt;
 170
 171 &lt;p&gt;To clean up, another shell script should go into
 172 /usr/lib/finish-install.d/ with code like this:
 173
 174 &lt;p&gt;&lt;blockquote&gt;&lt;pre&gt;
 175 #! /bin/sh -e
 176 . /usr/share/debconf/confmodule
 177 error() {
 178     logger -t my-finish-install &quot;error: $@&quot;
 179 }
 180 remove_install_override() {
 181     for bin in dpkg apt-get aptitude tasksel ; do
 182         file=/usr/bin/$bin
 183         if [ -x /target$file.edu ] ; then
 184             rm /target$file
 185             in-target dpkg-divert --package debian-edu-config \
 186                 --rename --quiet --remove $file
 187             rm /target$file.edu
 188         else
 189             error &quot;Missing divert for $file.&quot;
 190         fi
 191     done
 192     sync # Flush file buffers before continuing
 193 }
 194
 195 remove_install_override
 196 &lt;/pre&gt;&lt;/blockquote&gt;&lt;/p&gt;
 197
 198 &lt;p&gt;In Debian Edu, I placed both code fragments in a separate script
 199 edu-eatmydata-install and call it from the pre-pkgsel.d and
 200 finish-install.d scripts.&lt;/p&gt;
 201
 202 &lt;p&gt;By now you might ask if this change should get into the normal
 203 Debian installer too?  I suspect it should, but am not sure the
 204 current debian-installer coordinators find it useful enough.  It also
 205 depend on the side effects of the change.  I&#39;m not aware of any, but I
 206 guess we will see if the change is safe after some more testing.
 207 Perhaps there is some package in Debian depending on sync() and
 208 fsync() having effect?  Perhaps it should go into its own udeb, to
 209 allow those of us wanting to enable it to do so without affecting
 210 everyone.&lt;/p&gt;
 211 </description>
 212         </item>
 213
 214         <item>
 215                 <title>Good bye subkeys.pgp.net, welcome pool.sks-keyservers.net</title>
 216                 <link>http://people.skolelinux.org/pere/blog/Good_bye_subkeys_pgp_net__welcome_pool_sks_keyservers_net.html</link>
 217                 <guid isPermaLink="true">http://people.skolelinux.org/pere/blog/Good_bye_subkeys_pgp_net__welcome_pool_sks_keyservers_net.html</guid>
 218                 <pubDate>Wed, 10 Sep 2014 13:10:00 +0200</pubDate>
 219                 <description>&lt;p&gt;Yesterday, I had the pleasure of attending a talk with the
 220 &lt;a href=&quot;http://www.nuug.no/&quot;&gt;Norwegian Unix User Group&lt;/a&gt; about
 221 &lt;a href=&quot;http://www.nuug.no/aktiviteter/20140909-sks-keyservers/&quot;&gt;the
 222 OpenPGP keyserver pool sks-keyservers.net&lt;/a&gt;, and was very happy to
 223 learn that there is a large set of publicly available key servers to
 224 use when looking for peoples public key.  So far I have used
 225 subkeys.pgp.net, and some times wwwkeys.nl.pgp.net when the former
 226 were misbehaving, but those days are ended.  The servers I have used
 227 up until yesterday have been slow and some times unavailable.  I hope
 228 those problems are gone now.&lt;/p&gt;
 229
 230 &lt;p&gt;Behind the round robin DNS entry of the
 231 &lt;a href=&quot;https://sks-keyservers.net/&quot;&gt;sks-keyservers.net&lt;/a&gt; service
 232 there is a pool of more than 100 keyservers which are checked every
 233 day to ensure they are well connected and up to date.  It must be
 234 better than what I have used so far. :)&lt;/p&gt;
 235
 236 &lt;p&gt;Yesterdays speaker told me that the service is the default
 237 keyserver provided by the default configuration in GnuPG, but this do
 238 not seem to be used in Debian.  Perhaps it should?&lt;/p&gt;
 239
 240 &lt;p&gt;Anyway, I&#39;ve updated my ~/.gnupg/options file to now include this
 241 line:&lt;/p&gt;
 242
 243 &lt;p&gt;&lt;blockquote&gt;&lt;pre&gt;
 244 keyserver pool.sks-keyservers.net
 245 &lt;/pre&gt;&lt;/blockquote&gt;&lt;/p&gt;
 246
 247 &lt;p&gt;With GnuPG version 2 one can also locate the keyserver using SRV
 248 entries in DNS.  Just for fun, I did just that at work, so now every
 249 user of GnuPG at the University of Oslo should find a OpenGPG
 250 keyserver automatically should their need it:&lt;/p&gt;
 251
 252 &lt;p&gt;&lt;blockquote&gt;&lt;pre&gt;
 253 % host -t srv _pgpkey-http._tcp.uio.no
 254 _pgpkey-http._tcp.uio.no has SRV record 0 100 11371 pool.sks-keyservers.net.
 255 %
 256 &lt;/pre&gt;&lt;/blockquote&gt;&lt;/p&gt;
 257
 258 &lt;p&gt;Now if only
 259 &lt;a href=&quot;http://ietfreport.isoc.org/idref/draft-shaw-openpgp-hkp/&quot;&gt;the
 260 HKP lookup protocol&lt;/a&gt; supported finding signature paths, I would be
 261 very happy.  It can look up a given key or search for a user ID, but I
 262 normally do not want that, but to find a trust path from my key to
 263 another key.  Given a user ID or key ID, I would like to find (and
 264 download) the keys representing a signature path from my key to the
 265 key in question, to be able to get a trust path between the two keys.
 266 This is as far as I can tell not possible today.  Perhaps something
 267 for a future version of the protocol?&lt;/p&gt;
 268 </description>
 269         </item>
 270
 271         </channel>
 272 </rss>