Petter Reinholdtsen: Speeding up the Debian installer using eatmydata and dpkg-divert

Machine/setup	Original tasksel	Optimised tasksel	Reduction
Latitude D505 Main+LTSP LXDE	64 min (07:46-08:50)	<44 min (11:27-12:11)	>20 min 18%
Latitude D505 Roaming LXDE	57 min (08:48-09:45)	34 min (07:43-08:17)	23 min 40%
Latitude D505 Minimal	22 min (10:37-10:59)	11 min (11:16-11:27)	11 min 50%
Thinkpad X200 Minimal	6 min (08:19-08:25)	4 min (08:04-08:08)	2 min 33%
Thinkpad X200 Roaming KDE	19 min (09:21-09:40)	15 min (10:25-10:40)	4 min 21%

Machine/setup

Original tasksel

Optimised tasksel

Reduction

Latitude D505 Main+LTSP LXDE

64 min (07:46-08:50)

<44 min (11:27-12:11)

>20 min 18%

Latitude D505 Roaming LXDE

57 min (08:48-09:45)

34 min (07:43-08:17)

23 min 40%

Latitude D505 Minimal

22 min (10:37-10:59)

11 min (11:16-11:27)

11 min 50%

Thinkpad X200 Minimal

6 min (08:19-08:25)

4 min (08:04-08:08)

2 min 33%

Thinkpad X200 Roaming KDE

19 min (09:21-09:40)

15 min (10:25-10:40)

4 min 21%

+#!/bin/sh +set -e +. /usr/share/debconf/confmodule +info() { + logger -t my-pkgsel "info: $*" +} +error() { + logger -t my-pkgsel "error: $*" +} +override_install() { + apt-install eatmydata || true + if [ -x /target/usr/bin/eatmydata ] ; then + for bin in dpkg apt-get aptitude tasksel ; do + file=/usr/bin/$bin + # Test that the file exist and have not been diverted already. + if [ -f /target$file ] ; then + info "diverting $file using eatmydata" + printf "#!/bin/sh\neatmydata $bin.distrib \"\$@\"\n" \ + > /target$file.edu + chmod 755 /target$file.edu + in-target dpkg-divert --package debian-edu-config \ + --rename --quiet --add $file + ln -sf ./$bin.edu /target$file + else + error "unable to divert $file, as it is missing." + fi + done + else + error "unable to find /usr/bin/eatmydata after installing the eatmydata pacage" + fi +} + +override_install +

+#! /bin/sh -e +. /usr/share/debconf/confmodule +error() { + logger -t my-finish-install "error: $@" +} +remove_install_override() { + for bin in dpkg apt-get aptitude tasksel ; do + file=/usr/bin/$bin + if [ -x /target$file.edu ] ; then + rm /target$file + in-target dpkg-divert --package debian-edu-config \ + --rename --quiet --remove $file + rm /target$file.edu + else + error "Missing divert for $file." + fi + done + sync # Flush file buffers before continuing +} + +remove_install_override +

Machine/setup	Original tasksel	Optimised tasksel	Reduction
Latitude D505 Main+LTSP LXDE	64 min (07:46-08:50)	<44 min (11:27-12:11)	>20 min 18%
Latitude D505 Roaming LXDE	57 min (08:48-09:45)	34 min (07:43-08:17)	23 min 40%
Latitude D505 Minimal	22 min (10:37-10:59)	11 min (11:16-11:27)	11 min 50%
Thinkpad X200 Minimal	6 min (08:19-08:25)	4 min (08:04-08:08)	2 min 33%
Thinkpad X200 Roaming KDE	19 min (09:21-09:40)	15 min (10:25-10:40)	4 min 21%

Machine/setup

Original tasksel

Optimised tasksel

Reduction

Latitude D505 Main+LTSP LXDE

64 min (07:46-08:50)

<44 min (11:27-12:11)

>20 min 18%

Latitude D505 Roaming LXDE

57 min (08:48-09:45)

34 min (07:43-08:17)

23 min 40%

Latitude D505 Minimal

22 min (10:37-10:59)

11 min (11:16-11:27)

11 min 50%

Thinkpad X200 Minimal

6 min (08:19-08:25)

4 min (08:04-08:08)

2 min 33%

Thinkpad X200 Roaming KDE

19 min (09:21-09:40)

15 min (10:25-10:40)

4 min 21%

Good bye subkeys.pgp.net, welcome pool.sks-keyservers.net @@ -117,7 +334,7 @@ for a future version of the protocol?

August (2)

September (1)

September (2)

@@ -298,9 +515,9 @@ for a future version of the protocol?

@@ -310,7 +527,7 @@ for a future version of the protocol?

diff --git a/blog/data/2014-09-16-d-i-eatmydata.txt b/blog/data/2014-09-16-d-i-eatmydata.txt index 48f7cedb01..07550300e0 100644 --- a/blog/data/2014-09-16-d-i-eatmydata.txt +++ b/blog/data/2014-09-16-d-i-eatmydata.txt @@ -1,6 +1,6 @@ Title: Speeding up the Debian installer using eatmydata and dpkg-divert Tags: english, debian, debian edu -Date: 2014-09-16 13:30 +Date: 2014-09-16 14:00

The Debian installer could be a lot quicker. When we install more than 2000 packages in @@ -31,7 +31,7 @@ packages responsible for selecting, fetching and installing packages), it occurred to me that we could just divert the programs away, replace them with a simple shell wrapper calling "eatmydata $program $@", to get the same effect. -Yesterday I decided to test the idea, and wrapped up a simple +Two days ago I decided to test the idea, and wrapped up a simple implementation for the Debian Edu udeb.

The effect was stunning. In my first test it reduced the running @@ -186,6 +186,10 @@ remove_install_override() { remove_install_override

In Debian Edu, I placed both code fragments in a separate script +edu-eatmydata-install and call it from the pre-pkgsel.d and +finish-install.d scripts.

By now you might ask if this change should get into the normal Debian installer too? I suspect it should, but am not sure the current debian-installer coordinators find it useful enough. It also diff --git a/blog/index.html b/blog/index.html index 8fc1c5bb8c..a8a9674b3e 100644 --- a/blog/index.html +++ b/blog/index.html @@ -19,6 +19,217 @@ +

Speeding up the Debian installer using eatmydata and dpkg-divert

16th September 2014

The Debian installer could be +a lot quicker. When we install more than 2000 packages in +Skolelinux / Debian Edu using +tasksel in the installer, unpacking the binary packages take forever. +A part of the slow I/O issue was discussed in +bug #613428 about too +much file system sync-ing done by dpkg, which is the package +responsible for unpacking the binary packages. Other parts (like code +executed by postinst scripts) might also sync to disk during +installation. All this sync-ing to disk do not really make sense to +me. If the machine crash half-way through, I start over, I do not try +to salvage the half installed system. So the failure sync-ing is +supposed to protect against, hardware or system crash, is not really +relevant while the installer is running.

+ +

A few days ago, I thought of a way to get rid of all the file +system sync()-ing in a fairly non-intrusive way, without the need to +change the code in several packages. The idea is not new, but I have +not heard anyone propose the approach using dpkg-divert before. It +depend on the small and clever package +eatmydata, which +uses LD_PRELOAD to replace the system functions for syncing data to +disk with functions doing nothing, thus allowing programs to live +dangerous while speeding up disk I/O significantly. Instead of +modifying the implementation of dpkg, apt and tasksel (which are the +packages responsible for selecting, fetching and installing packages), +it occurred to me that we could just divert the programs away, replace +them with a simple shell wrapper calling +"eatmydata $program $@", to get the same effect. +Yesterday I decided to test the idea, and wrapped up a simple +implementation for the Debian Edu udeb.

+ +

The effect was stunning. In my first test it reduced the running +time of the pkgsel step (installing tasks) from 64 to less than 44 +minutes (20 minutes shaved off the installation) on an old Dell +Latitude D505 machine. I am not quite sure what the optimised time +would have been, as I messed up the testing a bit, causing the debconf +priority to get low enough for two questions to pop up during +installation. As soon as I saw the questions I moved the installation +along, but do not know how long the question were holding up the +installation. I did some more measurements using Debian Edu Jessie, +and got these results. The time measured is the time stamp in +/var/log/syslog between the "pkgsel: starting tasksel" and the +"pkgsel: finishing up" lines, if you want to do the same measurement +yourself. In Debian Edu, the tasksel dialog do not show up, and the +timing thus do not depend on how quickly the user handle the tasksel +dialog.

+ +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Machine/setup Original tasksel Optimised tasksel Reduction
Latitude D505 Main+LTSP LXDE 64 min (07:46-08:50) <44 min (11:27-12:11) >20 min 18%
Latitude D505 Roaming LXDE 57 min (08:48-09:45) 34 min (07:43-08:17) 23 min 40%
Latitude D505 Minimal 22 min (10:37-10:59) 11 min (11:16-11:27) 11 min 50%
Thinkpad X200 Minimal 6 min (08:19-08:25) 4 min (08:04-08:08) 2 min 33%
Thinkpad X200 Roaming KDE 19 min (09:21-09:40) 15 min (10:25-10:40) 4 min 21%

Machine/setup	Original tasksel	Optimised tasksel	Reduction
Latitude D505 Main+LTSP LXDE	64 min (07:46-08:50)	<44 min (11:27-12:11)	>20 min 18%
Latitude D505 Roaming LXDE	57 min (08:48-09:45)	34 min (07:43-08:17)	23 min 40%
Latitude D505 Minimal	22 min (10:37-10:59)	11 min (11:16-11:27)	11 min 50%
Thinkpad X200 Minimal	6 min (08:19-08:25)	4 min (08:04-08:08)	2 min 33%
Thinkpad X200 Roaming KDE	19 min (09:21-09:40)	15 min (10:25-10:40)	4 min 21%

+ +

The test is done using a netinst ISO on a USB stick, so some of the +time is spent downloading packages. The connection to the Internet +was 100Mbit/s during testing, so downloading should not be a +significant factor in the measurement. Download typically took a few +seconds to a few minutes, depending on the amount of packages being +installed.

+ +

The speedup is implemented by using two hooks in +Debian +Installer, the pre-pkgsel.d hook to set up the diverts, and the +finish-install.d hook to remove the divert at the end of the +installation. I picked the pre-pkgsel.d hook instead of the +post-base-installer.d hook because I test using an ISO without the +eatmydata package included, and the post-base-installer.d hook in +Debian Edu can only operate on packages included in the ISO. The +negative effect of this is that I am unable to activate this +optimization for the kernel installation step in d-i. If the code is +moved to the post-base-installer.d hook, the speedup would be larger +for the entire installation.

+ +

I've implemented this in the +debian-edu-install +git repository, and plan to provide the optimization as part of the +Debian Edu installation. If you want to test this yourself, you can +create two files in the installer (or in an udeb). One shell script +need do go into /usr/lib/pre-pkgsel.d/, with content like this:

+ +

+#!/bin/sh
+set -e
+. /usr/share/debconf/confmodule
+info() {
+    logger -t my-pkgsel "info: $*"
+}
+error() {
+    logger -t my-pkgsel "error: $*"
+}
+override_install() {
+    apt-install eatmydata || true
+    if [ -x /target/usr/bin/eatmydata ] ; then
+        for bin in dpkg apt-get aptitude tasksel ; do
+            file=/usr/bin/$bin
+            # Test that the file exist and have not been diverted already.
+            if [ -f /target$file ] ; then
+                info "diverting $file using eatmydata"
+                printf "#!/bin/sh\neatmydata $bin.distrib \"\$@\"\n" \
+                    > /target$file.edu
+                chmod 755 /target$file.edu
+                in-target dpkg-divert --package debian-edu-config \
+                    --rename --quiet --add $file
+                ln -sf ./$bin.edu /target$file
+            else
+                error "unable to divert $file, as it is missing."
+            fi
+        done
+    else
+        error "unable to find /usr/bin/eatmydata after installing the eatmydata pacage"
+    fi
+}
+
+override_install
+

+ +

To clean up, another shell script should go into +/usr/lib/finish-install.d/ with code like this: + +

+#! /bin/sh -e
+. /usr/share/debconf/confmodule
+error() {
+    logger -t my-finish-install "error: $@"
+}
+remove_install_override() {
+    for bin in dpkg apt-get aptitude tasksel ; do
+        file=/usr/bin/$bin
+        if [ -x /target$file.edu ] ; then
+            rm /target$file
+            in-target dpkg-divert --package debian-edu-config \
+                --rename --quiet --remove $file
+            rm /target$file.edu
+        else
+            error "Missing divert for $file."
+        fi
+    done
+    sync # Flush file buffers before continuing
+}
+
+remove_install_override
+

+ +

In Debian Edu, I placed both code fragments in a separate script +edu-eatmydata-install and call it from the pre-pkgsel.d and +finish-install.d scripts.

+ +

By now you might ask if this change should get into the normal +Debian installer too? I suspect it should, but am not sure the +current debian-installer coordinators find it useful enough. It also +depend on the side effects of the change. I'm not aware of any, but I +guess we will see if the change is safe after some more testing. +Perhaps there is some package in Debian depending on sync() and +fsync() having effect? Perhaps it should go into its own udeb, to +allow those of us wanting to enable it to do so without affecting +everyone.

+ + + Tags: debian, debian edu, english. + + +

Good bye subkeys.pgp.net, welcome pool.sks-keyservers.net

10th September 2014

@@ -787,101 +998,6 @@ irc.freenode.net IRC server.

Install hardware dependent packages using tasksel (Isenkram 0.7)

23rd April 2014

It would be nice if it was easier in Debian to get all the hardware -related packages relevant for the computer installed automatically. -So I implemented one, using -my Isenkram -package. To use it, install the tasksel and isenkram packages and -run tasksel as user root. You should be presented with a new option, -"Hardware specific packages (autodetected by isenkram)". When you -select it, tasksel will install the packages isenkram claim is fit for -the current hardware, hot pluggable or not.

- -

The implementation is in two files, one is the tasksel menu entry -description, and the other is the script used to extract the list of -packages to install. The first part is in -/usr/share/tasksel/descs/isenkram.desc and look like -this:

- -

-Task: isenkram
-Section: hardware
-Description: Hardware specific packages (autodetected by isenkram)
- Based on the detected hardware various hardware specific packages are
- proposed.
-Test-new-install: mark show
-Relevance: 8
-Packages: for-current-hardware
-

- -

The second part is in -/usr/lib/tasksel/packages/for-current-hardware and look like -this:

- -

-#!/bin/sh
-#
-(
-    isenkram-lookup
-    isenkram-autoinstall-firmware -l
-) | sort -u
-

- -

All in all, a very short and simple implementation making it -trivial to install the hardware dependent package we all may want to -have installed on our machines. I've not been able to find a way to -get tasksel to tell you exactly which packages it plan to install -before doing the installation. So if you are curious or careful, -check the output from the isenkram-* command line tools first.

- -

The information about which packages are handling which hardware is -fetched either from the isenkram package itself in -/usr/share/isenkram/, from git.debian.org or from the APT package -database (using the Modaliases header). The APT package database -parsing have caused a nasty resource leak in the isenkram daemon (bugs -#719837 and -#730704). The cause is in -the python-apt code (bug -#745487), but using a -workaround I was able to get rid of the file descriptor leak and -reduce the memory leak from ~30 MiB per hardware detection down to -around 2 MiB per hardware detection. It should make the desktop -daemon a lot more useful. The fix is in version 0.7 uploaded to -unstable today.

- -

I believe the current way of mapping hardware to packages in -Isenkram is is a good draft, but in the future I expect isenkram to -use the AppStream data source for this. A proposal for getting proper -AppStream support into Debian is floating around as -DEP-11, and -GSoC -project will take place this summer to improve the situation. I -look forward to seeing the result, and welcome patches for isenkram to -start using the information when it is ready.

- -

If you want your package to map to some specific hardware, either -add a "Xb-Modaliases" header to your control file like I did in -the pymissile -package or submit a bug report with the details to the isenkram -package. See also -all my -blog posts tagged isenkram for details on the notation. I expect -the information will be migrated to AppStream eventually, but for the -moment I got no better place to store it.

- - - Tags: debian, english, isenkram. - - -

@@ -909,7 +1025,7 @@ moment I got no better place to store it.

August (2)

September (1)

September (2)

@@ -1090,9 +1206,9 @@ moment I got no better place to store it.

@@ -1102,7 +1218,7 @@ moment I got no better place to store it.

diff --git a/blog/index.rss b/blog/index.rss index 4bf4664e98..1fb369c9e8 100644 --- a/blog/index.rss +++ b/blog/index.rss @@ -6,6 +6,211 @@ http://people.skolelinux.org/pere/blog/ + + Speeding up the Debian installer using eatmydata and dpkg-divert + http://people.skolelinux.org/pere/blog/Speeding_up_the_Debian_installer_using_eatmydata_and_dpkg_divert.html + http://people.skolelinux.org/pere/blog/Speeding_up_the_Debian_installer_using_eatmydata_and_dpkg_divert.html + Tue, 16 Sep 2014 14:00:00 +0200 + The <a href="https://www.debian.org/">Debian</a> installer could be +a lot quicker. When we install more than 2000 packages in +<a href="http://www.skolelinux.org/">Skolelinux / Debian Edu</a> using +tasksel in the installer, unpacking the binary packages take forever. +A part of the slow I/O issue was discussed in +<a href="https://bugs.debian.org/613428">bug #613428</a> about too +much file system sync-ing done by dpkg, which is the package +responsible for unpacking the binary packages. Other parts (like code +executed by postinst scripts) might also sync to disk during +installation. All this sync-ing to disk do not really make sense to +me. If the machine crash half-way through, I start over, I do not try +to salvage the half installed system. So the failure sync-ing is +supposed to protect against, hardware or system crash, is not really +relevant while the installer is running. + +A few days ago, I thought of a way to get rid of all the file +system sync()-ing in a fairly non-intrusive way, without the need to +change the code in several packages. The idea is not new, but I have +not heard anyone propose the approach using dpkg-divert before. It +depend on the small and clever package +<a href="https://packages.qa.debian.org/eatmydata">eatmydata</a>, which +uses LD_PRELOAD to replace the system functions for syncing data to +disk with functions doing nothing, thus allowing programs to live +dangerous while speeding up disk I/O significantly. Instead of +modifying the implementation of dpkg, apt and tasksel (which are the +packages responsible for selecting, fetching and installing packages), +it occurred to me that we could just divert the programs away, replace +them with a simple shell wrapper calling +"eatmydata $program $@", to get the same effect. +Yesterday I decided to test the idea, and wrapped up a simple +implementation for the Debian Edu udeb. + +The effect was stunning. In my first test it reduced the running +time of the pkgsel step (installing tasks) from 64 to less than 44 +minutes (20 minutes shaved off the installation) on an old Dell +Latitude D505 machine. I am not quite sure what the optimised time +would have been, as I messed up the testing a bit, causing the debconf +priority to get low enough for two questions to pop up during +installation. As soon as I saw the questions I moved the installation +along, but do not know how long the question were holding up the +installation. I did some more measurements using Debian Edu Jessie, +and got these results. The time measured is the time stamp in +/var/log/syslog between the "pkgsel: starting tasksel" and the +"pkgsel: finishing up" lines, if you want to do the same measurement +yourself. In Debian Edu, the tasksel dialog do not show up, and the +timing thus do not depend on how quickly the user handle the tasksel +dialog. + +<table> + +<tr> +<th>Machine/setup</th> +<th>Original tasksel</th> +<th>Optimised tasksel</th> +<th>Reduction</th> +</tr> + +<tr> +<td>Latitude D505 Main+LTSP LXDE</td> +<td>64 min (07:46-08:50)</td> +<td><44 min (11:27-12:11)</td> +<td>>20 min 18%</td> +</tr> + +<tr> +<td>Latitude D505 Roaming LXDE</td> +<td>57 min (08:48-09:45)</td> +<td>34 min (07:43-08:17)</td> +<td>23 min 40%</td> +</tr> + +<tr> +<td>Latitude D505 Minimal</td> +<td>22 min (10:37-10:59)</td> +<td>11 min (11:16-11:27)</td> +<td>11 min 50%</td> +</tr> + +<tr> +<td>Thinkpad X200 Minimal</td> +<td>6 min (08:19-08:25)</td> +<td>4 min (08:04-08:08)</td> +<td>2 min 33%</td> +</tr> + +<tr> +<td>Thinkpad X200 Roaming KDE</td> +<td>19 min (09:21-09:40)</td> +<td>15 min (10:25-10:40)</td> +<td>4 min 21%</td> +</tr> + +</table> + +The test is done using a netinst ISO on a USB stick, so some of the +time is spent downloading packages. The connection to the Internet +was 100Mbit/s during testing, so downloading should not be a +significant factor in the measurement. Download typically took a few +seconds to a few minutes, depending on the amount of packages being +installed. + +The speedup is implemented by using two hooks in +<a href="https://www.debian.org/devel/debian-installer/">Debian +Installer</a>, the pre-pkgsel.d hook to set up the diverts, and the +finish-install.d hook to remove the divert at the end of the +installation. I picked the pre-pkgsel.d hook instead of the +post-base-installer.d hook because I test using an ISO without the +eatmydata package included, and the post-base-installer.d hook in +Debian Edu can only operate on packages included in the ISO. The +negative effect of this is that I am unable to activate this +optimization for the kernel installation step in d-i. If the code is +moved to the post-base-installer.d hook, the speedup would be larger +for the entire installation. + +I've implemented this in the +<a href="https://packages.qa.debian.org/debian-edu-install">debian-edu-install</a> +git repository, and plan to provide the optimization as part of the +Debian Edu installation. If you want to test this yourself, you can +create two files in the installer (or in an udeb). One shell script +need do go into /usr/lib/pre-pkgsel.d/, with content like this: + +<blockquote><pre> +#!/bin/sh +set -e +. /usr/share/debconf/confmodule +info() { + logger -t my-pkgsel "info: $*" +} +error() { + logger -t my-pkgsel "error: $*" +} +override_install() { + apt-install eatmydata || true + if [ -x /target/usr/bin/eatmydata ] ; then + for bin in dpkg apt-get aptitude tasksel ; do + file=/usr/bin/$bin + # Test that the file exist and have not been diverted already. + if [ -f /target$file ] ; then + info "diverting $file using eatmydata" + printf "#!/bin/sh\neatmydata $bin.distrib \"\$@\"\n" \ + > /target$file.edu + chmod 755 /target$file.edu + in-target dpkg-divert --package debian-edu-config \ + --rename --quiet --add $file + ln -sf ./$bin.edu /target$file + else + error "unable to divert $file, as it is missing." + fi + done + else + error "unable to find /usr/bin/eatmydata after installing the eatmydata pacage" + fi +} + +override_install +</pre></blockquote> + +To clean up, another shell script should go into +/usr/lib/finish-install.d/ with code like this: + +<blockquote><pre> +#! /bin/sh -e +. /usr/share/debconf/confmodule +error() { + logger -t my-finish-install "error: $@" +} +remove_install_override() { + for bin in dpkg apt-get aptitude tasksel ; do + file=/usr/bin/$bin + if [ -x /target$file.edu ] ; then + rm /target$file + in-target dpkg-divert --package debian-edu-config \ + --rename --quiet --remove $file + rm /target$file.edu + else + error "Missing divert for $file." + fi + done + sync # Flush file buffers before continuing +} + +remove_install_override +</pre></blockquote> + +In Debian Edu, I placed both code fragments in a separate script +edu-eatmydata-install and call it from the pre-pkgsel.d and +finish-install.d scripts. + +By now you might ask if this change should get into the normal +Debian installer too? I suspect it should, but am not sure the +current debian-installer coordinators find it useful enough. It also +depend on the side effects of the change. I'm not aware of any, but I +guess we will see if the change is safe after some more testing. +Perhaps there is some package in Debian depending on sync() and +fsync() having effect? Perhaps it should go into its own udeb, to +allow those of us wanting to enable it to do so without affecting +everyone. + + + Good bye subkeys.pgp.net, welcome pool.sks-keyservers.net http://people.skolelinux.org/pere/blog/Good_bye_subkeys_pgp_net__welcome_pool_sks_keyservers_net.html @@ -720,94 +925,5 @@ irc.freenode.net IRC server</a>. - - Install hardware dependent packages using tasksel (Isenkram 0.7) - http://people.skolelinux.org/pere/blog/Install_hardware_dependent_packages_using_tasksel__Isenkram_0_7_.html - http://people.skolelinux.org/pere/blog/Install_hardware_dependent_packages_using_tasksel__Isenkram_0_7_.html - Wed, 23 Apr 2014 14:50:00 +0200 - It would be nice if it was easier in Debian to get all the hardware -related packages relevant for the computer installed automatically. -So I implemented one, using -<a href="http://packages.qa.debian.org/isenkram">my Isenkram -package</a>. To use it, install the tasksel and isenkram packages and -run tasksel as user root. You should be presented with a new option, -"Hardware specific packages (autodetected by isenkram)". When you -select it, tasksel will install the packages isenkram claim is fit for -the current hardware, hot pluggable or not. - -The implementation is in two files, one is the tasksel menu entry -description, and the other is the script used to extract the list of -packages to install. The first part is in -<tt>/usr/share/tasksel/descs/isenkram.desc</tt> and look like -this: - -<blockquote><pre> -Task: isenkram -Section: hardware -Description: Hardware specific packages (autodetected by isenkram) - Based on the detected hardware various hardware specific packages are - proposed. -Test-new-install: mark show -Relevance: 8 -Packages: for-current-hardware -</pre></blockquote> - -The second part is in -<tt>/usr/lib/tasksel/packages/for-current-hardware</tt> and look like -this: - -<blockquote><pre> -#!/bin/sh -# -( - isenkram-lookup - isenkram-autoinstall-firmware -l -) | sort -u -</pre></blockquote> - -All in all, a very short and simple implementation making it -trivial to install the hardware dependent package we all may want to -have installed on our machines. I've not been able to find a way to -get tasksel to tell you exactly which packages it plan to install -before doing the installation. So if you are curious or careful, -check the output from the isenkram-* command line tools first. - -The information about which packages are handling which hardware is -fetched either from the isenkram package itself in -/usr/share/isenkram/, from git.debian.org or from the APT package -database (using the Modaliases header). The APT package database -parsing have caused a nasty resource leak in the isenkram daemon (bugs -<a href="http://bugs.debian.org/719837">#719837</a> and -<a href="http://bugs.debian.org/730704">#730704</a>). The cause is in -the python-apt code (bug -<a href="http://bugs.debian.org/745487">#745487</a>), but using a -workaround I was able to get rid of the file descriptor leak and -reduce the memory leak from ~30 MiB per hardware detection down to -around 2 MiB per hardware detection. It should make the desktop -daemon a lot more useful. The fix is in version 0.7 uploaded to -unstable today. - -I believe the current way of mapping hardware to packages in -Isenkram is is a good draft, but in the future I expect isenkram to -use the AppStream data source for this. A proposal for getting proper -AppStream support into Debian is floating around as -<a href="https://wiki.debian.org/DEP-11">DEP-11</a>, and -<a href="https://wiki.debian.org/SummerOfCode2014/Projects#SummerOfCode2014.2FProjects.2FAppStreamDEP11Implementation.AppStream.2FDEP-11_for_the_Debian_Archive">GSoC -project</a> will take place this summer to improve the situation. I -look forward to seeing the result, and welcome patches for isenkram to -start using the information when it is ready. - -If you want your package to map to some specific hardware, either -add a "Xb-Modaliases" header to your control file like I did in -<a href="http://packages.qa.debian.org/pymissile">the pymissile -package</a> or submit a bug report with the details to the isenkram -package. See also -<a href="http://people.skolelinux.org/pere/blog/tags/isenkram/">all my -blog posts tagged isenkram</a> for details on the notation. I expect -the information will be migrated to AppStream eventually, but for the -moment I got no better place to store it. - - - diff --git a/blog/jXplorer__a_very_nice_LDAP_GUI.html b/blog/jXplorer__a_very_nice_LDAP_GUI.html index f45b83d40f..ad8dca410e 100644 --- a/blog/jXplorer__a_very_nice_LDAP_GUI.html +++ b/blog/jXplorer__a_very_nice_LDAP_GUI.html @@ -72,7 +72,7 @@ and remove the failing query. Nothing big, but very annoying.

August (2)

September (1)

September (2)

@@ -253,9 +253,9 @@ and remove the failing query. Nothing big, but very annoying.

@@ -265,7 +265,7 @@ and remove the failing query. Nothing big, but very annoying.

diff --git a/blog/sitemap.xml b/blog/sitemap.xml index 0f94c42fc2..9c9109e3c1 100644 --- a/blog/sitemap.xml +++ b/blog/sitemap.xml @@ -2125,6 +2125,11 @@ 0.50 weekly + + http://people.skolelinux.org/pere/blog/Speeding_up_the_Debian_installer_using_eatmydata_and_dpkg_divert.html + 0.50 + weekly + http://people.skolelinux.org/pere/blog/Spr_kkoder_for_POSIX_locale_i_Norge.html 0.50 diff --git a/blog/systemd__an_interesting_alternative_to_upstart.html b/blog/systemd__an_interesting_alternative_to_upstart.html index 688110644b..0eee2cc905 100644 --- a/blog/systemd__an_interesting_alternative_to_upstart.html +++ b/blog/systemd__an_interesting_alternative_to_upstart.html @@ -90,7 +90,7 @@ with parallel booting enabled by default.

August (2)

September (1)

September (2)

@@ -271,9 +271,9 @@ with parallel booting enabled by default.

@@ -283,7 +283,7 @@ with parallel booting enabled by default.

diff --git a/blog/tags/3d-printer/index.html b/blog/tags/3d-printer/index.html index a1ad789eff..29f18ba334 100644 --- a/blog/tags/3d-printer/index.html +++ b/blog/tags/3d-printer/index.html @@ -622,7 +622,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

August (2)

September (1)

September (2)

@@ -803,9 +803,9 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

@@ -815,7 +815,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

diff --git a/blog/tags/amiga/index.html b/blog/tags/amiga/index.html index 31d7d5b5ff..b431f1c4dc 100644 --- a/blog/tags/amiga/index.html +++ b/blog/tags/amiga/index.html @@ -76,7 +76,7 @@ pakke. Kanskje Aros kunne vÃ¦rt interessant for et NUUG-foredrag?

August (2)

September (1)

September (2)

@@ -257,9 +257,9 @@ pakke. Kanskje Aros kunne vÃ¦rt interessant for et NUUG-foredrag?

@@ -269,7 +269,7 @@ pakke. Kanskje Aros kunne vÃ¦rt interessant for et NUUG-foredrag?

diff --git a/blog/tags/aros/index.html b/blog/tags/aros/index.html index 84679ebb22..966422ca57 100644 --- a/blog/tags/aros/index.html +++ b/blog/tags/aros/index.html @@ -76,7 +76,7 @@ pakke. Kanskje Aros kunne vÃ¦rt interessant for et NUUG-foredrag?

August (2)

September (1)

September (2)

@@ -257,9 +257,9 @@ pakke. Kanskje Aros kunne vÃ¦rt interessant for et NUUG-foredrag?

@@ -269,7 +269,7 @@ pakke. Kanskje Aros kunne vÃ¦rt interessant for et NUUG-foredrag?

diff --git a/blog/tags/bankid/index.html b/blog/tags/bankid/index.html index 42bb8274e8..b28e288311 100644 --- a/blog/tags/bankid/index.html +++ b/blog/tags/bankid/index.html @@ -482,7 +482,7 @@ til BankID-modellen.

August (2)

September (1)

September (2)

@@ -663,9 +663,9 @@ til BankID-modellen.

@@ -675,7 +675,7 @@ til BankID-modellen.

diff --git a/blog/tags/bitcoin/index.html b/blog/tags/bitcoin/index.html index 5579d3388b..62c71af754 100644 --- a/blog/tags/bitcoin/index.html +++ b/blog/tags/bitcoin/index.html @@ -603,7 +603,7 @@ donations to the address

August (2)

September (1)

September (2)

@@ -784,9 +784,9 @@ donations to the address

@@ -796,7 +796,7 @@ donations to the address

diff --git a/blog/tags/bootsystem/index.html b/blog/tags/bootsystem/index.html index 0dd290151d..3cab302895 100644 --- a/blog/tags/bootsystem/index.html +++ b/blog/tags/bootsystem/index.html @@ -1114,7 +1114,7 @@ insserv'. Will need to test if that work. :)

August (2)

September (1)

September (2)

@@ -1295,9 +1295,9 @@ insserv'. Will need to test if that work. :)

@@ -1307,7 +1307,7 @@ insserv'. Will need to test if that work. :)

diff --git a/blog/tags/bsa/index.html b/blog/tags/bsa/index.html index 5078dcbed7..e6e6fb84bd 100644 --- a/blog/tags/bsa/index.html +++ b/blog/tags/bsa/index.html @@ -164,7 +164,7 @@ pÃ¥ Slashdot.

August (2)

September (1)

September (2)

@@ -345,9 +345,9 @@ pÃ¥ Slashdot.

@@ -357,7 +357,7 @@ pÃ¥ Slashdot.

diff --git a/blog/tags/chrpath/index.html b/blog/tags/chrpath/index.html index b2c6b677f8..0c459541f0 100644 --- a/blog/tags/chrpath/index.html +++ b/blog/tags/chrpath/index.html @@ -171,7 +171,7 @@ include a testsuite check.

August (2)

September (1)

September (2)

@@ -352,9 +352,9 @@ include a testsuite check.

@@ -364,7 +364,7 @@ include a testsuite check.

diff --git a/blog/tags/debian edu/debian edu.rss b/blog/tags/debian edu/debian edu.rss index dec8e63046..ce0141a0bd 100644 --- a/blog/tags/debian edu/debian edu.rss +++ b/blog/tags/debian edu/debian edu.rss @@ -6,6 +6,211 @@ http://people.skolelinux.org/pere/blog/ + + Speeding up the Debian installer using eatmydata and dpkg-divert + http://people.skolelinux.org/pere/blog/Speeding_up_the_Debian_installer_using_eatmydata_and_dpkg_divert.html + http://people.skolelinux.org/pere/blog/Speeding_up_the_Debian_installer_using_eatmydata_and_dpkg_divert.html + Tue, 16 Sep 2014 14:00:00 +0200 + The <a href="https://www.debian.org/">Debian</a> installer could be +a lot quicker. When we install more than 2000 packages in +<a href="http://www.skolelinux.org/">Skolelinux / Debian Edu</a> using +tasksel in the installer, unpacking the binary packages take forever. +A part of the slow I/O issue was discussed in +<a href="https://bugs.debian.org/613428">bug #613428</a> about too +much file system sync-ing done by dpkg, which is the package +responsible for unpacking the binary packages. Other parts (like code +executed by postinst scripts) might also sync to disk during +installation. All this sync-ing to disk do not really make sense to +me. If the machine crash half-way through, I start over, I do not try +to salvage the half installed system. So the failure sync-ing is +supposed to protect against, hardware or system crash, is not really +relevant while the installer is running. + +A few days ago, I thought of a way to get rid of all the file +system sync()-ing in a fairly non-intrusive way, without the need to +change the code in several packages. The idea is not new, but I have +not heard anyone propose the approach using dpkg-divert before. It +depend on the small and clever package +<a href="https://packages.qa.debian.org/eatmydata">eatmydata</a>, which +uses LD_PRELOAD to replace the system functions for syncing data to +disk with functions doing nothing, thus allowing programs to live +dangerous while speeding up disk I/O significantly. Instead of +modifying the implementation of dpkg, apt and tasksel (which are the +packages responsible for selecting, fetching and installing packages), +it occurred to me that we could just divert the programs away, replace +them with a simple shell wrapper calling +"eatmydata $program $@", to get the same effect. +Yesterday I decided to test the idea, and wrapped up a simple +implementation for the Debian Edu udeb. + +The effect was stunning. In my first test it reduced the running +time of the pkgsel step (installing tasks) from 64 to less than 44 +minutes (20 minutes shaved off the installation) on an old Dell +Latitude D505 machine. I am not quite sure what the optimised time +would have been, as I messed up the testing a bit, causing the debconf +priority to get low enough for two questions to pop up during +installation. As soon as I saw the questions I moved the installation +along, but do not know how long the question were holding up the +installation. I did some more measurements using Debian Edu Jessie, +and got these results. The time measured is the time stamp in +/var/log/syslog between the "pkgsel: starting tasksel" and the +"pkgsel: finishing up" lines, if you want to do the same measurement +yourself. In Debian Edu, the tasksel dialog do not show up, and the +timing thus do not depend on how quickly the user handle the tasksel +dialog. + +<table> + +<tr> +<th>Machine/setup</th> +<th>Original tasksel</th> +<th>Optimised tasksel</th> +<th>Reduction</th> +</tr> + +<tr> +<td>Latitude D505 Main+LTSP LXDE</td> +<td>64 min (07:46-08:50)</td> +<td><44 min (11:27-12:11)</td> +<td>>20 min 18%</td> +</tr> + +<tr> +<td>Latitude D505 Roaming LXDE</td> +<td>57 min (08:48-09:45)</td> +<td>34 min (07:43-08:17)</td> +<td>23 min 40%</td> +</tr> + +<tr> +<td>Latitude D505 Minimal</td> +<td>22 min (10:37-10:59)</td> +<td>11 min (11:16-11:27)</td> +<td>11 min 50%</td> +</tr> + +<tr> +<td>Thinkpad X200 Minimal</td> +<td>6 min (08:19-08:25)</td> +<td>4 min (08:04-08:08)</td> +<td>2 min 33%</td> +</tr> + +<tr> +<td>Thinkpad X200 Roaming KDE</td> +<td>19 min (09:21-09:40)</td> +<td>15 min (10:25-10:40)</td> +<td>4 min 21%</td> +</tr> + +</table> + +The test is done using a netinst ISO on a USB stick, so some of the +time is spent downloading packages. The connection to the Internet +was 100Mbit/s during testing, so downloading should not be a +significant factor in the measurement. Download typically took a few +seconds to a few minutes, depending on the amount of packages being +installed. + +The speedup is implemented by using two hooks in +<a href="https://www.debian.org/devel/debian-installer/">Debian +Installer</a>, the pre-pkgsel.d hook to set up the diverts, and the +finish-install.d hook to remove the divert at the end of the +installation. I picked the pre-pkgsel.d hook instead of the +post-base-installer.d hook because I test using an ISO without the +eatmydata package included, and the post-base-installer.d hook in +Debian Edu can only operate on packages included in the ISO. The +negative effect of this is that I am unable to activate this +optimization for the kernel installation step in d-i. If the code is +moved to the post-base-installer.d hook, the speedup would be larger +for the entire installation. + +I've implemented this in the +<a href="https://packages.qa.debian.org/debian-edu-install">debian-edu-install</a> +git repository, and plan to provide the optimization as part of the +Debian Edu installation. If you want to test this yourself, you can +create two files in the installer (or in an udeb). One shell script +need do go into /usr/lib/pre-pkgsel.d/, with content like this: + +<blockquote><pre> +#!/bin/sh +set -e +. /usr/share/debconf/confmodule +info() { + logger -t my-pkgsel "info: $*" +} +error() { + logger -t my-pkgsel "error: $*" +} +override_install() { + apt-install eatmydata || true + if [ -x /target/usr/bin/eatmydata ] ; then + for bin in dpkg apt-get aptitude tasksel ; do + file=/usr/bin/$bin + # Test that the file exist and have not been diverted already. + if [ -f /target$file ] ; then + info "diverting $file using eatmydata" + printf "#!/bin/sh\neatmydata $bin.distrib \"\$@\"\n" \ + > /target$file.edu + chmod 755 /target$file.edu + in-target dpkg-divert --package debian-edu-config \ + --rename --quiet --add $file + ln -sf ./$bin.edu /target$file + else + error "unable to divert $file, as it is missing." + fi + done + else + error "unable to find /usr/bin/eatmydata after installing the eatmydata pacage" + fi +} + +override_install +</pre></blockquote> + +To clean up, another shell script should go into +/usr/lib/finish-install.d/ with code like this: + +<blockquote><pre> +#! /bin/sh -e +. /usr/share/debconf/confmodule +error() { + logger -t my-finish-install "error: $@" +} +remove_install_override() { + for bin in dpkg apt-get aptitude tasksel ; do + file=/usr/bin/$bin + if [ -x /target$file.edu ] ; then + rm /target$file + in-target dpkg-divert --package debian-edu-config \ + --rename --quiet --remove $file + rm /target$file.edu + else + error "Missing divert for $file." + fi + done + sync # Flush file buffers before continuing +} + +remove_install_override +</pre></blockquote> + +In Debian Edu, I placed both code fragments in a separate script +edu-eatmydata-install and call it from the pre-pkgsel.d and +finish-install.d scripts. + +By now you might ask if this change should get into the normal +Debian installer too? I suspect it should, but am not sure the +current debian-installer coordinators find it useful enough. It also +depend on the side effects of the change. I'm not aware of any, but I +guess we will see if the change is safe after some more testing. +Perhaps there is some package in Debian depending on sync() and +fsync() having effect? Perhaps it should go into its own udeb, to +allow those of us wanting to enable it to do so without affecting +everyone. + + + Debian Edu interview: Bernd Zeitzen http://people.skolelinux.org/pere/blog/Debian_Edu_interview__Bernd_Zeitzen.html diff --git a/blog/tags/debian edu/index.html b/blog/tags/debian edu/index.html index 086d2b4914..3310ecc861 100644 --- a/blog/tags/debian edu/index.html +++ b/blog/tags/debian edu/index.html @@ -20,6 +20,223 @@

Entries tagged "debian edu".

+ Speeding up the Debian installer using eatmydata and dpkg-divert +

+ 16th September 2014 +

+ +

Machine/setup	Original tasksel	Optimised tasksel	Reduction
Latitude D505 Main+LTSP LXDE	64 min (07:46-08:50)	<44 min (11:27-12:11)	>20 min 18%
Latitude D505 Roaming LXDE	57 min (08:48-09:45)	34 min (07:43-08:17)	23 min 40%
Latitude D505 Minimal	22 min (10:37-10:59)	11 min (11:16-11:27)	11 min 50%
Thinkpad X200 Minimal	6 min (08:19-08:25)	4 min (08:04-08:08)	2 min 33%
Thinkpad X200 Roaming KDE	19 min (09:21-09:40)	15 min (10:25-10:40)	4 min 21%

+#!/bin/sh
+set -e
+. /usr/share/debconf/confmodule
+info() {
+    logger -t my-pkgsel "info: $*"
+}
+error() {
+    logger -t my-pkgsel "error: $*"
+}
+override_install() {
+    apt-install eatmydata || true
+    if [ -x /target/usr/bin/eatmydata ] ; then
+        for bin in dpkg apt-get aptitude tasksel ; do
+            file=/usr/bin/$bin
+            # Test that the file exist and have not been diverted already.
+            if [ -f /target$file ] ; then
+                info "diverting $file using eatmydata"
+                printf "#!/bin/sh\neatmydata $bin.distrib \"\$@\"\n" \
+                    > /target$file.edu
+                chmod 755 /target$file.edu
+                in-target dpkg-divert --package debian-edu-config \
+                    --rename --quiet --add $file
+                ln -sf ./$bin.edu /target$file
+            else
+                error "unable to divert $file, as it is missing."
+            fi
+        done
+    else
+        error "unable to find /usr/bin/eatmydata after installing the eatmydata pacage"
+    fi
+}
+
+override_install
+

+ +

To clean up, another shell script should go into +/usr/lib/finish-install.d/ with code like this: + +

+#! /bin/sh -e
+. /usr/share/debconf/confmodule
+error() {
+    logger -t my-finish-install "error: $@"
+}
+remove_install_override() {
+    for bin in dpkg apt-get aptitude tasksel ; do
+        file=/usr/bin/$bin
+        if [ -x /target$file.edu ] ; then
+            rm /target$file
+            in-target dpkg-divert --package debian-edu-config \
+                --rename --quiet --remove $file
+            rm /target$file.edu
+        else
+            error "Missing divert for $file."
+        fi
+    done
+    sync # Flush file buffers before continuing
+}
+
+remove_install_override
+

+ +

In Debian Edu, I placed both code fragments in a separate script +edu-eatmydata-install and call it from the pre-pkgsel.d and +finish-install.d scripts.

+ +

+ + + Tags: debian, debian edu, english. + + +

Debian Edu interview: Bernd Zeitzen @@ -13695,7 +13912,7 @@ be the only one fitting our needs. :/

August (2)

September (1)

September (2)

@@ -13876,9 +14093,9 @@ be the only one fitting our needs. :/

@@ -13888,7 +14105,7 @@ be the only one fitting our needs. :/

diff --git a/blog/tags/debian/debian.rss b/blog/tags/debian/debian.rss index 73536fc598..f55ec1e935 100644 --- a/blog/tags/debian/debian.rss +++ b/blog/tags/debian/debian.rss @@ -6,6 +6,211 @@ http://people.skolelinux.org/pere/blog/ + + Speeding up the Debian installer using eatmydata and dpkg-divert + http://people.skolelinux.org/pere/blog/Speeding_up_the_Debian_installer_using_eatmydata_and_dpkg_divert.html + http://people.skolelinux.org/pere/blog/Speeding_up_the_Debian_installer_using_eatmydata_and_dpkg_divert.html + Tue, 16 Sep 2014 14:00:00 +0200 + The <a href="https://www.debian.org/">Debian</a> installer could be +a lot quicker. When we install more than 2000 packages in +<a href="http://www.skolelinux.org/">Skolelinux / Debian Edu</a> using +tasksel in the installer, unpacking the binary packages take forever. +A part of the slow I/O issue was discussed in +<a href="https://bugs.debian.org/613428">bug #613428</a> about too +much file system sync-ing done by dpkg, which is the package +responsible for unpacking the binary packages. Other parts (like code +executed by postinst scripts) might also sync to disk during +installation. All this sync-ing to disk do not really make sense to +me. If the machine crash half-way through, I start over, I do not try +to salvage the half installed system. So the failure sync-ing is +supposed to protect against, hardware or system crash, is not really +relevant while the installer is running. + +A few days ago, I thought of a way to get rid of all the file +system sync()-ing in a fairly non-intrusive way, without the need to +change the code in several packages. The idea is not new, but I have +not heard anyone propose the approach using dpkg-divert before. It +depend on the small and clever package +<a href="https://packages.qa.debian.org/eatmydata">eatmydata</a>, which +uses LD_PRELOAD to replace the system functions for syncing data to +disk with functions doing nothing, thus allowing programs to live +dangerous while speeding up disk I/O significantly. Instead of +modifying the implementation of dpkg, apt and tasksel (which are the +packages responsible for selecting, fetching and installing packages), +it occurred to me that we could just divert the programs away, replace +them with a simple shell wrapper calling +"eatmydata $program $@", to get the same effect. +Yesterday I decided to test the idea, and wrapped up a simple +implementation for the Debian Edu udeb. + +The effect was stunning. In my first test it reduced the running +time of the pkgsel step (installing tasks) from 64 to less than 44 +minutes (20 minutes shaved off the installation) on an old Dell +Latitude D505 machine. I am not quite sure what the optimised time +would have been, as I messed up the testing a bit, causing the debconf +priority to get low enough for two questions to pop up during +installation. As soon as I saw the questions I moved the installation +along, but do not know how long the question were holding up the +installation. I did some more measurements using Debian Edu Jessie, +and got these results. The time measured is the time stamp in +/var/log/syslog between the "pkgsel: starting tasksel" and the +"pkgsel: finishing up" lines, if you want to do the same measurement +yourself. In Debian Edu, the tasksel dialog do not show up, and the +timing thus do not depend on how quickly the user handle the tasksel +dialog. + +<table> + +<tr> +<th>Machine/setup</th> +<th>Original tasksel</th> +<th>Optimised tasksel</th> +<th>Reduction</th> +</tr> + +<tr> +<td>Latitude D505 Main+LTSP LXDE</td> +<td>64 min (07:46-08:50)</td> +<td><44 min (11:27-12:11)</td> +<td>>20 min 18%</td> +</tr> + +<tr> +<td>Latitude D505 Roaming LXDE</td> +<td>57 min (08:48-09:45)</td> +<td>34 min (07:43-08:17)</td> +<td>23 min 40%</td> +</tr> + +<tr> +<td>Latitude D505 Minimal</td> +<td>22 min (10:37-10:59)</td> +<td>11 min (11:16-11:27)</td> +<td>11 min 50%</td> +</tr> + +<tr> +<td>Thinkpad X200 Minimal</td> +<td>6 min (08:19-08:25)</td> +<td>4 min (08:04-08:08)</td> +<td>2 min 33%</td> +</tr> + +<tr> +<td>Thinkpad X200 Roaming KDE</td> +<td>19 min (09:21-09:40)</td> +<td>15 min (10:25-10:40)</td> +<td>4 min 21%</td> +</tr> + +</table> + +The test is done using a netinst ISO on a USB stick, so some of the +time is spent downloading packages. The connection to the Internet +was 100Mbit/s during testing, so downloading should not be a +significant factor in the measurement. Download typically took a few +seconds to a few minutes, depending on the amount of packages being +installed. + +The speedup is implemented by using two hooks in +<a href="https://www.debian.org/devel/debian-installer/">Debian +Installer</a>, the pre-pkgsel.d hook to set up the diverts, and the +finish-install.d hook to remove the divert at the end of the +installation. I picked the pre-pkgsel.d hook instead of the +post-base-installer.d hook because I test using an ISO without the +eatmydata package included, and the post-base-installer.d hook in +Debian Edu can only operate on packages included in the ISO. The +negative effect of this is that I am unable to activate this +optimization for the kernel installation step in d-i. If the code is +moved to the post-base-installer.d hook, the speedup would be larger +for the entire installation. + +I've implemented this in the +<a href="https://packages.qa.debian.org/debian-edu-install">debian-edu-install</a> +git repository, and plan to provide the optimization as part of the +Debian Edu installation. If you want to test this yourself, you can +create two files in the installer (or in an udeb). One shell script +need do go into /usr/lib/pre-pkgsel.d/, with content like this: + +<blockquote><pre> +#!/bin/sh +set -e +. /usr/share/debconf/confmodule +info() { + logger -t my-pkgsel "info: $*" +} +error() { + logger -t my-pkgsel "error: $*" +} +override_install() { + apt-install eatmydata || true + if [ -x /target/usr/bin/eatmydata ] ; then + for bin in dpkg apt-get aptitude tasksel ; do + file=/usr/bin/$bin + # Test that the file exist and have not been diverted already. + if [ -f /target$file ] ; then + info "diverting $file using eatmydata" + printf "#!/bin/sh\neatmydata $bin.distrib \"\$@\"\n" \ + > /target$file.edu + chmod 755 /target$file.edu + in-target dpkg-divert --package debian-edu-config \ + --rename --quiet --add $file + ln -sf ./$bin.edu /target$file + else + error "unable to divert $file, as it is missing." + fi + done + else + error "unable to find /usr/bin/eatmydata after installing the eatmydata pacage" + fi +} + +override_install +</pre></blockquote> + +To clean up, another shell script should go into +/usr/lib/finish-install.d/ with code like this: + +<blockquote><pre> +#! /bin/sh -e +. /usr/share/debconf/confmodule +error() { + logger -t my-finish-install "error: $@" +} +remove_install_override() { + for bin in dpkg apt-get aptitude tasksel ; do + file=/usr/bin/$bin + if [ -x /target$file.edu ] ; then + rm /target$file + in-target dpkg-divert --package debian-edu-config \ + --rename --quiet --remove $file + rm /target$file.edu + else + error "Missing divert for $file." + fi + done + sync # Flush file buffers before continuing +} + +remove_install_override +</pre></blockquote> + +In Debian Edu, I placed both code fragments in a separate script +edu-eatmydata-install and call it from the pre-pkgsel.d and +finish-install.d scripts. + +By now you might ask if this change should get into the normal +Debian installer too? I suspect it should, but am not sure the +current debian-installer coordinators find it useful enough. It also +depend on the side effects of the change. I'm not aware of any, but I +guess we will see if the change is safe after some more testing. +Perhaps there is some package in Debian depending on sync() and +fsync() having effect? Perhaps it should go into its own udeb, to +allow those of us wanting to enable it to do so without affecting +everyone. + + + Good bye subkeys.pgp.net, welcome pool.sks-keyservers.net http://people.skolelinux.org/pere/blog/Good_bye_subkeys_pgp_net__welcome_pool_sks_keyservers_net.html diff --git a/blog/tags/debian/index.html b/blog/tags/debian/index.html index b471b97059..1e21b30ee2 100644 --- a/blog/tags/debian/index.html +++ b/blog/tags/debian/index.html @@ -20,6 +20,223 @@

Entries tagged "debian".

+ Speeding up the Debian installer using eatmydata and dpkg-divert +

+ 16th September 2014 +

+ +

Machine/setup	Original tasksel	Optimised tasksel	Reduction
Latitude D505 Main+LTSP LXDE	64 min (07:46-08:50)	<44 min (11:27-12:11)	>20 min 18%
Latitude D505 Roaming LXDE	57 min (08:48-09:45)	34 min (07:43-08:17)	23 min 40%
Latitude D505 Minimal	22 min (10:37-10:59)	11 min (11:16-11:27)	11 min 50%
Thinkpad X200 Minimal	6 min (08:19-08:25)	4 min (08:04-08:08)	2 min 33%
Thinkpad X200 Roaming KDE	19 min (09:21-09:40)	15 min (10:25-10:40)	4 min 21%

+#!/bin/sh
+set -e
+. /usr/share/debconf/confmodule
+info() {
+    logger -t my-pkgsel "info: $*"
+}
+error() {
+    logger -t my-pkgsel "error: $*"
+}
+override_install() {
+    apt-install eatmydata || true
+    if [ -x /target/usr/bin/eatmydata ] ; then
+        for bin in dpkg apt-get aptitude tasksel ; do
+            file=/usr/bin/$bin
+            # Test that the file exist and have not been diverted already.
+            if [ -f /target$file ] ; then
+                info "diverting $file using eatmydata"
+                printf "#!/bin/sh\neatmydata $bin.distrib \"\$@\"\n" \
+                    > /target$file.edu
+                chmod 755 /target$file.edu
+                in-target dpkg-divert --package debian-edu-config \
+                    --rename --quiet --add $file
+                ln -sf ./$bin.edu /target$file
+            else
+                error "unable to divert $file, as it is missing."
+            fi
+        done
+    else
+        error "unable to find /usr/bin/eatmydata after installing the eatmydata pacage"
+    fi
+}
+
+override_install
+

+ +

To clean up, another shell script should go into +/usr/lib/finish-install.d/ with code like this: + +

+#! /bin/sh -e
+. /usr/share/debconf/confmodule
+error() {
+    logger -t my-finish-install "error: $@"
+}
+remove_install_override() {
+    for bin in dpkg apt-get aptitude tasksel ; do
+        file=/usr/bin/$bin
+        if [ -x /target$file.edu ] ; then
+            rm /target$file
+            in-target dpkg-divert --package debian-edu-config \
+                --rename --quiet --remove $file
+            rm /target$file.edu
+        else
+            error "Missing divert for $file."
+        fi
+    done
+    sync # Flush file buffers before continuing
+}
+
+remove_install_override
+

+ +

In Debian Edu, I placed both code fragments in a separate script +edu-eatmydata-install and call it from the pre-pkgsel.d and +finish-install.d scripts.

+ +

+ + + Tags: debian, debian edu, english. + + +

Good bye subkeys.pgp.net, welcome pool.sks-keyservers.net @@ -8189,7 +8406,7 @@ be the only one fitting our needs. :/

August (2)

September (1)

September (2)

@@ -8370,9 +8587,9 @@ be the only one fitting our needs. :/

@@ -8382,7 +8599,7 @@ be the only one fitting our needs. :/

diff --git a/blog/tags/digistan/index.html b/blog/tags/digistan/index.html index fb9167d980..5a378a8c1b 100644 --- a/blog/tags/digistan/index.html +++ b/blog/tags/digistan/index.html @@ -1263,7 +1263,7 @@ produkter basert pÃ¥ standarden.

August (2)

September (1)

September (2)

@@ -1444,9 +1444,9 @@ produkter basert pÃ¥ standarden.

@@ -1456,7 +1456,7 @@ produkter basert pÃ¥ standarden.

diff --git a/blog/tags/dld/index.html b/blog/tags/dld/index.html index f9a982951b..b956d38bf2 100644 --- a/blog/tags/dld/index.html +++ b/blog/tags/dld/index.html @@ -1404,7 +1404,7 @@ fÃ¥r snart se om det stemmer.

August (2)

September (1)

September (2)

@@ -1585,9 +1585,9 @@ fÃ¥r snart se om det stemmer.

@@ -1597,7 +1597,7 @@ fÃ¥r snart se om det stemmer.

diff --git a/blog/tags/docbook/index.html b/blog/tags/docbook/index.html index c62af65efb..fe343ef4ae 100644 --- a/blog/tags/docbook/index.html +++ b/blog/tags/docbook/index.html @@ -952,7 +952,7 @@ slik at du kan oppdatere direkte.

August (2)

September (1)

September (2)

@@ -1133,9 +1133,9 @@ slik at du kan oppdatere direkte.

@@ -1145,7 +1145,7 @@ slik at du kan oppdatere direkte.

diff --git a/blog/tags/drivstoffpriser/index.html b/blog/tags/drivstoffpriser/index.html index 135763c7b6..32b6554777 100644 --- a/blog/tags/drivstoffpriser/index.html +++ b/blog/tags/drivstoffpriser/index.html @@ -505,7 +505,7 @@ en liste med stasjoner pÃ¥ samme format som PriserVedStasjoner.

August (2)

September (1)

September (2)

@@ -686,9 +686,9 @@ en liste med stasjoner pÃ¥ samme format som PriserVedStasjoner.

@@ -698,7 +698,7 @@ en liste med stasjoner pÃ¥ samme format som PriserVedStasjoner.

diff --git a/blog/tags/english/english.rss b/blog/tags/english/english.rss index c563fe82a7..7ddf617fd6 100644 --- a/blog/tags/english/english.rss +++ b/blog/tags/english/english.rss @@ -6,6 +6,211 @@ http://people.skolelinux.org/pere/blog/ + + Speeding up the Debian installer using eatmydata and dpkg-divert + http://people.skolelinux.org/pere/blog/Speeding_up_the_Debian_installer_using_eatmydata_and_dpkg_divert.html + http://people.skolelinux.org/pere/blog/Speeding_up_the_Debian_installer_using_eatmydata_and_dpkg_divert.html + Tue, 16 Sep 2014 14:00:00 +0200 + The <a href="https://www.debian.org/">Debian</a> installer could be +a lot quicker. When we install more than 2000 packages in +<a href="http://www.skolelinux.org/">Skolelinux / Debian Edu</a> using +tasksel in the installer, unpacking the binary packages take forever. +A part of the slow I/O issue was discussed in +<a href="https://bugs.debian.org/613428">bug #613428</a> about too +much file system sync-ing done by dpkg, which is the package +responsible for unpacking the binary packages. Other parts (like code +executed by postinst scripts) might also sync to disk during +installation. All this sync-ing to disk do not really make sense to +me. If the machine crash half-way through, I start over, I do not try +to salvage the half installed system. So the failure sync-ing is +supposed to protect against, hardware or system crash, is not really +relevant while the installer is running. + +A few days ago, I thought of a way to get rid of all the file +system sync()-ing in a fairly non-intrusive way, without the need to +change the code in several packages. The idea is not new, but I have +not heard anyone propose the approach using dpkg-divert before. It +depend on the small and clever package +<a href="https://packages.qa.debian.org/eatmydata">eatmydata</a>, which +uses LD_PRELOAD to replace the system functions for syncing data to +disk with functions doing nothing, thus allowing programs to live +dangerous while speeding up disk I/O significantly. Instead of +modifying the implementation of dpkg, apt and tasksel (which are the +packages responsible for selecting, fetching and installing packages), +it occurred to me that we could just divert the programs away, replace +them with a simple shell wrapper calling +"eatmydata $program $@", to get the same effect. +Yesterday I decided to test the idea, and wrapped up a simple +implementation for the Debian Edu udeb. + +The effect was stunning. In my first test it reduced the running +time of the pkgsel step (installing tasks) from 64 to less than 44 +minutes (20 minutes shaved off the installation) on an old Dell +Latitude D505 machine. I am not quite sure what the optimised time +would have been, as I messed up the testing a bit, causing the debconf +priority to get low enough for two questions to pop up during +installation. As soon as I saw the questions I moved the installation +along, but do not know how long the question were holding up the +installation. I did some more measurements using Debian Edu Jessie, +and got these results. The time measured is the time stamp in +/var/log/syslog between the "pkgsel: starting tasksel" and the +"pkgsel: finishing up" lines, if you want to do the same measurement +yourself. In Debian Edu, the tasksel dialog do not show up, and the +timing thus do not depend on how quickly the user handle the tasksel +dialog. + +<table> + +<tr> +<th>Machine/setup</th> +<th>Original tasksel</th> +<th>Optimised tasksel</th> +<th>Reduction</th> +</tr> + +<tr> +<td>Latitude D505 Main+LTSP LXDE</td> +<td>64 min (07:46-08:50)</td> +<td><44 min (11:27-12:11)</td> +<td>>20 min 18%</td> +</tr> + +<tr> +<td>Latitude D505 Roaming LXDE</td> +<td>57 min (08:48-09:45)</td> +<td>34 min (07:43-08:17)</td> +<td>23 min 40%</td> +</tr> + +<tr> +<td>Latitude D505 Minimal</td> +<td>22 min (10:37-10:59)</td> +<td>11 min (11:16-11:27)</td> +<td>11 min 50%</td> +</tr> + +<tr> +<td>Thinkpad X200 Minimal</td> +<td>6 min (08:19-08:25)</td> +<td>4 min (08:04-08:08)</td> +<td>2 min 33%</td> +</tr> + +<tr> +<td>Thinkpad X200 Roaming KDE</td> +<td>19 min (09:21-09:40)</td> +<td>15 min (10:25-10:40)</td> +<td>4 min 21%</td> +</tr> + +</table> + +The test is done using a netinst ISO on a USB stick, so some of the +time is spent downloading packages. The connection to the Internet +was 100Mbit/s during testing, so downloading should not be a +significant factor in the measurement. Download typically took a few +seconds to a few minutes, depending on the amount of packages being +installed. + +The speedup is implemented by using two hooks in +<a href="https://www.debian.org/devel/debian-installer/">Debian +Installer</a>, the pre-pkgsel.d hook to set up the diverts, and the +finish-install.d hook to remove the divert at the end of the +installation. I picked the pre-pkgsel.d hook instead of the +post-base-installer.d hook because I test using an ISO without the +eatmydata package included, and the post-base-installer.d hook in +Debian Edu can only operate on packages included in the ISO. The +negative effect of this is that I am unable to activate this +optimization for the kernel installation step in d-i. If the code is +moved to the post-base-installer.d hook, the speedup would be larger +for the entire installation. + +I've implemented this in the +<a href="https://packages.qa.debian.org/debian-edu-install">debian-edu-install</a> +git repository, and plan to provide the optimization as part of the +Debian Edu installation. If you want to test this yourself, you can +create two files in the installer (or in an udeb). One shell script +need do go into /usr/lib/pre-pkgsel.d/, with content like this: + +<blockquote><pre> +#!/bin/sh +set -e +. /usr/share/debconf/confmodule +info() { + logger -t my-pkgsel "info: $*" +} +error() { + logger -t my-pkgsel "error: $*" +} +override_install() { + apt-install eatmydata || true + if [ -x /target/usr/bin/eatmydata ] ; then + for bin in dpkg apt-get aptitude tasksel ; do + file=/usr/bin/$bin + # Test that the file exist and have not been diverted already. + if [ -f /target$file ] ; then + info "diverting $file using eatmydata" + printf "#!/bin/sh\neatmydata $bin.distrib \"\$@\"\n" \ + > /target$file.edu + chmod 755 /target$file.edu + in-target dpkg-divert --package debian-edu-config \ + --rename --quiet --add $file + ln -sf ./$bin.edu /target$file + else + error "unable to divert $file, as it is missing." + fi + done + else + error "unable to find /usr/bin/eatmydata after installing the eatmydata pacage" + fi +} + +override_install +</pre></blockquote> + +To clean up, another shell script should go into +/usr/lib/finish-install.d/ with code like this: + +<blockquote><pre> +#! /bin/sh -e +. /usr/share/debconf/confmodule +error() { + logger -t my-finish-install "error: $@" +} +remove_install_override() { + for bin in dpkg apt-get aptitude tasksel ; do + file=/usr/bin/$bin + if [ -x /target$file.edu ] ; then + rm /target$file + in-target dpkg-divert --package debian-edu-config \ + --rename --quiet --remove $file + rm /target$file.edu + else + error "Missing divert for $file." + fi + done + sync # Flush file buffers before continuing +} + +remove_install_override +</pre></blockquote> + +In Debian Edu, I placed both code fragments in a separate script +edu-eatmydata-install and call it from the pre-pkgsel.d and +finish-install.d scripts. + +By now you might ask if this change should get into the normal +Debian installer too? I suspect it should, but am not sure the +current debian-installer coordinators find it useful enough. It also +depend on the side effects of the change. I'm not aware of any, but I +guess we will see if the change is safe after some more testing. +Perhaps there is some package in Debian depending on sync() and +fsync() having effect? Perhaps it should go into its own udeb, to +allow those of us wanting to enable it to do so without affecting +everyone. + + + Good bye subkeys.pgp.net, welcome pool.sks-keyservers.net http://people.skolelinux.org/pere/blog/Good_bye_subkeys_pgp_net__welcome_pool_sks_keyservers_net.html diff --git a/blog/tags/english/index.html b/blog/tags/english/index.html index c9a52309bc..2fc435a6bc 100644 --- a/blog/tags/english/index.html +++ b/blog/tags/english/index.html @@ -20,6 +20,223 @@

Entries tagged "english".

+ Speeding up the Debian installer using eatmydata and dpkg-divert +

+ 16th September 2014 +

+ +

Machine/setup	Original tasksel	Optimised tasksel	Reduction
Latitude D505 Main+LTSP LXDE	64 min (07:46-08:50)	<44 min (11:27-12:11)	>20 min 18%
Latitude D505 Roaming LXDE	57 min (08:48-09:45)	34 min (07:43-08:17)	23 min 40%
Latitude D505 Minimal	22 min (10:37-10:59)	11 min (11:16-11:27)	11 min 50%
Thinkpad X200 Minimal	6 min (08:19-08:25)	4 min (08:04-08:08)	2 min 33%
Thinkpad X200 Roaming KDE	19 min (09:21-09:40)	15 min (10:25-10:40)	4 min 21%

+#!/bin/sh
+set -e
+. /usr/share/debconf/confmodule
+info() {
+    logger -t my-pkgsel "info: $*"
+}
+error() {
+    logger -t my-pkgsel "error: $*"
+}
+override_install() {
+    apt-install eatmydata || true
+    if [ -x /target/usr/bin/eatmydata ] ; then
+        for bin in dpkg apt-get aptitude tasksel ; do
+            file=/usr/bin/$bin
+            # Test that the file exist and have not been diverted already.
+            if [ -f /target$file ] ; then
+                info "diverting $file using eatmydata"
+                printf "#!/bin/sh\neatmydata $bin.distrib \"\$@\"\n" \
+                    > /target$file.edu
+                chmod 755 /target$file.edu
+                in-target dpkg-divert --package debian-edu-config \
+                    --rename --quiet --add $file
+                ln -sf ./$bin.edu /target$file
+            else
+                error "unable to divert $file, as it is missing."
+            fi
+        done
+    else
+        error "unable to find /usr/bin/eatmydata after installing the eatmydata pacage"
+    fi
+}
+
+override_install
+

+ +

To clean up, another shell script should go into +/usr/lib/finish-install.d/ with code like this: + +

+#! /bin/sh -e
+. /usr/share/debconf/confmodule
+error() {
+    logger -t my-finish-install "error: $@"
+}
+remove_install_override() {
+    for bin in dpkg apt-get aptitude tasksel ; do
+        file=/usr/bin/$bin
+        if [ -x /target$file.edu ] ; then
+            rm /target$file
+            in-target dpkg-divert --package debian-edu-config \
+                --rename --quiet --remove $file
+            rm /target$file.edu
+        else
+            error "Missing divert for $file."
+        fi
+    done
+    sync # Flush file buffers before continuing
+}
+
+remove_install_override
+

+ +

In Debian Edu, I placed both code fragments in a separate script +edu-eatmydata-install and call it from the pre-pkgsel.d and +finish-install.d scripts.

+ +

+ + + Tags: debian, debian edu, english. + + +

Good bye subkeys.pgp.net, welcome pool.sks-keyservers.net @@ -20475,7 +20692,7 @@ be the only one fitting our needs. :/

August (2)

September (1)

September (2)

@@ -20656,9 +20873,9 @@ be the only one fitting our needs. :/

@@ -20668,7 +20885,7 @@ be the only one fitting our needs. :/

diff --git a/blog/tags/fiksgatami/index.html b/blog/tags/fiksgatami/index.html index 3b639c70b9..579d819684 100644 --- a/blog/tags/fiksgatami/index.html +++ b/blog/tags/fiksgatami/index.html @@ -1279,7 +1279,7 @@ med dem. Dette blir bra.

August (2)

September (1)

September (2)

@@ -1460,9 +1460,9 @@ med dem. Dette blir bra.

@@ -1472,7 +1472,7 @@ med dem. Dette blir bra.

diff --git a/blog/tags/fildeling/index.html b/blog/tags/fildeling/index.html index 08e4d47d91..e65db8e8b5 100644 --- a/blog/tags/fildeling/index.html +++ b/blog/tags/fildeling/index.html @@ -649,7 +649,7 @@ og fildeling av slike filer er fullt ut lovlig.

August (2)

September (1)

September (2)

@@ -830,9 +830,9 @@ og fildeling av slike filer er fullt ut lovlig.

@@ -842,7 +842,7 @@ og fildeling av slike filer er fullt ut lovlig.

diff --git a/blog/tags/freeculture/index.html b/blog/tags/freeculture/index.html index 6d05e8cdbc..25816260e6 100644 --- a/blog/tags/freeculture/index.html +++ b/blog/tags/freeculture/index.html @@ -949,7 +949,7 @@ slik at du kan oppdatere direkte.

August (2)

September (1)

September (2)

@@ -1130,9 +1130,9 @@ slik at du kan oppdatere direkte.

@@ -1142,7 +1142,7 @@ slik at du kan oppdatere direkte.

diff --git a/blog/tags/freedombox/index.html b/blog/tags/freedombox/index.html index d16f841f12..5a7a224c07 100644 --- a/blog/tags/freedombox/index.html +++ b/blog/tags/freedombox/index.html @@ -862,7 +862,7 @@ default password is 'secret'.

August (2)

September (1)

September (2)

@@ -1043,9 +1043,9 @@ default password is 'secret'.

@@ -1055,7 +1055,7 @@ default password is 'secret'.

diff --git a/blog/tags/frikanalen/index.html b/blog/tags/frikanalen/index.html index 7e7244ab57..db8e9a211e 100644 --- a/blog/tags/frikanalen/index.html +++ b/blog/tags/frikanalen/index.html @@ -987,7 +987,7 @@ NUUG lykkes med Ã¥ fÃ¥ ut sine opptak med like stor suksess.

August (2)

September (1)

September (2)

@@ -1168,9 +1168,9 @@ NUUG lykkes med Ã¥ fÃ¥ ut sine opptak med like stor suksess.

@@ -1180,7 +1180,7 @@ NUUG lykkes med Ã¥ fÃ¥ ut sine opptak med like stor suksess.

diff --git a/blog/tags/intervju/index.html b/blog/tags/intervju/index.html index a0c23fd81a..024679eaf7 100644 --- a/blog/tags/intervju/index.html +++ b/blog/tags/intervju/index.html @@ -4944,7 +4944,7 @@ veldig bra utvalg av gratis spill som er av hÃ¸y kvalitet. Veldig lett

August (2)

September (1)

September (2)

@@ -5125,9 +5125,9 @@ veldig bra utvalg av gratis spill som er av hÃ¸y kvalitet. Veldig lett

@@ -5137,7 +5137,7 @@ veldig bra utvalg av gratis spill som er av hÃ¸y kvalitet. Veldig lett

diff --git a/blog/tags/isenkram/index.html b/blog/tags/isenkram/index.html index b084fe8db7..cc9ded9126 100644 --- a/blog/tags/isenkram/index.html +++ b/blog/tags/isenkram/index.html @@ -940,7 +940,7 @@ please send me an email. :)

August (2)

September (1)

September (2)

@@ -1121,9 +1121,9 @@ please send me an email. :)

@@ -1133,7 +1133,7 @@ please send me an email. :)

diff --git a/blog/tags/kart/index.html b/blog/tags/kart/index.html index 57343d50bd..fc43aa0cee 100644 --- a/blog/tags/kart/index.html +++ b/blog/tags/kart/index.html @@ -1148,7 +1148,7 @@ det viser at behovet for fribruks-sjÃ¸kart er til stedet.

August (2)

September (1)

September (2)

@@ -1329,9 +1329,9 @@ det viser at behovet for fribruks-sjÃ¸kart er til stedet.

@@ -1341,7 +1341,7 @@ det viser at behovet for fribruks-sjÃ¸kart er til stedet.

diff --git a/blog/tags/ldap/index.html b/blog/tags/ldap/index.html index c4a1b6ee9d..3b460e8d50 100644 --- a/blog/tags/ldap/index.html +++ b/blog/tags/ldap/index.html @@ -1066,7 +1066,7 @@ new IETF work group?

August (2)

September (1)

September (2)

@@ -1247,9 +1247,9 @@ new IETF work group?

@@ -1259,7 +1259,7 @@ new IETF work group?

diff --git a/blog/tags/lenker/index.html b/blog/tags/lenker/index.html index 47c34bc2b5..8aff843a4e 100644 --- a/blog/tags/lenker/index.html +++ b/blog/tags/lenker/index.html @@ -661,7 +661,7 @@ Word 2007 hÃ¥ndterer ODF dÃ¥rlig

August (2)

September (1)

September (2)

@@ -842,9 +842,9 @@ Word 2007 hÃ¥ndterer ODF dÃ¥rlig

@@ -854,7 +854,7 @@ Word 2007 hÃ¥ndterer ODF dÃ¥rlig

diff --git a/blog/tags/ltsp/index.html b/blog/tags/ltsp/index.html index 3f2c4134d7..b97a30ac59 100644 --- a/blog/tags/ltsp/index.html +++ b/blog/tags/ltsp/index.html @@ -81,7 +81,7 @@ of these cards.

August (2)

September (1)

September (2)

@@ -262,9 +262,9 @@ of these cards.

@@ -274,7 +274,7 @@ of these cards.

diff --git a/blog/tags/mesh network/index.html b/blog/tags/mesh network/index.html index 929a91732b..ba7a97d3ad 100644 --- a/blog/tags/mesh network/index.html +++ b/blog/tags/mesh network/index.html @@ -731,7 +731,7 @@ mesh system.

August (2)

September (1)

September (2)

@@ -912,9 +912,9 @@ mesh system.

@@ -924,7 +924,7 @@ mesh system.

diff --git a/blog/tags/multimedia/index.html b/blog/tags/multimedia/index.html index 0ab76f3c93..7d218243b3 100644 --- a/blog/tags/multimedia/index.html +++ b/blog/tags/multimedia/index.html @@ -2207,7 +2207,7 @@ be the only one fitting our needs. :/

August (2)

September (1)

September (2)

@@ -2388,9 +2388,9 @@ be the only one fitting our needs. :/

@@ -2400,7 +2400,7 @@ be the only one fitting our needs. :/

diff --git a/blog/tags/norsk/index.html b/blog/tags/norsk/index.html index f20a8b3d42..5ccc4468c3 100644 --- a/blog/tags/norsk/index.html +++ b/blog/tags/norsk/index.html @@ -18779,7 +18779,7 @@ forsÃ¸k.

August (2)

September (1)

September (2)

@@ -18960,9 +18960,9 @@ forsÃ¸k.

@@ -18972,7 +18972,7 @@ forsÃ¸k.

diff --git a/blog/tags/nuug/index.html b/blog/tags/nuug/index.html index 0c8885012a..516ee5f5d6 100644 --- a/blog/tags/nuug/index.html +++ b/blog/tags/nuug/index.html @@ -10719,7 +10719,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

August (2)

September (1)

September (2)

@@ -10900,9 +10900,9 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

@@ -10912,7 +10912,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

diff --git a/blog/tags/offentlig innsyn/index.html b/blog/tags/offentlig innsyn/index.html index 387ad19872..bbbd7d0524 100644 --- a/blog/tags/offentlig innsyn/index.html +++ b/blog/tags/offentlig innsyn/index.html @@ -1105,7 +1105,7 @@ til Ã¥ levere hver uke. Har ikke undersÃ¸kt noen av de andre.

August (2)

September (1)

September (2)

@@ -1286,9 +1286,9 @@ til Ã¥ levere hver uke. Har ikke undersÃ¸kt noen av de andre.

@@ -1298,7 +1298,7 @@ til Ã¥ levere hver uke. Har ikke undersÃ¸kt noen av de andre.

diff --git a/blog/tags/open311/index.html b/blog/tags/open311/index.html index cbd4d5f43a..a86bfb6150 100644 --- a/blog/tags/open311/index.html +++ b/blog/tags/open311/index.html @@ -176,7 +176,7 @@ work like the free software project communities I am used to.

August (2)

September (1)

September (2)

@@ -357,9 +357,9 @@ work like the free software project communities I am used to.

@@ -369,7 +369,7 @@ work like the free software project communities I am used to.

diff --git a/blog/tags/opphavsrett/index.html b/blog/tags/opphavsrett/index.html index 0d03155033..775c0279cf 100644 --- a/blog/tags/opphavsrett/index.html +++ b/blog/tags/opphavsrett/index.html @@ -3773,7 +3773,7 @@ database.

August (2)

September (1)

September (2)

@@ -3954,9 +3954,9 @@ database.

@@ -3966,7 +3966,7 @@ database.

diff --git a/blog/tags/personvern/index.html b/blog/tags/personvern/index.html index 6d17954076..a79ad2319c 100644 --- a/blog/tags/personvern/index.html +++ b/blog/tags/personvern/index.html @@ -6442,7 +6442,7 @@ kontanter for noen dager siden.

August (2)

September (1)

September (2)

@@ -6623,9 +6623,9 @@ kontanter for noen dager siden.

@@ -6635,7 +6635,7 @@ kontanter for noen dager siden.

diff --git a/blog/tags/raid/index.html b/blog/tags/raid/index.html index 44b83beb72..9e59a74559 100644 --- a/blog/tags/raid/index.html +++ b/blog/tags/raid/index.html @@ -117,7 +117,7 @@ disk(s) is failing when the RAID is running short on disks.

August (2)

September (1)

September (2)

@@ -298,9 +298,9 @@ disk(s) is failing when the RAID is running short on disks.

@@ -310,7 +310,7 @@ disk(s) is failing when the RAID is running short on disks.

diff --git a/blog/tags/reactos/index.html b/blog/tags/reactos/index.html index 2536ff1286..cb6deebd13 100644 --- a/blog/tags/reactos/index.html +++ b/blog/tags/reactos/index.html @@ -116,7 +116,7 @@ image.

August (2)

September (1)

September (2)

@@ -297,9 +297,9 @@ image.

@@ -309,7 +309,7 @@ image.

diff --git a/blog/tags/reprap/index.html b/blog/tags/reprap/index.html index 834a8e36ac..74c5982ea3 100644 --- a/blog/tags/reprap/index.html +++ b/blog/tags/reprap/index.html @@ -543,7 +543,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

August (2)

September (1)

September (2)

@@ -724,9 +724,9 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

@@ -736,7 +736,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

diff --git a/blog/tags/rfid/index.html b/blog/tags/rfid/index.html index 3ce913d293..0f0d54bda0 100644 --- a/blog/tags/rfid/index.html +++ b/blog/tags/rfid/index.html @@ -166,7 +166,7 @@ mer om de nye biometriske passene.

August (2)

September (1)

September (2)

@@ -347,9 +347,9 @@ mer om de nye biometriske passene.

@@ -359,7 +359,7 @@ mer om de nye biometriske passene.

diff --git a/blog/tags/robot/index.html b/blog/tags/robot/index.html index da57d841df..d9d6984bd4 100644 --- a/blog/tags/robot/index.html +++ b/blog/tags/robot/index.html @@ -481,7 +481,7 @@ firmwaren. :)

August (2)

September (1)

September (2)

@@ -662,9 +662,9 @@ firmwaren. :)

@@ -674,7 +674,7 @@ firmwaren. :)

diff --git a/blog/tags/rss/index.html b/blog/tags/rss/index.html index ec49526ee5..df9300e34a 100644 --- a/blog/tags/rss/index.html +++ b/blog/tags/rss/index.html @@ -70,7 +70,7 @@ forsÃ¸k.

August (2)

September (1)

September (2)

@@ -251,9 +251,9 @@ forsÃ¸k.

@@ -263,7 +263,7 @@ forsÃ¸k.

diff --git a/blog/tags/ruter/index.html b/blog/tags/ruter/index.html index 463879ca12..56191b9aee 100644 --- a/blog/tags/ruter/index.html +++ b/blog/tags/ruter/index.html @@ -224,7 +224,7 @@ OsloomrÃ¥det.

August (2)

September (1)

September (2)

@@ -405,9 +405,9 @@ OsloomrÃ¥det.

@@ -417,7 +417,7 @@ OsloomrÃ¥det.

diff --git a/blog/tags/scraperwiki/index.html b/blog/tags/scraperwiki/index.html index 854f3dd7f4..1fe8f20219 100644 --- a/blog/tags/scraperwiki/index.html +++ b/blog/tags/scraperwiki/index.html @@ -159,7 +159,7 @@ av Ã¥ dele informasjon med andre uten bruksbegresninger.

August (2)

September (1)

September (2)

@@ -340,9 +340,9 @@ av Ã¥ dele informasjon med andre uten bruksbegresninger.

@@ -352,7 +352,7 @@ av Ã¥ dele informasjon med andre uten bruksbegresninger.

diff --git a/blog/tags/sikkerhet/index.html b/blog/tags/sikkerhet/index.html index f9957305c5..bdd98e43af 100644 --- a/blog/tags/sikkerhet/index.html +++ b/blog/tags/sikkerhet/index.html @@ -3140,7 +3140,7 @@ betydelige.

August (2)

September (1)

September (2)

@@ -3321,9 +3321,9 @@ betydelige.

@@ -3333,7 +3333,7 @@ betydelige.

diff --git a/blog/tags/sitesummary/index.html b/blog/tags/sitesummary/index.html index 29d6dd3f66..b7776a1163 100644 --- a/blog/tags/sitesummary/index.html +++ b/blog/tags/sitesummary/index.html @@ -275,7 +275,7 @@ everything is taken care of.

August (2)

September (1)

September (2)

@@ -456,9 +456,9 @@ everything is taken care of.

@@ -468,7 +468,7 @@ everything is taken care of.

diff --git a/blog/tags/skepsis/index.html b/blog/tags/skepsis/index.html index b9ae923559..1351f53327 100644 --- a/blog/tags/skepsis/index.html +++ b/blog/tags/skepsis/index.html @@ -408,7 +408,7 @@ skyskrapere. Takke meg til en tur til mÃ¥nen.

August (2)

September (1)

September (2)

@@ -589,9 +589,9 @@ skyskrapere. Takke meg til en tur til mÃ¥nen.

@@ -601,7 +601,7 @@ skyskrapere. Takke meg til en tur til mÃ¥nen.

diff --git a/blog/tags/standard/index.html b/blog/tags/standard/index.html index 39ba844385..b374207b37 100644 --- a/blog/tags/standard/index.html +++ b/blog/tags/standard/index.html @@ -3773,7 +3773,7 @@ Kjenner kun til ufullstendige lÃ¸sninger for slikt.

August (2)

September (1)

September (2)

@@ -3954,9 +3954,9 @@ Kjenner kun til ufullstendige lÃ¸sninger for slikt.

@@ -3966,7 +3966,7 @@ Kjenner kun til ufullstendige lÃ¸sninger for slikt.

diff --git a/blog/tags/stavekontroll/index.html b/blog/tags/stavekontroll/index.html index bc20b7605f..0dc23d9977 100644 --- a/blog/tags/stavekontroll/index.html +++ b/blog/tags/stavekontroll/index.html @@ -254,7 +254,7 @@ stavekontrollen.

August (2)

September (1)

September (2)

@@ -435,9 +435,9 @@ stavekontrollen.

@@ -447,7 +447,7 @@ stavekontrollen.

diff --git a/blog/tags/stortinget/index.html b/blog/tags/stortinget/index.html index be7a93cf8f..a26e81953d 100644 --- a/blog/tags/stortinget/index.html +++ b/blog/tags/stortinget/index.html @@ -851,7 +851,7 @@ at vi i NUUG har fÃ¥tt operativ en norsk utgave av

August (2)

September (1)

September (2)

@@ -1032,9 +1032,9 @@ at vi i NUUG har fÃ¥tt operativ en norsk utgave av

@@ -1044,7 +1044,7 @@ at vi i NUUG har fÃ¥tt operativ en norsk utgave av

diff --git a/blog/tags/surveillance/index.html b/blog/tags/surveillance/index.html index 2e359d6025..f9b634abf6 100644 --- a/blog/tags/surveillance/index.html +++ b/blog/tags/surveillance/index.html @@ -1808,7 +1808,7 @@ automatisk over i spesialkartet.

August (2)

September (1)

September (2)

@@ -1989,9 +1989,9 @@ automatisk over i spesialkartet.

@@ -2001,7 +2001,7 @@ automatisk over i spesialkartet.

diff --git a/blog/tags/sysadmin/index.html b/blog/tags/sysadmin/index.html index 6ed02a3106..84adc0f56d 100644 --- a/blog/tags/sysadmin/index.html +++ b/blog/tags/sysadmin/index.html @@ -107,7 +107,7 @@ and DHCP updates I wanted to do.

August (2)

September (1)

September (2)

@@ -288,9 +288,9 @@ and DHCP updates I wanted to do.

@@ -300,7 +300,7 @@ and DHCP updates I wanted to do.

diff --git a/blog/tags/valg/index.html b/blog/tags/valg/index.html index 179d5b3aa9..1035848f07 100644 --- a/blog/tags/valg/index.html +++ b/blog/tags/valg/index.html @@ -957,7 +957,7 @@ inneholdt i Iran hvis de ikke hadde hemmelige valg?

August (2)

September (1)

September (2)

@@ -1138,9 +1138,9 @@ inneholdt i Iran hvis de ikke hadde hemmelige valg?

@@ -1150,7 +1150,7 @@ inneholdt i Iran hvis de ikke hadde hemmelige valg?

diff --git a/blog/tags/video/index.html b/blog/tags/video/index.html index 2bb3624fc9..401be41f2e 100644 --- a/blog/tags/video/index.html +++ b/blog/tags/video/index.html @@ -3430,7 +3430,7 @@ larger stick as well.

August (2)

September (1)

September (2)

@@ -3611,9 +3611,9 @@ larger stick as well.

@@ -3623,7 +3623,7 @@ larger stick as well.

diff --git a/blog/tags/vitenskap/index.html b/blog/tags/vitenskap/index.html index c33a47b4fd..e6df7d5284 100644 --- a/blog/tags/vitenskap/index.html +++ b/blog/tags/vitenskap/index.html @@ -411,7 +411,7 @@ skyskrapere. Takke meg til en tur til mÃ¥nen.

August (2)

September (1)

September (2)

@@ -592,9 +592,9 @@ skyskrapere. Takke meg til en tur til mÃ¥nen.

@@ -604,7 +604,7 @@ skyskrapere. Takke meg til en tur til mÃ¥nen.

diff --git a/blog/tags/web/index.html b/blog/tags/web/index.html index 14235020f9..39d740ee20 100644 --- a/blog/tags/web/index.html +++ b/blog/tags/web/index.html @@ -2629,7 +2629,7 @@ be the only one fitting our needs. :/

August (2)

September (1)

September (2)

@@ -2810,9 +2810,9 @@ be the only one fitting our needs. :/

@@ -2822,7 +2822,7 @@ be the only one fitting our needs. :/

drivstoffpriser (4)