Petter Reinholdtsen

It is strange to watch how a bug in Debian causing KDM to fail to start at boot when an NVidia video card is used is handled. The problem seem to be that the nvidia X.org driver uses a long time to initialize, and this duration is longer than kdm is configured to wait.

I came across two bugs related to this issue, #583312 initially filed against initscripts and passed on to nvidia-glx when it became obvious that the nvidia drivers were involved, and #524751 initially filed against kdm and passed on to src:nvidia-graphics-drivers for unknown reasons.

To me, it seem that no-one is interested in actually solving the problem nvidia video card owners experience and make sure the Debian distribution work out of the box for these users. The nvidia driver maintainers expect kdm to be set up to wait longer, while kdm expect the nvidia driver maintainers to fix the driver to start faster, and while they wait for each other I guess the users end up switching to a distribution that work for them. I have no idea what the solution is, but I am pretty sure that waiting for each other is not it.

I wonder why we end up handling bugs this way.

Det står dårlig til med toget når en finner på å la det kappkjøre med sykkel... Jeg tror det trengs strukturendringer for å få fikset på togproblemene i Norge.

Mon tro hva toglinje mellom Narvik og Tromsø ville hatt slags effekt på området der?

When using sitesummary at a site to track machines, it is possible to get a list of the machine types in use thanks to the DMI information extracted from each machine. The script to do so is included in the sitesummary package, and here is example output from the Skolelinux build servers:

maintainer:~# /usr/lib/sitesummary/hardware-model-summary
  vendor                    count
  Dell Computer Corporation     1
    PowerEdge 1750              1
  IBM                           1
    eserver xSeries 345 -[8670M1X]-     1
  Intel                         2
  [no-dmi-info]                 3
maintainer:~#

The quality of the report depend on the quality of the DMI tables provided in each machine. Here there are Intel machines without model information listed with Intel as vendor and mo model, and virtual Xen machines listed as [no-dmi-info]. One can add -l as a command line option to list the individual machines.

A larger list is available from the the city of Narvik, which uses Skolelinux on all their shools and also provide the basic sitesummary report publicly. In their report there are ~1400 machines. I know they use both Ubuntu and Skolelinux on their machines, and as sitesummary is available in both distributions, it is trivial to get all of them to report to the same central collector.

Via the blog of Rob Weir I came across the very interesting essay named The Art of Standards Wars (PDF 25 pages). I recommend it for everyone following the standards wars of today.

If Debian is to migrate to upstart on Linux, I expect some init.d scripts to migrate (some of) their operations to upstart job while keeping the init.d for hurd and kfreebsd. The packages with such needs will need a way to get their init.d scripts to behave differently when used with sysvinit and with upstart. Because of this, I had a look at the environment variables set when a init.d script is running under upstart, and when it is not.

With upstart, I notice these environment variables are set when a script is started from rcS.d/ (ignoring some irrelevant ones like COLUMNS):

DEFAULT_RUNLEVEL=2
previous=N
PREVLEVEL=
RUNLEVEL=
runlevel=S
UPSTART_EVENTS=startup
UPSTART_INSTANCE=
UPSTART_JOB=rc-sysinit

With sysvinit, these environment variables are set for the same script.

INIT_VERSION=sysvinit-2.88
previous=N
PREVLEVEL=N
RUNLEVEL=S
runlevel=S

The RUNLEVEL and PREVLEVEL environment variables passed on from sysvinit are not set by upstart. Not sure if it is intentional or not to not be compatible with sysvinit in this regard.

For scripts needing to behave differently when upstart is used, looking for the UPSTART_JOB environment variable seem to be a good choice.

Det er merkelig hvordan myter om Skolelinux overlever. En slik myte er at Skolelinux ikke kan sentraldriftes og ha sentralt plasserte tjenermaskiner. I siste Computerworld Norge er IT-sjef Viggo Billdal i Steinkjer intervjuet, og forteller uten blygsel:

Vi hadde Skolelinux, men det har vi sluttet med. Vi testet om det lønte seg med Microsoft eller en åpen plattform. Vi fant ut at Microsoft egentlig var totalt sett bedre egnet. Det var store driftskostnader med Skolelinux, blant annet på grunn av desentraliserte servere. Det var komplisert, så vi gikk vekk fra det og bruker nå bare Windows.

En rask sjekk mot den norske brukerlista i Skolelinuxprosjektet forteller at Steinkjers forsøk foregikk fram til 2004/2005, og at Røysing skole i Steinkjer skal ha vært svært fornøyd med Skolelinux men at kommunen overkjørte skolen og krevde at de gikk over til Windows. Et søk på nettet sendte meg til Dagens IT nr. 18 2005 hvor en kan lese på side 18:

Inge Tømmerås ved Røysing skole i Steinkjer kjører ennå Microsoft, men forteller at kompetanseutfordringen med Skolelinux ikke var så stor. ­ Jeg syntes Skolelinux var utrolig lett å drifte uten forkunnskaper. Men man må jo selvsagt ha tilgang på ekstern kompetanse til installasjoner og maskinvarefeil, sier Tømmerås.

Som systemarkitekten bak Skolelinux, kan jeg bare riste på hodet over påstanden om at Skolelinux krever desentraliserte tjenere. Skolelinux-arkitekturen er laget for sentralisert drift og plassering av tjenerne lokalt eller sentralt alt etter behov og nettkapasitet. Den er modellert på nettverks- og tjenerløsningen som brukes på Universitetet i Tromsø og Oslo, der jeg jobber med utvikling av driftstjenester. Dette er det heldigvis noen som har fått med seg, og jeg er glad for å kunne sitere fra en kommentar på den overnevnte artikkelen. Min venn og gamle kollega Sturle Sunde forteller der:

I Flora kommune køyrer vi Skulelinux på skular med alt frå 15 til meir enn 500 elevar. Dei store skulane har eigen tenar, for det er mest praktisk. Eg, som er driftsansvarleg for heile nettet, ser sjeldan dei tenarane fysisk, men at dei står der gjer skulane mindre avhengige av eksterne linjer som er trege eller dyre. Dei minste skulane har ikkje eigen tenar. Å bruke sentral tenar er heller ikkje noko problem. Småskulane klarar seg fint med 1 mbit-linje til ein sentral tenar eller tenaren på ein større skule.

Det beste med Skulelinux er halvtjukke klientar. Dei treng ikkje harddisk og brukar minimalt med ressursar på tenaren fordi dei køyrer programma lokalt. Eit klasserom med 30 sju-åtte år gamle maskiner har mykje meir CPU og RAM totalt enn nokon moderne tenar til under millionen. Det trengst to kommandoar på den sentrale tenaren for å oppdatere alle klientane, både tynne og halvtjukke. Vi har ingen problem med diskar som ryk heller, som var eit problem før fordi elevane sat og sparka i maskinene. Og dei krev lite bandbreidde i nettet, so det er fullt mogleg å køyre slike på småskular med trege linjer mot tenaren på ein større skule.

Flora kommune har nesten 800 Linux-maskiner i sitt skulenett, og ein person som tek seg av drift av heile nettet, inkludert tenarar, klientar, operativsystem, programvare, heimekontorløysing og administrasjon av brukarar.

No skal det seiast at vi ikkje køyrer rein Skulelinux ut av boksen. Vi har gjort ein del tilpassingar mot noko Novell-greier som var der frå før, og som har komplisert installasjonen vår. Etter at oppsettet var gjort har løysinga vore stabil og kravd minimalt med arbeid.

Jeg vet at Narvik, Harstad og Oslo er kommuner der Skolelinux sentraldriftes med sentrale tjenere. Det forteller meg at Steinkjers IT-sjef neppe bør skylde på Skolelinux-løsningen for sine 5 år gamle minner.

The last few days I have done some upgrade testing in Debian, to see if the upgrade from Lenny to Squeeze will go smoothly. A few bugs have been discovered and reported in the process (#585410 in nagios3-cgi, #584879 already fixed in enscript and #584861 in kdebase-workspace-data), and to get a more regular testing going on, I am working on a script to automate the test.

The idea is to create a Lenny chroot and use tasksel to install a Gnome or KDE desktop installation inside the chroot before upgrading it. To ensure no services are started in the chroot, a policy-rc.d script is inserted. To make sure tasksel believe it is to install a desktop on a laptop, the tasksel tests are replaced in the chroot (only acceptable because this is a throw-away chroot).

A naive upgrade from Lenny to Squeeze using aptitude dist-upgrade currently always fail because udev refuses to upgrade with the kernel in Lenny, so to avoid that problem the file /etc/udev/kernel-upgrade is created. The bug report #566000 make me suspect this problem do not trigger in a chroot, but I touch the file anyway to make sure the upgrade go well. Testing on virtual and real hardware have failed me because of udev so far, and creating this file do the trick in such settings anyway. This is a known issue and the current udev behaviour is intended by the udev maintainer because he lack the resources to rewrite udev to keep working with old kernels or something like that. I really wish the udev upstream would keep udev backwards compatible, to avoid such upgrade problem, but given that they fail to do so, I guess documenting the way out of this mess is the best option we got for Debian Squeeze.

Anyway, back to the task at hand, testing upgrades. This test script, which I call upgrade-test for now, is doing the trick:

#!/bin/sh
set -ex

if [ "$1" ] ; then
    desktop=$1
else
    desktop=gnome
fi

from=lenny
to=squeeze

exec < /dev/null
unset LANG
mirror=http://ftp.skolelinux.org/debian
tmpdir=chroot-$from-upgrade-$to-$desktop
fuser -mv .
debootstrap $from $tmpdir $mirror
chroot $tmpdir aptitude update
cat > $tmpdir/usr/sbin/policy-rc.d <<EOF
#!/bin/sh
exit 101
EOF
chmod a+rx $tmpdir/usr/sbin/policy-rc.d
exit_cleanup() {
    umount $tmpdir/proc
}
mount -t proc proc $tmpdir/proc
# Make sure proc is unmounted also on failure
trap exit_cleanup EXIT INT

chroot $tmpdir aptitude -y install debconf-utils

# Make sure tasksel autoselection trigger.  It need the test scripts
# to return the correct answers.
echo tasksel tasksel/desktop multiselect $desktop | \
    chroot $tmpdir debconf-set-selections

# Include the desktop and laptop task
for test in desktop laptop ; do
    echo > $tmpdir/usr/lib/tasksel/tests/$test <<EOF
#!/bin/sh
exit 2
EOF
    chmod a+rx $tmpdir/usr/lib/tasksel/tests/$test
done

DEBIAN_FRONTEND=noninteractive
DEBIAN_PRIORITY=critical
export DEBIAN_FRONTEND DEBIAN_PRIORITY
chroot $tmpdir tasksel --new-install

echo deb $mirror $to main > $tmpdir/etc/apt/sources.list
chroot $tmpdir aptitude update
touch $tmpdir/etc/udev/kernel-upgrade
chroot $tmpdir aptitude -y dist-upgrade
fuser -mv

I suspect it would be useful to test upgrades with both apt-get and with aptitude, but I have not had time to look at how they behave differently so far. I hope to get a cron job running to do the test regularly and post the result on the web. The Gnome upgrade currently work, while the KDE upgrade fail because of the bug in kdebase-workspace-data

I am not quite sure what kind of extract from the huge upgrade logs (KDE 167 KiB, Gnome 516 KiB) it make sense to include in this blog post, so I will refrain from trying. I can report that for Gnome, aptitude report 760 packages upgraded, 448 newly installed, 129 to remove and 1 not upgraded and 1024MB need to be downloaded while for KDE the same numbers are 702 packages upgraded, 507 newly installed, 193 to remove and 0 not upgraded and 1117MB need to be downloaded

I am very happy to notice that the Gnome desktop + laptop upgrade is able to migrate to dependency based boot sequencing and parallel booting without a hitch. Was unsure if there were still bugs with packages failing to clean up their obsolete init.d script during upgrades, and no such problem seem to affect the Gnome desktop+laptop packages.