+ The life and death of a laptop battery +

+ 24th September 2015 +

When I get a new laptop, the battery life time at the start is OK. +But this do not last. The last few laptops gave me a feeling that +within a year, the life time is just a fraction of what it used to be, +and it slowly become painful to use the laptop without power connected +all the time. Because of this, when I got a new Thinkpad X230 laptop +about two years ago, I decided to monitor its battery state to have +more hard facts when the battery started to fail.

+ +

First I tried to find a sensible Debian package to record the +battery status, assuming that this must be a problem already handled +by someone else. I found +battery-stats, +which collects statistics from the battery, but it was completely +broken. I sent a few suggestions to the maintainer, but decided to +write my own collector as a shell script while I waited for feedback +from him. Via +a +blog post about the battery development on a MacBook Air I also +discovered +batlog, not +available in Debian.

+ +

I started my collector 2013-07-15, and it has been collecting +battery stats ever since. Now my +/var/log/hjemmenett-battery-status.log file contain around 115,000 +measurements, from the time the battery was working great until now, +when it is unable to charge above 7% of original capasity. My +colletor shell script is quite simple and look like this:

+ +

+#!/bin/sh
+# Inspired by
+# http://www.ifweassume.com/2013/08/the-de-evolution-of-my-laptop-battery.html
+# See also
+# http://blog.sleeplessbeastie.eu/2013/01/02/debian-how-to-monitor-battery-capacity/
+logfile=/var/log/hjemmenett-battery-status.log
+
+files="manufacturer model_name technology serial_number \
+    energy_full energy_full_design energy_now cycle_count status"
+
+if [ ! -e "$logfile" ] ; then
+    (
+	printf "timestamp,"
+	for f in $files; do
+	    printf "%s," $f
+	done
+	echo
+    ) > "$logfile"
+fi
+
+log_battery() {
+    # Print complete message in one echo call, to avoid race condition
+    # when several log processes run in parallell.
+    msg=$(printf "%s," $(date +%s); \
+	for f in $files; do \
+	    printf "%s," $(cat $f); \
+	done)
+    echo "$msg"
+}
+
+cd /sys/class/power_supply
+
+for bat in BAT*; do
+    (cd $bat && log_battery >> "$logfile")
+done
+

+ +

The script is called when the power management system detect a +change in the power status (power plug in or out), and when going into +and out of hibernation and suspend. In addition, it collect a value +every 10 minutes. This make it possible for me know when the battery +is discharging, charging and how the maximum charge change over time. +The code for the Debian package +is now +available on github.

+ +

The collected log file look like this:

+ +

+timestamp,manufacturer,model_name,technology,serial_number,energy_full,energy_full_design,energy_now,cycle_count,status,
+1376591133,LGC,45N1025,Li-ion,974,62800000,62160000,39050000,0,Discharging,
+[...]
+1443090528,LGC,45N1025,Li-ion,974,4900000,62160000,4900000,0,Full,
+1443090601,LGC,45N1025,Li-ion,974,4900000,62160000,4900000,0,Full,
+

+ +

I wrote a small script to create a graph of the charge development +over time. This graph depicted above show the slow death of mylaptop +battery.

+ +

But why is this happening? Why are my laptop batteries always +dying in a year or two, while the batteries of space probes and +satellites keep working year after year. If we are to believe +Battery +University, the cause is me charging the battery whenever I have a +chance, and the fix is to not charge the Lithium-ion batteries to 100% +all the time, but to stay below 90% of full charge most of the time. +I've been told that the Tesla electric cars +limit +the charge of their batteries to 80%, with the option to charge to +100% when preparing for a longer trip (not that I would want a car +like Tesla where rights to privacy is abandoned, but that is another +story), which I guess is the option we should have for laptops on +Linux too.

+ +

Is there a good and generic way with Linux to tell the battery to +stop charging at 80%, unless requested to charge to 100% once in +preparation for a longer trip? I found +one +recipe on askubuntu for Ubuntu to limit charging on Thinkpad to +80%, but could not get it to work (kernel module refused to +load).

+ +

I wonder why the battery capacity was reported to be more than 100% +at the start. I also wonder why the "full capacity" increases some +times, and if it is possible to repeat the process to get the battery +back to design capacity. And I wonder if the discharge and charge +speed change over time, or if this stay the same. I did not yet try +to write a tool to calculate the derivative values of the battery +level, but suspect some interesting insights might be learned from +those.

+ +

+ + + Tags: debian, english. + + +

+ New laptop - some more clues and ideas based on feedback +

+ 5th July 2015 +

Several people contacted me after my previous blog post about my +need for a new laptop, and provided very useful feedback. I wish to +thank every one of these. Several pointed me to the possibility of +fixing my X230, and I am already in the process of getting Lenovo to +do so thanks to the on site, next day support contract covering the +machine. But the battery is almost useless (I expect to replace it +with a non-official battery) and I do not expect the machine to live +for many more years, so it is time to plan its replacement. If I did +not have a support contract, it was suggested to find replacement parts +using FrancEcrans, but it +might present a language barrier as I do not understand French.

+ +

One tip I got was to use the +Skinflint web service to +compare laptop models. It seem to have more models available than +prisjakt.no. Another tip I got from someone I know have similar +keyboard preferences was that the HP EliteBook 840 keyboard is not +very good, and this matches my experience with earlier EliteBook +keyboards I tested. Because of this, I will not consider it any further. + +

When I wrote my blog post, I was not aware of Thinkpad X250, the +newest Thinkpad X model. The keyboard reintroduces mouse buttons +(which is missing from the X240), and is working fairly well with +Debian Sid/Unstable according to +Corsac.net. The reports I +got on the keyboard quality are not consistent. Some say the keyboard +is good, others say it is ok, while others say it is not very good. +Those with experience from X41 and and X60 agree that the X250 +keyboard is not as good as those trusty old laptops, and suggest I +keep and fix my X230 instead of upgrading, or get a used X230 to +replace it. I'm also told that the X250 lack leds for caps lock, disk +activity and battery status, which is very convenient on my X230. I'm +also told that the CPU fan is running very often, making it a bit +noisy. In any case, the X250 do not work out of the box with Debian +Stable/Jessie, one of my requirements.

+ +

I have also gotten a few vendor proposals, one was +Pro-Star, another was +Libreboot. +The latter look very attractive to me.

+ +

Again, thank you all for the very useful feedback. It help a lot +as I keep looking for a replacement.

+ +

Update 2015-07-06: I was recommended to check out the +lapstore.de web shop for used laptops. They got several +different +old +thinkpad X models, and provide one year warranty.

+ +

+ + + Tags: debian, english. + + +

+ Time to find a new laptop, as the old one is broken after only two years +

+ 3rd July 2015 +

My primary work horse laptop is failing, and will need a +replacement soon. The left 5 cm of the screen on my Thinkpad X230 +started flickering yesterday, and I suspect the cause is a broken +cable, as changing the angle of the screen some times get rid of the +flickering.

+ +

My requirements have not really changed since I bought it, and is +still as +I +described them in 2013. The last time I bought a laptop, I had +good help from +prisjakt.no +where I could select at least a few of the requirements (mouse pin, +wifi, weight) and go through the rest manually. Three button mouse +and a good keyboard is not available as an option, and all the three +laptop models proposed today (Thinkpad X240, HP EliteBook 820 G1 and +G2) lack three mouse buttons). It is also unclear to me how good the +keyboard on the HP EliteBooks are. I hope Lenovo have not messed up +the keyboard, even if the quality and robustness in the X series have +deteriorated since X41.

+ +

I wonder how I can find a sensible laptop when none of the options +seem sensible to me? Are there better services around to search the +set of available laptops for features? Please send me an email if you +have suggestions.

+ +

Update 2015-07-23: I got a suggestion to check out the FSF +list +of endorsed hardware, which is useful background information.

+ +

+ + + Tags: debian, english. + + +

+ How to stay with sysvinit in Debian Jessie +

+ 22nd November 2014 +

By now, it is well known that Debian Jessie will not be using +sysvinit as its boot system by default. But how can one keep using +sysvinit in Jessie? It is fairly easy, and here are a few recipes, +courtesy of +Erich +Schubert and +Simon +McVittie. + +

If you already are using Wheezy and want to upgrade to Jessie and +keep sysvinit as your boot system, create a file +/etc/apt/preferences.d/use-sysvinit with this content before +you upgrade:

+ +

+Package: systemd-sysv
+Pin: release o=Debian
+Pin-Priority: -1
+

+ +

This file content will tell apt and aptitude to not consider +installing systemd-sysv as part of any installation and upgrade +solution when resolving dependencies, and thus tell it to avoid +systemd as a default boot system. The end result should be that the +upgraded system keep using sysvinit.

+ +

If you are installing Jessie for the first time, there is no way to +get sysvinit installed by default (debootstrap used by +debian-installer have no option for this), but one can tell the +installer to switch to sysvinit before the first boot. Either by +using a kernel argument to the installer, or by adding a line to the +preseed file used. First, the kernel command line argument: + +

+preseed/late_command="in-target apt-get install --purge -y sysvinit-core"
+

+ +

Next, the line to use in a preseed file:

+ +

+d-i preseed/late_command string in-target apt-get install -y sysvinit-core
+

+ +

One can of course also do this after the first boot by installing +the sysvinit-core package.

+ +

I recommend only using sysvinit if you really need it, as the +sysvinit boot sequence in Debian have several hardware specific bugs +on Linux caused by the fact that it is unpredictable when hardware +devices show up during boot. But on the other hand, the new default +boot system still have a few rough edges I hope will be fixed before +Jessie is released.

+ +

Update 2014-11-26: Inspired by +a +blog post by Torsten Glaser, added --purge to the preseed +line.

+ +

+ + + Tags: bootsystem, debian, english. + + +

+ A Debian package for SMTP via Tor (aka SMTorP) using exim4 +

+ 10th November 2014 +

The right to communicate with your friends and family in private, +without anyone snooping, is a right every citicen have in a liberal +democracy. But this right is under serious attack these days.

+ +

A while back it occurred to me that one way to make the dragnet +surveillance conducted by NSA, GCHQ, FRA and others (and confirmed by +the whisleblower Snowden) more expensive for Internet email, +is to deliver all email using SMTP via Tor. Such SMTP option would be +a nice addition to the FreedomBox project if we could send email +between FreedomBox machines without leaking metadata about the emails +to the people peeking on the wire. I +proposed +this on the FreedomBox project mailing list in October and got a +lot of useful feedback and suggestions. It also became obvious to me +that this was not a novel idea, as the same idea was tested and +documented by Johannes Berg as early as 2006, and both +the +Mailpile and the Cables systems +propose a similar method / protocol to pass emails between users.

+ +

To implement such system one need to set up a Tor hidden service +providing the SMTP protocol on port 25, and use email addresses +looking like username@hidden-service-name.onion. With such addresses +the connections to port 25 on hidden-service-name.onion using Tor will +go to the correct SMTP server. To do this, one need to configure the +Tor daemon to provide the hidden service and the mail server to accept +emails for this .onion domain. To learn more about Exim configuration +in Debian and test the design provided by Johannes Berg in his FAQ, I +set out yesterday to create a Debian package for making it trivial to +set up such SMTP over Tor service based on Debian. Getting it to work +were fairly easy, and +the +source code for the Debian package is available from github. I +plan to move it into Debian if further testing prove this to be a +useful approach.

+ +

If you want to test this, set up a blank Debian machine without any +mail system installed (or run apt-get purge exim4-config to +get rid of exim4). Install tor, clone the git repository mentioned +above, build the deb and install it on the machine. Next, run +/usr/lib/exim4-smtorp/setup-exim-hidden-service and follow +the instructions to get the service up and running. Restart tor and +exim when it is done, and test mail delivery using swaks like +this:

+ +

+torsocks swaks --server dutlqrrmjhtfa3vp.onion \
+  --to fbx@dutlqrrmjhtfa3vp.onion
+

+ +

This will test the SMTP delivery using tor. Replace the email +address with your own address to test your server. :)

+ +

The setup procedure is still to complex, and I hope it can be made +easier and more automatic. Especially the tor setup need more work. +Also, the package include a tor-smtp tool written in C, but its task +should probably be rewritten in some script language to make the deb +architecture independent. It would probably also make the code easier +to review. The tor-smtp tool currently need to listen on a socket for +exim to talk to it and is started using xinetd. It would be better if +no daemon and no socket is needed. I suspect it is possible to get +exim to run a command line tool for delivery instead of talking to a +socket, and hope to figure out how in a future version of this +system.

+ +

Until I wipe my test machine, I can be reached using the +fbx@dutlqrrmjhtfa3vp.onion mail address, deliverable over +SMTorP. :)

+ +

+ + + Tags: debian, english, freedombox, personvern, surveillance. + + +

+ listadmin, the quick way to moderate mailman lists - nice free software +

+ 22nd October 2014 +

If you ever had to moderate a mailman list, like the ones on +alioth.debian.org, you know the web interface is fairly slow to +operate. First you visit one web page, enter the moderation password +and get a new page shown with a list of all the messages to moderate +and various options for each email address. This take a while for +every list you moderate, and you need to do it regularly to do a good +job as a list moderator. But there is a quick alternative, +the +listadmin program. It allow you to check lists for new messages +to moderate in a fraction of a second. Here is a test run on two +lists I recently took over:

+ +

+% time listadmin xiph
+fetching data for pkg-xiph-commits@lists.alioth.debian.org ... nothing in queue
+fetching data for pkg-xiph-maint@lists.alioth.debian.org ... nothing in queue
+
+real    0m1.709s
+user    0m0.232s
+sys     0m0.012s
+%
+

+ +

In 1.7 seconds I had checked two mailing lists and confirmed that +there are no message in the moderation queue. Every morning I +currently moderate 68 mailman lists, and it normally take around two +minutes. When I took over the two pkg-xiph lists above a few days +ago, there were 400 emails waiting in the moderator queue. It took me +less than 15 minutes to process them all using the listadmin +program.

+ +

If you install +the listadmin +package from Debian and create a file ~/.listadmin.ini +with content like this, the moderation task is a breeze:

+ +

+username username@example.org
+spamlevel 23
+default discard
+discard_if_reason "Posting restricted to members only. Remove us from your mail list."
+
+password secret
+adminurl https://{domain}/mailman/admindb/{list}
+mailman-list@lists.example.com
+
+password hidden
+other-list@otherserver.example.org
+

+ +

There are other options to set as well. Check the manual page to +learn the details.

+ +

If you are forced to moderate lists on a mailman installation where +the SSL certificate is self signed or not properly signed by a +generally accepted signing authority, you can set a environment +variable when calling listadmin to disable SSL verification:

+ +

+PERL_LWP_SSL_VERIFY_HOSTNAME=0 listadmin
+

+ +

If you want to moderate a subset of the lists you take care of, you +can provide an argument to the listadmin script like I do in the +initial screen dump (the xiph argument). Using an argument, only +lists matching the argument string will be processed. This make it +quick to accept messages if you notice the moderation request in your +email.

+ +

Without the listadmin program, I would never be the moderator of 68 +mailing lists, as I simply do not have time to spend on that if the +process was any slower. The listadmin program have saved me hours of +time I could spend elsewhere over the years. It truly is nice free +software.

+ +

As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

+ +

Update 2014-10-27: Added missing 'username' statement in +configuration example. Also, I've been told that the +PERL_LWP_SSL_VERIFY_HOSTNAME=0 setting do not work for everyone. Not +sure why.

+ +

+ + + Tags: debian, english. + + +

+ Debian Jessie, PXE and automatic firmware installation +

+ 17th October 2014 +

When PXE installing laptops with Debian, I often run into the +problem that the WiFi card require some firmware to work properly. +And it has been a pain to fix this using preseeding in Debian. +Normally something more is needed. But thanks to +my isenkram +package and its recent tasksel extension, it has now become easy +to do this using simple preseeding.

+ +

The isenkram-cli package provide tasksel tasks which will install +firmware for the hardware found in the machine (actually, requested by +the kernel modules for the hardware). (It can also install user space +programs supporting the hardware detected, but that is not the focus +of this story.)

+ +

To get this working in the default installation, two preeseding +values are needed. First, the isenkram-cli package must be installed +into the target chroot (aka the hard drive) before tasksel is executed +in the pkgsel step of the debian-installer system. This is done by +preseeding the base-installer/includes debconf value to include the +isenkram-cli package. The package name is next passed to debootstrap +for installation. With the isenkram-cli package in place, tasksel +will automatically use the isenkram tasks to detect hardware specific +packages for the machine being installed and install them, because +isenkram-cli contain tasksel tasks.

+ +

Second, one need to enable the non-free APT repository, because +most firmware unfortunately is non-free. This is done by preseeding +the apt-mirror-setup step. This is unfortunate, but for a lot of +hardware it is the only option in Debian.

+ +

The end result is two lines needed in your preseeding file to get +firmware installed automatically by the installer:

+ +

+base-installer base-installer/includes string isenkram-cli
+apt-mirror-setup apt-setup/non-free boolean true
+

+ +

The current version of isenkram-cli in testing/jessie will install +both firmware and user space packages when using this method. It also +do not work well, so use version 0.15 or later. Installing both +firmware and user space packages might give you a bit more than you +want, so I decided to split the tasksel task in two, one for firmware +and one for user space programs. The firmware task is enabled by +default, while the one for user space programs is not. This split is +implemented in the package currently in unstable.

+ +

If you decide to give this a go, please let me know (via email) how +this recipe work for you. :)

+ +

So, I bet you are wondering, how can this work. First and +foremost, it work because tasksel is modular, and driven by whatever +files it find in /usr/lib/tasksel/ and /usr/share/tasksel/. So the +isenkram-cli package place two files for tasksel to find. First there +is the task description file (/usr/share/tasksel/descs/isenkram.desc):

+ +

+Task: isenkram-packages
+Section: hardware
+Description: Hardware specific packages (autodetected by isenkram)
+ Based on the detected hardware various hardware specific packages are
+ proposed.
+Test-new-install: show show
+Relevance: 8
+Packages: for-current-hardware
+
+Task: isenkram-firmware
+Section: hardware
+Description: Hardware specific firmware packages (autodetected by isenkram)
+ Based on the detected hardware various hardware specific firmware
+ packages are proposed.
+Test-new-install: mark show
+Relevance: 8
+Packages: for-current-hardware-firmware
+

+ +

The key parts are Test-new-install which indicate how the task +should be handled and the Packages line referencing to a script in +/usr/lib/tasksel/packages/. The scripts use other scripts to get a +list of packages to install. The for-current-hardware-firmware script +look like this to list relevant firmware for the machine: + +

+#!/bin/sh
+#
+PATH=/usr/sbin:$PATH
+export PATH
+isenkram-autoinstall-firmware -l
+

+ +

With those two pieces in place, the firmware is installed by +tasksel during the normal d-i run. :)

+ +

If you want to test what tasksel will install when isenkram-cli is +installed, run DEBIAN_PRIORITY=critical tasksel --test +--new-install to get the list of packages that tasksel would +install.

+ +

Debian Edu will be +pilots in testing this feature, as isenkram is used there now to +install firmware, replacing the earlier scripts.

+ +

+ + + Tags: debian, english, isenkram, sysadmin. + + +

+ Ubuntu used to show the bread prizes at ICA Storo +

+ 4th October 2014 +

Today I came across an unexpected Ubuntu boot screen. Above the +bread shelf on the ICA shop at Storo in Oslo, the grub menu of Ubuntu +with Linux kernel 3.2.0-23 (ie probably version 12.04 LTS) was stuck +on a screen normally showing the bread types and prizes:

+ +

If it had booted as it was supposed to, I would never had known +about this hidden Linux installation. It is interesting what +errors can reveal.

+ +

+ + + Tags: debian, english. + + +

+ New lsdvd release version 0.17 is ready +

+ 4th October 2014 +

The lsdvd project +got a new set of developers a few weeks ago, after the original +developer decided to step down and pass the project to fresh blood. +This project is now maintained by Petter Reinholdtsen and Steve +Dibb.

+ +

I just wrapped up +a +new lsdvd release, available in git or from +the +download page. This is the changelog dated 2014-10-03 for version +0.17.

+ +

Ignore 'phantom' audio, subtitle tracks
Check for garbage in the program chains, which indicate that a track is + non-existant, to work around additional copy protection
Fix displaying content type for audio tracks, subtitles
Fix pallete display of first entry
Fix include orders
Ignore read errors in titles that would not be displayed anyway
Fix the chapter count
Make sure the array size and the array limit used when initialising + the palette size is the same.
Fix array printing.
Correct subsecond calculations.
Add sector information to the output format.
Clean up code to be closer to ANSI C and compile without warnings + with more GCC compiler warnings.

+ +

This change bring together patches for lsdvd in use in various +Linux and Unix distributions, as well as patches submitted to the +project the last nine years. Please check it out. :)

+ +

+ + + Tags: debian, english, lsdvd, multimedia. + + +

+ How to test Debian Edu Jessie despite some fatal problems with the installer +

+ 26th September 2014 +

The Debian Edu / Skolelinux +project provide a Linux solution for schools, including a +powerful desktop with education software, a central server providing +web pages, user database, user home directories, central login and PXE +boot of both clients without disk and the installation to install Debian +Edu on machines with disk (and a few other services perhaps to small +to mention here). We in the Debian Edu team are currently working on +the Jessie based version, trying to get everything in shape before the +freeze, to avoid having to maintain our own package repository in the +future. The +current +status can be seen on the Debian wiki, and there is still heaps of +work left. Some fatal problems block testing, breaking the installer, +but it is possible to work around these to get anyway. Here is a +recipe on how to get the installation limping along.

+ +

First, download the test ISO via +ftp, +http +or rsync (use +ftp.skolelinux.org::cd-edu-testing-nolocal-netinst/debian-edu-amd64-i386-NETINST-1.iso). +The ISO build was broken on Tuesday, so we do not get a new ISO every +12 hours or so, but thankfully the ISO we already got we are able to +install with some tweaking.

+ +

When you get to the Debian Edu profile question, go to tty2 +(use Alt-Ctrl-F2), run

+ +

+nano /usr/bin/edu-eatmydata-install
+

+ +

and add 'exit 0' as the second line, disabling the eatmydata +optimization. Return to the installation, select the profile you want +and continue. Without this change, exim4-config will fail to install +due to a known bug in eatmydata.

+ +

When you get the grub question at the end, answer /dev/sda (or if +this do not work, figure out what your correct value would be. All my +test machines need /dev/sda, so I have no advice if it do not fit +your need.

+ +

If you installed a profile including a graphical desktop, log in as +root after the initial boot from hard drive, and install the +education-desktop-XXX metapackage. XXX can be kde, gnome, lxde, xfce +or mate. If you want several desktop options, install more than one +metapackage. Once this is done, reboot and you should have a working +graphical login screen. This workaround should no longer be needed +once the education-tasks package version 1.801 enter testing in two +days.

+ +

I believe the ISO build will start working on two days when the new +tasksel package enter testing and Steve McIntyre get a chance to +update the debian-cd git repository. The eatmydata, grub and desktop +issues are already fixed in unstable and testing, and should show up +on the ISO as soon as the ISO build start working again. Well the +eatmydata optimization is really just disabled. The proper fix +require an upload by the eatmydata maintainer applying the patch +provided in bug #702711. +The rest have proper fixes in unstable.

+ +

I hope this get you going with the installation testing, as we are +quickly running out of time trying to get our Jessie based +installation ready before the distribution freeze in a month.

+ +

+ + + Tags: debian, debian edu, english. + + +

+ Suddenly I am the new upstream of the lsdvd command line tool +

+ 25th September 2014 +

I use the lsdvd tool +to handle my fairly large DVD collection. It is a nice command line +tool to get details about a DVD, like title, tracks, track length, +etc, in XML, Perl or human readable format. But lsdvd have not seen +any new development since 2006 and had a few irritating bugs affecting +its use with some DVDs. Upstream seemed to be dead, and in January I +sent a small probe asking for a version control repository for the +project, without any reply. But I use it regularly and would like to +get an updated version +into Debian. So two weeks ago I tried harder to get in touch with +the project admin, and after getting a reply from him explaining that +he was no longer interested in the project, I asked if I could take +over. And yesterday, I became project admin.

+ +

I've been in touch with a Gentoo developer and the Debian +maintainer interested in joining forces to maintain the upstream +project, and I hope we can get a new release out fairly quickly, +collecting the patches spread around on the internet into on place. +I've added the relevant Debian patches to the freshly created git +repository, and expect the Gentoo patches to make it too. If you got +a DVD collection and care about command line tools, check out +the git source and join +the project mailing +list. :)

+ +

+ + + Tags: debian, english, lsdvd, multimedia. + + +

+ Speeding up the Debian installer using eatmydata and dpkg-divert +

+ 16th September 2014 +

The Debian installer could be +a lot quicker. When we install more than 2000 packages in +Skolelinux / Debian Edu using +tasksel in the installer, unpacking the binary packages take forever. +A part of the slow I/O issue was discussed in +bug #613428 about too +much file system sync-ing done by dpkg, which is the package +responsible for unpacking the binary packages. Other parts (like code +executed by postinst scripts) might also sync to disk during +installation. All this sync-ing to disk do not really make sense to +me. If the machine crash half-way through, I start over, I do not try +to salvage the half installed system. So the failure sync-ing is +supposed to protect against, hardware or system crash, is not really +relevant while the installer is running.

+ +

A few days ago, I thought of a way to get rid of all the file +system sync()-ing in a fairly non-intrusive way, without the need to +change the code in several packages. The idea is not new, but I have +not heard anyone propose the approach using dpkg-divert before. It +depend on the small and clever package +eatmydata, which +uses LD_PRELOAD to replace the system functions for syncing data to +disk with functions doing nothing, thus allowing programs to live +dangerous while speeding up disk I/O significantly. Instead of +modifying the implementation of dpkg, apt and tasksel (which are the +packages responsible for selecting, fetching and installing packages), +it occurred to me that we could just divert the programs away, replace +them with a simple shell wrapper calling +"eatmydata $program $@", to get the same effect. +Two days ago I decided to test the idea, and wrapped up a simple +implementation for the Debian Edu udeb.

+ +

The effect was stunning. In my first test it reduced the running +time of the pkgsel step (installing tasks) from 64 to less than 44 +minutes (20 minutes shaved off the installation) on an old Dell +Latitude D505 machine. I am not quite sure what the optimised time +would have been, as I messed up the testing a bit, causing the debconf +priority to get low enough for two questions to pop up during +installation. As soon as I saw the questions I moved the installation +along, but do not know how long the question were holding up the +installation. I did some more measurements using Debian Edu Jessie, +and got these results. The time measured is the time stamp in +/var/log/syslog between the "pkgsel: starting tasksel" and the +"pkgsel: finishing up" lines, if you want to do the same measurement +yourself. In Debian Edu, the tasksel dialog do not show up, and the +timing thus do not depend on how quickly the user handle the tasksel +dialog.

+ +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Machine/setup Original tasksel Optimised tasksel Reduction
Latitude D505 Main+LTSP LXDE 64 min (07:46-08:50) <44 min (11:27-12:11) >20 min 18%
Latitude D505 Roaming LXDE 57 min (08:48-09:45) 34 min (07:43-08:17) 23 min 40%
Latitude D505 Minimal 22 min (10:37-10:59) 11 min (11:16-11:27) 11 min 50%
Thinkpad X200 Minimal 6 min (08:19-08:25) 4 min (08:04-08:08) 2 min 33%
Thinkpad X200 Roaming KDE 19 min (09:21-09:40) 15 min (10:25-10:40) 4 min 21%

Machine/setup	Original tasksel	Optimised tasksel	Reduction
Latitude D505 Main+LTSP LXDE	64 min (07:46-08:50)	<44 min (11:27-12:11)	>20 min 18%
Latitude D505 Roaming LXDE	57 min (08:48-09:45)	34 min (07:43-08:17)	23 min 40%
Latitude D505 Minimal	22 min (10:37-10:59)	11 min (11:16-11:27)	11 min 50%
Thinkpad X200 Minimal	6 min (08:19-08:25)	4 min (08:04-08:08)	2 min 33%
Thinkpad X200 Roaming KDE	19 min (09:21-09:40)	15 min (10:25-10:40)	4 min 21%

+ +

The test is done using a netinst ISO on a USB stick, so some of the +time is spent downloading packages. The connection to the Internet +was 100Mbit/s during testing, so downloading should not be a +significant factor in the measurement. Download typically took a few +seconds to a few minutes, depending on the amount of packages being +installed.

+ +

The speedup is implemented by using two hooks in +Debian +Installer, the pre-pkgsel.d hook to set up the diverts, and the +finish-install.d hook to remove the divert at the end of the +installation. I picked the pre-pkgsel.d hook instead of the +post-base-installer.d hook because I test using an ISO without the +eatmydata package included, and the post-base-installer.d hook in +Debian Edu can only operate on packages included in the ISO. The +negative effect of this is that I am unable to activate this +optimization for the kernel installation step in d-i. If the code is +moved to the post-base-installer.d hook, the speedup would be larger +for the entire installation.

+ +

I've implemented this in the +debian-edu-install +git repository, and plan to provide the optimization as part of the +Debian Edu installation. If you want to test this yourself, you can +create two files in the installer (or in an udeb). One shell script +need do go into /usr/lib/pre-pkgsel.d/, with content like this:

+ +

+#!/bin/sh
+set -e
+. /usr/share/debconf/confmodule
+info() {
+    logger -t my-pkgsel "info: $*"
+}
+error() {
+    logger -t my-pkgsel "error: $*"
+}
+override_install() {
+    apt-install eatmydata || true
+    if [ -x /target/usr/bin/eatmydata ] ; then
+        for bin in dpkg apt-get aptitude tasksel ; do
+            file=/usr/bin/$bin
+            # Test that the file exist and have not been diverted already.
+            if [ -f /target$file ] ; then
+                info "diverting $file using eatmydata"
+                printf "#!/bin/sh\neatmydata $bin.distrib \"\$@\"\n" \
+                    > /target$file.edu
+                chmod 755 /target$file.edu
+                in-target dpkg-divert --package debian-edu-config \
+                    --rename --quiet --add $file
+                ln -sf ./$bin.edu /target$file
+            else
+                error "unable to divert $file, as it is missing."
+            fi
+        done
+    else
+        error "unable to find /usr/bin/eatmydata after installing the eatmydata pacage"
+    fi
+}
+
+override_install
+

+ +

To clean up, another shell script should go into +/usr/lib/finish-install.d/ with code like this: + +

+#! /bin/sh -e
+. /usr/share/debconf/confmodule
+error() {
+    logger -t my-finish-install "error: $@"
+}
+remove_install_override() {
+    for bin in dpkg apt-get aptitude tasksel ; do
+        file=/usr/bin/$bin
+        if [ -x /target$file.edu ] ; then
+            rm /target$file
+            in-target dpkg-divert --package debian-edu-config \
+                --rename --quiet --remove $file
+            rm /target$file.edu
+        else
+            error "Missing divert for $file."
+        fi
+    done
+    sync # Flush file buffers before continuing
+}
+
+remove_install_override
+

+ +

In Debian Edu, I placed both code fragments in a separate script +edu-eatmydata-install and call it from the pre-pkgsel.d and +finish-install.d scripts.

+ +

By now you might ask if this change should get into the normal +Debian installer too? I suspect it should, but am not sure the +current debian-installer coordinators find it useful enough. It also +depend on the side effects of the change. I'm not aware of any, but I +guess we will see if the change is safe after some more testing. +Perhaps there is some package in Debian depending on sync() and +fsync() having effect? Perhaps it should go into its own udeb, to +allow those of us wanting to enable it to do so without affecting +everyone.

+ +

Update 2014-09-24: Since a few days ago, enabling this optimization +will break installation of all programs using gnutls because of +bug #702711. An updated +eatmydata package in Debian will solve it.

+ +

Update 2014-10-17: The bug mentioned above is fixed in testing and +the optimization work again. And I have discovered that the +dpkg-divert trick is not really needed and implemented a slightly +simpler approach as part of the debian-edu-install package. See +tools/edu-eatmydata-install in the source package.

+ +

Update 2014-11-11: Unfortunately, a new +bug #765738 in eatmydata only +triggering on i386 made it into testing, and broke this installation +optimization again. If unblock +request 768893 is accepted, it should be working again.

+ +

+ + + Tags: debian, debian edu, english. + + +

+ Good bye subkeys.pgp.net, welcome pool.sks-keyservers.net +

+ 10th September 2014 +

Yesterday, I had the pleasure of attending a talk with the +Norwegian Unix User Group about +the +OpenPGP keyserver pool sks-keyservers.net, and was very happy to +learn that there is a large set of publicly available key servers to +use when looking for peoples public key. So far I have used +subkeys.pgp.net, and some times wwwkeys.nl.pgp.net when the former +were misbehaving, but those days are ended. The servers I have used +up until yesterday have been slow and some times unavailable. I hope +those problems are gone now.

+ +

Behind the round robin DNS entry of the +sks-keyservers.net service +there is a pool of more than 100 keyservers which are checked every +day to ensure they are well connected and up to date. It must be +better than what I have used so far. :)

+ +

Yesterdays speaker told me that the service is the default +keyserver provided by the default configuration in GnuPG, but this do +not seem to be used in Debian. Perhaps it should?

+ +

Anyway, I've updated my ~/.gnupg/options file to now include this +line:

+ +

+keyserver pool.sks-keyservers.net
+

+ +

With GnuPG version 2 one can also locate the keyserver using SRV +entries in DNS. Just for fun, I did just that at work, so now every +user of GnuPG at the University of Oslo should find a OpenGPG +keyserver automatically should their need it:

+ +

+% host -t srv _pgpkey-http._tcp.uio.no
+_pgpkey-http._tcp.uio.no has SRV record 0 100 11371 pool.sks-keyservers.net.
+%
+

+ +

Now if only +the +HKP lookup protocol supported finding signature paths, I would be +very happy. It can look up a given key or search for a user ID, but I +normally do not want that, but to find a trust path from my key to +another key. Given a user ID or key ID, I would like to find (and +download) the keys representing a signature path from my key to the +key in question, to be able to get a trust path between the two keys. +This is as far as I can tell not possible today. Perhaps something +for a future version of the protocol?

+ +

+ + + Tags: debian, english, personvern, sikkerhet. + + +

+ From English wiki to translated PDF and epub via Docbook +

+ 17th June 2014 +

The Debian Edu / Skolelinux +project provide an instruction manual for teachers, system +administrators and other users that contain useful tips for setting up +and maintaining a Debian Edu installation. This text is about how the +text processing of this manual is handled in the project.

+ +

One goal of the project is to provide information in the native +language of its users, and for this we need to handle translations. +But we also want to make sure each language contain the same +information, so for this we need a good way to keep the translations +in sync. And we want it to be easy for our users to improve the +documentation, avoiding the need to learn special formats or tools to +contribute, and the obvious way to do this is to make it possible to +edit the documentation using a web browser. We also want it to be +easy for translators to keep the translation up to date, and give them +help in figuring out what need to be translated. Here is the list of +tools and the process we have found trying to reach all these +goals.

+ +

We maintain the authoritative source of our manual in the +Debian +wiki, as several wiki pages written in English. It consist of one +front page with references to the different chapters, several pages +for each chapter, and finally one "collection page" gluing all the +chapters together into one large web page (aka +the +AllInOne page). The AllInOne page is the one used for further +processing and translations. Thanks to the fact that the +MoinMoin installation on +wiki.debian.org support exporting pages in +the Docbook format, we can fetch +the list of pages to export using the raw version of the AllInOne +page, loop over each of them to generate a Docbook XML version of the +manual. This process also download images and transform image +references to use the locally downloaded images. The generated +Docbook XML files are slightly broken, so some post-processing is done +using the documentation/scripts/get_manual program, and the +result is a nice Docbook XML file (debian-edu-wheezy-manual.xml) and +a handfull of images. The XML file can now be used to generate PDF, HTML +and epub versions of the English manual. This is the basic step of +our process, making PDF (using dblatex), HTML (using xsltproc) and +epub (using dbtoepub) version from Docbook XML, and the resulting files +are placed in the debian-edu-doc-en binary package.

+ +

But English documentation is not enough for us. We want translated +documentation too, and we want to make it easy for translators to +track the English original. For this we use the +poxml package, +which allow us to transform the English Docbook XML file into a +translation file (a .pot file), usable with the normal gettext based +translation tools used by those translating free software. The pot +file is used to create and maintain translation files (several .po +files), which the translations update with the native language +translations of all titles, paragraphs and blocks of text in the +original. The next step is combining the original English Docbook XML +and the translation file (say debian-edu-wheezy-manual.nb.po), to +create a translated Docbook XML file (in this case +debian-edu-wheezy-manual.nb.xml). This translated (or partly +translated, if the translation is not complete) Docbook XML file can +then be used like the original to create a PDF, HTML and epub version +of the documentation.

+ +

The translators use different tools to edit the .po files. We +recommend using +lokalize, +while some use emacs and vi, others can use web based editors like +Poodle or +Transifex. All we care about +is where the .po file end up, in our git repository. Updated +translations can either be committed directly to git, or submitted as +bug reports +against the debian-edu-doc package.

+ +

One challenge is images, which both might need to be translated (if +they show translated user applications), and are needed in different +formats when creating PDF and HTML versions (epub is a HTML version in +this regard). For this we transform the original PNG images to the +needed density and format during build, and have a way to provide +translated images by storing translated versions in +images/$LANGUAGECODE/. I am a bit unsure about the details here. The +package maintainers know more.

+ +

If you wonder what the result look like, we provide +the content +of the documentation packages on the web. See for example the +Italian +PDF version or the +German +HTML version. We do not yet build the epub version by default, +but perhaps it will be done in the future.

+ +

To learn more, check out +the +debian-edu-doc package, +the +manual on the wiki and +the +translation instructions in the manual.

+ +

+ + + Tags: debian, debian edu, docbook, english. + + +

+ Install hardware dependent packages using tasksel (Isenkram 0.7) +

+ 23rd April 2014 +

It would be nice if it was easier in Debian to get all the hardware +related packages relevant for the computer installed automatically. +So I implemented one, using +my Isenkram +package. To use it, install the tasksel and isenkram packages and +run tasksel as user root. You should be presented with a new option, +"Hardware specific packages (autodetected by isenkram)". When you +select it, tasksel will install the packages isenkram claim is fit for +the current hardware, hot pluggable or not.

+ +

The implementation is in two files, one is the tasksel menu entry +description, and the other is the script used to extract the list of +packages to install. The first part is in +/usr/share/tasksel/descs/isenkram.desc and look like +this:

+ +

+Task: isenkram
+Section: hardware
+Description: Hardware specific packages (autodetected by isenkram)
+ Based on the detected hardware various hardware specific packages are
+ proposed.
+Test-new-install: mark show
+Relevance: 8
+Packages: for-current-hardware
+

+ +

The second part is in +/usr/lib/tasksel/packages/for-current-hardware and look like +this:

+ +

+#!/bin/sh
+#
+(
+    isenkram-lookup
+    isenkram-autoinstall-firmware -l
+) | sort -u
+

+ +

All in all, a very short and simple implementation making it +trivial to install the hardware dependent package we all may want to +have installed on our machines. I've not been able to find a way to +get tasksel to tell you exactly which packages it plan to install +before doing the installation. So if you are curious or careful, +check the output from the isenkram-* command line tools first.

+ +

The information about which packages are handling which hardware is +fetched either from the isenkram package itself in +/usr/share/isenkram/, from git.debian.org or from the APT package +database (using the Modaliases header). The APT package database +parsing have caused a nasty resource leak in the isenkram daemon (bugs +#719837 and +#730704). The cause is in +the python-apt code (bug +#745487), but using a +workaround I was able to get rid of the file descriptor leak and +reduce the memory leak from ~30 MiB per hardware detection down to +around 2 MiB per hardware detection. It should make the desktop +daemon a lot more useful. The fix is in version 0.7 uploaded to +unstable today.

+ +

I believe the current way of mapping hardware to packages in +Isenkram is is a good draft, but in the future I expect isenkram to +use the AppStream data source for this. A proposal for getting proper +AppStream support into Debian is floating around as +DEP-11, and +GSoC +project will take place this summer to improve the situation. I +look forward to seeing the result, and welcome patches for isenkram to +start using the information when it is ready.

+ +

If you want your package to map to some specific hardware, either +add a "Xb-Modaliases" header to your control file like I did in +the pymissile +package or submit a bug report with the details to the isenkram +package. See also +all my +blog posts tagged isenkram for details on the notation. I expect +the information will be migrated to AppStream eventually, but for the +moment I got no better place to store it.

+ +

+ + + Tags: debian, english, isenkram. + + +

+ FreedomBox milestone - all packages now in Debian Sid +

+ 15th April 2014 +

The Freedombox +project is working on providing the software and hardware to make +it easy for non-technical people to host their data and communication +at home, and being able to communicate with their friends and family +encrypted and away from prying eyes. It is still going strong, and +today a major mile stone was reached.

+ +

Today, the last of the packages currently used by the project to +created the system images were accepted into Debian Unstable. It was +the freedombox-setup package, which is used to configure the images +during build and on the first boot. Now all one need to get going is +the build code from the freedom-maker git repository and packages from +Debian. And once the freedombox-setup package enter testing, we can +build everything directly from Debian. :)

+ +

Some key packages used by Freedombox are +freedombox-setup, +plinth, +pagekite, +tor, +privoxy, +owncloud and +dnsmasq. There +are plans to integrate more packages into the setup. User +documentation is maintained on the Debian wiki. Please +check out +the manual and help us improve it.

+ +

To test for yourself and create boot images with the FreedomBox +setup, run this on a Debian machine using a user with sudo rights to +become root:

+ +

+sudo apt-get install git vmdebootstrap mercurial python-docutils \
+  mktorrent extlinux virtualbox qemu-user-static binfmt-support \
+  u-boot-tools
+git clone http://anonscm.debian.org/git/freedombox/freedom-maker.git \
+  freedom-maker
+make -C freedom-maker dreamplug-image raspberry-image virtualbox-image
+

+ +

Root access is needed to run debootstrap and mount loopback +devices. See the README in the freedom-maker git repo for more +details on the build. If you do not want all three images, trim the +make line. Note that the virtualbox-image target is not really +virtualbox specific. It create a x86 image usable in kvm, qemu, +vmware and any other x86 virtual machine environment. You might need +the version of vmdebootstrap in Jessie to get the build working, as it +include fixes for a race condition with kpartx.

+ +

If you instead want to install using a Debian CD and the preseed +method, boot a Debian Wheezy ISO and use this boot argument to load +the preseed values:

+ +

+url=http://www.reinholdtsen.name/freedombox/preseed-jessie.dat
+

+ +

I have not tested it myself the last few weeks, so I do not know if +it still work.

+ +

If you wonder how to help, one task you could look at is using +systemd as the boot system. It will become the default for Linux in +Jessie, so we need to make sure it is usable on the Freedombox. I did +a simple test a few weeks ago, and noticed dnsmasq failed to start +during boot when using systemd. I suspect there are other problems +too. :) To detect problems, there is a test suite included, which can +be run from the plinth web interface.

+ +

Give it a go and let us know how it goes on the mailing list, and help +us get the new release published. :) Please join us on +IRC (#freedombox on +irc.debian.org) and +the +mailing list if you want to help make this vision come true.

+ +

+ + + Tags: debian, english, freedombox, sikkerhet, surveillance, web. + + +

+ S3QL, a locally mounted cloud file system - nice free software +

+ 9th April 2014 +

For a while now, I have been looking for a sensible offsite backup +solution for use at home. My requirements are simple, it must be +cheap and locally encrypted (in other words, I keep the encryption +keys, the storage provider do not have access to my private files). +One idea me and my friends had many years ago, before the cloud +storage providers showed up, was to use Google mail as storage, +writing a Linux block device storing blocks as emails in the mail +service provided by Google, and thus get heaps of free space. On top +of this one can add encryption, RAID and volume management to have +lots of (fairly slow, I admit that) cheap and encrypted storage. But +I never found time to implement such system. But the last few weeks I +have looked at a system called +S3QL, a locally +mounted network backed file system with the features I need.

+ +

S3QL is a fuse file system with a local cache and cloud storage, +handling several different storage providers, any with Amazon S3, +Google Drive or OpenStack API. There are heaps of such storage +providers. S3QL can also use a local directory as storage, which +combined with sshfs allow for file storage on any ssh server. S3QL +include support for encryption, compression, de-duplication, snapshots +and immutable file systems, allowing me to mount the remote storage as +a local mount point, look at and use the files as if they were local, +while the content is stored in the cloud as well. This allow me to +have a backup that should survive fire. The file system can not be +shared between several machines at the same time, as only one can +mount it at the time, but any machine with the encryption key and +access to the storage service can mount it if it is unmounted.

+ +

It is simple to use. I'm using it on Debian Wheezy, where the +package is included already. So to get started, run apt-get +install s3ql. Next, pick a storage provider. I ended up picking +Greenqloud, after reading their nice recipe on +how +to use S3QL with their Amazon S3 service, because I trust the laws +in Iceland more than those in USA when it come to keeping my personal +data safe and private, and thus would rather spend money on a company +in Iceland. Another nice recipe is available from the article +S3QL +Filesystem for HPC Storage by Jeff Layton in the HPC section of +Admin magazine. When the provider is picked, figure out how to get +the API key needed to connect to the storage API. With Greencloud, +the key did not show up until I had added payment details to my +account.

+ +

Armed with the API access details, it is time to create the file +system. First, create a new bucket in the cloud. This bucket is the +file system storage area. I picked a bucket name reflecting the +machine that was going to store data there, but any name will do. +I'll refer to it as bucket-name below. In addition, one need +the API login and password, and a locally created password. Store it +all in ~root/.s3ql/authinfo2 like this: + +

+[s3c]
+storage-url: s3c://s.greenqloud.com:443/bucket-name
+backend-login: API-login
+backend-password: API-password
+fs-passphrase: local-password
+

+ +

I create my local passphrase using pwget 50 or similar, +but any sensible way to create a fairly random password should do it. +Armed with these details, it is now time to run mkfs, entering the API +details and password to create it:

+ +

+# mkdir -m 700 /var/lib/s3ql-cache
+# mkfs.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl s3c://s.greenqloud.com:443/bucket-name
+Enter backend login: 
+Enter backend password: 
+Before using S3QL, make sure to read the user's guide, especially
+the 'Important Rules to Avoid Loosing Data' section.
+Enter encryption password: 
+Confirm encryption password: 
+Generating random encryption key...
+Creating metadata tables...
+Dumping metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Compressing and uploading metadata...
+Wrote 0.00 MB of compressed metadata.
+#

+ +

The next step is mounting the file system to make the storage available. + +

+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+Using 4 upload threads.
+Downloading and decompressing metadata...
+Reading metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Mounting filesystem...
+# df -h /s3ql
+Filesystem                              Size  Used Avail Use% Mounted on
+s3c://s.greenqloud.com:443/bucket-name  1.0T     0  1.0T   0% /s3ql
+#
+

+ +

The file system is now ready for use. I use rsync to store my +backups in it, and as the metadata used by rsync is downloaded at +mount time, no network traffic (and storage cost) is triggered by +running rsync. To unmount, one should not use the normal umount +command, as this will not flush the cache to the cloud storage, but +instead running the umount.s3ql command like this: + +

+# umount.s3ql /s3ql
+# 
+

+ +

There is a fsck command available to check the file system and +correct any problems detected. This can be used if the local server +crashes while the file system is mounted, to reset the "already +mounted" flag. This is what it look like when processing a working +file system:

+ +

+# fsck.s3ql --force --ssl s3c://s.greenqloud.com:443/bucket-name
+Using cached metadata.
+File system seems clean, checking anyway.
+Checking DB integrity...
+Creating temporary extra indices...
+Checking lost+found...
+Checking cached objects...
+Checking names (refcounts)...
+Checking contents (names)...
+Checking contents (inodes)...
+Checking contents (parent inodes)...
+Checking objects (reference counts)...
+Checking objects (backend)...
+..processed 5000 objects so far..
+..processed 10000 objects so far..
+..processed 15000 objects so far..
+Checking objects (sizes)...
+Checking blocks (referenced objects)...
+Checking blocks (refcounts)...
+Checking inode-block mapping (blocks)...
+Checking inode-block mapping (inodes)...
+Checking inodes (refcounts)...
+Checking inodes (sizes)...
+Checking extended attributes (names)...
+Checking extended attributes (inodes)...
+Checking symlinks (inodes)...
+Checking directory reachability...
+Checking unix conventions...
+Checking referential integrity...
+Dropping temporary indices...
+Backing up old metadata...
+Dumping metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Compressing and uploading metadata...
+Wrote 0.89 MB of compressed metadata.
+# 
+

+ +

Thanks to the cache, working on files that fit in the cache is very +quick, about the same speed as local file access. Uploading large +amount of data is to me limited by the bandwidth out of and into my +house. Uploading 685 MiB with a 100 MiB cache gave me 305 kiB/s, +which is very close to my upload speed, and downloading the same +Debian installation ISO gave me 610 kiB/s, close to my download speed. +Both were measured using dd. So for me, the bottleneck is my +network, not the file system code. I do not know what a good cache +size would be, but suspect that the cache should e larger than your +working set.

+ +

I mentioned that only one machine can mount the file system at the +time. If another machine try, it is told that the file system is +busy:

+ +

+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+Using 8 upload threads.
+Backend reports that fs is still mounted elsewhere, aborting.
+#
+

+ +

The file content is uploaded when the cache is full, while the +metadata is uploaded once every 24 hour by default. To ensure the +file system content is flushed to the cloud, one can either umount the +file system, or ask S3QL to flush the cache and metadata using +s3qlctrl: + +

+# s3qlctrl upload-meta /s3ql
+# s3qlctrl flushcache /s3ql
+# 
+

+ +

If you are curious about how much space your data uses in the +cloud, and how much compression and deduplication cut down on the +storage usage, you can use s3qlstat on the mounted file system to get +a report:

+ +

+# s3qlstat /s3ql
+Directory entries:    9141
+Inodes:               9143
+Data blocks:          8851
+Total data size:      22049.38 MB
+After de-duplication: 21955.46 MB (99.57% of total)
+After compression:    21877.28 MB (99.22% of total, 99.64% of de-duplicated)
+Database size:        2.39 MB (uncompressed)
+(some values do not take into account not-yet-uploaded dirty blocks in cache)
+#
+

+ +

I mentioned earlier that there are several possible suppliers of +storage. I did not try to locate them all, but am aware of at least +Greenqloud, +Google Drive, +Amazon S3 web serivces, +Rackspace and +Crowncloud. The latter even +accept payment in Bitcoin. Pick one that suit your need. Some of +them provide several GiB of free storage, but the prize models are +quite different and you will have to figure out what suits you +best.

+ +

While researching this blog post, I had a look at research papers +and posters discussing the S3QL file system. There are several, which +told me that the file system is getting a critical check by the +science community and increased my confidence in using it. One nice +poster is titled +"An +Innovative Parallel Cloud Storage System using OpenStackâs SwiftObject +Store and Transformative Parallel I/O Approach" by Hsing-Bung +Chen, Benjamin McClelland, David Sherrill, Alfred Torrez, Parks Fields +and Pamela Smith. Please have a look.

+ +

Given my problems with different file systems earlier, I decided to +check out the mounted S3QL file system to see if it would be usable as +a home directory (in other word, that it provided POSIX semantics when +it come to locking and umask handling etc). Running +my +test code to check file system semantics, I was happy to discover that +no error was found. So the file system can be used for home +directories, if one chooses to do so.

+ +

If you do not want a locally file system, and want something that +work without the Linux fuse file system, I would like to mention the +Tarsnap service, which also +provide locally encrypted backup using a command line client. It have +a nicer access control system, where one can split out read and write +access, allowing some systems to write to the backup and others to +only read from it.

+ +

As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

+ +

+ + + Tags: debian, english, personvern, sikkerhet. + + +

Entries tagged "debian".

Archive