Petter Reinholdtsen

Today NUUG and Hackeriet announced our plans to join forces and create a wireless community network in Oslo. The workshop to help people get started will take place Thursday 2013-11-28, but we already are collecting the geolocation of people joining forces to make this happen. We have 9 locations plotted on the map, but we will need more before we have a connected mesh spread across Oslo. If this sound interesting to you, please join us at the workshop. If you are too impatient to wait 15 days, please join us on the IRC channel #nuug on irc.freenode.net right away. :)

Continuing my research into mesh networking, I was recommended to use TP-Link 3040 and 3600 access points as mesh nodes, and the pair I bought arrived on Friday. Here are my notes on how to set up the MR3040 as a mesh node using OpenWrt.

I started by following the instructions on the OpenWRT wiki for TL-MR3040, and downloaded the recommended firmware image (openwrt-ar71xx-generic-tl-mr3040-v2-squashfs-factory.bin) and uploaded it into the original web interface. The flashing went fine, and the machine was available via telnet on the ethernet port. After logging in and setting the root password, ssh was available and I could start to set it up as a batman-adv mesh node.

I started off by reading the instructions from Wireless Africa, which had quite a lot of useful information, but eventually I followed the recipe from the Open Mesh wiki for using batman-adv on OpenWrt. A small snag was the fact that the opkg install kmod-batman-adv command did not work as it should. The batman-adv kernel module would fail to load because its dependency crc16 was not already loaded. I reported the bug to the openwrt project and hope it will be fixed soon. But the problem only seem to affect initial testing of batman-adv, as configuration seem to work when booting from scratch.

The setup is done using files in /etc/config/. I did not bridge the Ethernet and mesh interfaces this time, to be able to hook up the box on my local network and log into it for configuration updates. The following files were changed and look like this after modifying them:

/etc/config/network

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fdbf:4c12:3fed::/48'

config interface 'lan'
        option ifname 'eth0'
        option type 'bridge'
        option proto 'dhcp'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option hostname 'tl-mr3040'
        option ip6assign '60'

config interface 'mesh'
        option ifname 'adhoc0'
        option mtu '1528'
        option proto 'batadv'
        option mesh 'bat0'

/etc/config/wireless

config wifi-device 'radio0'
        option type 'mac80211'
        option channel '11'
        option hwmode '11ng'
        option path 'platform/ar933x_wmac'
        option htmode 'HT20'
        list ht_capab 'SHORT-GI-20'
        list ht_capab 'SHORT-GI-40'
        list ht_capab 'RX-STBC1'
        list ht_capab 'DSSS_CCK-40'
        option disabled '0'

config wifi-iface 'wmesh'
        option device 'radio0'
        option ifname 'adhoc0'
        option network 'mesh'
        option encryption 'none'
        option mode 'adhoc'
        option bssid '02:BA:00:00:00:01'
        option ssid 'meshfx@hackeriet'

/etc/config/batman-adv

config 'mesh' 'bat0'
        option interfaces 'adhoc0'
        option 'aggregated_ogms'
        option 'ap_isolation'
        option 'bonding'
        option 'fragmentation'
        option 'gw_bandwidth'
        option 'gw_mode'
        option 'gw_sel_class'
        option 'log_level'
        option 'orig_interval'
        option 'vis_mode'
        option 'bridge_loop_avoidance'
        option 'distributed_arp_table'
        option 'network_coding'
        option 'hop_penalty'

# yet another batX instance
# config 'mesh' 'bat5'
#       option 'interfaces' 'second_mesh'

The mesh node is now operational. I have yet to test its range, but I hope it is good. I have not yet tested the TP-Link 3600 box still wrapped up in plastic.

If one of the points of switching to a new init system in Debian is to get rid of huge init.d scripts, I doubt we need to switch away from sysvinit and init.d scripts at all. Here is an example init.d script, ie a rewrite of /etc/init.d/rsyslog:

#!/lib/init/init-d-script
### BEGIN INIT INFO
# Provides:          rsyslog
# Required-Start:    $remote_fs $time
# Required-Stop:     umountnfs $time
# X-Stop-After:      sendsigs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: enhanced syslogd
# Description:       Rsyslog is an enhanced multi-threaded syslogd.
#                    It is quite compatible to stock sysklogd and can be 
#                    used as a drop-in replacement.
### END INIT INFO
DESC="enhanced syslogd"
DAEMON=/usr/sbin/rsyslogd

Pretty minimalistic to me... For the record, the original sysv-rc script was 137 lines, and the above is just 15 lines, most of it meta info/comments.

How to do this, you ask? Well, one create a new script /lib/init/init-d-script looking something like this:

#!/bin/sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.2-14) to ensure that this file is present
# and status_of_proc is working.
. /lib/lsb/init-functions

#
# Function that starts the daemon/service

#
do_start()
{
	# Return
	#   0 if daemon has been started
	#   1 if daemon was already running
	#   2 if daemon could not be started
	start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null \
		|| return 1
	start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- \
		$DAEMON_ARGS \
		|| return 2
	# Add code here, if necessary, that waits for the process to be ready
	# to handle requests from services started subsequently which depend
	# on this one.  As a last resort, sleep for some time.
}

#
# Function that stops the daemon/service
#
do_stop()
{
	# Return
	#   0 if daemon has been stopped
	#   1 if daemon was already stopped
	#   2 if daemon could not be stopped
	#   other if a failure occurred
	start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME
	RETVAL="$?"
	[ "$RETVAL" = 2 ] && return 2
	# Wait for children to finish too if this is a daemon that forks
	# and if the daemon is only ever run from this initscript.
	# If the above conditions are not satisfied then add some other code
	# that waits for the process to drop all resources that could be
	# needed by services started subsequently.  A last resort is to
	# sleep for some time.
	start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
	[ "$?" = 2 ] && return 2
	# Many daemons don't delete their pidfiles when they exit.
	rm -f $PIDFILE
	return "$RETVAL"
}

#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
	#
	# If the daemon can reload its configuration without
	# restarting (for example, when it is sent a SIGHUP),
	# then implement that here.
	#
	start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME
	return 0
}

SCRIPTNAME=$1
scriptbasename="$(basename $1)"
echo "SN: $scriptbasename"
if [ "$scriptbasename" != "init-d-library" ] ; then
    script="$1"
    shift
    . $script
else
    exit 0
fi

NAME=$(basename $DAEMON)
PIDFILE=/var/run/$NAME.pid

# Exit if the package is not installed
#[ -x "$DAEMON" ] || exit 0

# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

case "$1" in
  start)
	[ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
	do_start
	case "$?" in
		0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
		2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
	esac
	;;
  stop)
	[ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
	do_stop
	case "$?" in
		0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
		2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
	esac
	;;
  status)
	status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $?
	;;
  #reload|force-reload)
	#
	# If do_reload() is not implemented then leave this commented out
	# and leave 'force-reload' as an alias for 'restart'.
	#
	#log_daemon_msg "Reloading $DESC" "$NAME"
	#do_reload
	#log_end_msg $?
	#;;
  restart|force-reload)
	#
	# If the "reload" option is implemented then remove the
	# 'force-reload' alias
	#
	log_daemon_msg "Restarting $DESC" "$NAME"
	do_stop
	case "$?" in
	  0|1)
		do_start
		case "$?" in
			0) log_end_msg 0 ;;
			1) log_end_msg 1 ;; # Old process is still running
			*) log_end_msg 1 ;; # Failed to start
		esac
		;;
	  *)
		# Failed to stop
		log_end_msg 1
		;;
	esac
	;;
  *)
	echo "Usage: $SCRIPTNAME {start|stop|status|restart|force-reload}" >&2
	exit 3
	;;
esac

:

It is based on /etc/init.d/skeleton, and could be improved quite a lot. I did not really polish the approach, so it might not always work out of the box, but you get the idea. I did not try very hard to optimize it nor make it more robust either.

A better argument for switching init system in Debian than reducing the size of init scripts (which is a good thing to do anyway), is to get boot system that is able to handle the kernel events sensibly and robustly, and do not depend on the boot to run sequentially. The boot and the kernel have not behaved sequentially in years.

The SPICE protocol for remote display access is the preferred solution with oVirt and RedHat Enterprise Virtualization, and I was sad to discover the other day that the browser plugin needed to use these systems seamlessly was missing in Debian. The request for a package was from 2012-04-10 with no progress since 2013-04-01, so I decided to wrap up a package based on the great work from Cajus Pollmeier and put it in a collab-maint maintained git repository to get a package I could use. I would very much like others to help me maintain the package (or just take over, I do not mind), but as no-one had volunteered so far, I just uploaded it to NEW. I hope it will be available in Debian in a few days.

The source is now available from http://anonscm.debian.org/gitweb/?p=collab-maint/spice-xpi.git;a=summary.

The vmdebootstrap program is a a very nice system to create virtual machine images. It create a image file, add a partition table, mount it and run debootstrap in the mounted directory to create a Debian system on a stick. Yesterday, I decided to try to teach it how to make images for Raspberry Pi, as part of a plan to simplify the build system for the FreedomBox project. The FreedomBox project already uses vmdebootstrap for the virtualbox images, but its current build system made multistrap based system for Dreamplug images, and it is lacking support for Raspberry Pi.

Armed with the knowledge on how to build "foreign" (aka non-native architecture) chroots for Raspberry Pi, I dived into the vmdebootstrap code and adjusted it to be able to build armel images on my amd64 Debian laptop. I ended up giving vmdebootstrap five new options, allowing me to replicate the image creation process I use to make Debian Jessie based mesh node images for the Raspberry Pi. First, the --foreign /path/to/binfm_handler option tell vmdebootstrap to call debootstrap with --foreign and to copy the handler into the generated chroot before running the second stage. This allow vmdebootstrap to create armel images on an amd64 host. Next I added two new options --bootsize size and --boottype fstype to teach it to create a separate /boot/ partition with the given file system type, allowing me to create an image with a vfat partition for the /boot/ stuff. I also added a --variant variant option to allow me to create smaller images without the Debian base system packages installed. Finally, I added an option --no-extlinux to tell vmdebootstrap to not install extlinux as a boot loader. It is not needed on the Raspberry Pi and probably most other non-x86 architectures. The changes were accepted by the upstream author of vmdebootstrap yesterday and today, and is now available from the upstream project page.

To use it to build a Raspberry Pi image using Debian Jessie, first create a small script (the customize script) to add the non-free binary blob needed to boot the Raspberry Pi and the APT source list:

#!/bin/sh
set -e # Exit on first error
rootdir="$1"
cd "$rootdir"
cat <<EOF > etc/apt/sources.list
deb http://http.debian.net/debian/ jessie main contrib non-free
EOF
# Install non-free binary blob needed to boot Raspberry Pi.  This
# install a kernel somewhere too.
wget https://raw.github.com/Hexxeh/rpi-update/master/rpi-update \
    -O $rootdir/usr/bin/rpi-update
chmod a+x $rootdir/usr/bin/rpi-update
mkdir -p $rootdir/lib/modules
touch $rootdir/boot/start.elf
chroot $rootdir rpi-update

Next, fetch the latest vmdebootstrap script and call it like this to build the image:

sudo ./vmdebootstrap \
    --variant minbase \
    --arch armel \
    --distribution jessie \
    --mirror http://http.debian.net/debian \
    --image test.img \
    --size 600M \
    --bootsize 64M \
    --boottype vfat \
    --log-level debug \
    --verbose \
    --no-kernel \
    --no-extlinux \
    --root-password raspberry \
    --hostname raspberrypi \
    --foreign /usr/bin/qemu-arm-static \
    --customize `pwd`/customize \
    --package netbase \
    --package git-core \
    --package binutils \
    --package ca-certificates \
    --package wget \
    --package kmod

The list of packages being installed are the ones needed by rpi-update to make the image bootable on the Raspberry Pi, with the exception of netbase, which is needed by debootstrap to find /etc/hosts with the minbase variant. I really wish there was a way to set up an Raspberry Pi using only packages in the Debian archive, but that is not possible as far as I know, because it boots from the GPU using a non-free binary blob.

The build host need debootstrap, kpartx and qemu-user-static and probably a few others installed. I have not checked the complete build dependency list.

The resulting image will not use the hardware floating point unit on the Raspberry PI, because the armel architecture in Debian is not optimized for that use. So the images created will be a bit slower than Raspbian based images.

De siste måneders eksponering av den totale overvåkningen som foregår i den vestlige verden dokumenterer hvor sårbare vi er. Men det slår meg at de som er mest sårbare for dette, myndighetspersoner på alle nivåer, neppe har innsett at de selv er de mest interessante personene å lage profiler på, for å kunne påvirke dem.

For å ta et lite eksempel: Stortingets nettsted, www.stortinget.no (og forsåvidt også data.stortinget.no), inneholder informasjon om det som foregår på Stortinget, og jeg antar de største brukerne av informasjonen der er representanter og rådgivere på Stortinget. Intet overraskende med det. Det som derimot er mer skjult er at Stortingets nettsted bruker Google Analytics, hvilket gjør at enhver som besøker nettsidene der også rapporterer om besøket via Internett-linjer som passerer Sverige, England og videre til USA. Det betyr at informasjon om ethvert besøk på stortingets nettsider kan snappes opp av svensk, britisk og USAs etterretningsvesen. De kan dermed holde et øye med hvilke Stortingssaker stortingsrepresentantene synes er interessante å sjekke ut, og hvilke sider rådgivere og andre på stortinget synes er interessant å besøke, når de gjør det og hvilke andre representanter som sjekker de samme sidene omtrent samtidig. Stortingets bruk av Google Analytics gjør det dermed enkelt for utenlands etteretning å spore representantenes aktivitet og interesse. Hvis noen av representantene bruker Google Mail eller noen andre tjenestene som krever innlogging, så vil det være enda enklere å finne ut nøyaktig hvilke personer som bruker hvilke nettlesere og dermed knytte informasjonen opp til enkeltpersoner på Stortinget.

Og jo flere nettsteder som bruker Google Analytics, jo bedre oversikt over stortingsrepresentantenes lesevaner og interesse blir tilgjengelig for svensk, britisk og USAs etterretning. Hva de kan bruke den informasjonen til overlater jeg til leseren å undres over.

The last few days I have been experimenting with the batman-adv mesh technology. I want to gain some experience to see if it will fit the Freedombox project, and together with my neighbors try to build a mesh network around the park where I live. Batman-adv is a layer 2 mesh system ("ethernet" in other words), where the mesh network appear as if all the mesh clients are connected to the same switch.

My hardware of choice was the Linksys WRT54GL routers I had lying around, but I've been unable to get them working with batman-adv. So instead, I started playing with a Raspberry Pi, and tried to get it working as a mesh node. My idea is to use it to create a mesh node which function as a switch port, where everything connected to the Raspberry Pi ethernet plug is connected (bridged) to the mesh network. This allow me to hook a wifi base station like the Linksys WRT54GL to the mesh by plugging it into a Raspberry Pi, and allow non-mesh clients to hook up to the mesh. This in turn is useful for Android phones using the Serval Project voip client, allowing every one around the playground to phone and message each other for free. The reason is that Android phones do not see ad-hoc wifi networks (they are filtered away from the GUI view), and can not join the mesh without being rooted. But if they are connected using a normal wifi base station, they can talk to every client on the local network.

To get this working, I've created a debian package meshfx-node and a script build-rpi-mesh-node to create the Raspberry Pi boot image. I'm using Debian Jessie (and not Raspbian), to get more control over the packages available. Unfortunately a huge binary blob need to be inserted into the boot image to get it booting, but I'll ignore that for now. Also, as Debian lack support for the CPU features available in the Raspberry Pi, the system do not use the hardware floating point unit. I hope the routing performance isn't affected by the lack of hardware FPU support.

To create an image, run the following with a sudo enabled user after inserting the target SD card into the build machine:

% wget -O build-rpi-mesh-node \
    https://raw.github.com/petterreinholdtsen/meshfx-node/master/build-rpi-mesh-node
% sudo bash -x ./build-rpi-mesh-node > build.log 2>&1
% dd if=/root/rpi/rpi_basic_jessie_$(date +%Y%m%d).img of=/dev/mmcblk0 bs=1M
%

Booting with the resulting SD card on a Raspberry PI with a USB wifi card inserted should give you a mesh node. At least it does for me with a the wifi card I am using. The default mesh settings are the ones used by the Oslo mesh project at Hackeriet, as I mentioned in an earlier blog post about this mesh testing.

The mesh node was not horribly expensive either. I bought everything over the counter in shops nearby. If I had ordered online from the lowest bidder, the price should be significantly lower:

Now my mesh network at home consist of one laptop in the basement connected to my production network, one Raspberry Pi node on the 1th floor that can be seen by my neighbor across the park, and one play-node I use to develop the image building script. And some times I hook up my work horse laptop to the mesh to test it. I look forward to figuring out what kind of latency the batman-adv setup will give, and how much packet loss we will experience around the park. :)

Back in 2010, I created a Perl library to talk to the Spykee robot (with two belts, wifi, USB and Linux) and made it available from my web page. Today I concluded that it should move to a site that is easier to use to cooperate with others, and moved it to github. If you got a Spykee robot, you might want to check out the libspykee-perl github repository.

The last few days I came across a few good causes that should get wider attention. I recommend signing and donating to each one of these. :)

Via Debian Project News for 2013-10-14 I came across the Outreach Program for Women program which is a Google Summer of Code like initiative to get more women involved in free software. One debian sponsor has offered to match any donation done to Debian earmarked for this initiative. I donated a few minutes ago, and hope you will to. :)

And the Electronic Frontier Foundation just announced plans to create video documentaries about the excessive spying on every Internet user that take place these days, and their need to fund the work. I've already donated. Are you next?

For my Norwegian audience, the organisation Studentenes og Akademikernes Internasjonale Hjelpefond is collecting signatures for a statement under the heading Bloggers United for Open Access for those of us asking for more focus on open access in the Norwegian government. So far 499 signatures. I hope you will sign it too.

Wireless mesh networks are self organising and self healing networks that can be used to connect computers across small and large areas, depending on the radio technology used. Normal wifi equipment can be used to create home made radio networks, and there are several successful examples like Freifunk and Athens Wireless Metropolitan Network (see wikipedia for a large list) around the globe. To give you an idea how it work, check out the nice overview of the Kiel Freifunk community which can be seen from their dynamically updated node graph and map, where one can see how the mesh nodes automatically handle routing and recover from nodes disappearing. There is also a small community mesh network group in Oslo, Norway, and that is the main topic of this blog post.

I've wanted to check out mesh networks for a while now, and hoped to do it as part of my involvement with the NUUG member organisation community, and my recent involvement in the Freedombox project finally lead me to give mesh networks some priority, as I suspect a Freedombox should use mesh networks to connect neighbours and family when possible, given that most communication between people are between those nearby (as shown for example by research on Facebook communication patterns). It also allow people to communicate without any central hub to tap into for those that want to listen in on the private communication of citizens, which have become more and more important over the years.

So far I have only been able to find one group of people in Oslo working on community mesh networks, over at the hack space Hackeriet at Husmania. They seem to have started with some Freifunk based effort using OLSR, called the Oslo Freifunk project, but that effort is now dead and the people behind it have moved on to a batman-adv based system called meshfx. Unfortunately the wiki site for the Oslo Freifunk project is no longer possible to update to reflect this fact, so the old project page can't be updated to point to the new project. A while back, the people at Hackeriet invited people from the Freifunk community to Oslo to talk about mesh networks. I came across this video where Hans Jørgen Lysglimt interview the speakers about this talk (from youtube):

I mentioned OLSR and batman-adv, which are mesh routing protocols. There are heaps of different protocols, and I am still struggling to figure out which one would be "best" for some definitions of best, but given that the community mesh group in Oslo is so small, I believe it is best to hook up with the existing one instead of trying to create a completely different setup, and thus I have decided to focus on batman-adv for now. It sure help me to know that the very cool Serval project in Australia is using batman-adv as their meshing technology when it create a self organizing and self healing telephony system for disaster areas and less industrialized communities. Check out this cool video presenting that project (from youtube):

According to the wikipedia page on Wireless mesh network there are around 70 competing schemes for routing packets across mesh networks, and OLSR, B.A.T.M.A.N. and B.A.T.M.A.N. advanced are protocols used by several free software based community mesh networks.

The batman-adv protocol is a bit special, as it provide layer 2 (as in ethernet ) routing, allowing ipv4 and ipv6 to work on the same network. One way to think about it is that it provide a mesh based vlan you can bridge to or handle like any other vlan connected to your computer. The required drivers are already in the Linux kernel at least since Debian Wheezy, and it is fairly easy to set up. A good introduction is available from the Open Mesh project. These are the key settings needed to join the Oslo meshfx network:

The reason for setting ad-hoc wifi Cell ID is to work around bugs in firmware used in wifi card and wifi drivers. (See a nice post from VillageTelco about "Information about cell-id splitting, stuck beacons, and failed IBSS merges! for details.) When these settings are activated and you have some other mesh node nearby, your computer will be connected to the mesh network and can communicate with any mesh node that is connected to any of the nodes in your network of nodes. :)

My initial plan was to reuse my old Linksys WRT54GL as a mesh node, but that seem to be very hard, as I have not been able to locate a firmware supporting batman-adv. If anyone know how to use that old wifi access point with batman-adv these days, please let me know.

If you find this project interesting and want to join, please join us on IRC, either channel #oslohackerspace or #nuug on irc.freenode.net.

While investigating mesh networks in Oslo, I came across an old research paper from the university of Stavanger and Telenor Research and Innovation called The reliability of wireless backhaul mesh networks and elsewhere learned that Telenor have been experimenting with mesh networks at Grünerløkka in Oslo. So mesh networks are also interesting for commercial companies, even though Telenor discovered that it was hard to figure out a good business plan for mesh networking and as far as I know have closed down the experiment. Perhaps Telenor or others would be interested in a cooperation?

Update 2013-10-12: I was just told by the Serval project developers that they no longer use batman-adv (but are compatible with it), but their own crypto based mesh system.