X-Git-Url: http://pere.pagekite.me/gitweb/homepage.git/blobdiff_plain/673eb6dabf42b96e6d6e49f66b8b206ac28ff917..1e3f26d5d9e14795929f28bd8016e6fbb7ead365:/blog/index.rss diff --git a/blog/index.rss b/blog/index.rss index 3b3c3bc599..1ad8c05fe6 100644 --- a/blog/index.rss +++ b/blog/index.rss @@ -6,6 +6,82 @@ http://people.skolelinux.org/pere/blog/ + + A Debian package for SMTP via Tor (aka SMTorP) using exim4 + http://people.skolelinux.org/pere/blog/A_Debian_package_for_SMTP_via_Tor__aka_SMTorP__using_exim4.html + http://people.skolelinux.org/pere/blog/A_Debian_package_for_SMTP_via_Tor__aka_SMTorP__using_exim4.html + Mon, 10 Nov 2014 13:40:00 +0100 + <p>The right to communicate with your friends and family in private, +without anyone snooping, is a right every citicen have in a liberal +democracy. But this right is under serious attack these days.</p> + +<p>A while back it occurred to me that one way to make the dragnet +surveillance conducted by NSA, GCHQ, FRA and others (and confirmed by +the whisleblower Snowden) more expensive for Internet email, +is to deliver all email using SMTP via Tor. Such SMTP option would be +a nice addition to the FreedomBox project if we could send email +between FreedomBox machines without leaking metadata about the emails +to the people peeking on the wire. I +<a href="http://lists.alioth.debian.org/pipermail/freedombox-discuss/2014-October/006493.html">proposed +this on the FreedomBox project mailing list in October</a> and got a +lot of useful feedback and suggestions. It also became obvious to me +that this was not a novel idea, as the same idea was tested and +documented by Johannes Berg as early as 2006, and both +<a href="https://github.com/pagekite/Mailpile/wiki/SMTorP">the +Mailpile</a> and <a href="http://dee.su/cables">the Cables</a> systems +propose a similar method / protocol to pass emails between users.</p> + +<p>To implement such system one need to set up a Tor hidden service +providing the SMTP protocol on port 25, and use email addresses +looking like username@hidden-service-name.onion. With such addresses +the connections to port 25 on hidden-service-name.onion using Tor will +go to the correct SMTP server. To do this, one need to configure the +Tor daemon to provide the hidden service and the mail server to accept +emails for this .onion domain. To learn more about Exim configuration +in Debian and test the design provided by Johannes Berg in his FAQ, I +set out yesterday to create a Debian package for making it trivial to +set up such SMTP over Tor service based on Debian. Getting it to work +were fairly easy, and +<a href="https://github.com/petterreinholdtsen/exim4-smtorp">the +source code for the Debian package</a> is available from github. I +plan to move it into Debian if further testing prove this to be a +useful approach.</p> + +<p>If you want to test this, set up a blank Debian machine without any +mail system installed (or run <tt>apt-get purge exim4-config</tt> to +get rid of exim4). Install tor, clone the git repository mentioned +above, build the deb and install it on the machine. Next, run +<tt>/usr/lib/exim4-smtorp/setup-exim-hidden-service</tt> and follow +the instructions to get the service up and running. Restart tor and +exim when it is done, and test mail delivery using swaks like +this:</p> + +<p><blockquote><pre> +torsocks swaks --server dutlqrrmjhtfa3vp.onion \ + --to fbx@dutlqrrmjhtfa3vp.onion +</pre></blockquote></p> + +<p>This will test the SMTP delivery using tor. Replace the email +address with your own address to test your server. :)</p> + +<p>The setup procedure is still to complex, and I hope it can be made +easier and more automatic. Especially the tor setup need more work. +Also, the package include a tor-smtp tool written in C, but its task +should probably be rewritten in some script language to make the deb +architecture independent. It would probably also make the code easier +to review. The tor-smtp tool currently need to listen on a socket for +exim to talk to it and is started using xinetd. It would be better if +no daemon and no socket is needed. I suspect it is possible to get +exim to run a command line tool for delivery instead of talking to a +socket, and hope to figure out how in a future version of this +system.</p> + +<p>Until I wipe my test machine, I can be reached using the +<tt>fbx@dutlqrrmjhtfa3vp.onion</tt> mail address, deliverable over +SMTorP. :)</p> + + + First Jessie based Debian Edu released (alpha0) http://people.skolelinux.org/pere/blog/First_Jessie_based_Debian_Edu_released__alpha0_.html @@ -844,221 +920,5 @@ dokumentasjon på vegvesenets REST-API.</p> - - Speeding up the Debian installer using eatmydata and dpkg-divert - http://people.skolelinux.org/pere/blog/Speeding_up_the_Debian_installer_using_eatmydata_and_dpkg_divert.html - http://people.skolelinux.org/pere/blog/Speeding_up_the_Debian_installer_using_eatmydata_and_dpkg_divert.html - Tue, 16 Sep 2014 14:00:00 +0200 - <p>The <a href="https://www.debian.org/">Debian</a> installer could be -a lot quicker. When we install more than 2000 packages in -<a href="http://www.skolelinux.org/">Skolelinux / Debian Edu</a> using -tasksel in the installer, unpacking the binary packages take forever. -A part of the slow I/O issue was discussed in -<a href="https://bugs.debian.org/613428">bug #613428</a> about too -much file system sync-ing done by dpkg, which is the package -responsible for unpacking the binary packages. Other parts (like code -executed by postinst scripts) might also sync to disk during -installation. All this sync-ing to disk do not really make sense to -me. If the machine crash half-way through, I start over, I do not try -to salvage the half installed system. So the failure sync-ing is -supposed to protect against, hardware or system crash, is not really -relevant while the installer is running.</p> - -<p>A few days ago, I thought of a way to get rid of all the file -system sync()-ing in a fairly non-intrusive way, without the need to -change the code in several packages. The idea is not new, but I have -not heard anyone propose the approach using dpkg-divert before. It -depend on the small and clever package -<a href="https://packages.qa.debian.org/eatmydata">eatmydata</a>, which -uses LD_PRELOAD to replace the system functions for syncing data to -disk with functions doing nothing, thus allowing programs to live -dangerous while speeding up disk I/O significantly. Instead of -modifying the implementation of dpkg, apt and tasksel (which are the -packages responsible for selecting, fetching and installing packages), -it occurred to me that we could just divert the programs away, replace -them with a simple shell wrapper calling -"eatmydata&nbsp;$program&nbsp;$@", to get the same effect. -Two days ago I decided to test the idea, and wrapped up a simple -implementation for the Debian Edu udeb.</p> - -<p>The effect was stunning. In my first test it reduced the running -time of the pkgsel step (installing tasks) from 64 to less than 44 -minutes (20 minutes shaved off the installation) on an old Dell -Latitude D505 machine. I am not quite sure what the optimised time -would have been, as I messed up the testing a bit, causing the debconf -priority to get low enough for two questions to pop up during -installation. As soon as I saw the questions I moved the installation -along, but do not know how long the question were holding up the -installation. I did some more measurements using Debian Edu Jessie, -and got these results. The time measured is the time stamp in -/var/log/syslog between the "pkgsel: starting tasksel" and the -"pkgsel: finishing up" lines, if you want to do the same measurement -yourself. In Debian Edu, the tasksel dialog do not show up, and the -timing thus do not depend on how quickly the user handle the tasksel -dialog.</p> - -<p><table> - -<tr> -<th>Machine/setup</th> -<th>Original tasksel</th> -<th>Optimised tasksel</th> -<th>Reduction</th> -</tr> - -<tr> -<td>Latitude D505 Main+LTSP LXDE</td> -<td>64 min (07:46-08:50)</td> -<td><44 min (11:27-12:11)</td> -<td>>20 min 18%</td> -</tr> - -<tr> -<td>Latitude D505 Roaming LXDE</td> -<td>57 min (08:48-09:45)</td> -<td>34 min (07:43-08:17)</td> -<td>23 min 40%</td> -</tr> - -<tr> -<td>Latitude D505 Minimal</td> -<td>22 min (10:37-10:59)</td> -<td>11 min (11:16-11:27)</td> -<td>11 min 50%</td> -</tr> - -<tr> -<td>Thinkpad X200 Minimal</td> -<td>6 min (08:19-08:25)</td> -<td>4 min (08:04-08:08)</td> -<td>2 min 33%</td> -</tr> - -<tr> -<td>Thinkpad X200 Roaming KDE</td> -<td>19 min (09:21-09:40)</td> -<td>15 min (10:25-10:40)</td> -<td>4 min 21%</td> -</tr> - -</table></p> - -<p>The test is done using a netinst ISO on a USB stick, so some of the -time is spent downloading packages. The connection to the Internet -was 100Mbit/s during testing, so downloading should not be a -significant factor in the measurement. Download typically took a few -seconds to a few minutes, depending on the amount of packages being -installed.</p> - -<p>The speedup is implemented by using two hooks in -<a href="https://www.debian.org/devel/debian-installer/">Debian -Installer</a>, the pre-pkgsel.d hook to set up the diverts, and the -finish-install.d hook to remove the divert at the end of the -installation. I picked the pre-pkgsel.d hook instead of the -post-base-installer.d hook because I test using an ISO without the -eatmydata package included, and the post-base-installer.d hook in -Debian Edu can only operate on packages included in the ISO. The -negative effect of this is that I am unable to activate this -optimization for the kernel installation step in d-i. If the code is -moved to the post-base-installer.d hook, the speedup would be larger -for the entire installation.</p> - -<p>I've implemented this in the -<a href="https://packages.qa.debian.org/debian-edu-install">debian-edu-install</a> -git repository, and plan to provide the optimization as part of the -Debian Edu installation. If you want to test this yourself, you can -create two files in the installer (or in an udeb). One shell script -need do go into /usr/lib/pre-pkgsel.d/, with content like this:</p> - -<p><blockquote><pre> -#!/bin/sh -set -e -. /usr/share/debconf/confmodule -info() { - logger -t my-pkgsel "info: $*" -} -error() { - logger -t my-pkgsel "error: $*" -} -override_install() { - apt-install eatmydata || true - if [ -x /target/usr/bin/eatmydata ] ; then - for bin in dpkg apt-get aptitude tasksel ; do - file=/usr/bin/$bin - # Test that the file exist and have not been diverted already. - if [ -f /target$file ] ; then - info "diverting $file using eatmydata" - printf "#!/bin/sh\neatmydata $bin.distrib \"\$@\"\n" \ - > /target$file.edu - chmod 755 /target$file.edu - in-target dpkg-divert --package debian-edu-config \ - --rename --quiet --add $file - ln -sf ./$bin.edu /target$file - else - error "unable to divert $file, as it is missing." - fi - done - else - error "unable to find /usr/bin/eatmydata after installing the eatmydata pacage" - fi -} - -override_install -</pre></blockquote></p> - -<p>To clean up, another shell script should go into -/usr/lib/finish-install.d/ with code like this: - -<p><blockquote><pre> -#! /bin/sh -e -. /usr/share/debconf/confmodule -error() { - logger -t my-finish-install "error: $@" -} -remove_install_override() { - for bin in dpkg apt-get aptitude tasksel ; do - file=/usr/bin/$bin - if [ -x /target$file.edu ] ; then - rm /target$file - in-target dpkg-divert --package debian-edu-config \ - --rename --quiet --remove $file - rm /target$file.edu - else - error "Missing divert for $file." - fi - done - sync # Flush file buffers before continuing -} - -remove_install_override -</pre></blockquote></p> - -<p>In Debian Edu, I placed both code fragments in a separate script -edu-eatmydata-install and call it from the pre-pkgsel.d and -finish-install.d scripts.</p> - -<p>By now you might ask if this change should get into the normal -Debian installer too? I suspect it should, but am not sure the -current debian-installer coordinators find it useful enough. It also -depend on the side effects of the change. I'm not aware of any, but I -guess we will see if the change is safe after some more testing. -Perhaps there is some package in Debian depending on sync() and -fsync() having effect? Perhaps it should go into its own udeb, to -allow those of us wanting to enable it to do so without affecting -everyone.</p> - -<p>Update 2014-09-24: Since a few days ago, enabling this optimization -will break installation of all programs using gnutls because of -<a href="https://bugs.debian.org/702711">bug #702711</a>. An updated -eatmydata package in Debian will solve it.</p> - -<p>Update 2014-10-17: The bug mentioned above is fixed in testing and -the optimization work again. And I have discovered that the -dpkg-divert trick is not really needed and implemented a slightly -simpler approach as part of the debian-edu-install package. See -tools/edu-eatmydata-install in the source package.</p> - - -