X-Git-Url: https://pere.pagekite.me/gitweb/homepage.git/blobdiff_plain/b0d2b6f11893ad06e848fd2f198ad85d9f2eb928..b279793fc3681f16b43cc7dba38f0d6f4e22d247:/blog/tags/english/index.html diff --git a/blog/tags/english/index.html b/blog/tags/english/index.html index 79e4ea7eec..8e6c64df82 100644 --- a/blog/tags/english/index.html +++ b/blog/tags/english/index.html @@ -20,6 +20,2046 @@

Entries tagged "english".

+
+
+ Release 0.1.1 of free software archive system Nikita announced +
+
+ 10th June 2017 +
+
+

I am very happy to report that the +Nikita Noark 5 +core project tagged its second release today. The free software +solution is an implementation of the Norwegian archive standard Noark +5 used by government offices in Norway. These were the changes in +version 0.1.1 since version 0.1.0 (from NEWS.md): + +

+ +

If this sound interesting to you, please contact us on IRC (#nikita +on irc.freenode.net) or email +(nikita-noark +mailing list).

+ +
+
+ + + Tags: english, nuug, offentlig innsyn, standard. + + +
+
+
+ +
+
+ Idea for storing trusted timestamps in a Noark 5 archive +
+
+ 7th June 2017 +
+
+

This is a copy of +an +email I posted to the nikita-noark mailing list. Please follow up +there if you would like to discuss this topic. The background is that +we are making a free software archive system based on the Norwegian +Noark +5 standard for government archives.

+ +

I've been wondering a bit lately how trusted timestamps could be +stored in Noark 5. +Trusted +timestamps can be used to verify that some information +(document/file/checksum/metadata) have not been changed since a +specific time in the past. This is useful to verify the integrity of +the documents in the archive.

+ +

Then it occured to me, perhaps the trusted timestamps could be +stored as dokument variants (ie dokumentobjekt referered to from +dokumentbeskrivelse) with the filename set to the hash it is +stamping?

+ +

Given a "dokumentbeskrivelse" with an associated "dokumentobjekt", +a new dokumentobjekt is associated with "dokumentbeskrivelse" with the +same attributes as the stamped dokumentobjekt except these +attributes:

+ + + +

This assume a service following +IETF RFC 3161 is +used, which specifiy the given MIME type for replies and the .tsr file +ending for the content of such trusted timestamp. As far as I can +tell from the Noark 5 specifications, it is OK to have several +variants/renderings of a dokument attached to a given +dokumentbeskrivelse objekt. It might be stretching it a bit to make +some of these variants represent crypto-signatures useful for +verifying the document integrity instead of representing the dokument +itself.

+ +

Using the source of the service in formatDetaljer allow several +timestamping services to be used. This is useful to spread the risk +of key compromise over several organisations. It would only be a +problem to trust the timestamps if all of the organisations are +compromised.

+ +

The following oneliner on Linux can be used to generate the tsr +file. $input is the path to the file to checksum, and $sha256 is the +SHA-256 checksum of the file (ie the ".tsr" value mentioned +above).

+ +

+openssl ts -query -data "$inputfile" -cert -sha256 -no_nonce \
+  | curl -s -H "Content-Type: application/timestamp-query" \
+      --data-binary "@-" http://zeitstempel.dfn.de > $sha256.tsr
+

+ +

To verify the timestamp, you first need to download the public key +of the trusted timestamp service, for example using this command:

+ +

+wget -O ca-cert.txt \
+  https://pki.pca.dfn.de/global-services-ca/pub/cacert/chain.txt
+

+ +

Note, the public key should be stored alongside the timestamps in +the archive to make sure it is also available 100 years from now. It +is probably a good idea to standardise how and were to store such +public keys, to make it easier to find for those trying to verify +documents 100 or 1000 years from now. :)

+ +

The verification itself is a simple openssl command:

+ +

+openssl ts -verify -data $inputfile -in $sha256.tsr \
+  -CAfile ca-cert.txt -text
+

+ +

Is there any reason this approach would not work? Is it somehow against +the Noark 5 specification?

+ +
+
+ + + Tags: english, offentlig innsyn, standard. + + +
+
+
+ +
+
+ Free software archive system Nikita now able to store documents +
+
+ 19th March 2017 +
+
+

The Nikita +Noark 5 core project is implementing the Norwegian standard for +keeping an electronic archive of government documents. +The +Noark 5 standard document the requirement for data systems used by +the archives in the Norwegian government, and the Noark 5 web interface +specification document a REST web service for storing, searching and +retrieving documents and metadata in such archive. I've been involved +in the project since a few weeks before Christmas, when the Norwegian +Unix User Group +announced +it supported the project. I believe this is an important project, +and hope it can make it possible for the government archives in the +future to use free software to keep the archives we citizens depend +on. But as I do not hold such archive myself, personally my first use +case is to store and analyse public mail journal metadata published +from the government. I find it useful to have a clear use case in +mind when developing, to make sure the system scratches one of my +itches.

+ +

If you would like to help make sure there is a free software +alternatives for the archives, please join our IRC channel +(#nikita on +irc.freenode.net) and +the +project mailing list.

+ +

When I got involved, the web service could store metadata about +documents. But a few weeks ago, a new milestone was reached when it +became possible to store full text documents too. Yesterday, I +completed an implementation of a command line tool +archive-pdf to upload a PDF file to the archive using this +API. The tool is very simple at the moment, and find existing +fonds, series and +files while asking the user to select which one to use if more than +one exist. Once a file is identified, the PDF is associated with the +file and uploaded, using the title extracted from the PDF itself. The +process is fairly similar to visiting the archive, opening a cabinet, +locating a file and storing a piece of paper in the archive. Here is +a test run directly after populating the database with test data using +our API tester:

+ +

+~/src//noark5-tester$ ./archive-pdf mangelmelding/mangler.pdf
+using arkiv: Title of the test fonds created 2017-03-18T23:49:32.103446
+using arkivdel: Title of the test series created 2017-03-18T23:49:32.103446
+
+ 0 - Title of the test case file created 2017-03-18T23:49:32.103446
+ 1 - Title of the test file created 2017-03-18T23:49:32.103446
+Select which mappe you want (or search term): 0
+Uploading mangelmelding/mangler.pdf
+  PDF title: Mangler i spesifikasjonsdokumentet for NOARK 5 Tjenestegrensesnitt
+  File 2017/1: Title of the test case file created 2017-03-18T23:49:32.103446
+~/src//noark5-tester$
+

+ +

You can see here how the fonds (arkiv) and serie (arkivdel) only had +one option, while the user need to choose which file (mappe) to use +among the two created by the API tester. The archive-pdf +tool can be found in the git repository for the API tester.

+ +

In the project, I have been mostly working on +the API +tester so far, while getting to know the code base. The API +tester currently use +the HATEOAS links +to traverse the entire exposed service API and verify that the exposed +operations and objects match the specification, as well as trying to +create objects holding metadata and uploading a simple XML file to +store. The tester has proved very useful for finding flaws in our +implementation, as well as flaws in the reference site and the +specification.

+ +

The test document I uploaded is a summary of all the specification +defects we have collected so far while implementing the web service. +There are several unclear and conflicting parts of the specification, +and we have +started +writing down the questions we get from implementing it. We use a +format inspired by how The +Austin Group collect defect reports for the POSIX standard with +their +instructions for the MANTIS defect tracker system, in lack of an official way to structure defect reports for Noark 5 (our first submitted defect report was a request for a procedure for submitting defect reports :). + +

The Nikita project is implemented using Java and Spring, and is +fairly easy to get up and running using Docker containers for those +that want to test the current code base. The API tester is +implemented in Python.

+ +
+
+ + + Tags: english, nuug, offentlig innsyn, standard. + + +
+
+
+ +
+
+ Detecting NFS hangs on Linux without hanging yourself... +
+
+ 9th March 2017 +
+
+

Over the years, administrating thousand of NFS mounting linux +computers at the time, I often needed a way to detect if the machine +was experiencing NFS hang. If you try to use df or look at a +file or directory affected by the hang, the process (and possibly the +shell) will hang too. So you want to be able to detect this without +risking the detection process getting stuck too. It has not been +obvious how to do this. When the hang has lasted a while, it is +possible to find messages like these in dmesg:

+ +

+nfs: server nfsserver not responding, still trying +
nfs: server nfsserver OK +

+ +

It is hard to know if the hang is still going on, and it is hard to +be sure looking in dmesg is going to work. If there are lots of other +messages in dmesg the lines might have rotated out of site before they +are noticed.

+ +

While reading through the nfs client implementation in linux kernel +code, I came across some statistics that seem to give a way to detect +it. The om_timeouts sunrpc value in the kernel will increase every +time the above log entry is inserted into dmesg. And after digging a +bit further, I discovered that this value show up in +/proc/self/mountstats on Linux.

+ +

The mountstats content seem to be shared between files using the +same file system context, so it is enough to check one of the +mountstats files to get the state of the mount point for the machine. +I assume this will not show lazy umounted NFS points, nor NFS mount +points in a different process context (ie with a different filesystem +view), but that does not worry me.

+ +

The content for a NFS mount point look similar to this:

+ +

+[...]
+device /dev/mapper/Debian-var mounted on /var with fstype ext3
+device nfsserver:/mnt/nfsserver/home0 mounted on /mnt/nfsserver/home0 with fstype nfs statvers=1.1
+        opts:   rw,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,soft,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=129.240.3.145,mountvers=3,mountport=4048,mountproto=udp,local_lock=all
+        age:    7863311
+        caps:   caps=0x3fe7,wtmult=4096,dtsize=8192,bsize=0,namlen=255
+        sec:    flavor=1,pseudoflavor=1
+        events: 61063112 732346265 1028140 35486205 16220064 8162542 761447191 71714012 37189 3891185 45561809 110486139 4850138 420353 15449177 296502 52736725 13523379 0 52182 9016896 1231 0 0 0 0 0 
+        bytes:  166253035039 219519120027 0 0 40783504807 185466229638 11677877 45561809 
+        RPC iostats version: 1.0  p/v: 100003/3 (nfs)
+        xprt:   tcp 925 1 6810 0 0 111505412 111480497 109 2672418560317 0 248 53869103 22481820
+        per-op statistics
+                NULL: 0 0 0 0 0 0 0 0
+             GETATTR: 61063106 61063108 0 9621383060 6839064400 453650 77291321 78926132
+             SETATTR: 463469 463470 0 92005440 66739536 63787 603235 687943
+              LOOKUP: 17021657 17021657 0 3354097764 4013442928 57216 35125459 35566511
+              ACCESS: 14281703 14290009 5 2318400592 1713803640 1709282 4865144 7130140
+            READLINK: 125 125 0 20472 18620 0 1112 1118
+                READ: 4214236 4214237 0 715608524 41328653212 89884 22622768 22806693
+               WRITE: 8479010 8494376 22 187695798568 1356087148 178264904 51506907 231671771
+              CREATE: 171708 171708 0 38084748 46702272 873 1041833 1050398
+               MKDIR: 3680 3680 0 773980 993920 26 23990 24245
+             SYMLINK: 903 903 0 233428 245488 6 5865 5917
+               MKNOD: 80 80 0 20148 21760 0 299 304
+              REMOVE: 429921 429921 0 79796004 61908192 3313 2710416 2741636
+               RMDIR: 3367 3367 0 645112 484848 22 5782 6002
+              RENAME: 466201 466201 0 130026184 121212260 7075 5935207 5961288
+                LINK: 289155 289155 0 72775556 67083960 2199 2565060 2585579
+             READDIR: 2933237 2933237 0 516506204 13973833412 10385 3190199 3297917
+         READDIRPLUS: 1652839 1652839 0 298640972 6895997744 84735 14307895 14448937
+              FSSTAT: 6144 6144 0 1010516 1032192 51 9654 10022
+              FSINFO: 2 2 0 232 328 0 1 1
+            PATHCONF: 1 1 0 116 140 0 0 0
+              COMMIT: 0 0 0 0 0 0 0 0
+
+device binfmt_misc mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc
+[...]
+

+ +

The key number to look at is the third number in the per-op list. +It is the number of NFS timeouts experiences per file system +operation. Here 22 write timeouts and 5 access timeouts. If these +numbers are increasing, I believe the machine is experiencing NFS +hang. Unfortunately the timeout value do not start to increase right +away. The NFS operations need to time out first, and this can take a +while. The exact timeout value depend on the setup. For example the +defaults for TCP and UDP mount points are quite different, and the +timeout value is affected by the soft, hard, timeo and retrans NFS +mount options.

+ +

The only way I have been able to get working on Debian and RedHat +Enterprise Linux for getting the timeout count is to peek in /proc/. +But according to +Solaris +10 System Administration Guide: Network Services, the 'nfsstat -c' +command can be used to get these timeout values. But this do not work +on Linux, as far as I can tell. I +asked Debian about this, +but have not seen any replies yet.

+ +

Is there a better way to figure out if a Linux NFS client is +experiencing NFS hangs? Is there a way to detect which processes are +affected? Is there a way to get the NFS mount going quickly once the +network problem causing the NFS hang has been cleared? I would very +much welcome some clues, as we regularly run into NFS hangs.

+ +
+
+ + + Tags: debian, english, sysadmin. + + +
+
+
+ +
+
+ How does it feel to be wiretapped, when you should be doing the wiretapping... +
+
+ 8th March 2017 +
+
+

So the new president in the United States of America claim to be +surprised to discover that he was wiretapped during the election +before he was elected president. He even claim this must be illegal. +Well, doh, if it is one thing the confirmations from Snowden +documented, it is that the entire population in USA is wiretapped, one +way or another. Of course the president candidates were wiretapped, +alongside the senators, judges and the rest of the people in USA.

+ +

Next, the Federal Bureau of Investigation ask the Department of +Justice to go public rejecting the claims that Donald Trump was +wiretapped illegally. I fail to see the relevance, given that I am +sure the surveillance industry in USA believe they have all the legal +backing they need to conduct mass surveillance on the entire +world.

+ +

There is even the director of the FBI stating that he never saw an +order requesting wiretapping of Donald Trump. That is not very +surprising, given how the FISA court work, with all its activity being +secret. Perhaps he only heard about it?

+ +

What I find most sad in this story is how Norwegian journalists +present it. In a news reports the other day in the radio from the +Norwegian National broadcasting Company (NRK), I heard the journalist +claim that 'the FBI denies any wiretapping', while the reality is that +'the FBI denies any illegal wiretapping'. There is a fundamental and +important difference, and it make me sad that the journalists are +unable to grasp it.

+ +

Update 2017-03-13: Look like +The +Intercept report that US Senator Rand Paul confirm what I state above.

+ +
+
+ + + Tags: english, surveillance. + + +
+
+
+ +
+
+ Norwegian Bokmål translation of The Debian Administrator's Handbook complete, proofreading in progress +
+
+ 3rd March 2017 +
+
+

For almost a year now, we have been working on making a Norwegian +Bokmål edition of The Debian +Administrator's Handbook. Now, thanks to the tireless effort of +Ole-Erik, Ingrid and Andreas, the initial translation is complete, and +we are working on the proof reading to ensure consistent language and +use of correct computer science terms. The plan is to make the book +available on paper, as well as in electronic form. For that to +happen, the proof reading must be completed and all the figures need +to be translated. If you want to help out, get in touch.

+ +

A + +fresh PDF edition in A4 format (the final book will have smaller +pages) of the book created every morning is available for +proofreading. If you find any errors, please +visit +Weblate and correct the error. The +state +of the translation including figures is a useful source for those +provide Norwegian bokmål screen shots and figures.

+ +
+
+ + + Tags: debian, debian-handbook, english. + + +
+
+
+ +
+
+ Unlimited randomness with the ChaosKey? +
+
+ 1st March 2017 +
+
+

A few days ago I ordered a small batch of +the ChaosKey, a small +USB dongle for generating entropy created by Bdale Garbee and Keith +Packard. Yesterday it arrived, and I am very happy to report that it +work great! According to its designers, to get it to work out of the +box, you need the Linux kernel version 4.1 or later. I tested on a +Debian Stretch machine (kernel version 4.9), and there it worked just +fine, increasing the available entropy very quickly. I wrote a small +test oneliner to test. It first print the current entropy level, +drain /dev/random, and then print the entropy level for five seconds. +Here is the situation without the ChaosKey inserted:

+ +
+% cat /proc/sys/kernel/random/entropy_avail; \
+  dd bs=1M if=/dev/random of=/dev/null count=1; \
+  for n in $(seq 1 5); do \
+     cat /proc/sys/kernel/random/entropy_avail; \
+     sleep 1; \
+  done
+300
+0+1 oppføringer inn
+0+1 oppføringer ut
+28 byte kopiert, 0,000264565 s, 106 kB/s
+4
+8
+12
+17
+21
+%
+
+ +

The entropy level increases by 3-4 every second. In such case any +application requiring random bits (like a HTTPS enabled web server) +will halt and wait for more entrpy. And here is the situation with +the ChaosKey inserted:

+ +
+% cat /proc/sys/kernel/random/entropy_avail; \
+  dd bs=1M if=/dev/random of=/dev/null count=1; \
+  for n in $(seq 1 5); do \
+     cat /proc/sys/kernel/random/entropy_avail; \
+     sleep 1; \
+  done
+1079
+0+1 oppføringer inn
+0+1 oppføringer ut
+104 byte kopiert, 0,000487647 s, 213 kB/s
+433
+1028
+1031
+1035
+1038
+%
+
+ +

Quite the difference. :) I bought a few more than I need, in case +someone want to buy one here in Norway. :)

+ +

Update: The dongle was presented at Debconf last year. You might +find the talk +recording illuminating. It explains exactly what the source of +randomness is, if you are unable to spot it from the schema drawing +available from the ChaosKey web site linked at the start of this blog +post.

+ +
+
+ + + Tags: debian, english. + + +
+
+
+ +
+
+ Detect OOXML files with undefined behaviour? +
+
+ 21st February 2017 +
+
+

I just noticed +the +new Norwegian proposal for archiving rules in the goverment list +ECMA-376 +/ ISO/IEC 29500 (aka OOXML) as valid formats to put in long term +storage. Luckily such files will only be accepted based on +pre-approval from the National Archive. Allowing OOXML files to be +used for long term storage might seem like a good idea as long as we +forget that there are plenty of ways for a "valid" OOXML document to +have content with no defined interpretation in the standard, which +lead to a question and an idea.

+ +

Is there any tool to detect if a OOXML document depend on such +undefined behaviour? It would be useful for the National Archive (and +anyone else interested in verifying that a document is well defined) +to have such tool available when considering to approve the use of +OOXML. I'm aware of the +officeotron OOXML +validator, but do not know how complete it is nor if it will +report use of undefined behaviour. Are there other similar tools +available? Please send me an email if you know of any such tool.

+ +
+
+ + + Tags: english, nuug, standard. + + +
+
+
+ +
+
+ Ruling ignored our objections to the seizure of popcorn-time.no (#domstolkontroll) +
+
+ 13th February 2017 +
+
+

A few days ago, we received the ruling from +my +day in court. The case in question is a challenge of the seizure +of the DNS domain popcorn-time.no. The ruling simply did not mention +most of our arguments, and seemed to take everything ØKOKRIM said at +face value, ignoring our demonstration and explanations. But it is +hard to tell for sure, as we still have not seen most of the documents +in the case and thus were unprepared and unable to contradict several +of the claims made in court by the opposition. We are considering an +appeal, but it is partly a question of funding, as it is costing us +quite a bit to pay for our lawyer. If you want to help, please +donate to the +NUUG defense fund.

+ +

The details of the case, as far as we know it, is available in +Norwegian from +the NUUG +blog. This also include +the +ruling itself.

+ +
+
+ + + Tags: english, nuug, offentlig innsyn, opphavsrett. + + +
+
+
+ +
+
+ A day in court challenging seizure of popcorn-time.no for #domstolkontroll +
+
+ 3rd February 2017 +
+
+

+ +

On Wednesday, I spent the entire day in court in Follo Tingrett +representing the member association +NUUG, alongside the member +association EFN and the DNS registrar +IMC, challenging the seizure of the DNS name popcorn-time.no. It +was interesting to sit in a court of law for the first time in my +life. Our team can be seen in the picture above: attorney Ola +Tellesbø, EFN board member Tom Fredrik Blenning, IMC CEO Morten Emil +Eriksen and NUUG board member Petter Reinholdtsen.

+ +

The +case at hand is that the Norwegian National Authority for +Investigation and Prosecution of Economic and Environmental Crime (aka +Økokrim) decided on their own, to seize a DNS domain early last +year, without following +the +official policy of the Norwegian DNS authority which require a +court decision. The web site in question was a site covering Popcorn +Time. And Popcorn Time is the name of a technology with both legal +and illegal applications. Popcorn Time is a client combining +searching a Bittorrent directory available on the Internet with +downloading/distribute content via Bittorrent and playing the +downloaded content on screen. It can be used illegally if it is used +to distribute content against the will of the right holder, but it can +also be used legally to play a lot of content, for example the +millions of movies +available from the +Internet Archive or the collection +available from Vodo. We created +a +video demonstrating legally use of Popcorn Time and played it in +Court. It can of course be downloaded using Bittorrent.

+ +

I did not quite know what to expect from a day in court. The +government held on to their version of the story and we held on to +ours, and I hope the judge is able to make sense of it all. We will +know in two weeks time. Unfortunately I do not have high hopes, as +the Government have the upper hand here with more knowledge about the +case, better training in handling criminal law and in general higher +standing in the courts than fairly unknown DNS registrar and member +associations. It is expensive to be right also in Norway. So far the +case have cost more than NOK 70 000,-. To help fund the case, NUUG +and EFN have asked for donations, and managed to collect around NOK 25 +000,- so far. Given the presentation from the Government, I expect +the government to appeal if the case go our way. And if the case do +not go our way, I hope we have enough funding to appeal.

+ +

From the other side came two people from Økokrim. On the benches, +appearing to be part of the group from the government were two people +from the Simonsen Vogt Wiik lawyer office, and three others I am not +quite sure who was. Økokrim had proposed to present two witnesses +from The Motion Picture Association, but this was rejected because +they did not speak Norwegian and it was a bit late to bring in a +translator, but perhaps the two from MPA were present anyway. All +seven appeared to know each other. Good to see the case is take +seriously.

+ +

If you, like me, believe the courts should be involved before a DNS +domain is hijacked by the government, or you believe the Popcorn Time +technology have a lot of useful and legal applications, I suggest you +too donate to +the NUUG defense fund. Both Bitcoin and bank transfer are +available. If NUUG get more than we need for the legal action (very +unlikely), the rest will be spend promoting free software, open +standards and unix-like operating systems in Norway, so no matter what +happens the money will be put to good use.

+ +

If you want to lean more about the case, I recommend you check out +the blog +posts from NUUG covering the case. They cover the legal arguments +on both sides.

+ +
+
+ + + Tags: english, nuug, offentlig innsyn, opphavsrett. + + +
+
+
+ +
+
+ Where did that package go? — geolocated IP traceroute +
+
+ 9th January 2017 +
+
+

Did you ever wonder where the web trafic really flow to reach the +web servers, and who own the network equipment it is flowing through? +It is possible to get a glimpse of this from using traceroute, but it +is hard to find all the details. Many years ago, I wrote a system to +map the Norwegian Internet (trying to figure out if our plans for a +network game service would get low enough latency, and who we needed +to talk to about setting up game servers close to the users. Back +then I used traceroute output from many locations (I asked my friends +to run a script and send me their traceroute output) to create the +graph and the map. The output from traceroute typically look like +this: + +

+traceroute to www.stortinget.no (85.88.67.10), 30 hops max, 60 byte packets
+ 1  uio-gw10.uio.no (129.240.202.1)  0.447 ms  0.486 ms  0.621 ms
+ 2  uio-gw8.uio.no (129.240.24.229)  0.467 ms  0.578 ms  0.675 ms
+ 3  oslo-gw1.uninett.no (128.39.65.17)  0.385 ms  0.373 ms  0.358 ms
+ 4  te3-1-2.br1.fn3.as2116.net (193.156.90.3)  1.174 ms  1.172 ms  1.153 ms
+ 5  he16-1-1.cr1.san110.as2116.net (195.0.244.234)  2.627 ms he16-1-1.cr2.oslosda310.as2116.net (195.0.244.48)  3.172 ms he16-1-1.cr1.san110.as2116.net (195.0.244.234)  2.857 ms
+ 6  ae1.ar8.oslosda310.as2116.net (195.0.242.39)  0.662 ms  0.637 ms ae0.ar8.oslosda310.as2116.net (195.0.242.23)  0.622 ms
+ 7  89.191.10.146 (89.191.10.146)  0.931 ms  0.917 ms  0.955 ms
+ 8  * * *
+ 9  * * *
+[...]
+

+ +

This show the DNS names and IP addresses of (at least some of the) +network equipment involved in getting the data traffic from me to the +www.stortinget.no server, and how long it took in milliseconds for a +package to reach the equipment and return to me. Three packages are +sent, and some times the packages do not follow the same path. This +is shown for hop 5, where three different IP addresses replied to the +traceroute request.

+ +

There are many ways to measure trace routes. Other good traceroute +implementations I use are traceroute (using ICMP packages) mtr (can do +both ICMP, UDP and TCP) and scapy (python library with ICMP, UDP, TCP +traceroute and a lot of other capabilities). All of them are easily +available in Debian.

+ +

This time around, I wanted to know the geographic location of +different route points, to visualize how visiting a web page spread +information about the visit to a lot of servers around the globe. The +background is that a web site today often will ask the browser to get +from many servers the parts (for example HTML, JSON, fonts, +JavaScript, CSS, video) required to display the content. This will +leak information about the visit to those controlling these servers +and anyone able to peek at the data traffic passing by (like your ISP, +the ISPs backbone provider, FRA, GCHQ, NSA and others).

+ +

Lets pick an example, the Norwegian parliament web site +www.stortinget.no. It is read daily by all members of parliament and +their staff, as well as political journalists, activits and many other +citizens of Norway. A visit to the www.stortinget.no web site will +ask your browser to contact 8 other servers: ajax.googleapis.com, +insights.hotjar.com, script.hotjar.com, static.hotjar.com, +stats.g.doubleclick.net, www.google-analytics.com, +www.googletagmanager.com and www.netigate.se. I extracted this by +asking PhantomJS to visit the +Stortinget web page and tell me all the URLs PhantomJS downloaded to +render the page (in HAR format using +their +netsniff example. I am very grateful to Gorm for showing me how +to do this). My goal is to visualize network traces to all IP +addresses behind these DNS names, do show where visitors personal +information is spread when visiting the page.

+ +

map of combined traces for URLs used by www.stortinget.no using GeoIP

+ +

When I had a look around for options, I could not find any good +free software tools to do this, and decided I needed my own traceroute +wrapper outputting KML based on locations looked up using GeoIP. KML +is easy to work with and easy to generate, and understood by several +of the GIS tools I have available. I got good help from by NUUG +colleague Anders Einar with this, and the result can be seen in +my +kmltraceroute git repository. Unfortunately, the quality of the +free GeoIP databases I could find (and the for-pay databases my +friends had access to) is not up to the task. The IP addresses of +central Internet infrastructure would typically be placed near the +controlling companies main office, and not where the router is really +located, as you can see from the +KML file I created using the GeoLite City dataset from MaxMind. + +

scapy traceroute graph for URLs used by www.stortinget.no

+ +

I also had a look at the visual traceroute graph created by +the scrapy project, +showing IP network ownership (aka AS owner) for the IP address in +question. +The +graph display a lot of useful information about the traceroute in SVG +format, and give a good indication on who control the network +equipment involved, but it do not include geolocation. This graph +make it possible to see the information is made available at least for +UNINETT, Catchcom, Stortinget, Nordunet, Google, Amazon, Telia, Level +3 Communications and NetDNA.

+ +

example geotraceroute view for www.stortinget.no

+ +

In the process, I came across the +web service GeoTraceroute by +Salim Gasmi. Its methology of combining guesses based on DNS names, +various location databases and finally use latecy times to rule out +candidate locations seemed to do a very good job of guessing correct +geolocation. But it could only do one trace at the time, did not have +a sensor in Norway and did not make the geolocations easily available +for postprocessing. So I contacted the developer and asked if he +would be willing to share the code (he refused until he had time to +clean it up), but he was interested in providing the geolocations in a +machine readable format, and willing to set up a sensor in Norway. So +since yesterday, it is possible to run traces from Norway in this +service thanks to a sensor node set up by +the NUUG assosiation, and get the +trace in KML format for further processing.

+ +

map of combined traces for URLs used by www.stortinget.no using geotraceroute

+ +

Here we can see a lot of trafic passes Sweden on its way to +Denmark, Germany, Holland and Ireland. Plenty of places where the +Snowden confirmations verified the traffic is read by various actors +without your best interest as their top priority.

+ +

Combining KML files is trivial using a text editor, so I could loop +over all the hosts behind the urls imported by www.stortinget.no and +ask for the KML file from GeoTraceroute, and create a combined KML +file with all the traces (unfortunately only one of the IP addresses +behind the DNS name is traced this time. To get them all, one would +have to request traces using IP number instead of DNS names from +GeoTraceroute). That might be the next step in this project.

+ +

Armed with these tools, I find it a lot easier to figure out where +the IP traffic moves and who control the boxes involved in moving it. +And every time the link crosses for example the Swedish border, we can +be sure Swedish Signal Intelligence (FRA) is listening, as GCHQ do in +Britain and NSA in USA and cables around the globe. (Hm, what should +we tell them? :) Keep that in mind if you ever send anything +unencrypted over the Internet.

+ +

PS: KML files are drawn using +the KML viewer from Ivan +Rublev, as it was less cluttered than the local Linux application +Marble. There are heaps of other options too.

+ +

As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

+ +
+
+ + + Tags: debian, english, kart, nuug, personvern, stortinget, surveillance, web. + + +
+
+
+ +
+
+ Introducing ical-archiver to split out old iCalendar entries +
+
+ 4th January 2017 +
+
+

Do you have a large iCalendar +file with lots of old entries, and would like to archive them to save +space and resources? At least those of us using KOrganizer know that +turning on and off an event set become slower and slower the more +entries are in the set. While working on migrating our calendars to a +Radicale CalDAV server on our +Freedombox server, my +loved one wondered if I could find a way to split up the calendar file +she had in KOrganizer, and I set out to write a tool. I spent a few +days writing and polishing the system, and it is now ready for general +consumption. The +code for +ical-archiver is publicly available from a git repository on +github. The system is written in Python and depend on +the vobject Python +module.

+ +

To use it, locate the iCalendar file you want to operate on and +give it as an argument to the ical-archiver script. This will +generate a set of new files, one file per component type per year for +all components expiring more than two years in the past. The vevent, +vtodo and vjournal entries are handled by the script. The remaining +entries are stored in a 'remaining' file.

+ +

This is what a test run can look like: + +

+% ical-archiver t/2004-2016.ics 
+Found 3612 vevents
+Found 6 vtodos
+Found 2 vjournals
+Writing t/2004-2016.ics-subset-vevent-2004.ics
+Writing t/2004-2016.ics-subset-vevent-2005.ics
+Writing t/2004-2016.ics-subset-vevent-2006.ics
+Writing t/2004-2016.ics-subset-vevent-2007.ics
+Writing t/2004-2016.ics-subset-vevent-2008.ics
+Writing t/2004-2016.ics-subset-vevent-2009.ics
+Writing t/2004-2016.ics-subset-vevent-2010.ics
+Writing t/2004-2016.ics-subset-vevent-2011.ics
+Writing t/2004-2016.ics-subset-vevent-2012.ics
+Writing t/2004-2016.ics-subset-vevent-2013.ics
+Writing t/2004-2016.ics-subset-vevent-2014.ics
+Writing t/2004-2016.ics-subset-vjournal-2007.ics
+Writing t/2004-2016.ics-subset-vjournal-2011.ics
+Writing t/2004-2016.ics-subset-vtodo-2012.ics
+Writing t/2004-2016.ics-remaining.ics
+%
+

+ +

As you can see, the original file is untouched and new files are +written with names derived from the original file. If you are happy +with their content, the *-remaining.ics file can replace the original +the the others can be archived or imported as historical calendar +collections.

+ +

The script should probably be improved a bit. The error handling +when discovering broken entries is not good, and I am not sure yet if +it make sense to split different entry types into separate files or +not. The program is thus likely to change. If you find it +interesting, please get in touch. :)

+ +

As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

+ +
+
+ + + Tags: english, standard. + + +
+
+
+ +
+
+ Appstream just learned how to map hardware to packages too! +
+
+ 23rd December 2016 +
+
+

I received a very nice Christmas present today. As my regular +readers probably know, I have been working on the +the Isenkram +system for many years. The goal of the Isenkram system is to make +it easier for users to figure out what to install to get a given piece +of hardware to work in Debian, and a key part of this system is a way +to map hardware to packages. Isenkram have its own mapping database, +and also uses data provided by each package using the AppStream +metadata format. And today, +AppStream in +Debian learned to look up hardware the same way Isenkram is doing it, +ie using fnmatch():

+ +

+% appstreamcli what-provides modalias \
+  usb:v1130p0202d0100dc00dsc00dp00ic03isc00ip00in00
+Identifier: pymissile [generic]
+Name: pymissile
+Summary: Control original Striker USB Missile Launcher
+Package: pymissile
+% appstreamcli what-provides modalias usb:v0694p0002d0000
+Identifier: libnxt [generic]
+Name: libnxt
+Summary: utility library for talking to the LEGO Mindstorms NXT brick
+Package: libnxt
+---
+Identifier: t2n [generic]
+Name: t2n
+Summary: Simple command-line tool for Lego NXT
+Package: t2n
+---
+Identifier: python-nxt [generic]
+Name: python-nxt
+Summary: Python driver/interface/wrapper for the Lego Mindstorms NXT robot
+Package: python-nxt
+---
+Identifier: nbc [generic]
+Name: nbc
+Summary: C compiler for LEGO Mindstorms NXT bricks
+Package: nbc
+%
+

+ +

A similar query can be done using the combined AppStream and +Isenkram databases using the isenkram-lookup tool:

+ +

+% isenkram-lookup usb:v1130p0202d0100dc00dsc00dp00ic03isc00ip00in00
+pymissile
+% isenkram-lookup usb:v0694p0002d0000
+libnxt
+nbc
+python-nxt
+t2n
+%
+

+ +

You can find modalias values relevant for your machine using +cat $(find /sys/devices/ -name modalias). + +

If you want to make this system a success and help Debian users +make the most of the hardware they have, please +helpadd +AppStream metadata for your package following the guidelines +documented in the wiki. So far only 11 packages provide such +information, among the several hundred hardware specific packages in +Debian. The Isenkram database on the other hand contain 101 packages, +mostly related to USB dongles. Most of the packages with hardware +mapping in AppStream are LEGO Mindstorms related, because I have, as +part of my involvement in +the Debian LEGO +team given priority to making sure LEGO users get proposed the +complete set of packages in Debian for that particular hardware. The +team also got a nice Christmas present today. The +nxt-firmware +package made it into Debian. With this package in place, it is +now possible to use the LEGO Mindstorms NXT unit with only free +software, as the nxt-firmware package contain the source and firmware +binaries for the NXT brick.

+ +

As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

+ +
+
+ + + Tags: debian, english, isenkram. + + +
+
+
+ +
+
+ Isenkram updated with a lot more hardware-package mappings +
+
+ 20th December 2016 +
+
+

The Isenkram +system I wrote two years ago to make it easier in Debian to find +and install packages to get your hardware dongles to work, is still +going strong. It is a system to look up the hardware present on or +connected to the current system, and map the hardware to Debian +packages. It can either be done using the tools in isenkram-cli or +using the user space daemon in the isenkram package. The latter will +notify you, when inserting new hardware, about what packages to +install to get the dongle working. It will even provide a button to +click on to ask packagekit to install the packages.

+ +

Here is an command line example from my Thinkpad laptop:

+ +

+% isenkram-lookup  
+bluez
+cheese
+ethtool
+fprintd
+fprintd-demo
+gkrellm-thinkbat
+hdapsd
+libpam-fprintd
+pidgin-blinklight
+thinkfan
+tlp
+tp-smapi-dkms
+tp-smapi-source
+tpb
+%
+

+ +

It can also list the firware package providing firmware requested +by the load kernel modules, which in my case is an empty list because +I have all the firmware my machine need: + +

+% /usr/sbin/isenkram-autoinstall-firmware -l
+info: did not find any firmware files requested by loaded kernel modules.  exiting
+%
+

+ +

The last few days I had a look at several of the around 250 +packages in Debian with udev rules. These seem like good candidates +to install when a given hardware dongle is inserted, and I found +several that should be proposed by isenkram. I have not had time to +check all of them, but am happy to report that now there are 97 +packages packages mapped to hardware by Isenkram. 11 of these +packages provide hardware mapping using AppStream, while the rest are +listed in the modaliases file provided in isenkram.

+ +

These are the packages with hardware mappings at the moment. The +marked packages are also announcing their hardware +support using AppStream, for everyone to use:

+ +

air-quality-sensor, alsa-firmware-loaders, argyll, +array-info, avarice, avrdude, b43-fwcutter, +bit-babbler, bluez, bluez-firmware, brltty, +broadcom-sta-dkms, calibre, cgminer, cheese, colord, +colorhug-client, dahdi-firmware-nonfree, dahdi-linux, +dfu-util, dolphin-emu, ekeyd, ethtool, firmware-ipw2x00, fprintd, +fprintd-demo, galileo, gkrellm-thinkbat, gphoto2, +gpsbabel, gpsbabel-gui, gpsman, gpstrans, gqrx-sdr, gr-fcdproplus, +gr-osmosdr, gtkpod, hackrf, hdapsd, hdmi2usb-udev, hpijs-ppds, hplip, +ipw3945-source, ipw3945d, kde-config-tablet, kinect-audio-setup, +libnxt, libpam-fprintd, lomoco, +madwimax, minidisc-utils, mkgmap, msi-keyboard, mtkbabel, +nbc, nqc, nut-hal-drivers, ola, +open-vm-toolbox, open-vm-tools, openambit, pcgminer, pcmciautils, +pcscd, pidgin-blinklight, printer-driver-splix, +pymissile, python-nxt, qlandkartegt, +qlandkartegt-garmin, rosegarden, rt2x00-source, sispmctl, +soapysdr-module-hackrf, solaar, squeak-plugins-scratch, sunxi-tools, +t2n, thinkfan, thinkfinger-tools, tlp, tp-smapi-dkms, +tp-smapi-source, tpb, tucnak, uhd-host, usbmuxd, viking, +virtualbox-ose-guest-x11, w1retap, xawtv, xserver-xorg-input-vmmouse, +xserver-xorg-input-wacom, xserver-xorg-video-qxl, +xserver-xorg-video-vmware, yubikey-personalization and +zd1211-firmware

+ +

If you know of other packages, please let me know with a wishlist +bug report against the isenkram-cli package, and ask the package +maintainer to +add AppStream +metadata according to the guidelines to provide the information +for everyone. In time, I hope to get rid of the isenkram specific +hardware mapping and depend exclusively on AppStream.

+ +

Note, the AppStream metadata for broadcom-sta-dkms is matching too +much hardware, and suggest that the package with with any ethernet +card. See bug #838735 for +the details. I hope the maintainer find time to address it soon. In +the mean time I provide an override in isenkram.

+ +
+
+ + + Tags: debian, english, isenkram. + + +
+
+
+ +
+
+ Oolite, a life in space as vagabond and mercenary - nice free software +
+
+ 11th December 2016 +
+
+

+ +

In my early years, I played +the epic game +Elite on my PC. I spent many months trading and fighting in +space, and reached the 'elite' fighting status before I moved on. The +original Elite game was available on Commodore 64 and the IBM PC +edition I played had a 64 KB executable. I am still impressed today +that the authors managed to squeeze both a 3D engine and details about +more than 2000 planet systems across 7 galaxies into a binary so +small.

+ +

I have known about the free +software game Oolite inspired by Elite for a while, but did not +really have time to test it properly until a few days ago. It was +great to discover that my old knowledge about trading routes were +still valid. But my fighting and flying abilities were gone, so I had +to retrain to be able to dock on a space station. And I am still not +able to make much resistance when I am attacked by pirates, so I +bougth and mounted the most powerful laser in the rear to be able to +put up at least some resistance while fleeing for my life. :)

+ +

When playing Elite in the late eighties, I had to discover +everything on my own, and I had long lists of prices seen on different +planets to be able to decide where to trade what. This time I had the +advantages of the +Elite wiki, +where information about each planet is easily available with common +price ranges and suggested trading routes. This improved my ability +to earn money and I have been able to earn enough to buy a lot of +useful equipent in a few days. I believe I originally played for +months before I could get a docking computer, while now I could get it +after less then a week.

+ +

If you like science fiction and dreamed of a life as a vagabond in +space, you should try out Oolite. It is available for Linux, MacOSX +and Windows, and is included in Debian and derivatives since 2011.

+ +

As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

+ +
+
+ + + Tags: debian, english, nice free software. + + +
+
+
+ +
+
+ Quicker Debian installations using eatmydata +
+
+ 25th November 2016 +
+
+

Two years ago, I did some experiments with eatmydata and the Debian +installation system, observing how using +eatmydata +could speed up the installation quite a bit. My testing measured +speedup around 20-40 percent for Debian Edu, where we install around +1000 packages from within the installer. The eatmydata package +provide a way to disable/delay file system flushing. This is a bit +risky in the general case, as files that should be stored on disk will +stay only in memory a bit longer than expected, causing problems if a +machine crashes at an inconvenient time. But for an installation, if +the machine crashes during installation the process is normally +restarted, and avoiding disk operations as much as possible to speed +up the process make perfect sense. + +

I added code in the Debian Edu specific installation code to enable +eatmydata, +but did not have time to push it any further. But a few months ago I +picked it up again and worked with the libeatmydata package maintainer +Mattia Rizzolo to make it easier for everyone to get this installation +speedup in Debian. Thanks to our cooperation There is now an +eatmydata-udeb package in Debian testing and unstable, and simply +enabling/installing it in debian-installer (d-i) is enough to get the +quicker installations. It can be enabled using preseeding. The +following untested kernel argument should do the trick:

+ +
+preseed/early_command="anna-install eatmydata-udeb"
+
+ +

This should ask d-i to install the package inside the d-i +environment early in the installation sequence. Having it installed +in d-i in turn will make sure the relevant scripts are called just +after debootstrap filled /target/ with the freshly installed Debian +system to configure apt to run dpkg with eatmydata. This is enough to +speed up the installation process. There is a proposal to +extend the idea a bit further +by using /etc/ld.so.preload instead of apt.conf, but I have not +tested its impact.

+ + +
+
+ + + Tags: debian, debian edu, english. + + +
+
+
+ +
+
+ Coz profiler for multi-threaded software is now in Debian +
+
+ 13th November 2016 +
+
+

The Coz profiler, a nice +profiler able to run benchmarking experiments on the instrumented +multi-threaded program, finally +made it into +Debian unstable yesterday. Lluís Vilanova and I have spent many +months since +I +blogged about the coz tool in August working with upstream to make +it suitable for Debian. There are still issues with clang +compatibility, inline assembly only working x86 and minimized +JavaScript libraries.

+ +

To test it, install 'coz-profiler' using apt and run it like this:

+ +

+coz run --- /path/to/binary-with-debug-info +

+ +

This will produce a profile.coz file in the current working +directory with the profiling information. This is then given to a +JavaScript application provided in the package and available from +a project web page. +To start the local copy, invoke it in a browser like this:

+ +

+sensible-browser /usr/share/coz-profiler/viewer/index.htm +

+ +

See the project home page and the +USENIX +;login: article on Coz for more information on how it is +working.

+ +
+
+ + + Tags: debian, english. + + +
+
+
+ +
+
+ How to talk with your loved ones in private +
+
+ 7th November 2016 +
+
+

A few days ago I ran a very biased and informal survey to get an +idea about what options are being used to communicate with end to end +encryption with friends and family. I explicitly asked people not to +list options only used in a work setting. The background is the +uneasy feeling I get when using Signal, a feeling shared by others as +a blog post from Sander Venima about +why +he do not recommend Signal anymore (with +feedback from +the Signal author available from ycombinator). I wanted an +overview of the options being used, and hope to include those options +in a less biased survey later on. So far I have not taken the time to +look into the individual proposed systems. They range from text +sharing web pages, via file sharing and email to instant messaging, +VOIP and video conferencing. For those considering which system to +use, it is also useful to have a look at +the EFF Secure +messaging scorecard which is slightly out of date but still +provide valuable information.

+ +

So, on to the list. There were some used by many, some used by a +few, some rarely used ones and a few mentioned but without anyone +claiming to use them. Notice the grouping is in reality quite random +given the biased self selected set of participants. First the ones +used by many:

+ + + +

Then the ones used by a few.

+ + + +

Then the ones used by even fewer people

+ + + +

And finally the ones mentioned by not marked as used by +anyone. This might be a mistake, perhaps the person adding the entry +forgot to flag it as used?

+ + + +

Given the network effect it seem obvious to me that we as a society +have been divided and conquered by those interested in keeping +encrypted and secure communication away from the masses. The +finishing remarks from Aral Balkan +in his talk "Free is a lie" about the usability of free software +really come into effect when you want to communicate in private with +your friends and family. We can not expect them to allow the +usability of communication tool to block their ability to talk to +their loved ones.

+ +

Note for example the option IRC w/OTR. Most IRC clients do not +have OTR support, so in most cases OTR would not be an option, even if +you wanted to. In my personal experience, about 1 in 20 I talk to +have a IRC client with OTR. For private communication to really be +available, most people to talk to must have the option in their +currently used client. I can not simply ask my family to install an +IRC client. I need to guide them through a technical multi-step +process of adding extensions to the client to get them going. This is +a non-starter for most.

+ +

I would like to be able to do video phone calls, audio phone calls, +exchange instant messages and share files with my loved ones, without +being forced to share with people I do not know. I do not want to +share the content of the conversations, and I do not want to share who +I communicate with or the fact that I communicate with someone. +Without all these factors in place, my private life is being more or +less invaded.

+ +
+
+ + + Tags: english, personvern, sikkerhet, surveillance. + + +
+
+
+ +
+
+ My own self balancing Lego Segway +
+
+ 4th November 2016 +
+
+

A while back I received a Gyro sensor for the NXT +Mindstorms controller as a birthday +present. It had been on my wishlist for a while, because I wanted to +build a Segway like balancing lego robot. I had already built +a simple balancing +robot with the kids, using the light/color sensor included in the +NXT kit as the balance sensor, but it was not working very well. It +could balance for a while, but was very sensitive to the light +condition in the room and the reflective properties of the surface and +would fall over after a short while. I wanted something more robust, +and had +the +gyro sensor from HiTechnic I believed would solve it on my +wishlist for some years before it suddenly showed up as a gift from my +loved ones. :)

+ +

Unfortunately I have not had time to sit down and play with it +since then. But that changed some days ago, when I was searching for +lego segway information and came across a recipe from HiTechnic for +building +the +HTWay, a segway like balancing robot. Build instructions and +source +code was included, so it was just a question of putting it all +together. And thanks to the great work of many Debian developers, the +compiler needed to build the source for the NXT is already included in +Debian, so I was read to go in less than an hour. The resulting robot +do not look very impressive in its simplicity:

+ +

+ +

Because I lack the infrared sensor used to control the robot in the +design from HiTechnic, I had to comment out the last task +(taskControl). I simply placed /* and */ around it get the program +working without that sensor present. Now it balances just fine until +the battery status run low:

+ +

+ +

Now we would like to teach it how to follow a line and take remote +control instructions using the included Bluetooth receiver in the NXT.

+ +

If you, like me, love LEGO and want to make sure we find the tools +they need to work with LEGO in Debian and all our derivative +distributions like Ubuntu, check out +the LEGO designers +project page and join the Debian LEGO team. Personally I own a +RCX and NXT controller (no EV3), and would like to make sure the +Debian tools needed to program the systems I own work as they +should.

+ +
+
+ + + Tags: debian, english, robot. + + +
+
+
+ +
+
+ Experience and updated recipe for using the Signal app without a mobile phone +
+
+ 10th October 2016 +
+
+

In July +I +wrote how to get the Signal Chrome/Chromium app working without +the ability to receive SMS messages (aka without a cell phone). It is +time to share some experiences and provide an updated setup.

+ +

The Signal app have worked fine for several months now, and I use +it regularly to chat with my loved ones. I had a major snag at the +end of my summer vacation, when the the app completely forgot my +setup, identity and keys. The reason behind this major mess was +running out of disk space. To avoid that ever happening again I have +started storing everything in userdata/ in git, to be able to +roll back to an earlier version if the files are wiped by mistake. I +had to use it once after introducing the git backup. When rolling +back to an earlier version, one need to use the 'reset session' option +in Signal to get going, and notify the people you talk with about the +problem. I assume there is some sequence number tracking in the +protocol to detect rollback attacks. The git repository is rather big +(674 MiB so far), but I have not tried to figure out if some of the +content can be added to a .gitignore file due to lack of spare +time.

+ +

I've also hit the 90 days timeout blocking, and noticed that this +make it impossible to send messages using Signal. I could still +receive them, but had to patch the code with a new timestamp to send. +I believe the timeout is added by the developers to force people to +upgrade to the latest version of the app, even when there is no +protocol changes, to reduce the version skew among the user base and +thus try to keep the number of support requests down.

+ +

Since my original recipe, the Signal source code changed slightly, +making the old patch fail to apply cleanly. Below is an updated +patch, including the shell wrapper I use to start Signal. The +original version required a new user to locate the JavaScript console +and call a function from there. I got help from a friend with more +JavaScript knowledge than me to modify the code to provide a GUI +button instead. This mean that to get started you just need to run +the wrapper and click the 'Register without mobile phone' to get going +now. I've also modified the timeout code to always set it to 90 days +in the future, to avoid having to patch the code regularly.

+ +

So, the updated recipe for Debian Jessie:

+ +
    + +
  1. First, install required packages to get the source code and the +browser you need. Signal only work with Chrome/Chromium, as far as I +know, so you need to install it. + +
    +apt install git tor chromium
+git clone https://github.com/WhisperSystems/Signal-Desktop.git
+
    2. + +
  2. Modify the source code using command listed in the the patch +block below.
    3. + +
  3. Start Signal using the run-signal-app wrapper (for example using +`pwd`/run-signal-app). + +
  4. Click on the 'Register without mobile phone', will in a phone +number you can receive calls to the next minute, receive the +verification code and enter it into the form field and press +'Register'. Note, the phone number you use will be user Signal +username, ie the way others can find you on Signal.
    5. + +
  5. You can now use Signal to contact others. Note, new contacts do +not show up in the contact list until you restart Signal, and there is +no way to assign names to Contacts. There is also no way to create or +update chat groups. I suspect this is because the web app do not have +a associated contact database.
    6. + +
+ +

I am still a bit uneasy about using Signal, because of the way its +main author moxie0 reject federation and accept dependencies to major +corporations like Google (part of the code is fetched from Google) and +Amazon (the central coordination point is owned by Amazon). See for +example +the +LibreSignal issue tracker for a thread documenting the authors +view on these issues. But the network effect is strong in this case, +and several of the people I want to communicate with already use +Signal. Perhaps we can all move to Ring +once it work on my +laptop? It already work on Windows and Android, and is included +in Debian and +Ubuntu, but not +working on Debian Stable.

+ +

Anyway, this is the patch I apply to the Signal code to get it +working. It switch to the production servers, disable to timeout, +make registration easier and add the shell wrapper:

+ +
+cd Signal-Desktop; cat <<EOF | patch -p1
+diff --git a/js/background.js b/js/background.js
+index 24b4c1d..579345f 100644
+--- a/js/background.js
++++ b/js/background.js
+@@ -33,9 +33,9 @@
+         });
+     });
+ 
+-    var SERVER_URL = 'https://textsecure-service-staging.whispersystems.org';
++    var SERVER_URL = 'https://textsecure-service-ca.whispersystems.org';
+     var SERVER_PORTS = [80, 4433, 8443];
+-    var ATTACHMENT_SERVER_URL = 'https://whispersystems-textsecure-attachments-staging.s3.amazonaws.com';
++    var ATTACHMENT_SERVER_URL = 'https://whispersystems-textsecure-attachments.s3.amazonaws.com';
+     var messageReceiver;
+     window.getSocketStatus = function() {
+         if (messageReceiver) {
+diff --git a/js/expire.js b/js/expire.js
+index 639aeae..beb91c3 100644
+--- a/js/expire.js
++++ b/js/expire.js
+@@ -1,6 +1,6 @@
+ ;(function() {
+     'use strict';
+-    var BUILD_EXPIRATION = 0;
++    var BUILD_EXPIRATION = Date.now() + (90 * 24 * 60 * 60 * 1000);
+ 
+     window.extension = window.extension || {};
+ 
+diff --git a/js/views/install_view.js b/js/views/install_view.js
+index 7816f4f..1d6233b 100644
+--- a/js/views/install_view.js
++++ b/js/views/install_view.js
+@@ -38,7 +38,8 @@
+             return {
+                 'click .step1': this.selectStep.bind(this, 1),
+                 'click .step2': this.selectStep.bind(this, 2),
+-                'click .step3': this.selectStep.bind(this, 3)
++                'click .step3': this.selectStep.bind(this, 3),
++                'click .callreg': function() { extension.install('standalone') },
+             };
+         },
+         clearQR: function() {
+diff --git a/options.html b/options.html
+index dc0f28e..8d709f6 100644
+--- a/options.html
++++ b/options.html
+@@ -14,7 +14,10 @@
+         <div class='nav'>
+           <h1>{{ installWelcome }}</h1>
+           <p>{{ installTagline }}</p>
+-          <div> <a class='button step2'>{{ installGetStartedButton }}</a> </div>
++          <div> <a class='button step2'>{{ installGetStartedButton }}</a>
++	    <br> <a class="button callreg">Register without mobile phone</a>
++
++	  </div>
+           <span class='dot step1 selected'></span>
+           <span class='dot step2'></span>
+           <span class='dot step3'></span>
+--- /dev/null   2016-10-07 09:55:13.730181472 +0200
++++ b/run-signal-app   2016-10-10 08:54:09.434172391 +0200
+@@ -0,0 +1,12 @@
++#!/bin/sh
++set -e
++cd $(dirname $0)
++mkdir -p userdata
++userdata="`pwd`/userdata"
++if [ -d "$userdata" ] && [ ! -d "$userdata/.git" ] ; then
++    (cd $userdata && git init)
++fi
++(cd $userdata && git add . && git commit -m "Current status." || true)
++exec chromium \
++  --proxy-server="socks://localhost:9050" \
++  --user-data-dir=$userdata --load-and-launch-app=`pwd`
+EOF
+chmod a+rx run-signal-app
+
+ +

As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

+ +
+
+ + + Tags: debian, english, sikkerhet, surveillance. + + +
+
+
+ +
+
+ Isenkram, Appstream and udev make life as a LEGO builder easier +
+
+ 7th October 2016 +
+
+

The Isenkram +system provide a practical and easy way to figure out which +packages support the hardware in a given machine. The command line +tool isenkram-lookup and the tasksel options provide a +convenient way to list and install packages relevant for the current +hardware during system installation, both user space packages and +firmware packages. The GUI background daemon on the other hand provide +a pop-up proposing to install packages when a new dongle is inserted +while using the computer. For example, if you plug in a smart card +reader, the system will ask if you want to install pcscd if +that package isn't already installed, and if you plug in a USB video +camera the system will ask if you want to install cheese if +cheese is currently missing. This already work just fine.

+ +

But Isenkram depend on a database mapping from hardware IDs to +package names. When I started no such database existed in Debian, so +I made my own data set and included it with the isenkram package and +made isenkram fetch the latest version of this database from git using +http. This way the isenkram users would get updated package proposals +as soon as I learned more about hardware related packages.

+ +

The hardware is identified using modalias strings. The modalias +design is from the Linux kernel where most hardware descriptors are +made available as a strings that can be matched using filename style +globbing. It handle USB, PCI, DMI and a lot of other hardware related +identifiers.

+ +

The downside to the Isenkram specific database is that there is no +information about relevant distribution / Debian version, making +isenkram propose obsolete packages too. But along came AppStream, a +cross distribution mechanism to store and collect metadata about +software packages. When I heard about the proposal, I contacted the +people involved and suggested to add a hardware matching rule using +modalias strings in the specification, to be able to use AppStream for +mapping hardware to packages. This idea was accepted and AppStream is +now a great way for a package to announce the hardware it support in a +distribution neutral way. I wrote +a +recipe on how to add such meta-information in a blog post last +December. If you have a hardware related package in Debian, please +announce the relevant hardware IDs using AppStream.

+ +

In Debian, almost all packages that can talk to a LEGO Mindestorms +RCX or NXT unit, announce this support using AppStream. The effect is +that when you insert such LEGO robot controller into your Debian +machine, Isenkram will propose to install the packages needed to get +it working. The intention is that this should allow the local user to +start programming his robot controller right away without having to +guess what packages to use or which permissions to fix.

+ +

But when I sat down with my son the other day to program our NXT +unit using his Debian Stretch computer, I discovered something +annoying. The local console user (ie my son) did not get access to +the USB device for programming the unit. This used to work, but no +longer in Jessie and Stretch. After some investigation and asking +around on #debian-devel, I discovered that this was because udev had +changed the mechanism used to grant access to local devices. The +ConsoleKit mechanism from /lib/udev/rules.d/70-udev-acl.rules +no longer applied, because LDAP users no longer was added to the +plugdev group during login. Michael Biebl told me that this method +was obsolete and the new method used ACLs instead. This was good +news, as the plugdev mechanism is a mess when using a remote user +directory like LDAP. Using ACLs would make sure a user lost device +access when she logged out, even if the user left behind a background +process which would retain the plugdev membership with the ConsoleKit +setup. Armed with this knowledge I moved on to fix the access problem +for the LEGO Mindstorms related packages.

+ +

The new system uses a udev tag, 'uaccess'. It can either be +applied directly for a device, or is applied in +/lib/udev/rules.d/70-uaccess.rules for classes of devices. As the +LEGO Mindstorms udev rules did not have a class, I decided to add the +tag directly in the udev rules files included in the packages. Here +is one example. For the nqc C compiler for the RCX, the +/lib/udev/rules.d/60-nqc.rules file now look like this: + +

+SUBSYSTEM=="usb", ACTION=="add", ATTR{idVendor}=="0694", ATTR{idProduct}=="0001", \
+    SYMLINK+="rcx-%k", TAG+="uaccess"
+

+ +

The key part is the 'TAG+="uaccess"' at the end. I suspect all +packages using plugdev in their /lib/udev/rules.d/ files should be +changed to use this tag (either directly or indirectly via +70-uaccess.rules). Perhaps a lintian check should be created +to detect this?

+ +

I've been unable to find good documentation on the uaccess feature. +It is unclear to me if the uaccess tag is an internal implementation +detail like the udev-acl tag used by +/lib/udev/rules.d/70-udev-acl.rules. If it is, I guess the +indirect method is the preferred way. Michael +asked for more +documentation from the systemd project and I hope it will make +this clearer. For now I use the generic classes when they exist and +is already handled by 70-uaccess.rules, and add the tag +directly if no such class exist.

+ +

To learn more about the isenkram system, please check out +my +blog posts tagged isenkram.

+ +

To help out making life for LEGO constructors in Debian easier, +please join us on our IRC channel +#debian-lego and join +the Debian +LEGO team in the Alioth project we created yesterday. A mailing +list is not yet created, but we are working on it. :)

+ +

As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

+ +
+
+ + + Tags: debian, english, isenkram. + + +
+
+
+ +
+
+ First draft Norwegian Bokmål edition of The Debian Administrator's Handbook now public +
+
+ 30th August 2016 +
+
+

In April we +started +to work on a Norwegian Bokmål edition of the "open access" book on +how to set up and administrate a Debian system. Today I am happy to +report that the first draft is now publicly available. You can find +it on get the Debian +Administrator's Handbook page (under Other languages). The first +eight chapters have a first draft translation, and we are working on +proofreading the content. If you want to help out, please start +contributing using +the +hosted weblate project page, and get in touch using +the +translators mailing list. Please also check out +the instructions for +contributors. A good way to contribute is to proofread the text +and update weblate if you find errors.

+ +

Our goal is still to make the Norwegian book available on paper as well as +electronic form.

+ +
+
+ + + Tags: debian, debian-handbook, english. + + +
+
+
+ +
+
+ Coz can help you find bottlenecks in multi-threaded software - nice free software +
+
+ 11th August 2016 +
+
+

This summer, I read a great article +"coz: +This Is the Profiler You're Looking For" in USENIX ;login: about +how to profile multi-threaded programs. It presented a system for +profiling software by running experiences in the running program, +testing how run time performance is affected by "speeding up" parts of +the code to various degrees compared to a normal run. It does this by +slowing down parallel threads while the "faster up" code is running +and measure how this affect processing time. The processing time is +measured using probes inserted into the code, either using progress +counters (COZ_PROGRESS) or as latency meters (COZ_BEGIN/COZ_END). It +can also measure unmodified code by measuring complete the program +runtime and running the program several times instead.

+ +

The project and presentation was so inspiring that I would like to +get the system into Debian. I +created +a WNPP request for it and contacted upstream to try to make the +system ready for Debian by sending patches. The build process need to +be changed a bit to avoid running 'git clone' to get dependencies, and +to include the JavaScript web page used to visualize the collected +profiling information included in the source package. +But I expect that should work out fairly soon.

+ +

The way the system work is fairly simple. To run an coz experiment +on a binary with debug symbols available, start the program like this: + +

+coz run --- program-to-run
+

+ +

This will create a text file profile.coz with the instrumentation +information. To show what part of the code affect the performance +most, use a web browser and either point it to +http://plasma-umass.github.io/coz/ +or use the copy from git (in the gh-pages branch). Check out this web +site to have a look at several example profiling runs and get an idea what the end result from the profile runs look like. To make the +profiling more useful you include <coz.h> and insert the +COZ_PROGRESS or COZ_BEGIN and COZ_END at appropriate places in the +code, rebuild and run the profiler. This allow coz to do more +targeted experiments.

+ +

A video published by ACM +presenting the +Coz profiler is available from Youtube. There is also a paper +from the 25th Symposium on Operating Systems Principles available +titled +Coz: +finding code that counts with causal profiling.

+ +

The source code +for Coz is available from github. It will only build with clang +because it uses a +C++ +feature missing in GCC, but I've submitted +a patch to solve +it and hope it will be included in the upstream source soon.

+ +

Please get in touch if you, like me, would like to see this piece +of software in Debian. I would very much like some help with the +packaging effort, as I lack the in depth knowledge on how to package +C++ libraries.

+ +
+
+ + + Tags: debian, english, nice free software. + + +
+
+
+
Sales number for the Free Culture translation, first half of 2016 @@ -33,12 +2073,13 @@ a French and Norwegian translation of the classic Free Culture book by the founder of the Creative Commons movement, Lawrence Lessig. A bit less known is the fact that due to the way I created the translations, -using docbook and po4a, and also recreated the English original. And +using docbook and po4a, I also recreated the English original. And because I already had created a new the PDF edition, I published it -too. The revenue from the book is sent to the Creative Commons -Corporation. So I do not earn any money from the project, I just earn -the warm fuzzy feeling that the text is available for a wider audience -and more people can learn why the Creative Commons is needed.

+too. The revenue from the books are sent to the Creative Commons +Corporation. In other words, I do not earn any money from this +project, I just earn the warm fuzzy feeling that the text is available +for a wider audience and more people can learn why the Creative +Commons is needed.

Today, just for fun, I had a look at the sales number over at Lulu.com, which take care of payment, printing and shipping. Much to @@ -49,9 +2090,9 @@ books was sold for USD $19.99 between 2016-01-01 and 2016-07-31:

- - - + + +
Title / languageQuantity
Culture Libre / French3
Fri kultur / Norwegian7
Free Culture / English14
Culture Libre / French3
Fri kultur / Norwegian7
Free Culture / English14

The books are available both from Lulu.com and from large book @@ -347,6 +2388,12 @@ those I knew could tell who I was on Signal. There are options avoiding such information leakage, but most of my friends are not using them, so I am stuck with Signal for now.

+

Update 2017-01-10: There is an updated blog post +on this topic in +Experience +and updated recipe for using the Signal app without a mobile +phone.

+
@@ -1137,7 +3184,7 @@ available for many more languages.

- Tags: debian, english. + Tags: debian, debian-handbook, english.
@@ -26274,6 +28321,21 @@ be the only one fitting our needs. :/

Archive