Since this morning, the battery-stats package in Debian include an -extended collector that will collect the complete battery history for -later processing and graphing. The original collector store the -battery level as percentage of last full level, while the new -collector also record battery vendor, model, serial number, design -full level, last full level and current battery level. This make it -possible to predict the lifetime of the battery as well as visualise -the energy flow when the battery is charging or discharging.
- -The new tools are available in /usr/share/battery-stats/ -in the version 0.5.1 package in unstable. Get the new battery level graph -and lifetime prediction by running: +
+Did you ever wonder where the web trafic really flow to reach the +web servers, and who own the network equipment it is flowing through? +It is possible to get a glimpse of this from using traceroute, but it +is hard to find all the details. Many years ago, I wrote a system to +map the Norwegian Internet (trying to figure out if our plans for a +network game service would get low enough latency, and who we needed +to talk to about setting up game servers close to the users. Back +then I used traceroute output from many locations (I asked my friends +to run a script and send me their traceroute output) to create the +graph and the map. The output from traceroute typically look like +this:
-/usr/share/battery-stats/battery-stats-graph /var/log/battery-stats.csv +traceroute to www.stortinget.no (85.88.67.10), 30 hops max, 60 byte packets + 1 uio-gw10.uio.no (129.240.202.1) 0.447 ms 0.486 ms 0.621 ms + 2 uio-gw8.uio.no (129.240.24.229) 0.467 ms 0.578 ms 0.675 ms + 3 oslo-gw1.uninett.no (128.39.65.17) 0.385 ms 0.373 ms 0.358 ms + 4 te3-1-2.br1.fn3.as2116.net (193.156.90.3) 1.174 ms 1.172 ms 1.153 ms + 5 he16-1-1.cr1.san110.as2116.net (195.0.244.234) 2.627 ms he16-1-1.cr2.oslosda310.as2116.net (195.0.244.48) 3.172 ms he16-1-1.cr1.san110.as2116.net (195.0.244.234) 2.857 ms + 6 ae1.ar8.oslosda310.as2116.net (195.0.242.39) 0.662 ms 0.637 ms ae0.ar8.oslosda310.as2116.net (195.0.242.23) 0.622 ms + 7 89.191.10.146 (89.191.10.146) 0.931 ms 0.917 ms 0.955 ms + 8 * * * + 9 * * * +[...]-
Or select the 'Battery Level Graph' from your application menu.
- -The flow in/out of the battery can be seen by running (no menu -entry yet):
+This show the DNS names and IP addresses of (at least some of the) +network equipment involved in getting the data traffic from me to the +www.stortinget.no server, and how long it took in milliseconds for a +package to reach the equipment and return to me. Three packages are +sent, and some times the packages do not follow the same path. This +is shown for hop 5, where three different IP addresses replied to the +traceroute request.
+ +There are many ways to measure trace routes. Other good traceroute +implementations I use are traceroute (using ICMP packages) mtr (can do +both ICMP, UDP and TCP) and scapy (python library with ICMP, UDP, TCP +traceroute and a lot of other capabilities). All of them are easily +available in Debian.
+ +This time around, I wanted to know the geographic location of +different route points, to visualize how visiting a web page spread +information about the visit to a lot of servers around the globe. The +background is that a web site today often will ask the browser to get +from many servers the parts (for example HTML, JSON, fonts, +JavaScript, CSS, video) required to display the content. This will +leak information about the visit to those controlling these servers +and anyone able to peek at the data traffic passing by (like your ISP, +the ISPs backbone provider, FRA, GCHQ, NSA and others).
+ +Lets pick an example, the Norwegian parliament web site +www.stortinget.no. It is read daily by all members of parliament and +their staff, as well as political journalists, activits and many other +citizens of Norway. A visit to the www.stortinget.no web site will +ask your browser to contact 8 other servers: ajax.googleapis.com, +insights.hotjar.com, script.hotjar.com, static.hotjar.com, +stats.g.doubleclick.net, www.google-analytics.com, +www.googletagmanager.com and www.netigate.se. I extracted this by +asking PhantomJS to visit the +Stortinget web page and tell me all the URLs PhantomJS downloaded to +render the page (in HAR format using +their +netsniff example. I am very grateful to Gorm for showing me how +to do this). My goal is to visualize network traces to all IP +addresses behind these DNS names, do show where visitors personal +information is spread when visiting the page.
+ + + +When I had a look around for options, I could not find any good +free software tools to do this, and decided I needed my own traceroute +wrapper outputting KML based on locations looked up using GeoIP. KML +is easy to work with and easy to generate, and understood by several +of the GIS tools I have available. I got good help from by NUUG +colleague Anders Einar with this, and the result can be seen in +my +kmltraceroute git repository. Unfortunately, the quality of the +free GeoIP databases I could find (and the for-pay databases my +friends had access to) is not up to the task. The IP addresses of +central Internet infrastructure would typically be placed near the +controlling companies main office, and not where the router is really +located, as you can see from the +KML file I created using the GeoLite City dataset from MaxMind. + +
+ +I also had a look at the visual traceroute graph created by +the scrapy project, +showing IP network ownership (aka AS owner) for the IP address in +question. +The +graph display a lot of useful information about the traceroute in SVG +format, and give a good indication on who control the network +equipment involved, but it do not include geolocation. This graph +make it possible to see the information is made available at least for +UNINETT, Catchcom, Stortinget, Nordunet, Google, Amazon, Telia, Level +3 Communications and NetDNA.
+ + + +In the process, I came across the +web service GeoTraceRoute by +Salim Gasmi. Its methology of combining guesses based on DNS names, +various location databases and finally use latecy times to rule out +candidate locations seemed to do a very good job of guessing correct +geolocation. But it could only do one trace at the time, did not have +a sensor in Norway and did not make the geolocations easily available +for postprocessing. So I contacted the developer and asked if he +would be willing to share the code (he refused until he had time to +clean it up), but he was interested in providing the geolocations in a +machine readable format, and willing to set up a sensor in Norway. So +since yesterday, it is possible to run traces from Norway in this +service thanks to a sensor node set up by +the NUUG assosiation, and get the +trace in KML format for further processing.
+ + + +Here we can see a lot of trafic passes Sweden on its way to +Denmark, Germany, Holland and Ireland. Plenty of places where the +Snowden confirmations verified the traffic is read by various actors +without your best interest as their top priority.
+ +Combining KML files is trivial using a text editor, so I could loop +over all the hosts behind the urls imported by www.stortinget.no and +ask for the KML file from geotraceroute, and create a combined KML +file with all the traces (unfortunately only one of the IP addresses +behind the DNS name is traced this time. To get them all, one would +have to request traces using IP number instead of DNS names from +geotraceroute). That might be the next step in this project.
+ +Armed with these tools, I find it a lot easier to figure out where +the IP traffic moves and who control the boxes involved in moving it. +And every time the link crosses for example the Swedish border, we can +be sure Swedish Signal Intelligence (FRA) is listening, as GCHQ do in +Britain and NSA in USA and cables around the globe. (Hm, what should +we tell them? :) Keep that in mind if you ever send anything +unencrypted over the Internet.
+ +PS: KML files are drawn using +the KML viewer from Ivan +Rublev, as it was less cluttered than the local Linux application +Marble. There are heaps of other options too.
--/usr/share/battery-stats/battery-stats-graph-flow -- -
I'm not quite happy with the way the data is visualised, at least -when there are few data points. The graphs look a bit better with a -few years of data.
- -A while back one important feature I use in the battery stats -collector broke in Debian. The scripts in -/usr/lib/pm-utils/power.d/ were no longer executed. I -suspect it happened when Jessie started using systemd, but I do not -know. The issue is reported as bug #818649 against pm-utils. I -managed to work around it by adding an udev rule to call the collector -script every time the power connector is connected and disconnected. -With this fix in place it was finally time to make a new release of -the package, and get it into Debian.
- -If you are interested in how your laptop battery is doing, please -check out the -battery-stats -in Debian unstable, or rebuild it on Jessie to get it working on -Debian stable. :) The upstream source is available from -github. -As always, patches are very welcome.
+As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.
Back in 2013 I proposed -a -way to make paper and PDF invoices easier to process electronically by -adding a QR code with the key information about the invoice. I -suggested using vCard field definition, to get some standard format -for name and address, but any format would work. I did not do -anything about the proposal, but hoped someone one day would make -something like it. It would make it possible to efficiently send -machine readable invoices directly between seller and buyer.
- -This was the background when I came across a proposal and -specification from the web based accounting and invoicing supplier -Visma in Sweden called -UsingQR. Their PDF invoices contain -a QR code with the key information of the invoice in JSON format. -This is the typical content of a QR code following the UsingQR -specification (based on a real world example, some numbers replaced to -get a more bogus entry). I've reformatted the JSON to make it easier -to read. Normally this is all on one long line:
- --{ - "vh":500.00, - "vm":0, - "vl":0, - "uqr":1, - "tp":1, - "nme":"Din Leverandør", - "cc":"NO", - "cid":"997912345 MVA", - "iref":"12300001", - "idt":"20151022", - "ddt":"20151105", - "due":2500.0000, - "cur":"NOK", - "pt":"BBAN", - "acc":"17202612345", - "bc":"BIENNOK1", - "adr":"0313 OSLO" -} + +4th January 2017+Do you have a large iCalendar +file with lots of old entries, and would like to archive them to save +space and resources? At least those of us using KOrganizer know that +turning on and off an event set become slower and slower the more +entries are in the set. While working on migrating our calendars to a +Radicale CalDAV server on our +Freedombox server, my +loved one wondered if I could find a way to split up the calendar file +she had in KOrganizer, and I set out to write a tool. I spent a few +days writing and polishing the system, and it is now ready for general +consumption. The +code for +ical-archiver is publicly available from a git repository on +github. The system is written in Python and depend on +the vobject Python +module.
+ +To use it, locate the iCalendar file you want to operate on and +give it as an argument to the ical-archiver script. This will +generate a set of new files, one file per component type per year for +all components expiring more than two years in the past. The vevent, +vtodo and vjournal entries are handled by the script. The remaining +entries are stored in a 'remaining' file.
+ +This is what a test run can look like: + +
+% ical-archiver t/2004-2016.ics +Found 3612 vevents +Found 6 vtodos +Found 2 vjournals +Writing t/2004-2016.ics-subset-vevent-2004.ics +Writing t/2004-2016.ics-subset-vevent-2005.ics +Writing t/2004-2016.ics-subset-vevent-2006.ics +Writing t/2004-2016.ics-subset-vevent-2007.ics +Writing t/2004-2016.ics-subset-vevent-2008.ics +Writing t/2004-2016.ics-subset-vevent-2009.ics +Writing t/2004-2016.ics-subset-vevent-2010.ics +Writing t/2004-2016.ics-subset-vevent-2011.ics +Writing t/2004-2016.ics-subset-vevent-2012.ics +Writing t/2004-2016.ics-subset-vevent-2013.ics +Writing t/2004-2016.ics-subset-vevent-2014.ics +Writing t/2004-2016.ics-subset-vjournal-2007.ics +Writing t/2004-2016.ics-subset-vjournal-2011.ics +Writing t/2004-2016.ics-subset-vtodo-2012.ics +Writing t/2004-2016.ics-remaining.ics +%-The interpretation of the fields can be found in the -format -specification (revision 2 from june 2014). The format seem to -have most of the information needed to handle accounting and payment -of invoices, at least the fields I have needed so far here in -Norway. - -Unfortunately, the site and document do not mention anything about -the patent, trademark and copyright status of the format and the -specification. Because of this, I asked the people behind it back in -November to clarify. Ann-Christine Savlid (ann-christine.savlid (at) -visma.com) replied that Visma had not applied for patent or trademark -protection for this format, and that there were no copyright based -usage limitations for the format. I urged her to make sure this was -explicitly written on the web pages and in the specification, but -unfortunately this has not happened yet. So I guess if there is -submarine patents, hidden trademarks or a will to sue for copyright -infringements, those starting to use the UsingQR format might be at -risk, but if this happen there is some legal defense in the fact that -the people behind the format claimed it was safe to do so. At least -with patents, there is always -a -chance of getting sued...
- -I also asked if they planned to maintain the format in an -independent standard organization to give others more confidence that -they would participate in the standardization process on equal terms -with Visma, but they had no immediate plans for this. Their plan was -to work with banks to try to get more users of the format, and -evaluate the way forward if the format proved to be popular. I hope -they conclude that using an open standard organisation like -IETF is the correct place to -maintain such specification.
- -Update 2016-03-20: Via Twitter I became aware of -some comments -about this blog post that had several useful links and references to -similar systems. In the Czech republic, the Czech Banking Association -standard #26, with short name SPAYD, uses QR codes with payment -information. More information is available from the Wikipedia page on -Short -Payment Descriptor. And in Germany, there is a system named -BezahlCode, -(specification -v1.8 2013-12-05 available as PDF), which uses QR codes with -URL-like formatting using "bank:" as the URI schema/protocol to -provide the payment information. There is also the -ZUGFeRD -file format that perhaps could be transfered using QR codes, but I am -not sure if it is done already. Last, in Bolivia there are reports -that tax information since november 2014 need to be printed in QR -format on invoices. I have not been able to track down a -specification for this format, because of my limited language skill -sets.
+As you can see, the original file is untouched and new files are +written with names derived from the original file. If you are happy +with their content, the *-remaining.ics file can replace the original +the the others can be archived or imported as historical calendar +collections.
+ +The script should probably be improved a bit. The error handling +when discovering broken entries is not good, and I am not sure yet if +it make sense to split different entry types into separate files or +not. The program is thus likely to change. If you find it +interesting, please get in touch. :)
+ +As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.