When I set out a few weeks ago to figure out -which -multimedia player in Debian claimed to support most file formats / -MIME types, I was a bit surprised how varied the sets of MIME types -the various players claimed support for. The range was from 55 to 130 -MIME types. I suspect most media formats are supported by all -players, but this is not really reflected in the MimeTypes values in -their desktop files. There are probably also some bogus MIME types -listed, but it is hard to identify which one this is.
- -Anyway, in the mean time I got in touch with upstream for some of -the players suggesting to add more MIME types to their desktop files, -and decided to spend some time myself improving the situation for my -favorite media player VLC. The fixes for VLC entered Debian unstable -yesterday. The complete list of MIME types can be seen on the -Multimedia -player MIME type support status Debian wiki page.
- -The new "best" multimedia player in Debian? It is VLC, followed by -totem, parole, kplayer, gnome-mpv, mpv, smplayer, mplayer-gui and -kmplayer. I am sure some of the other players desktop files support -several of the formats currently listed as working only with vlc, -toten and parole.
- -A sad observation is that only 14 MIME types are listed as -supported by all the tested multimedia players in Debian in their -desktop files: audio/mpeg, audio/vnd.rn-realaudio, audio/x-mpegurl, -audio/x-ms-wma, audio/x-scpls, audio/x-wav, video/mp4, video/mpeg, -video/quicktime, video/vnd.rn-realvideo, video/x-matroska, -video/x-ms-asf, video/x-ms-wmv and video/x-msvideo. Personally I find -it sad that video/ogg and video/webm is not supported by all the media -players in Debian. As far as I can tell, all of them can handle both -formats.
+ +Did you ever wonder where the web trafic really flow to reach the +web servers, and who own the network equipment it is flowing through? +It is possible to get a glimpse of this from using traceroute, but it +is hard to find all the details. Many years ago, I wrote a system to +map the Norwegian Internet (trying to figure out if our plans for a +network game service would get low enough latency, and who we needed +to talk to about setting up game servers close to the users. Back +then I used traceroute output from many locations (I asked my friends +to run a script and send me their traceroute output) to create the +graph and the map. The output from traceroute typically look like +this: + +
+traceroute to www.stortinget.no (85.88.67.10), 30 hops max, 60 byte packets + 1 uio-gw10.uio.no (129.240.202.1) 0.447 ms 0.486 ms 0.621 ms + 2 uio-gw8.uio.no (129.240.24.229) 0.467 ms 0.578 ms 0.675 ms + 3 oslo-gw1.uninett.no (128.39.65.17) 0.385 ms 0.373 ms 0.358 ms + 4 te3-1-2.br1.fn3.as2116.net (193.156.90.3) 1.174 ms 1.172 ms 1.153 ms + 5 he16-1-1.cr1.san110.as2116.net (195.0.244.234) 2.627 ms he16-1-1.cr2.oslosda310.as2116.net (195.0.244.48) 3.172 ms he16-1-1.cr1.san110.as2116.net (195.0.244.234) 2.857 ms + 6 ae1.ar8.oslosda310.as2116.net (195.0.242.39) 0.662 ms 0.637 ms ae0.ar8.oslosda310.as2116.net (195.0.242.23) 0.622 ms + 7 89.191.10.146 (89.191.10.146) 0.931 ms 0.917 ms 0.955 ms + 8 * * * + 9 * * * +[...] ++ +
This show the DNS names and IP addresses of (at least some of the) +network equipment involved in getting the data traffic from me to the +www.stortinget.no server, and how long it took in milliseconds for a +package to reach the equipment and return to me. Three packages are +sent, and some times the packages do not follow the same path. This +is shown for hop 5, where three different IP addresses replied to the +traceroute request.
+ +There are many ways to measure trace routes. Other good traceroute +implementations I use are traceroute (using ICMP packages) mtr (can do +both ICMP, UDP and TCP) and scapy (python library with ICMP, UDP, TCP +traceroute and a lot of other capabilities). All of them are easily +available in Debian.
+ +This time around, I wanted to know the geographic location of +different route points, to visualize how visiting a web page spread +information about the visit to a lot of servers around the globe. The +background is that a web site today often will ask the browser to get +from many servers the parts (for example HTML, JSON, fonts, +JavaScript, CSS, video) required to display the content. This will +leak information about the visit to those controlling these servers +and anyone able to peek at the data traffic passing by (like your ISP, +the ISPs backbone provider, FRA, GCHQ, NSA and others).
+ +Lets pick an example, the Norwegian parliament web site +www.stortinget.no. It is read daily by all members of parliament and +their staff, as well as political journalists, activits and many other +citizens of Norway. A visit to the www.stortinget.no web site will +ask your browser to contact 8 other servers: ajax.googleapis.com, +insights.hotjar.com, script.hotjar.com, static.hotjar.com, +stats.g.doubleclick.net, www.google-analytics.com, +www.googletagmanager.com and www.netigate.se. I extracted this by +asking PhantomJS to visit the +Stortinget web page and tell me all the URLs PhantomJS downloaded to +render the page (in HAR format using +their +netsniff example. I am very grateful to Gorm for showing me how +to do this). My goal is to visualize network traces to all IP +addresses behind these DNS names, do show where visitors personal +information is spread when visiting the page.
+ + + +When I had a look around for options, I could not find any good +free software tools to do this, and decided I needed my own traceroute +wrapper outputting KML based on locations looked up using GeoIP. KML +is easy to work with and easy to generate, and understood by several +of the GIS tools I have available. I got good help from by NUUG +colleague Anders Einar with this, and the result can be seen in +my +kmltraceroute git repository. Unfortunately, the quality of the +free GeoIP databases I could find (and the for-pay databases my +friends had access to) is not up to the task. The IP addresses of +central Internet infrastructure would typically be placed near the +controlling companies main office, and not where the router is really +located, as you can see from the +KML file I created using the GeoLite City dataset from MaxMind. + +
+ +I also had a look at the visual traceroute graph created by +the scrapy project, +showing IP network ownership (aka AS owner) for the IP address in +question. +The +graph display a lot of useful information about the traceroute in SVG +format, and give a good indication on who control the network +equipment involved, but it do not include geolocation. This graph +make it possible to see the information is made available at least for +UNINETT, Catchcom, Stortinget, Nordunet, Google, Amazon, Telia, Level +3 Communications and NetDNA.
+ + + +In the process, I came across the +web service GeoTraceRoute by +Salim Gasmi. Its methology of combining guesses based on DNS names, +various location databases and finally use latecy times to rule out +candidate locations seemed to do a very good job of guessing correct +geolocation. But it could only do one trace at the time, did not have +a sensor in Norway and did not make the geolocations easily available +for postprocessing. So I contacted the developer and asked if he +would be willing to share the code (he refused until he had time to +clean it up), but he was interested in providing the geolocations in a +machine readable format, and willing to set up a sensor in Norway. So +since yesterday, it is possible to run traces from Norway in this +service thanks to a sensor node set up by +the NUUG assosiation, and get the +trace in KML format for further processing.
+ + + +Here we can see a lot of trafic passes Sweden on its way to +Denmark, Germany, Holland and Ireland. Plenty of places where the +Snowden confirmations verified the traffic is read by various actors +without your best interest as their top priority.
+ +Combining KML files is trivial using a text editor, so I could loop +over all the hosts behind the urls imported by www.stortinget.no and +ask for the KML file from geotraceroute, and create a combined KML +file with all the traces (unfortunately only one of the IP addresses +behind the DNS name is traced this time. To get them all, one would +have to request traces using IP number instead of DNS names from +geotraceroute). That might be the next step in this project.
+ +Armed with these tools, I find it a lot easier to figure out where +the IP traffic moves and who control the boxes involved in moving it. +And every time the link crosses for example the Swedish border, we can +be sure Swedish Signal Intelligence (FRA) is listening, as GCHQ do in +Britain and NSA in USA and cables around the globe. (Hm, what should +we tell them? :) Keep that in mind if you ever send anything +unencrypted over the Internet.
+ +PS: KML files are drawn using +the KML viewer from Ivan +Rublev, as it was less cluttered than the local Linux application +Marble. There are heaps of other options too.
+ +As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.
Many years ago, when koffice was fresh and with few users, I -decided to test its presentation tool when making the slides for a -talk I was giving for NUUG on Japhar, a free Java virtual machine. I -wrote the first draft of the slides, saved the result and went to bed -the day before I would give the talk. The next day I took a plane to -the location where the meeting should take place, and on the plane I -started up koffice again to polish the talk a bit, only to discover -that kpresenter refused to load its own data file. I cursed a bit and -started making the slides again from memory, to have something to -present when I arrived. I tested that the saved files could be -loaded, and the day seemed to be rescued. I continued to polish the -slides until I suddenly discovered that the saved file could no longer -be loaded into kpresenter. In the end I had to rewrite the slides -three times, condensing the content until the talk became shorter and -shorter. After the talk I was able to pinpoint the problem – -kpresenter wrote inline images in a way itself could not understand. -Eventually that bug was fixed and kpresenter ended up being a great -program to make slides. The point I'm trying to make is that we -expect a program to be able to load its own data files, and it is -embarrassing to its developers if it can't.
- -Did you ever experience a program failing to load its own data -files from the desktop file browser? It is not a uncommon problem. A -while back I discovered that the screencast recorder -gtk-recordmydesktop would save an Ogg Theora video file the KDE file -browser would refuse to open. No video player claimed to understand -such file. I tracked down the cause being file --mime-type -returning the application/ogg MIME type, which no video player I had -installed listed as a MIME type they would understand. I asked for -file to change its -behavour and use the MIME type video/ogg instead. I also asked -several video players to add video/ogg to their desktop files, to give -the file browser an idea what to do about Ogg Theora files. After a -while, the desktop file browsers in Debian started to handle the -output from gtk-recordmydesktop properly.
- -But history repeats itself. A few days ago I tested the music -system Rosegarden again, and I discovered that the KDE and xfce file -browsers did not know what to do with the Rosegarden project files -(*.rg). I've reported the -rosegarden problem to BTS and a fix is commited to git and will be -included in the next upload. To increase the chance of me remembering -how to fix the problem next time some program fail to load its files -from the file browser, here are some notes on how to fix it.
- -The file browsers in Debian in general operates on MIME types. -There are two sources for the MIME type of a given file. The output from -file --mime-type mentioned above, and the content of the -shared MIME type registry (under /usr/share/mime/). The file MIME -type is mapped to programs supporting the MIME type, and this -information is collected from -the -desktop files available in /usr/share/applications/. If there is -one desktop file claiming support for the MIME type of the file, it is -activated when asking to open a given file. If there are more, one -can normally select which one to use by right-clicking on the file and -selecting the wanted one using 'Open with' or similar. In general -this work well. But it depend on each program picking a good MIME -type (preferably -a -MIME type registered with IANA), file and/or the shared MIME -registry recognizing the file and the desktop file to list the MIME -type in its list of supported MIME types.
- -The /usr/share/mime/packages/rosegarden.xml entry for -the -Shared MIME database look like this:
- -- --<?xml version="1.0" encoding="UTF-8"?> -<mime-info xmlns="http://www.freedesktop.org/standards/shared-mime-info"> - <mime-type type="audio/x-rosegarden"> - <sub-class-of type="application/x-gzip"/> - <comment>Rosegarden project file</comment> - <glob pattern="*.rg"/> - </mime-type> -</mime-info> -
This states that audio/x-rosegarden is a kind of application/x-gzip -(it is a gzipped XML file). Note, it is much better to use an -official MIME type registered with IANA than it is to make up ones own -unofficial ones like the x-rosegarden type used by rosegarden.
- -The desktop file of the rosegarden program failed to list -audio/x-rosegarden in its list of supported MIME types, causing the -file browsers to have no idea what to do with *.rg files:
- --% grep Mime /usr/share/applications/rosegarden.desktop -MimeType=audio/x-rosegarden-composition;audio/x-rosegarden-device;audio/x-rosegarden-project;audio/x-rosegarden-template;audio/midi; -X-KDE-NativeMimeType=audio/x-rosegarden-composition + +4th January 2017+@@ -186,36 +263,96 @@ fixed. :)Do you have a large iCalendar +file with lots of old entries, and would like to archive them to save +space and resources? At least those of us using KOrganizer know that +turning on and off an event set become slower and slower the more +entries are in the set. While working on migrating our calendars to a +Radicale CalDAV server on our +Freedombox server, my +loved one wondered if I could find a way to split up the calendar file +she had in KOrganizer, and I set out to write a tool. I spent a few +days writing and polishing the system, and it is now ready for general +consumption. The +code for +ical-archiver is publicly available from a git repository on +github. The system is written in Python and depend on +the vobject Python +module.
+ +To use it, locate the iCalendar file you want to operate on and +give it as an argument to the ical-archiver script. This will +generate a set of new files, one file per component type per year for +all components expiring more than two years in the past. The vevent, +vtodo and vjournal entries are handled by the script. The remaining +entries are stored in a 'remaining' file.
+ +This is what a test run can look like: + +
+% ical-archiver t/2004-2016.ics +Found 3612 vevents +Found 6 vtodos +Found 2 vjournals +Writing t/2004-2016.ics-subset-vevent-2004.ics +Writing t/2004-2016.ics-subset-vevent-2005.ics +Writing t/2004-2016.ics-subset-vevent-2006.ics +Writing t/2004-2016.ics-subset-vevent-2007.ics +Writing t/2004-2016.ics-subset-vevent-2008.ics +Writing t/2004-2016.ics-subset-vevent-2009.ics +Writing t/2004-2016.ics-subset-vevent-2010.ics +Writing t/2004-2016.ics-subset-vevent-2011.ics +Writing t/2004-2016.ics-subset-vevent-2012.ics +Writing t/2004-2016.ics-subset-vevent-2013.ics +Writing t/2004-2016.ics-subset-vevent-2014.ics +Writing t/2004-2016.ics-subset-vjournal-2007.ics +Writing t/2004-2016.ics-subset-vjournal-2011.ics +Writing t/2004-2016.ics-subset-vtodo-2012.ics +Writing t/2004-2016.ics-remaining.ics % -- -The fix was to add "audio/x-rosegarden;" at the end of the -MimeType= line.
- -If you run into a file which fail to open the correct program when -selected from the file browser, please check out the output from -file --mime-type for the file, ensure the file ending and -MIME type is registered somewhere under /usr/share/mime/ and check -that some desktop file under /usr/share/applications/ is claiming -support for this MIME type. If not, please report a bug to have it -fixed. :)
+ + +As you can see, the original file is untouched and new files are +written with names derived from the original file. If you are happy +with their content, the *-remaining.ics file can replace the original +the the others can be archived or imported as historical calendar +collections.
+ +The script should probably be improved a bit. The error handling +when discovering broken entries is not good, and I am not sure yet if +it make sense to split different entry types into separate files or +not. The program is thus likely to change. If you find it +interesting, please get in touch. :)
+ +As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.
- -28th May 2016-A little more than 11 years ago, one of the creators of Tor, and -the current President of the Tor -project, Roger Dingledine, gave a talk for the members of the -Norwegian Unix User group (NUUG). A -video of the talk was recorded, and today, thanks to the great help -from David Noble, I finally was able to publish the video of the talk -on Frikanalen, the Norwegian open channel TV station where NUUG -currently publishes its talks. You can -watch the live stream using a web -browser with WebM support, or check out the recording on the video -on demand page for the talk -"Tor: Anonymous -communication for the US Department of Defence...and you.".
- -Here is the video included for those of you using browsers with -HTML video and Ogg Theora support:
- - - -I guess the gist of the talk can be summarised quite simply: If you -want to help the military in USA (and everyone else), use Tor. :)
+ +23rd December 2016+@@ -223,41 +360,26 @@ want to help the military in USA (and everyone else), use Tor. :)I received a very nice Christmas present today. As my regular +readers probably know, I have been working on the +the Isenkram +system for many years. The goal of the Isenkram system is to make +it easier for users to figure out what to install to get a given piece +of hardware to work in Debian, and a key part of this system is a way +to map hardware to packages. Isenkram have its own mapping database, +and also uses data provided by each package using the AppStream +metadata format. And today, +AppStream in +Debian learned to look up hardware the same way Isenkram is doing it, +ie using fnmatch():
+ ++% appstreamcli what-provides modalias \ + usb:v1130p0202d0100dc00dsc00dp00ic03isc00ip00in00 +Identifier: pymissile [generic] +Name: pymissile +Summary: Control original Striker USB Missile Launcher +Package: pymissile +% appstreamcli what-provides modalias usb:v0694p0002d0000 +Identifier: libnxt [generic] +Name: libnxt +Summary: utility library for talking to the LEGO Mindstorms NXT brick +Package: libnxt +--- +Identifier: t2n [generic] +Name: t2n +Summary: Simple command-line tool for Lego NXT +Package: t2n +--- +Identifier: python-nxt [generic] +Name: python-nxt +Summary: Python driver/interface/wrapper for the Lego Mindstorms NXT robot +Package: python-nxt +--- +Identifier: nbc [generic] +Name: nbc +Summary: C compiler for LEGO Mindstorms NXT bricks +Package: nbc +% ++ +A similar query can be done using the combined AppStream and +Isenkram databases using the isenkram-lookup tool:
+ ++% isenkram-lookup usb:v1130p0202d0100dc00dsc00dp00ic03isc00ip00in00 +pymissile +% isenkram-lookup usb:v0694p0002d0000 +libnxt +nbc +python-nxt +t2n +% ++ +You can find modalias values relevant for your machine using +cat $(find /sys/devices/ -name modalias). + +
If you want to make this system a success and help Debian users +make the most of the hardware they have, please +helpadd +AppStream metadata for your package following the guidelines +documented in the wiki. So far only 11 packages provide such +information, among the several hundred hardware specific packages in +Debian. The Isenkram database on the other hand contain 101 packages, +mostly related to USB dongles. Most of the packages with hardware +mapping in AppStream are LEGO Mindstorms related, because I have, as +part of my involvement in +the Debian LEGO +team given priority to making sure LEGO users get proposed the +complete set of packages in Debian for that particular hardware. The +team also got a nice Christmas present today. The +nxt-firmware +package made it into Debian. With this package in place, it is +now possible to use the LEGO Mindstorms NXT unit with only free +software, as the nxt-firmware package contain the source and firmware +binaries for the NXT brick.
+ +As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.
- -25th May 2016-The isenkram -system is a user-focused solution in Debian for handling hardware -related packages. The idea is to have a database of mappings between -hardware and packages, and pop up a dialog suggesting for the user to -install the packages to use a given hardware dongle. Some use cases -are when you insert a Yubikey, it proposes to install the software -needed to control it; when you insert a braille reader list it -proposes to install the packages needed to send text to the reader; -and when you insert a ColorHug screen calibrator it suggests to -install the driver for it. The system work well, and even have a few -command line tools to install firmware packages and packages for the -hardware already in the machine (as opposed to hotpluggable hardware).
- -The system was initially written using aptdaemon, because I found -good documentation and example code on how to use it. But aptdaemon -is going away and is generally being replaced by -PackageKit, -so Isenkram needed a rewrite. And today, thanks to the great patch -from my college Sunil Mohan Adapa in the FreedomBox project, the -rewrite finally took place. I've just uploaded a new version of -Isenkram into Debian Unstable with the patch included, and the default -for the background daemon is now to use PackageKit. To check it out, -install the isenkram package and insert some hardware dongle -and see if it is recognised.
- -If you want to know what kind of packages isenkram would propose for -the machine it is running on, you can check out the isenkram-lookup -program. This is what it look like on a Thinkpad X230:
- --% isenkram-lookup + +20th December 2016+The Isenkram +system I wrote two years ago to make it easier in Debian to find +and install packages to get your hardware dongles to work, is still +going strong. It is a system to look up the hardware present on or +connected to the current system, and map the hardware to Debian +packages. It can either be done using the tools in isenkram-cli or +using the user space daemon in the isenkram package. The latter will +notify you, when inserting new hardware, about what packages to +install to get the dongle working. It will even provide a button to +click on to ask packagekit to install the packages.
+ +Here is an command line example from my Thinkpad laptop:
+ ++% isenkram-lookup bluez cheese +ethtool fprintd fprintd-demo gkrellm-thinkbat @@ -265,20 +387,74 @@ hdapsd libpam-fprintd pidgin-blinklight thinkfan -tleds +tlp tp-smapi-dkms tp-smapi-source tpb -%p -- -The hardware mappings come from several places. The preferred way -is for packages to announce their hardware support using -the -cross distribution appstream system. -See -previous -blog posts about isenkram to learn how to do that.
+% + + +It can also list the firware package providing firmware requested +by the load kernel modules, which in my case is an empty list because +I have all the firmware my machine need: + +
+% /usr/sbin/isenkram-autoinstall-firmware -l +info: did not find any firmware files requested by loaded kernel modules. exiting +% ++ +The last few days I had a look at several of the around 250 +packages in Debian with udev rules. These seem like good candidates +to install when a given hardware dongle is inserted, and I found +several that should be proposed by isenkram. I have not had time to +check all of them, but am happy to report that now there are 97 +packages packages mapped to hardware by Isenkram. 11 of these +packages provide hardware mapping using AppStream, while the rest are +listed in the modaliases file provided in isenkram.
+ +These are the packages with hardware mappings at the moment. The +marked packages are also announcing their hardware +support using AppStream, for everyone to use:
+ +air-quality-sensor, alsa-firmware-loaders, argyll, +array-info, avarice, avrdude, b43-fwcutter, +bit-babbler, bluez, bluez-firmware, brltty, +broadcom-sta-dkms, calibre, cgminer, cheese, colord, +colorhug-client, dahdi-firmware-nonfree, dahdi-linux, +dfu-util, dolphin-emu, ekeyd, ethtool, firmware-ipw2x00, fprintd, +fprintd-demo, galileo, gkrellm-thinkbat, gphoto2, +gpsbabel, gpsbabel-gui, gpsman, gpstrans, gqrx-sdr, gr-fcdproplus, +gr-osmosdr, gtkpod, hackrf, hdapsd, hdmi2usb-udev, hpijs-ppds, hplip, +ipw3945-source, ipw3945d, kde-config-tablet, kinect-audio-setup, +libnxt, libpam-fprintd, lomoco, +madwimax, minidisc-utils, mkgmap, msi-keyboard, mtkbabel, +nbc, nqc, nut-hal-drivers, ola, +open-vm-toolbox, open-vm-tools, openambit, pcgminer, pcmciautils, +pcscd, pidgin-blinklight, printer-driver-splix, +pymissile, python-nxt, qlandkartegt, +qlandkartegt-garmin, rosegarden, rt2x00-source, sispmctl, +soapysdr-module-hackrf, solaar, squeak-plugins-scratch, sunxi-tools, +t2n, thinkfan, thinkfinger-tools, tlp, tp-smapi-dkms, +tp-smapi-source, tpb, tucnak, uhd-host, usbmuxd, viking, +virtualbox-ose-guest-x11, w1retap, xawtv, xserver-xorg-input-vmmouse, +xserver-xorg-input-wacom, xserver-xorg-video-qxl, +xserver-xorg-video-vmware, yubikey-personalization and +zd1211-firmware
+ +If you know of other packages, please let me know with a wishlist +bug report against the isenkram-cli package, and ask the package +maintainer to +add AppStream +metadata according to the guidelines to provide the information +for everyone. In time, I hope to get rid of the isenkram specific +hardware mapping and depend exclusively on AppStream.
+ +Note, the AppStream metadata for broadcom-sta-dkms is matching too +much hardware, and suggest that the package with with any ethernet +card. See bug #838735 for +the details. I hope the maintainer find time to address it soon. In +the mean time I provide an override in isenkram.