X-Git-Url: https://pere.pagekite.me/gitweb/homepage.git/blobdiff_plain/848bb0953a36f422b0ca8bf2750e8b144aed48ae..d7229285e9b6e0c99611af5fc2c6f5a669e5826e:/blog/tags/isenkram/index.html diff --git a/blog/tags/isenkram/index.html b/blog/tags/isenkram/index.html index 6992c93513..fd9ebed2a1 100644 --- a/blog/tags/isenkram/index.html +++ b/blog/tags/isenkram/index.html @@ -20,6 +20,677 @@

Entries tagged "isenkram".

+
+
+ What is the most supported MIME type in Debian in 2018? +
+
+ 9th July 2018 +
+
+

Five years ago, +I +measured what the most supported MIME type in Debian was, by +analysing the desktop files in all packages in the archive. Since +then, the DEP-11 AppStream system has been put into production, making +the task a lot easier. This made me want to repeat the measurement, +to see how much things changed. Here are the new numbers, for +unstable only this time: + +

Debian Unstable:

+ +
+  count MIME type
+  ----- -----------------------
+     56 image/jpeg
+     55 image/png
+     49 image/tiff
+     48 image/gif
+     39 image/bmp
+     38 text/plain
+     37 audio/mpeg
+     34 application/ogg
+     33 audio/x-flac
+     32 audio/x-mp3
+     30 audio/x-wav
+     30 audio/x-vorbis+ogg
+     29 image/x-portable-pixmap
+     27 inode/directory
+     27 image/x-portable-bitmap
+     27 audio/x-mpeg
+     26 application/x-ogg
+     25 audio/x-mpegurl
+     25 audio/ogg
+     24 text/html
+
+ +

The list was created like this using a sid chroot: "cat +/var/lib/apt/lists/*sid*_dep11_Components-amd64.yml.gz| zcat | awk '/^ +- \S+\/\S+$/ {print $2 }' | sort | uniq -c | sort -nr | head -20"

+ +

It is interesting to see how image formats have passed text/plain +as the most announced supported MIME type. These days, thanks to the +AppStream system, if you run into a file format you do not know, and +want to figure out which packages support the format, you can find the +MIME type of the file using "file --mime <filename>", and then +look up all packages announcing support for this format in their +AppStream metadata (XML or .desktop file) using "appstreamcli +what-provides mimetype <mime-type>. For example if you, like +me, want to know which packages support inode/directory, you can get a +list like this:

+ +

+% appstreamcli what-provides mimetype inode/directory | grep Package: | sort
+Package: anjuta
+Package: audacious
+Package: baobab
+Package: cervisia
+Package: chirp
+Package: dolphin
+Package: doublecmd-common
+Package: easytag
+Package: enlightenment
+Package: ephoto
+Package: filelight
+Package: gwenview
+Package: k4dirstat
+Package: kaffeine
+Package: kdesvn
+Package: kid3
+Package: kid3-qt
+Package: nautilus
+Package: nemo
+Package: pcmanfm
+Package: pcmanfm-qt
+Package: qweborf
+Package: ranger
+Package: sirikali
+Package: spacefm
+Package: spacefm
+Package: vifm
+%
+

+ +

Using the same method, I can quickly discover that the Sketchup file +format is not yet supported by any package in Debian:

+ +

+% appstreamcli what-provides mimetype  application/vnd.sketchup.skp
+Could not find component providing 'mimetype::application/vnd.sketchup.skp'.
+%
+

+ +

Yesterday I used it to figure out which packages support the STL 3D +format:

+ +

+% appstreamcli what-provides mimetype  application/sla|grep Package
+Package: cura
+Package: meshlab
+Package: printrun
+%
+

+ +

PS: A new version of Cura was uploaded to Debian yesterday.

+ +

As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

+ +
+
+ + + Tags: debian, english, isenkram. + + +
+
+
+ +
+
+ Appstream just learned how to map hardware to packages too! +
+
+ 23rd December 2016 +
+
+

I received a very nice Christmas present today. As my regular +readers probably know, I have been working on the +the Isenkram +system for many years. The goal of the Isenkram system is to make +it easier for users to figure out what to install to get a given piece +of hardware to work in Debian, and a key part of this system is a way +to map hardware to packages. Isenkram have its own mapping database, +and also uses data provided by each package using the AppStream +metadata format. And today, +AppStream in +Debian learned to look up hardware the same way Isenkram is doing it, +ie using fnmatch():

+ +

+% appstreamcli what-provides modalias \
+  usb:v1130p0202d0100dc00dsc00dp00ic03isc00ip00in00
+Identifier: pymissile [generic]
+Name: pymissile
+Summary: Control original Striker USB Missile Launcher
+Package: pymissile
+% appstreamcli what-provides modalias usb:v0694p0002d0000
+Identifier: libnxt [generic]
+Name: libnxt
+Summary: utility library for talking to the LEGO Mindstorms NXT brick
+Package: libnxt
+---
+Identifier: t2n [generic]
+Name: t2n
+Summary: Simple command-line tool for Lego NXT
+Package: t2n
+---
+Identifier: python-nxt [generic]
+Name: python-nxt
+Summary: Python driver/interface/wrapper for the Lego Mindstorms NXT robot
+Package: python-nxt
+---
+Identifier: nbc [generic]
+Name: nbc
+Summary: C compiler for LEGO Mindstorms NXT bricks
+Package: nbc
+%
+

+ +

A similar query can be done using the combined AppStream and +Isenkram databases using the isenkram-lookup tool:

+ +

+% isenkram-lookup usb:v1130p0202d0100dc00dsc00dp00ic03isc00ip00in00
+pymissile
+% isenkram-lookup usb:v0694p0002d0000
+libnxt
+nbc
+python-nxt
+t2n
+%
+

+ +

You can find modalias values relevant for your machine using +cat $(find /sys/devices/ -name modalias). + +

If you want to make this system a success and help Debian users +make the most of the hardware they have, please +helpadd +AppStream metadata for your package following the guidelines +documented in the wiki. So far only 11 packages provide such +information, among the several hundred hardware specific packages in +Debian. The Isenkram database on the other hand contain 101 packages, +mostly related to USB dongles. Most of the packages with hardware +mapping in AppStream are LEGO Mindstorms related, because I have, as +part of my involvement in +the Debian LEGO +team given priority to making sure LEGO users get proposed the +complete set of packages in Debian for that particular hardware. The +team also got a nice Christmas present today. The +nxt-firmware +package made it into Debian. With this package in place, it is +now possible to use the LEGO Mindstorms NXT unit with only free +software, as the nxt-firmware package contain the source and firmware +binaries for the NXT brick.

+ +

As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

+ +
+
+ + + Tags: debian, english, isenkram. + + +
+
+
+ +
+
+ Isenkram updated with a lot more hardware-package mappings +
+
+ 20th December 2016 +
+
+

The Isenkram +system I wrote two years ago to make it easier in Debian to find +and install packages to get your hardware dongles to work, is still +going strong. It is a system to look up the hardware present on or +connected to the current system, and map the hardware to Debian +packages. It can either be done using the tools in isenkram-cli or +using the user space daemon in the isenkram package. The latter will +notify you, when inserting new hardware, about what packages to +install to get the dongle working. It will even provide a button to +click on to ask packagekit to install the packages.

+ +

Here is an command line example from my Thinkpad laptop:

+ +

+% isenkram-lookup  
+bluez
+cheese
+ethtool
+fprintd
+fprintd-demo
+gkrellm-thinkbat
+hdapsd
+libpam-fprintd
+pidgin-blinklight
+thinkfan
+tlp
+tp-smapi-dkms
+tp-smapi-source
+tpb
+%
+

+ +

It can also list the firware package providing firmware requested +by the load kernel modules, which in my case is an empty list because +I have all the firmware my machine need: + +

+% /usr/sbin/isenkram-autoinstall-firmware -l
+info: did not find any firmware files requested by loaded kernel modules.  exiting
+%
+

+ +

The last few days I had a look at several of the around 250 +packages in Debian with udev rules. These seem like good candidates +to install when a given hardware dongle is inserted, and I found +several that should be proposed by isenkram. I have not had time to +check all of them, but am happy to report that now there are 97 +packages packages mapped to hardware by Isenkram. 11 of these +packages provide hardware mapping using AppStream, while the rest are +listed in the modaliases file provided in isenkram.

+ +

These are the packages with hardware mappings at the moment. The +marked packages are also announcing their hardware +support using AppStream, for everyone to use:

+ +

air-quality-sensor, alsa-firmware-loaders, argyll, +array-info, avarice, avrdude, b43-fwcutter, +bit-babbler, bluez, bluez-firmware, brltty, +broadcom-sta-dkms, calibre, cgminer, cheese, colord, +colorhug-client, dahdi-firmware-nonfree, dahdi-linux, +dfu-util, dolphin-emu, ekeyd, ethtool, firmware-ipw2x00, fprintd, +fprintd-demo, galileo, gkrellm-thinkbat, gphoto2, +gpsbabel, gpsbabel-gui, gpsman, gpstrans, gqrx-sdr, gr-fcdproplus, +gr-osmosdr, gtkpod, hackrf, hdapsd, hdmi2usb-udev, hpijs-ppds, hplip, +ipw3945-source, ipw3945d, kde-config-tablet, kinect-audio-setup, +libnxt, libpam-fprintd, lomoco, +madwimax, minidisc-utils, mkgmap, msi-keyboard, mtkbabel, +nbc, nqc, nut-hal-drivers, ola, +open-vm-toolbox, open-vm-tools, openambit, pcgminer, pcmciautils, +pcscd, pidgin-blinklight, printer-driver-splix, +pymissile, python-nxt, qlandkartegt, +qlandkartegt-garmin, rosegarden, rt2x00-source, sispmctl, +soapysdr-module-hackrf, solaar, squeak-plugins-scratch, sunxi-tools, +t2n, thinkfan, thinkfinger-tools, tlp, tp-smapi-dkms, +tp-smapi-source, tpb, tucnak, uhd-host, usbmuxd, viking, +virtualbox-ose-guest-x11, w1retap, xawtv, xserver-xorg-input-vmmouse, +xserver-xorg-input-wacom, xserver-xorg-video-qxl, +xserver-xorg-video-vmware, yubikey-personalization and +zd1211-firmware

+ +

If you know of other packages, please let me know with a wishlist +bug report against the isenkram-cli package, and ask the package +maintainer to +add AppStream +metadata according to the guidelines to provide the information +for everyone. In time, I hope to get rid of the isenkram specific +hardware mapping and depend exclusively on AppStream.

+ +

Note, the AppStream metadata for broadcom-sta-dkms is matching too +much hardware, and suggest that the package with with any ethernet +card. See bug #838735 for +the details. I hope the maintainer find time to address it soon. In +the mean time I provide an override in isenkram.

+ +
+
+ + + Tags: debian, english, isenkram. + + +
+
+
+ +
+
+ Isenkram, Appstream and udev make life as a LEGO builder easier +
+
+ 7th October 2016 +
+
+

The Isenkram +system provide a practical and easy way to figure out which +packages support the hardware in a given machine. The command line +tool isenkram-lookup and the tasksel options provide a +convenient way to list and install packages relevant for the current +hardware during system installation, both user space packages and +firmware packages. The GUI background daemon on the other hand provide +a pop-up proposing to install packages when a new dongle is inserted +while using the computer. For example, if you plug in a smart card +reader, the system will ask if you want to install pcscd if +that package isn't already installed, and if you plug in a USB video +camera the system will ask if you want to install cheese if +cheese is currently missing. This already work just fine.

+ +

But Isenkram depend on a database mapping from hardware IDs to +package names. When I started no such database existed in Debian, so +I made my own data set and included it with the isenkram package and +made isenkram fetch the latest version of this database from git using +http. This way the isenkram users would get updated package proposals +as soon as I learned more about hardware related packages.

+ +

The hardware is identified using modalias strings. The modalias +design is from the Linux kernel where most hardware descriptors are +made available as a strings that can be matched using filename style +globbing. It handle USB, PCI, DMI and a lot of other hardware related +identifiers.

+ +

The downside to the Isenkram specific database is that there is no +information about relevant distribution / Debian version, making +isenkram propose obsolete packages too. But along came AppStream, a +cross distribution mechanism to store and collect metadata about +software packages. When I heard about the proposal, I contacted the +people involved and suggested to add a hardware matching rule using +modalias strings in the specification, to be able to use AppStream for +mapping hardware to packages. This idea was accepted and AppStream is +now a great way for a package to announce the hardware it support in a +distribution neutral way. I wrote +a +recipe on how to add such meta-information in a blog post last +December. If you have a hardware related package in Debian, please +announce the relevant hardware IDs using AppStream.

+ +

In Debian, almost all packages that can talk to a LEGO Mindestorms +RCX or NXT unit, announce this support using AppStream. The effect is +that when you insert such LEGO robot controller into your Debian +machine, Isenkram will propose to install the packages needed to get +it working. The intention is that this should allow the local user to +start programming his robot controller right away without having to +guess what packages to use or which permissions to fix.

+ +

But when I sat down with my son the other day to program our NXT +unit using his Debian Stretch computer, I discovered something +annoying. The local console user (ie my son) did not get access to +the USB device for programming the unit. This used to work, but no +longer in Jessie and Stretch. After some investigation and asking +around on #debian-devel, I discovered that this was because udev had +changed the mechanism used to grant access to local devices. The +ConsoleKit mechanism from /lib/udev/rules.d/70-udev-acl.rules +no longer applied, because LDAP users no longer was added to the +plugdev group during login. Michael Biebl told me that this method +was obsolete and the new method used ACLs instead. This was good +news, as the plugdev mechanism is a mess when using a remote user +directory like LDAP. Using ACLs would make sure a user lost device +access when she logged out, even if the user left behind a background +process which would retain the plugdev membership with the ConsoleKit +setup. Armed with this knowledge I moved on to fix the access problem +for the LEGO Mindstorms related packages.

+ +

The new system uses a udev tag, 'uaccess'. It can either be +applied directly for a device, or is applied in +/lib/udev/rules.d/70-uaccess.rules for classes of devices. As the +LEGO Mindstorms udev rules did not have a class, I decided to add the +tag directly in the udev rules files included in the packages. Here +is one example. For the nqc C compiler for the RCX, the +/lib/udev/rules.d/60-nqc.rules file now look like this: + +

+SUBSYSTEM=="usb", ACTION=="add", ATTR{idVendor}=="0694", ATTR{idProduct}=="0001", \
+    SYMLINK+="rcx-%k", TAG+="uaccess"
+

+ +

The key part is the 'TAG+="uaccess"' at the end. I suspect all +packages using plugdev in their /lib/udev/rules.d/ files should be +changed to use this tag (either directly or indirectly via +70-uaccess.rules). Perhaps a lintian check should be created +to detect this?

+ +

I've been unable to find good documentation on the uaccess feature. +It is unclear to me if the uaccess tag is an internal implementation +detail like the udev-acl tag used by +/lib/udev/rules.d/70-udev-acl.rules. If it is, I guess the +indirect method is the preferred way. Michael +asked for more +documentation from the systemd project and I hope it will make +this clearer. For now I use the generic classes when they exist and +is already handled by 70-uaccess.rules, and add the tag +directly if no such class exist.

+ +

To learn more about the isenkram system, please check out +my +blog posts tagged isenkram.

+ +

To help out making life for LEGO constructors in Debian easier, +please join us on our IRC channel +#debian-lego and join +the Debian +LEGO team in the Alioth project we created yesterday. A mailing +list is not yet created, but we are working on it. :)

+ +

As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

+ +
+
+ + + Tags: debian, english, isenkram, lego. + + +
+
+
+ +
+
+ Isenkram with PackageKit support - new version 0.23 available in Debian unstable +
+
+ 25th May 2016 +
+
+

The isenkram +system is a user-focused solution in Debian for handling hardware +related packages. The idea is to have a database of mappings between +hardware and packages, and pop up a dialog suggesting for the user to +install the packages to use a given hardware dongle. Some use cases +are when you insert a Yubikey, it proposes to install the software +needed to control it; when you insert a braille reader list it +proposes to install the packages needed to send text to the reader; +and when you insert a ColorHug screen calibrator it suggests to +install the driver for it. The system work well, and even have a few +command line tools to install firmware packages and packages for the +hardware already in the machine (as opposed to hotpluggable hardware).

+ +

The system was initially written using aptdaemon, because I found +good documentation and example code on how to use it. But aptdaemon +is going away and is generally being replaced by +PackageKit, +so Isenkram needed a rewrite. And today, thanks to the great patch +from my college Sunil Mohan Adapa in the FreedomBox project, the +rewrite finally took place. I've just uploaded a new version of +Isenkram into Debian Unstable with the patch included, and the default +for the background daemon is now to use PackageKit. To check it out, +install the isenkram package and insert some hardware dongle +and see if it is recognised.

+ +

If you want to know what kind of packages isenkram would propose for +the machine it is running on, you can check out the isenkram-lookup +program. This is what it look like on a Thinkpad X230:

+ +

+% isenkram-lookup 
+bluez
+cheese
+fprintd
+fprintd-demo
+gkrellm-thinkbat
+hdapsd
+libpam-fprintd
+pidgin-blinklight
+thinkfan
+tleds
+tp-smapi-dkms
+tp-smapi-source
+tpb
+%p
+

+ +

The hardware mappings come from several places. The preferred way +is for packages to announce their hardware support using +the +cross distribution appstream system. +See +previous +blog posts about isenkram to learn how to do that.

+ +
+
+ + + Tags: debian, english, isenkram. + + +
+
+
+ +
+
+ Using appstream with isenkram to install hardware related packages in Debian +
+
+ 20th December 2015 +
+
+

Around three years ago, I created +the isenkram +system to get a more practical solution in Debian for handing +hardware related packages. A GUI system in the isenkram package will +present a pop-up dialog when some hardware dongle supported by +relevant packages in Debian is inserted into the machine. The same +lookup mechanism to detect packages is available as command line +tools in the isenkram-cli package. In addition to mapping hardware, +it will also map kernel firmware files to packages and make it easy to +install needed firmware packages automatically. The key for this +system to work is a good way to map hardware to packages, in other +words, allow packages to announce what hardware they will work +with.

+ +

I started by providing data files in the isenkram source, and +adding code to download the latest version of these data files at run +time, to ensure every user had the most up to date mapping available. +I also added support for storing the mapping in the Packages file in +the apt repositories, but did not push this approach because while I +was trying to figure out how to best store hardware/package mappings, +the +appstream system was announced. I got in touch and suggested to +add the hardware mapping into that data set to be able to use +appstream as a data source, and this was accepted at least for the +Debian version of appstream.

+ +

A few days ago using appstream in Debian for this became possible, +and today I uploaded a new version 0.20 of isenkram adding support for +appstream as a data source for mapping hardware to packages. The only +package so far using appstream to announce its hardware support is my +pymissile package. I got help from Matthias Klumpp with figuring out +how do add the required +metadata +in pymissile. I added a file debian/pymissile.metainfo.xml with +this content:

+ +
+<?xml version="1.0" encoding="UTF-8"?>
+<component>
+  <id>pymissile</id>
+  <metadata_license>MIT</metadata_license>
+  <name>pymissile</name>
+  <summary>Control original Striker USB Missile Launcher</summary>
+  <description>
+    <p>
+      Pymissile provides a curses interface to control an original
+      Marks and Spencer / Striker USB Missile Launcher, as well as a
+      motion control script to allow a webcamera to control the
+      launcher.
+    </p>
+  </description>
+  <provides>
+    <modalias>usb:v1130p0202d*</modalias>
+  </provides>
+</component>
+
+ +

The key for isenkram is the component/provides/modalias value, +which is a glob style match rule for hardware specific strings +(modalias strings) provided by the Linux kernel. In this case, it +will map to all USB devices with vendor code 1130 and product code +0202.

+ +

Note, it is important that the license of all the metadata files +are compatible to have permissions to aggregate them into archive wide +appstream files. Matthias suggested to use MIT or BSD licenses for +these files. A challenge is figuring out a good id for the data, as +it is supposed to be globally unique and shared across distributions +(in other words, best to coordinate with upstream what to use). But +it can be changed later or, so we went with the package name as +upstream for this project is dormant.

+ +

To get the metadata file installed in the correct location for the +mirror update scripts to pick it up and include its content the +appstream data source, the file must be installed in the binary +package under /usr/share/appdata/. I did this by adding the following +line to debian/pymissile.install:

+ +
+debian/pymissile.metainfo.xml usr/share/appdata
+
+ +

With that in place, the command line tool isenkram-lookup will list +all packages useful on the current computer automatically, and the GUI +pop-up handler will propose to install the package not already +installed if a hardware dongle is inserted into the machine in +question.

+ +

Details of the modalias field in appstream is available from the +DEP-11 proposal.

+ +

To locate the modalias values of all hardware present in a machine, +try running this command on the command line:

+ +
+cat $(find /sys/devices/|grep modalias)
+
+ +

To learn more about the isenkram system, please check out +my +blog posts tagged isenkram.

+ +
+
+ + + Tags: debian, english, isenkram. + + +
+
+
+
Debian Jessie, PXE and automatic firmware installation @@ -1042,12 +1713,122 @@ please send me an email. :)

Archive