While looking into how to look up Debian packages based on hardware
-information, to find the packages that support a given piece of
-hardware, I refreshed my memory regarding modalias values, and decided
-to document the details. Here are my findings so far, also available
-in
-the
-Debian Edu subversion repository:
-
-
Modalias decoded
-
-
This document try to explain what the different types of modalias
-values stands for. It is in part based on information from
-<URL: https://wiki.archlinux.org/index.php/Modalias >,
-<URL: http://unix.stackexchange.com/questions/26132/how-to-assign-usb-driver-to-device >,
-<URL: http://code.metager.de/source/history/linux/stable/scripts/mod/file2alias.c > and
-<URL: http://cvs.savannah.gnu.org/viewvc/dmidecode/dmidecode.c?root=dmidecode&view=markup >.
-
-
The modalias entries for a given Linux machine can be found using
-this shell script:
-
-
-find /sys -name modalias -print0 | xargs -0 cat | sort -u
-
-
-
The supported modalias globs for a given kernel module can be found
-using modinfo:
-
-
-% /sbin/modinfo psmouse | grep alias:
-alias: serio:ty05pr*id*ex*
-alias: serio:ty01pr*id*ex*
-%
-
-
-
PCI subtype
-
-
A typical PCI entry can look like this. This is an Intel Host
-Bridge memory controller:
-
-
-pci:v00008086d00002770sv00001028sd000001ADbc06sc00i00
-
-
-
This represent these values:
-
-
- v 00008086 (vendor)
- d 00002770 (device)
- sv 00001028 (subvendor)
- sd 000001AD (subdevice)
- bc 06 (bus class)
- sc 00 (bus subclass)
- i 00 (interface)
-
-
-
The vendor/device values are the same values outputted from 'lspci
--n' as 8086:2770. The bus class/subclass is also shown by lspci as
-0600. The 0600 class is a host bridge. Other useful bus values are
-0300 (VGA compatible card) and 0200 (Ethernet controller).
-
-
Not sure how to figure out the interface value, nor what it
-means.
-
-
USB subtype
-
-
Some typical USB entries can look like this. This is an internal
-USB hub in a laptop:
-
-
-usb:v1D6Bp0001d0206dc09dsc00dp00ic09isc00ip00
-
-
-
Here is the values included in this alias:
-
-
- v 1D6B (device vendor)
- p 0001 (device product)
- d 0206 (bcddevice)
- dc 09 (device class)
- dsc 00 (device subclass)
- dp 00 (device protocol)
- ic 09 (interface class)
- isc 00 (interface subclass)
- ip 00 (interface protocol)
-
-
-
The 0900 device class/subclass means hub. Some times the relevant
-class is in the interface class section. For a simple USB web camera,
-these alias entries show up:
-
-
-usb:v0AC8p3420d5000dcEFdsc02dp01ic01isc01ip00
-
usb:v0AC8p3420d5000dcEFdsc02dp01ic01isc02ip00
-
usb:v0AC8p3420d5000dcEFdsc02dp01ic0Eisc01ip00
-
usb:v0AC8p3420d5000dcEFdsc02dp01ic0Eisc02ip00
-
-
-
Interface class 0E01 is video control, 0E02 is video streaming (aka
-camera), 0101 is audio control device and 0102 is audio streaming (aka
-microphone). Thus this is a camera with microphone included.
-
-
ACPI subtype
-
-
The ACPI type is used for several non-PCI/USB stuff. This is an IR
-receiver in a Thinkpad X40:
-
-
-acpi:IBM0071:PNP0511:
-
-
-
The values between the colons are IDs.
-
-
DMI subtype
-
-
The DMI table contain lots of information about the computer case
-and model. This is an entry for a IBM Thinkpad X40, fetched from
-/sys/devices/virtual/dmi/id/modalias:
-
-
-dmi:bvnIBM:bvr1UETB6WW(1.66):bd06/15/2005:svnIBM:pn2371H4G:pvrThinkPadX40:rvnIBM:rn2371H4G:rvrNotAvailable:cvnIBM:ct10:cvrNotAvailable:
-
-
-
The values present are
-
-
- bvn IBM (BIOS vendor)
- bvr 1UETB6WW(1.66) (BIOS version)
- bd 06/15/2005 (BIOS date)
- svn IBM (system vendor)
- pn 2371H4G (product name)
- pvr ThinkPadX40 (product version)
- rvn IBM (board vendor)
- rn 2371H4G (board name)
- rvr NotAvailable (board version)
- cvn IBM (chassis vendor)
- ct 10 (chassis type)
- cvr NotAvailable (chassis version)
-
-
-
The chassis type 10 is Notebook. Other interesting values can be
-found in the dmidecode source:
-
-
- 3 Desktop
- 4 Low Profile Desktop
- 5 Pizza Box
- 6 Mini Tower
- 7 Tower
- 8 Portable
- 9 Laptop
- 10 Notebook
- 11 Hand Held
- 12 Docking Station
- 13 All In One
- 14 Sub Notebook
- 15 Space-saving
- 16 Lunch Box
- 17 Main Server Chassis
- 18 Expansion Chassis
- 19 Sub Chassis
- 20 Bus Expansion Chassis
- 21 Peripheral Chassis
- 22 RAID Chassis
- 23 Rack Mount Chassis
- 24 Sealed-case PC
- 25 Multi-system
- 26 CompactPCI
- 27 AdvancedTCA
- 28 Blade
- 29 Blade Enclosing
-
-
-
The chassis type values are not always accurately set in the DMI
-table. For example my home server is a tower, but the DMI modalias
-claim it is a desktop.
-
-
SerIO subtype
-
-
This type is used for PS/2 mouse plugs. One example is from my
-test machine:
-
-
-serio:ty01pr00id00ex00
-
-
-
The values present are
-
-
- ty 01 (type)
- pr 00 (prototype)
- id 00 (id)
- ex 00 (extra)
-
-
-
This type is supported by the psmouse driver. I am not sure what
-the valid values are.
-
-
Other subtypes
-
-
There are heaps of other modalias subtypes according to
-file2alias.c. There is the rest of the list from that source: amba,
-ap, bcma, ccw, css, eisa, hid, i2c, ieee1394, input, ipack, isapnp,
-mdio, of, parisc, pcmcia, platform, scsi, sdio, spi, ssb, vio, virtio,
-vmbus, x86cpu and zorro. I did not spend time documenting all of
-these, as they do not seem relevant for my intended use with mapping
-hardware to packages when new stuff is inserted during run time.
-
-
Looking up kernel modules using modalias values
-
-
To check which kernel modules provide support for a given modalias,
-one can use the following shell script:
-
-
- for id in $(find /sys -name modalias -print0 | xargs -0 cat | sort -u); do \
- echo "$id" ; \
- /sbin/modprobe --show-depends "$id"|sed 's/^/ /' ; \
- done
-
-
-
The output can look like this (only the first few entries as the
-list is very long on my test machine):
-
-
- acpi:ACPI0003:
- insmod /lib/modules/2.6.32-5-686/kernel/drivers/acpi/ac.ko
- acpi:device:
- FATAL: Module acpi:device: not found.
- acpi:IBM0068:
- insmod /lib/modules/2.6.32-5-686/kernel/drivers/char/nvram.ko
- insmod /lib/modules/2.6.32-5-686/kernel/drivers/leds/led-class.ko
- insmod /lib/modules/2.6.32-5-686/kernel/net/rfkill/rfkill.ko
- insmod /lib/modules/2.6.32-5-686/kernel/drivers/platform/x86/thinkpad_acpi.ko
- acpi:IBM0071:PNP0511:
- insmod /lib/modules/2.6.32-5-686/kernel/lib/crc-ccitt.ko
- insmod /lib/modules/2.6.32-5-686/kernel/net/irda/irda.ko
- insmod /lib/modules/2.6.32-5-686/kernel/drivers/net/irda/nsc-ircc.ko
- [...]
-
-
-
If you want to help implementing a system to let us propose what
-packages to install when new hardware is plugged into a Debian
-machine, please send me an email or talk to me on
-#debian-devel.
-
-
Update 2013-01-15: Rewrite "cat $(find ...)" to
-"find ... -print0 | xargs -0 cat" to make sure it handle directories
-in /sys/ with space in them.
+
+
9th April 2014
+
For a while now, I have been looking for a sensible offsite backup
+solution for use at home. My requirements are simple, it must be
+cheap and locally encrypted (in other words, I keep the encryption
+keys, the storage provider do not have access to my private files).
+One idea me and my friends had many years ago, before the cloud
+storage providers showed up, was to use Google mail as storage,
+writing a Linux block device storing blocks as emails in the mail
+service provided by Google, and thus get heaps of free space. On top
+of this one can add encryption, RAID and volume management to have
+lots of (fairly slow, I admit that) cheap and encrypted storage. But
+I never found time to implement such system. But the last few weeks I
+have looked at a system called
+S3QL, a locally
+mounted network backed file system with the features I need.
+
+
S3QL is a fuse file system with a local cache and cloud storage,
+handling several different storage providers, any with Amazon S3,
+Google Drive or OpenStack API. There are heaps of such storage
+providers. S3QL can also use a local directory as storage, which
+combined with sshfs allow for file storage on any ssh server. S3QL
+include support for encryption, compression, de-duplication, snapshots
+and immutable file systems, allowing me to mount the remote storage as
+a local mount point, look at and use the files as if they were local,
+while the content is stored in the cloud as well. This allow me to
+have a backup that should survive fire. The file system can not be
+shared between several machines at the same time, as only one can
+mount it at the time, but any machine with the encryption key and
+access to the storage service can mount it if it is unmounted.
+
+
It is simple to use. I'm using it on Debian Wheezy, where the
+package is included already. So to get started, run apt-get
+install s3ql. Next, pick a storage provider. I ended up picking
+Greenqloud, after reading their nice recipe on
+how
+to use S3QL with their Amazon S3 service, because I trust the laws
+in Iceland more than those in USA when it come to keeping my personal
+data safe and private, and thus would rather spend money on a company
+in Iceland. Another nice recipe is available from the article
+S3QL
+Filesystem for HPC Storage by Jeff Layton in the HPC section of
+Admin magazine. When the provider is picked, figure out how to get
+the API key needed to connect to the storage API. With Greencloud,
+the key did not show up until I had added payment details to my
+account.
+
+
Armed with the API access details, it is time to create the file
+system. First, create a new bucket in the cloud. This bucket is the
+file system storage area. I picked a bucket name reflecting the
+machine that was going to store data there, but any name will do.
+I'll refer to it as bucket-name below. In addition, one need
+the API login and password, and a locally created password. Store it
+all in ~root/.s3ql/authinfo2 like this:
+
+
+[s3c]
+storage-url: s3c://s.greenqloud.com:443/bucket-name
+backend-login: API-login
+backend-password: API-password
+fs-passphrase: local-password
+
+
+
I create my local passphrase using pwget 50 or similar,
+but any sensible way to create a fairly random password should do it.
+Armed with these details, it is now time to run mkfs, entering the API
+details and password to create it:
+
+
+# mkdir -m 700 /var/lib/s3ql-cache
+# mkfs.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+ --ssl s3c://s.greenqloud.com:443/bucket-name
+Enter backend login:
+Enter backend password:
+Before using S3QL, make sure to read the user's guide, especially
+the 'Important Rules to Avoid Loosing Data' section.
+Enter encryption password:
+Confirm encryption password:
+Generating random encryption key...
+Creating metadata tables...
+Dumping metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Compressing and uploading metadata...
+Wrote 0.00 MB of compressed metadata.
+#
+
+
The next step is mounting the file system to make the storage available.
+
+
+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+ --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+Using 4 upload threads.
+Downloading and decompressing metadata...
+Reading metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Mounting filesystem...
+# df -h /s3ql
+Filesystem Size Used Avail Use% Mounted on
+s3c://s.greenqloud.com:443/bucket-name 1.0T 0 1.0T 0% /s3ql
+#
+
+
+
The file system is now ready for use. I use rsync to store my
+backups in it, and as the metadata used by rsync is downloaded at
+mount time, no network traffic (and storage cost) is triggered by
+running rsync. To unmount, one should not use the normal umount
+command, as this will not flush the cache to the cloud storage, but
+instead running the umount.s3ql command like this:
+
+
+# umount.s3ql /s3ql
+#
+
+
+
There is a fsck command available to check the file system and
+correct any problems detected. This can be used if the local server
+crashes while the file system is mounted, to reset the "already
+mounted" flag. This is what it look like when processing a working
+file system:
+
+
+# fsck.s3ql --force --ssl s3c://s.greenqloud.com:443/bucket-name
+Using cached metadata.
+File system seems clean, checking anyway.
+Checking DB integrity...
+Creating temporary extra indices...
+Checking lost+found...
+Checking cached objects...
+Checking names (refcounts)...
+Checking contents (names)...
+Checking contents (inodes)...
+Checking contents (parent inodes)...
+Checking objects (reference counts)...
+Checking objects (backend)...
+..processed 5000 objects so far..
+..processed 10000 objects so far..
+..processed 15000 objects so far..
+Checking objects (sizes)...
+Checking blocks (referenced objects)...
+Checking blocks (refcounts)...
+Checking inode-block mapping (blocks)...
+Checking inode-block mapping (inodes)...
+Checking inodes (refcounts)...
+Checking inodes (sizes)...
+Checking extended attributes (names)...
+Checking extended attributes (inodes)...
+Checking symlinks (inodes)...
+Checking directory reachability...
+Checking unix conventions...
+Checking referential integrity...
+Dropping temporary indices...
+Backing up old metadata...
+Dumping metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Compressing and uploading metadata...
+Wrote 0.89 MB of compressed metadata.
+#
+
+
+
Thanks to the cache, working on files that fit in the cache is very
+quick, about the same speed as local file access. Uploading large
+amount of data is to me limited by the bandwidth out of and into my
+house. Uploading 685 MiB with a 100 MiB cache gave me 305 kiB/s,
+which is very close to my upload speed, and downloading the same
+Debian installation ISO gave me 610 kiB/s, close to my download speed.
+Both were measured using dd. So for me, the bottleneck is my
+network, not the file system code. I do not know what a good cache
+size would be, but suspect that the cache should e larger than your
+working set.
+
+
I mentioned that only one machine can mount the file system at the
+time. If another machine try, it is told that the file system is
+busy:
+
+
+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+ --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+Using 8 upload threads.
+Backend reports that fs is still mounted elsewhere, aborting.
+#
+
+
+
The file content is uploaded when the cache is full, while the
+metadata is uploaded once every 24 hour by default. To ensure the
+file system content is flushed to the cloud, one can either umount the
+file system, or ask S3QL to flush the cache and metadata using
+s3qlctrl:
+
+
+# s3qlctrl upload-meta /s3ql
+# s3qlctrl flushcache /s3ql
+#
+
+
+
If you are curious about how much space your data uses in the
+cloud, and how much compression and deduplication cut down on the
+storage usage, you can use s3qlstat on the mounted file system to get
+a report:
+
+
+# s3qlstat /s3ql
+Directory entries: 9141
+Inodes: 9143
+Data blocks: 8851
+Total data size: 22049.38 MB
+After de-duplication: 21955.46 MB (99.57% of total)
+After compression: 21877.28 MB (99.22% of total, 99.64% of de-duplicated)
+Database size: 2.39 MB (uncompressed)
+(some values do not take into account not-yet-uploaded dirty blocks in cache)
+#
+
+
+
I mentioned earlier that there are several possible suppliers of
+storage. I did not try to locate them all, but am aware of at least
+Greenqloud,
+Google Drive,
+Amazon S3 web serivces,
+Rackspace and
+Crowncloud. The latter even
+accept payment in Bitcoin. Pick one that suit your need. Some of
+them provide several GiB of free storage, but the prize models are
+quite different and you will have to figure out what suits you
+best.
+
+
While researching this blog post, I had a look at research papers
+and posters discussing the S3QL file system. There are several, which
+told me that the file system is getting a critical check by the
+science community and increased my confidence in using it. One nice
+poster is titled
+"An
+Innovative Parallel Cloud Storage System using OpenStackâs SwiftObject
+Store and Transformative Parallel I/O Approach" by Hsing-Bung
+Chen, Benjamin McClelland, David Sherrill, Alfred Torrez, Parks Fields
+and Pamela Smith. Please have a look.
+
+
Given my problems with different file systems earlier, I decided to
+check out the mounted S3QL file system to see if it would be usable as
+a home directory (in other word, that it provided POSIX semantics when
+it come to locking and umask handling etc). Running
+my
+test code to check file system semantics, I was happy to discover that
+no error was found. So the file system can be used for home
+directories, if one chooses to do so.
+
+
If you do not want a locally file system, and want something that
+work without the Linux fuse file system, I would like to mention the
+Tarsnap service, which also
+provide locally encrypted backup using a command line client. It have
+a nicer access control system, where one can split out read and write
+access, allowing some systems to write to the backup and others to
+only read from it.
+
+
As usual, if you use Bitcoin and want to show your support of my
+activities, please send Bitcoin donations to my address
+15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.
+
-
-
-
-
