Petter Reinholdtsen

Fetching trusted timestamps using the rfc3161ng python module
8th October 2018

I have earlier covered the basics of trusted timestamping using the 'openssl ts' client. See blog post for 2014, 2016 and 2017 for those stories. But some times I want to integrate the timestamping in other code, and recently I needed to integrate it into Python. After searching a bit, I found the rfc3161 library which seemed like a good fit, but I soon discovered it only worked for python version 2, and I needed something that work with python version 3. Luckily I next came across the rfc3161ng library, a fork of the original rfc3161 library. Not only is it working with python 3, it have fixed a few of the bugs in the original library, and it has an active maintainer. I decided to wrap it up and make it available in Debian, and a few days ago it entered Debian unstable and testing.

Using the library is fairly straight forward. The only slightly problematic step is to fetch the required certificates to verify the timestamp. For some services it is straight forward, while for others I have not yet figured out how to do it. Here is a small standalone code example based on of the integration tests in the library code:

#!/usr/bin/python3

"""

Python 3 script demonstrating how to use the rfc3161ng module to
get trusted timestamps.

The license of this code is the same as the license of the rfc3161ng
library, ie MIT/BSD.

"""

import os
import pyasn1.codec.der
import rfc3161ng
import subprocess
import tempfile
import urllib.request

def store(f, data):
    f.write(data)
    f.flush()
    f.seek(0)

def fetch(url, f=None):
    response = urllib.request.urlopen(url)
    data = response.read()
    if f:
        store(f, data)
    return data

def main():
    with tempfile.NamedTemporaryFile() as cert_f,\
    	 tempfile.NamedTemporaryFile() as ca_f,\
    	 tempfile.NamedTemporaryFile() as msg_f,\
    	 tempfile.NamedTemporaryFile() as tsr_f:

        # First fetch certificates used by service
        certificate_data = fetch('https://freetsa.org/files/tsa.crt', cert_f)
        ca_data_data = fetch('https://freetsa.org/files/cacert.pem', ca_f)

        # Then timestamp the message
        timestamper = \
            rfc3161ng.RemoteTimestamper('http://freetsa.org/tsr',
                                        certificate=certificate_data)
        data = b"Python forever!\n"
        tsr = timestamper(data=data, return_tsr=True)

        # Finally, convert message and response to something 'openssl ts' can verify
        store(msg_f, data)
        store(tsr_f, pyasn1.codec.der.encoder.encode(tsr))
        args = ["openssl", "ts", "-verify",
                "-data", msg_f.name,
	        "-in", tsr_f.name,
		"-CAfile", ca_f.name,
                "-untrusted", cert_f.name]
        subprocess.check_call(args)

if '__main__' == __name__:
   main()

The code fetches the required certificates, store them as temporary files, timestamp a simple message, store the message and timestamp to disk and ask 'openssl ts' to verify the timestamp. A timestamp is around 1.5 kiB in size, and should be fairly easy to store for future use.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Tags: english, sikkerhet.
Automatic Google Drive sync using grive in Debian
4th October 2018

A few days, I rescued a Windows victim over to Debian. To try to rescue the remains, I helped set up automatic sync with Google Drive. I did not find any sensible Debian package handling this automatically, so I rebuild the grive2 source from the Ubuntu UPD8 PPA to do the task and added a autostart desktop entry and a small shell script to run in the background while the user is logged in to do the sync. Here is a sketch of the setup for future reference.

I first created ~/googledrive, entered the directory and ran 'grive -a' to authenticate the machine/user. Next, I created a autostart hook in ~/.config/autostart/grive.desktop to start the sync when the user log in:

[Desktop Entry]
Name=Google drive autosync
Type=Application
Exec=/home/user/bin/grive-sync

Finally, I wrote the ~/bin/grive-sync script to sync ~/googledrive/ with the files in Google Drive.

#!/bin/sh
set -e
cd ~/
cleanup() {
    if [ "$syncpid" ] ; then
        kill $syncpid
    fi
}
trap cleanup EXIT INT QUIT
/usr/lib/grive/grive-sync.sh listen googledrive 2>&1 | sed "s%^%$0:%" &
syncpdi=$!
while true; do
    if ! xhost >/dev/null 2>&1 ; then
        echo "no DISPLAY, exiting as the user probably logged out"
        exit 1
    fi
    if [ ! -e /run/user/1000/grive-sync.sh_googledrive ] ; then
        /usr/lib/grive/grive-sync.sh sync googledrive
    fi
    sleep 300
done 2>&1 | sed "s%^%$0:%"

Feel free to use the setup if you want. It can be assumed to be GNU GPL v2 licensed (or any later version, at your leisure), but I doubt this code is possible to claim copyright on.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Tags: debian, english.
Valutakrambod - A python and bitcoin love story
29th September 2018

It would come as no surprise to anyone that I am interested in bitcoins and virtual currencies. I've been keeping an eye on virtual currencies for many years, and it is part of the reason a few months ago, I started writing a python library for collecting currency exchange rates and trade on virtual currency exchanges. I decided to name the end result valutakrambod, which perhaps can be translated to small currency shop.

The library uses the tornado python library to handle HTTP and websocket connections, and provide a asynchronous system for connecting to and tracking several services. The code is available from github.

There are two example clients of the library. One is very simple and list every updated buy/sell price received from the various services. This code is started by running bin/btc-rates and call the client code in valutakrambod/client.py. The simple client look like this:

import functools
import tornado.ioloop
import valutakrambod
class SimpleClient(object):
    def __init__(self):
        self.services = []
        self.streams = []
        pass
    def newdata(self, service, pair, changed):
        print("%-15s %s-%s: %8.3f %8.3f" % (
            service.servicename(),
            pair[0],
            pair[1],
            service.rates[pair]['ask'],
            service.rates[pair]['bid'])
        )
    async def refresh(self, service):
        await service.fetchRates(service.wantedpairs)
    def run(self):
        self.ioloop = tornado.ioloop.IOLoop.current()
        self.services = valutakrambod.service.knownServices()
        for e in self.services:
            service = e()
            service.subscribe(self.newdata)
            stream = service.websocket()
            if stream:
                self.streams.append(stream)
            else:
                # Fetch information from non-streaming services immediately
                self.ioloop.call_later(len(self.services),
                                       functools.partial(self.refresh, service))
                # as well as regularly
                service.periodicUpdate(60)
        for stream in self.streams:
            stream.connect()
        try:
            self.ioloop.start()
        except KeyboardInterrupt:
            print("Interrupted by keyboard, closing all connections.")
            pass
        for stream in self.streams:
            stream.close()

The library client loops over all known "public" services, initialises it, subscribes to any updates from the service, checks and activates websocket streaming if the service provide it, and if no streaming is supported, fetches information from the service and sets up a periodic update every 60 seconds. The output from this client can look like this:

Bl3p            BTC-EUR: 5687.110 5653.690
Bl3p            BTC-EUR: 5687.110 5653.690
Bl3p            BTC-EUR: 5687.110 5653.690
Hitbtc          BTC-USD: 6594.560 6593.690
Hitbtc          BTC-USD: 6594.560 6593.690
Bl3p            BTC-EUR: 5687.110 5653.690
Hitbtc          BTC-USD: 6594.570 6593.690
Bitstamp        EUR-USD:    1.159    1.154
Hitbtc          BTC-USD: 6594.570 6593.690
Hitbtc          BTC-USD: 6594.580 6593.690
Hitbtc          BTC-USD: 6594.580 6593.690
Hitbtc          BTC-USD: 6594.580 6593.690
Bl3p            BTC-EUR: 5687.110 5653.690
Paymium         BTC-EUR: 5680.000 5620.240

The exchange order book is tracked in addition to the best buy/sell price, for those that need to know the details.

The other example client is focusing on providing a curses view with updated buy/sell prices as soon as they are received from the services. This code is located in bin/btc-rates-curses and activated by using the '-c' argument. Without the argument the "curses" output is printed without using curses, which is useful for debugging. The curses view look like this:

           Name Pair   Bid         Ask         Spr    Ftcd    Age
 BitcoinsNorway BTCEUR   5591.8400   5711.0800   2.1%   16    nan     60
       Bitfinex BTCEUR   5671.0000   5671.2000   0.0%   16     22     59
        Bitmynt BTCEUR   5580.8000   5807.5200   3.9%   16     41     60
         Bitpay BTCEUR   5663.2700         nan   nan%   15    nan     60
       Bitstamp BTCEUR   5664.8400   5676.5300   0.2%    0      1      1
           Bl3p BTCEUR   5653.6900   5684.9400   0.5%    0    nan     19
       Coinbase BTCEUR   5600.8200   5714.9000   2.0%   15    nan    nan
         Kraken BTCEUR   5670.1000   5670.2000   0.0%   14     17     60
        Paymium BTCEUR   5620.0600   5680.0000   1.1%    1   7515    nan
 BitcoinsNorway BTCNOK  52898.9700  54034.6100   2.1%   16    nan     60
        Bitmynt BTCNOK  52960.3200  54031.1900   2.0%   16     41     60
         Bitpay BTCNOK  53477.7833         nan   nan%   16    nan     60
       Coinbase BTCNOK  52990.3500  54063.0600   2.0%   15    nan    nan
        MiraiEx BTCNOK  52856.5300  54100.6000   2.3%   16    nan    nan
 BitcoinsNorway BTCUSD   6495.5300   6631.5400   2.1%   16    nan     60
       Bitfinex BTCUSD   6590.6000   6590.7000   0.0%   16     23     57
         Bitpay BTCUSD   6564.1300         nan   nan%   15    nan     60
       Bitstamp BTCUSD   6561.1400   6565.6200   0.1%    0      2      1
       Coinbase BTCUSD   6504.0600   6635.9700   2.0%   14    nan    117
         Gemini BTCUSD   6567.1300   6573.0700   0.1%   16     89    nan
         Hitbtc+BTCUSD   6592.6200   6594.2100   0.0%    0      0      0
         Kraken BTCUSD   6565.2000   6570.9000   0.1%   15     17     58
  Exchangerates EURNOK      9.4665      9.4665   0.0%   16 107789    nan
     Norgesbank EURNOK      9.4665      9.4665   0.0%   16 107789    nan
       Bitstamp EURUSD      1.1537      1.1593   0.5%    4      5      1
  Exchangerates EURUSD      1.1576      1.1576   0.0%   16 107789    nan
 BitcoinsNorway LTCEUR      1.0000     49.0000  98.0%   16    nan    nan
 BitcoinsNorway LTCNOK    492.4800    503.7500   2.2%   16    nan     60
 BitcoinsNorway LTCUSD      1.0221     49.0000  97.9%   15    nan    nan
     Norgesbank USDNOK      8.1777      8.1777   0.0%   16 107789    nan

The code for this client is too complex for a simple blog post, so you will have to check out the git repository to figure out how it work. What I can tell is how the three last numbers on each line should be interpreted. The first is how many seconds ago information was received from the service. The second is how long ago, according to the service, the provided information was updated. The last is an estimate on how often the buy/sell values change.

If you find this library useful, or would like to improve it, I would love to hear from you. Note that for some of the services I've implemented a trading API. It might be the topic of a future blog post.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Tags: bitcoin, english.
VLC in Debian now can do bittorrent streaming
24th September 2018

Back in February, I got curious to see if VLC now supported Bittorrent streaming. It did not, despite the fact that the idea and code to handle such streaming had been floating around for years. I did however find a standalone plugin for VLC to do it, and half a year later I decided to wrap up the plugin and get it into Debian. I uploaded it to NEW a few days ago, and am very happy to report that it entered Debian a few hours ago, and should be available in Debian/Unstable tomorrow, and Debian/Testing in a few days.

With the vlc-plugin-bittorrent package installed you should be able to stream videos using a simple call to

vlc https://archive.org/download/TheGoat/TheGoat_archive.torrent

It can handle magnet links too. Now if only native vlc had bittorrent support. Then a lot more would be helping each other to share public domain and creative commons movies. The plugin need some stability work with seeking and picking the right file in a torrent with many files, but is already usable. Please note that the plugin is not removing downloaded files when vlc is stopped, so it can fill up your disk if you are not careful. Have fun. :)

I would love to get help maintaining this package. Get in touch if you are interested.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Tags: english, verkidetfri, video.
Using the Kodi API to play Youtube videos
2nd September 2018

I continue to explore my Kodi installation, and today I wanted to tell it to play a youtube URL I received in a chat, without having to insert search terms using the on-screen keyboard. After searching the web for API access to the Youtube plugin and testing a bit, I managed to find a recipe that worked. If you got a kodi instance with its API available from http://kodihost/jsonrpc, you can try the following to have check out a nice cover band.

curl --silent --header 'Content-Type: application/json' \
  --data-binary '{ "id": 1, "jsonrpc": "2.0", "method": "Player.Open",
  "params": {"item": { "file":
  "plugin://plugin.video.youtube/play/?video_id=LuRGVM9O0qg" } } }' \
  http://projector.local/jsonrpc

I've extended kodi-stream program to take a video source as its first argument. It can now handle direct video links, youtube links and 'desktop' to stream my desktop to Kodi. It is almost like a Chromecast. :)

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Tags: debian, english, kodi, video.
Software created using taxpayers’ money should be Free Software
30th August 2018

It might seem obvious that software created using tax money should be available for everyone to use and improve. Free Software Foundation Europe recentlystarted a campaign to help get more people to understand this, and I just signed the petition on Public Money, Public Code to help them. I hope you too will do the same.

Tags: english, opphavsrett.
A bit more on privacy respecting health monitor / fitness tracker
13th August 2018

A few days ago, I wondered if there are any privacy respecting health monitors and/or fitness trackers available for sale these days. I would like to buy one, but do not want to share my personal data with strangers, nor be forced to have a mobile phone to get data out of the unit. I've received some ideas, and would like to share them with you. One interesting data point was a pointer to a Free Software app for Android named Gadgetbridge. It provide cloudless collection and storing of data from a variety of trackers. Its list of supported devices is a good indicator for units where the protocol is fairly open, as it is obviously being handled by Free Software. Other units are reportedly encrypting the collected information with their own public key, making sure only the vendor cloud service is able to extract data from the unit. The people contacting me about Gadgetbirde said they were using Amazfit Bip and Xiaomi Band 3.

I also got a suggestion to look at some of the units from Garmin. I was told their GPS watches can be connected via USB and show up as a USB storage device with Garmin FIT files containing the collected measurements. While proprietary, FIT files apparently can be read at least by GPSBabel and the GpxPod Nextcloud app. It is unclear to me if they can read step count and heart rate data. The person I talked to was using a Garmin Forerunner 935, which is a fairly expensive unit. I doubt it is worth it for a unit where the vendor clearly is trying its best to move from open to closed systems. I still remember when Garmin dropped NMEA support in its GPSes.

A final idea was to build ones own unit, perhaps by basing it on a wearable hardware platforms like the Flora Geo Watch. Sound like fun, but I had more money than time to spend on the topic, so I suspect it will have to wait for another time.

While I was working on tracking down links, I came across an inspiring TED talk by Dave Debronkart about being a e-patient, and discovered the web site Participatory Medicine. If you too want to track your own health and fitness without having information about your private life floating around on computers owned by others, I recommend checking it out.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Tags: english.
Privacy respecting health monitor / fitness tracker?
7th August 2018

Dear lazyweb,

I wonder, is there a fitness tracker / health monitor available for sale today that respect the users privacy? With this I mean a watch/bracelet capable of measuring pulse rate and other fitness/health related values (and by all means, also the correct time and location if possible), which is only provided for me to extract/read from the unit with computer without a radio beacon and Internet connection. In other words, it do not depend on a cell phone app, and do make the measurements available via other peoples computer (aka "the cloud"). The collected data should be available using only free software. I'm not interested in depending on some non-free software that will leave me high and dry some time in the future. I've been unable to find any such unit. I would like to buy it. The ones I have seen for sale here in Norway are proud to report that they share my health data with strangers (aka "cloud enabled"). Is there an alternative? I'm not interested in giving money to people requiring me to accept "privacy terms" to allow myself to measure my own health.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Tags: english.
Sharing images with friends and family using RSS and EXIF/XMP metadata
31st July 2018

For a while now, I have looked for a sensible way to share images with my family using a self hosted solution, as it is unacceptable to place images from my personal life under the control of strangers working for data hoarders like Google or Dropbox. The last few days I have drafted an approach that might work out, and I would like to share it with you. I would like to publish images on a server under my control, and point some Internet connected display units using some free and open standard to the images I published. As my primary language is not limited to ASCII, I need to store metadata using UTF-8. Many years ago, I hoped to find a digital photo frame capable of reading a RSS feed with image references (aka using the <enclosure> RSS tag), but was unable to find a current supplier of such frames. In the end I gave up that approach.

Some months ago, I discovered that XScreensaver is able to read images from a RSS feed, and used it to set up a screen saver on my home info screen, showing images from the Daily images feed from NASA. This proved to work well. More recently I discovered that Kodi (both using OpenELEC and LibreELEC) provide the Feedreader screen saver capable of reading a RSS feed with images and news. For fun, I used it this summer to test Kodi on my parents TV by hooking up a Raspberry PI unit with LibreELEC, and wanted to provide them with a screen saver showing selected pictures from my selection.

Armed with motivation and a test photo frame, I set out to generate a RSS feed for the Kodi instance. I adjusted my Freedombox instance, created /var/www/html/privatepictures/, wrote a small Perl script to extract title and description metadata from the photo files and generate the RSS file. I ended up using Perl instead of python, as the libimage-exiftool-perl Debian package seemed to handle the EXIF/XMP tags I ended up using, while python3-exif did not. The relevant EXIF tags only support ASCII, so I had to find better alternatives. XMP seem to have the support I need.

I am a bit unsure which EXIF/XMP tags to use, as I would like to use tags that can be easily added/updated using normal free software photo managing software. I ended up using the tags set using this exiftool command, as these tags can also be set using digiKam:

exiftool -headline='The RSS image title' \
  -description='The RSS image description.' \
  -subject+=for-family photo.jpeg

I initially tried the "-title" and "keyword" tags, but they were invisible in digiKam, so I changed to "-headline" and "-subject". I use the keyword/subject 'for-family' to flag that the photo should be shared with my family. Images with this keyword set are located and copied into my Freedombox for the RSS generating script to find.

Are there better ways to do this? Get in touch if you have better suggestions.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Tags: debian, english.
Simple streaming the Linux desktop to Kodi using GStreamer and RTP
12th July 2018

Last night, I wrote a recipe to stream a Linux desktop using VLC to a instance of Kodi. During the day I received valuable feedback, and thanks to the suggestions I have been able to rewrite the recipe into a much simpler approach requiring no setup at all. It is a single script that take care of it all.

This new script uses GStreamer instead of VLC to capture the desktop and stream it to Kodi. This fixed the video quality issue I saw initially. It further removes the need to add a m3u file on the Kodi machine, as it instead connects to the JSON-RPC API in Kodi and simply ask Kodi to play from the stream created using GStreamer. Streaming the desktop to Kodi now become trivial. Copy the script below, run it with the DNS name or IP address of the kodi server to stream to as the only argument, and watch your screen show up on the Kodi screen. Note, it depend on multicast on the local network, so if you need to stream outside the local network, the script must be modified. Also note, I have no idea if audio work, as I only care about the picture part.

#!/bin/sh
#
# Stream the Linux desktop view to Kodi.  See
# http://people.skolelinux.org/pere/blog/Streaming_the_Linux_desktop_to_Kodi_using_VLC_and_RTSP.html
# for backgorund information.

# Make sure the stream is stopped in Kodi and the gstreamer process is
# killed if something go wrong (for example if curl is unable to find the
# kodi server).  Do the same when interrupting this script.
kodicmd() {
    host="$1"
    cmd="$2"
    params="$3"
    curl --silent --header 'Content-Type: application/json' \
	 --data-binary "{ \"id\": 1, \"jsonrpc\": \"2.0\", \"method\": \"$cmd\", \"params\": $params }" \
	 "http://$host/jsonrpc"
}
cleanup() {
    if [ -n "$kodihost" ] ; then
	# Stop the playing when we end
	playerid=$(kodicmd "$kodihost" Player.GetActivePlayers "{}" |
			    jq .result[].playerid)
	kodicmd "$kodihost" Player.Stop "{ \"playerid\" : $playerid }" > /dev/null
    fi
    if [ "$gstpid" ] && kill -0 "$gstpid" >/dev/null 2>&1; then
	kill "$gstpid"
    fi
}
trap cleanup EXIT INT

if [ -n "$1" ]; then
    kodihost=$1
    shift
else
    kodihost=kodi.local
fi

mcast=239.255.0.1
mcastport=1234
mcastttl=1

pasrc=$(pactl list | grep -A2 'Source #' | grep 'Name: .*\.monitor$' | \
  cut -d" " -f2|head -1)
gst-launch-1.0 ximagesrc use-damage=0 ! video/x-raw,framerate=30/1 ! \
  videoconvert ! queue2 ! \
  x264enc bitrate=8000 speed-preset=superfast tune=zerolatency qp-min=30 \
  key-int-max=15 bframes=2 ! video/x-h264,profile=high ! queue2 ! \
  mpegtsmux alignment=7 name=mux ! rndbuffersize max=1316 min=1316 ! \
  udpsink host=$mcast port=$mcastport ttl-mc=$mcastttl auto-multicast=1 sync=0 \
  pulsesrc device=$pasrc ! audioconvert ! queue2 ! avenc_aac ! queue2 ! mux. \
  > /dev/null 2>&1 &
gstpid=$!

# Give stream a second to get going
sleep 1

# Ask kodi to start streaming using its JSON-RPC API
kodicmd "$kodihost" Player.Open \
	"{\"item\": { \"file\": \"udp://@$mcast:$mcastport\" } }" > /dev/null

# wait for gst to end
wait "$gstpid"

I hope you find the approach useful. I know I do.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

Tags: debian, english, kodi, video.

RSS feed

Created by Chronicle v4.6