Generated.

[homepage.git] / blog / index.rss

<?xml version="1.0" encoding="utf-8"?>
<rss version='2.0' xmlns:lj='http://www.livejournal.org/rss/lj/1.0/' xmlns:atom="http://www.w3.org/2005/Atom">
        <channel>
                <title>Petter Reinholdtsen</title>
                <description></description>
                <link>https://people.skolelinux.org/pere/blog/</link>
                <atom:link href="https://people.skolelinux.org/pere/blog/index.rss" rel="self" type="application/rss+xml" />
        
        <item>
                <title>Speech to text, she APTly whispered, how hard can it be?</title>
                <link>https://people.skolelinux.org/pere/blog/Speech_to_text__she_APTly_whispered__how_hard_can_it_be_.html</link>
                <guid isPermaLink="true">https://people.skolelinux.org/pere/blog/Speech_to_text__she_APTly_whispered__how_hard_can_it_be_.html</guid>
                <pubDate>Sun, 23 Apr 2023 09:40:00 +0200</pubDate>
                <description>&lt;p&gt;While visiting a convention during Eastern, it occurred to me that
it would be great if I could have a digital Dictaphone with
transcribing capabilities, providing me with texts to cut-n-paste into
stuff I need to write.  The background is that long drives often bring
up the urge to write on texts I am working on, which of course is out
of the question while driving.  With the release of
&lt;a href=&quot;https://github.com/openai/whisper/&quot;&gt;OpenAI Whisper&lt;/a&gt;, this
seem to be within reach with Free Software, so I decided to give it a
go.  OpenAI Whisper is a Linux based neural network system to read in
audio files and provide text representation of the speech in that
audio recording.  It handle multiple languages and according to its
creators even can translate into a different language than the spoken
one.  I have not tested the latter feature.  It can either use the CPU
or a GPU with CODA support.  As far as I can tell, CODA in practice
limit that feature to NVidia graphics cards.  I have few of those, as
they do not work great with free software drivers, and have not tested
the GPU option.  While looking into the matter, I did discover some
work to provide CODA support on non-NVidia GPUs, and some work with
the library used by Whisper to port it to other GPUs, but have not
spent much time looking into GPU support yet.  I&#39;ve so far used an old
X220 laptop as my test machine, and only transcribed using its
CPU.&lt;/p&gt;

&lt;p&gt;As it from a privacy standpoint is unthinkable to use computers
under control of someone else (aka a &quot;cloud&quot; service) to transcribe
ones thoughts and personal notes, I want to run the transcribing
system locally on my own computers.  The only sensible approach to me
is to make the effort I put into this available for any Linux user and
to upload the needed packages into Debian.  Looking at Debian Bookworm, I
discovered that only three packages were missing,
&lt;a href=&quot;https://bugs.debian.org/1034307&quot;&gt;tiktoken&lt;/a&gt;,
&lt;a href=&quot;https://bugs.debian.org/1034144&quot;&gt;triton&lt;/a&gt;, and
&lt;a href=&quot;https://bugs.debian.org/1034091&quot;&gt;openai-whisper&lt;/a&gt;.  For a while
I also believed
&lt;a href=&quot;https://bugs.debian.org/1034286&quot;&gt;ffmpeg-python&lt;/a&gt; was
needed, but as its
&lt;a href=&quot;https://github.com/kkroening/ffmpeg-python/issues/760&quot;&gt;upstream
seem to have vanished&lt;/a&gt; I found it safer
&lt;a href=&quot;https://github.com/openai/whisper/pull/1242&quot;&gt;to rewrite
whisper&lt;/a&gt; to stop depending on in than to introduce ffmpeg-python
into Debian.  I decided to place these packages under the umbrella of
&lt;a href=&quot;https://salsa.debian.org/deeplearning-team&quot;&gt;the Debian Deep
Learning Team&lt;/a&gt;, which seem like the best team to look after such
packages.  Discussing the topic within the group also made me aware
that the triton package was already a future dependency of newer
versions of the torch package being planned, and would be needed after
Bookworm is released.&lt;/p&gt;

&lt;p&gt;All required code packages have been now waiting in
&lt;a href=&quot;https://ftp-master.debian.org/new.html&quot;&gt;the Debian NEW
queue&lt;/a&gt; since Wednesday, heading for Debian Experimental until
Bookworm is released.  An unsolved issue is how to handle the neural
network models used by Whisper.  The default behaviour of Whisper is
to require Internet connectivity and download the model requested to
&lt;tt&gt;~/.cache/whisper/&lt;/tt&gt; on first invocation.  This obviously would
fail &lt;a href=&quot;https://people.debian.org/~bap/dfsg-faq.html&quot;&gt;the
deserted island test of free software&lt;/a&gt; as the Debian packages would
be unusable for someone stranded with only the Debian archive and solar
powered computer on a deserted island.&lt;/p&gt;

&lt;p&gt;Because of this, I would love to include the models in the Debian
mirror system.  This is problematic, as the models are very large
files, which would put a heavy strain on the Debian mirror
infrastructure around the globe.  The strain would be even higher if
the models change often, which luckily as far as I can tell they do
not.  The small model, which according to its creator is most useful
for English and in my experience is not doing a great job there
either, is 462 MiB (deb is 414 MiB).  The medium model, which to me
seem to handle English speech fairly well is 1.5 GiB (deb is 1.3 GiB)
and the large model is 2.9 GiB (deb is 2.6 GiB).  I would assume
everyone with enough resources would prefer to use the large model for
highest quality.  I believe the models themselves would have to go
into the non-free part of the Debian archive, as they are not really
including any useful source code for updating the models.  The
&quot;source&quot;, aka the model training set, according to the creators
consist of &quot;680,000 hours of multilingual and multitask supervised
data collected from the web&quot;, which to me reads material with both
unknown copyright terms, unavailable to the general public.  In other
words, the source is not available according to the Debian Free
Software Guidelines and the model should be considered non-free.&lt;/p&gt;

&lt;p&gt;I asked the Debian FTP masters for advice regarding uploading a
model package on their IRC channel, and based on the feedback there it
is still unclear to me if such package would be accepted into the
archive.  In any case I wrote build rules for a
&lt;a href=&quot;https://salsa.debian.org/deeplearning-team/openai-whisper-model&quot;&gt;OpenAI
Whisper model package&lt;/a&gt; and
&lt;a href=&quot;https://github.com/openai/whisper/pull/1257&quot;&gt;modified the
Whisper code base&lt;/a&gt; to prefer shared files under &lt;tt&gt;/usr/&lt;/tt&gt; and
&lt;tt&gt;/var/&lt;/tt&gt; over user specific files in &lt;tt&gt;~/.cache/whisper/&lt;/tt&gt;
to be able to use these model packages, to prepare for such
possibility.  One solution might be to include only one of the models
(small or medium, I guess) in the Debian archive, and ask people to
download the others from the Internet.  Not quite sure what to do
here, and advice is most welcome (use the debian-ai mailing list).&lt;/p&gt;

&lt;p&gt;To make it easier to test the new packages while I wait for them to
clear the NEW queue, I created an APT source targeting bookworm.  I
selected Bookworm instead of Bullseye, even though I know the latter
would reach more users, is that some of the required dependencies are
missing from Bullseye and I during this phase of testing did not want
to backport a lot of packages just to get up and running.&lt;/p&gt;

&lt;p&gt;Here is a recipe to run as user root if you want to test OpenAI
Whisper using Debian packages on your Debian Bookworm installation,
first adding the APT repository GPG key to the list of trusted keys,
then setting up the APT repository and finally installing the packages
and one of the models:&lt;/p&gt;

&lt;p&gt;&lt;pre&gt;
curl https://geekbay.nuug.no/~pere/openai-whisper/D78F5C4796F353D211B119E28200D9B589641240.asc \
  -o /etc/apt/trusted.gpg.d/pere-whisper.asc
mkdir -p /etc/apt/sources.list.d
cat &gt; /etc/apt/sources.list.d/pere-whisper.list &amp;lt;&amp;lt;EOF
deb https://geekbay.nuug.no/~pere/openai-whisper/ bookworm main
deb-src https://geekbay.nuug.no/~pere/openai-whisper/ bookworm main
EOF
apt update
apt install openai-whisper
&lt;/pre&gt;&lt;/p&gt;

&lt;p&gt;The package work for me, but have not yet been tested on any other
computer than my own.  With it, I have been able to (badly) transcribe
a 2 minute 40 second Norwegian audio clip to test using the small
model.  This took 11 minutes and around 2.2 GiB of RAM.  Transcribing
the same file with the medium model gave a accurate text in 77 minutes
using around 5.2 GiB of RAM.  My test machine had too little memory to
test the large model, which I believe require 11 GiB of RAM.  In
short, this now work for me using Debian packages, and I hope it will
for you and everyone else once the packages enter Debian.&lt;/p&gt;

&lt;p&gt;Now I can start on the audio recording part of this project.&lt;/p&gt;
 
&lt;p&gt;As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
&lt;b&gt;&lt;a href=&quot;bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&quot;&gt;15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&lt;/a&gt;&lt;/b&gt;.&lt;/p&gt;
</description>
        </item>
        
        <item>
                <title>rtlsdr-scanner, software defined radio frequency scanner for Linux  - nice free software</title>
                <link>https://people.skolelinux.org/pere/blog/rtlsdr_scanner__software_defined_radio_frequency_scanner_for_Linux____nice_free_software.html</link>
                <guid isPermaLink="true">https://people.skolelinux.org/pere/blog/rtlsdr_scanner__software_defined_radio_frequency_scanner_for_Linux____nice_free_software.html</guid>
                <pubDate>Fri, 7 Apr 2023 23:10:00 +0200</pubDate>
                <description>&lt;p&gt;Today I finally found time to track down a useful radio frequency
scanner for my software defined radio.  Just for fun I tried to locate
the radios used in the areas, and a good start would be to scan all
the frequencies to see what is in use.  I&#39;ve tried to find a useful
program earlier, but ran out of time before I managed to find a useful
tool.  This time I was more successful, and after a few false leads I
found a description of
&lt;a href=&quot;https://www.kali.org/tools/rtlsdr-scanner/&quot;&gt;rtlsdr-scanner
over at the Kali site&lt;/a&gt;, and was able to track down
&lt;a href=&quot;https://gitlab.com/kalilinux/packages/rtlsdr-scanner.git&quot;&gt;the
Kali package git repository&lt;/a&gt; to build a deb package for the
scanner.  Sadly the package is missing from the Debian project itself,
at least in Debian Bullseye.  Two runtime dependencies,
&lt;a href=&quot;https://gitlab.com/kalilinux/packages/python-visvis.git&quot;&gt;python-visvis&lt;/a&gt;
and
&lt;a href=&quot;https://gitlab.com/kalilinux/packages/python-rtlsdr.git&quot;&gt;python-rtlsdr&lt;/a&gt;
had to be built and installed separately.  Luckily &#39;&lt;tt&gt;gbp
buildpackage&lt;/tt&gt;&#39; handled them just fine and no further packages had
to be manually built.  The end result worked out of the box after
installation.&lt;/p&gt;

&lt;p&gt;My initial scans for FM channels worked just fine, so I knew the
scanner was functioning.  But when I tried to scan every frequency
from 100 to 1000 MHz, the program stopped unexpectedly near the
completion.  After some debugging I discovered USB software radio I
used rejected frequencies above 948 MHz, triggering a unreported
exception breaking the scan.  Changing the scan to end at 957 worked
better.  I similarly found the lower limit to be around 15, and ended
up with the following full scan:&lt;/p&gt;

&lt;p&gt;&lt;a href=&quot;https://people.skolelinux.org/pere/blog/images/2023-04-07-radio-freq-scanning.png&quot;&gt;&lt;img src=&quot;https://people.skolelinux.org/pere/blog/images/2023-04-07-radio-freq-scanning.png&quot; width=&quot;100%&quot;&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Saving the scan did not work, but exporting it as a CSV file worked
just fine.  I ended up with around 477k CVS lines with dB level for
the given frequency.&lt;/p&gt;

&lt;p&gt;The save failure seem to be a missing UTF-8 encoding issue in the
python code.  Will see if I can find time to send a patch
&lt;a href=&quot;https://github.com/CdeMills/RTLSDR-Scanner/&quot;&gt;upstream&lt;/a&gt;
later to fix this exception:&lt;/p&gt;

&lt;pre&gt;
Traceback (most recent call last):
  File &quot;/usr/lib/python3/dist-packages/rtlsdr_scanner/main_window.py&quot;, line 485, in __on_save
    save_plot(fullName, self.scanInfo, self.spectrum, self.locations)
  File &quot;/usr/lib/python3/dist-packages/rtlsdr_scanner/file.py&quot;, line 408, in save_plot
    handle.write(json.dumps(data, indent=4))
TypeError: a bytes-like object is required, not &#39;str&#39;
Traceback (most recent call last):
  File &quot;/usr/lib/python3/dist-packages/rtlsdr_scanner/main_window.py&quot;, line 485, in __on_save
    save_plot(fullName, self.scanInfo, self.spectrum, self.locations)
  File &quot;/usr/lib/python3/dist-packages/rtlsdr_scanner/file.py&quot;, line 408, in save_plot
    handle.write(json.dumps(data, indent=4))
TypeError: a bytes-like object is required, not &#39;str&#39;
&lt;/pre&gt;
 
&lt;p&gt;As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
&lt;b&gt;&lt;a href=&quot;bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&quot;&gt;15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&lt;/a&gt;&lt;/b&gt;.&lt;/p&gt;
</description>
        </item>
        
        <item>
                <title>OpenSnitch available in Debian Sid and Bookworm</title>
                <link>https://people.skolelinux.org/pere/blog/OpenSnitch_available_in_Debian_Sid_and_Bookworm.html</link>
                <guid isPermaLink="true">https://people.skolelinux.org/pere/blog/OpenSnitch_available_in_Debian_Sid_and_Bookworm.html</guid>
                <pubDate>Sat, 25 Feb 2023 20:30:00 +0100</pubDate>
                <description>&lt;p&gt;Thanks to the efforts of the OpenSnitch lead developer Gustavo
Iñiguez Goya allowing me to sponsor the upload,
&lt;a href=&quot;https://tracker.debian.org/pkg/opensnitch&quot;&gt;the interactive
application firewall OpenSnitch&lt;/a&gt; is now available in Debian
Testing, soon to become the next stable release of Debian.&lt;/p&gt;

&lt;p&gt;This is a package which set up a network firewall on one or more
machines, which is controlled by a graphical user interface that will
ask the user if a program should be allowed to connect to the local
network or the Internet.  If some background daemon is trying to dial
home, it can be blocked from doing so with a simple mouse click, or by
default simply by not doing anything when the GUI question dialog pop
up.  A list of all programs discovered using the network is provided
in the GUI, giving the user an overview of how the machine(s) programs
use the network.&lt;/p&gt;

&lt;p&gt;OpenSnitch was uploaded for NEW processing about a month ago, and I
had little hope of it getting accepted and shaping up in time for the
package freeze, but the Debian ftpmasters proved to be amazingly quick
at checking out the package and it was accepted into the archive about
week after the first upload.  It is now team maintained under the Go
language team umbrella.  A few fixes to the default setup is only in
Sid, and should migrate to Testing/Bookworm in a week.&lt;/p&gt;

&lt;p&gt;During testing I ran into an
&lt;a href=&quot;https://github.com/evilsocket/opensnitch/issues/813&quot;&gt;issue
with Minecraft server broadcasts disappearing&lt;/a&gt;, which was quickly
resolved by the developer with a patch and a proposed configuration
change.  I&#39;ve been told this was caused by the Debian packages default
use if /proc/ information to track down kernel status, instead of the
newer eBPF module that can be used.  The reason is simply that
upstream and I have failed to find a way to build the eBPF modules for
OpenSnitch without a complete configured Linux kernel source tree,
which as far as we can tell is unavailable as a build dependency in
Debian.  We tried unsuccessfully so far to use the kernel-headers
package.  It would be great if someone could provide some clues how to
build eBPF modules on build daemons in Debian, possibly without the full
kernel source.&lt;/p&gt;
 
&lt;p&gt;As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
&lt;b&gt;&lt;a href=&quot;bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&quot;&gt;15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&lt;/a&gt;&lt;/b&gt;.&lt;/p&gt;
</description>
        </item>
        
        <item>
                <title>Is the desktop recommending your program for opening its files?</title>
                <link>https://people.skolelinux.org/pere/blog/Is_the_desktop_recommending_your_program_for_opening_its_files_.html</link>
                <guid isPermaLink="true">https://people.skolelinux.org/pere/blog/Is_the_desktop_recommending_your_program_for_opening_its_files_.html</guid>
                <pubDate>Sun, 29 Jan 2023 11:00:00 +0100</pubDate>
                <description>&lt;p&gt;Linux desktop systems
&lt;a href=&quot;https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html&quot;&gt;have
standardized&lt;/a&gt; how programs present themselves to the desktop
system.  If a package include a .desktop file in
/usr/share/applications/, Gnome, KDE, LXDE, Xfce and the other desktop
environments will pick up the file and use its content to generate the
menu of available programs in the system.  A lesser known fact is that
a package can also explain to the desktop system how to recognize the
files created by the program in question, and use it to open these
files on request, for example via a GUI file browser.&lt;/p&gt;

&lt;p&gt;A while back I ran into a package that did not tell the desktop
system how to recognize its files and was not used to open its files
in the file browser and fixed it.  In the process I wrote a simple
debian/tests/ script to ensure the setup keep working.  It might be
useful for other packages too, to ensure any future version of the
package keep handling its own files.&lt;/p&gt;

&lt;p&gt;For this to work the file format need a useful MIME type that can
be used to identify the format.  If the file format do not yet have a
MIME type, it should define one and preferably also
&lt;a href=&quot;https://www.iana.org/assignments/media-types/media-types.xhtml&quot;&gt;register
it with IANA&lt;/a&gt; to ensure the MIME type string is reserved.&lt;/p&gt;

&lt;p&gt;The script uses the &lt;tt&gt;xdg-mime&lt;/tt&gt; program from xdg-utils to
query the database of standardized package information and ensure it
return sensible values.  It also need the location of an example file
for xdg-mime to guess the format of.&lt;/p&gt;

&lt;pre&gt;
#!/bin/sh
#
# Author: Petter Reinholdtsen
# License: GPL v2 or later at your choice.
#
# Validate the MIME setup, making sure motor types have
# application/vnd.openmotor+yaml associated with them and is connected
# to the openmotor desktop file.

retval=0

mimetype=&quot;application/vnd.openmotor+yaml&quot;
testfile=&quot;test/data/real/o3100/motor.ric&quot;
mydesktopfile=&quot;openmotor.desktop&quot;

filemime=&quot;$(xdg-mime query filetype &quot;$testfile&quot;)&quot;

if [ &quot;$mimetype&quot; != &quot;$filemime&quot; ] ; then
    retval=1
    echo &quot;error: xdg-mime claim motor file MIME type is $filemine, not $mimetype&quot;
else
    echo &quot;success: xdg-mime report correct mime type $mimetype for motor file&quot;
fi

desktop=$(xdg-mime query default &quot;$mimetype&quot;)

if [ &quot;$mydesktopfile&quot; != &quot;$desktop&quot; ]; then
    retval=1
    echo &quot;error: xdg-mime claim motor file should be handled by $desktop, not $mydesktopfile&quot;
else
    echo &quot;success: xdg-mime agree motor file should be handled by $mydesktopfile&quot;
fi

exit $retval
&lt;/pre&gt;

&lt;p&gt;It is a simple way to ensure your users are not very surprised when
they try to open one of your file formats in their file browser.&lt;/p&gt;

&lt;p&gt;As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
&lt;b&gt;&lt;a href=&quot;bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&quot;&gt;15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&lt;/a&gt;&lt;/b&gt;.&lt;/p&gt;
</description>
        </item>
        
        <item>
                <title>Opensnitch, the application level interactive firewall, heading into the Debian archive</title>
                <link>https://people.skolelinux.org/pere/blog/Opensnitch__the_application_level_interactive_firewall__heading_into_the_Debian_archive.html</link>
                <guid isPermaLink="true">https://people.skolelinux.org/pere/blog/Opensnitch__the_application_level_interactive_firewall__heading_into_the_Debian_archive.html</guid>
                <pubDate>Sun, 22 Jan 2023 23:55:00 +0100</pubDate>
                <description>&lt;p&gt;While reading a
&lt;a href=&quot;https://sneak.berlin/20230115/macos-scans-your-local-files-now/&quot;&gt;blog
post claiming MacOS X recently started scanning local files and
reporting information about them to Apple&lt;/a&gt;, even on a machine where
all such callback features had been disabled, I came across a
description of the Little Snitch application for MacOS X.  It seemed
like a very nice tool to have in the tool box, and I decided to see if
something similar was available for Linux.&lt;/p&gt;

&lt;p&gt;It did not take long to find
&lt;a href=&quot;https://github.com/evilsocket/opensnitch&quot;&gt;the OpenSnitch
package&lt;/a&gt;, which has been in development since 2017, and now is in
version 1.5.0.  It has had a
&lt;a href=&quot;https://bugs.debian.org/909567&quot;&gt;request for Debian
packaging&lt;/a&gt; since 2018, but no-one completed the job so far.  Just
for fun, I decided to see if I could help, and I was very happy to
discover that
&lt;a href=&quot;https://github.com/evilsocket/opensnitch/issues/304&quot;&gt;upstream
want a Debian package too&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;After struggling a bit with getting the program to run, figuring
out building Go programs (and a little failed detour to look at eBPF
builds too - help needed), I am very happy to report that I am
sponsoring upstream to maintain the package in Debian, and it has
since this morning been waiting in NEW for the ftpmasters to have a
look.  Perhaps it can get into the archive in time for the Bookworm
release?&lt;/p&gt;
 
&lt;p&gt;As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
&lt;b&gt;&lt;a href=&quot;bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&quot;&gt;15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&lt;/a&gt;&lt;/b&gt;.&lt;/p&gt;
</description>
        </item>
        
        <item>
                <title>LinuxCNC MQTT publisher component</title>
                <link>https://people.skolelinux.org/pere/blog/LinuxCNC_MQTT_publisher_component.html</link>
                <guid isPermaLink="true">https://people.skolelinux.org/pere/blog/LinuxCNC_MQTT_publisher_component.html</guid>
                <pubDate>Sun, 8 Jan 2023 19:30:00 +0100</pubDate>
                <description>&lt;p&gt;I watched &lt;a href=&quot;https://yewtu.be/watch?v=jmKUV3aNLjk&quot;&gt;a 2015
video from Andreas Schiffler&lt;/a&gt; the other day, where he set up
&lt;a href=&quot;https://linuxcnc.org/&quot;&gt;LinuxCNC&lt;/a&gt; to send status
information to the MQTT broker IBM Bluemix.  As I also use MQTT for
graphing, it occured to me that a generic MQTT LinuxCNC component
would be useful and I set out to implement it.  Today I got the first
draft limping along and submitted as
&lt;a href=&quot;https://github.com/LinuxCNC/linuxcnc/pull/2253&quot;&gt;a patch to the
LinuxCNC project&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The simple part was setting up the MQTT publishing code in Python.
I already have set up other parts submitting data to my Mosquito MQTT
broker, so I could reuse that code.  Writing a LinuxCNC component in
Python as new to me, but using existing examples in the code
repository and the extensive documentation, this was fairly straight
forward.  The hardest part was creating a automated test for the
component to ensure it was working.  Testing it in a simulated
LinuxCNC machine proved very useful, as I discovered features I needed
that I had not thought of yet, and adjusted the code quite a bit to
make it easier to test without a operational MQTT broker
available.&lt;/p&gt;

&lt;p&gt;The draft is ready and working, but I am unsure which LinuxCNC HAL
pins I should collect and publish by default (in other words, the
default set of information pieces published), and how to get the
machine name from the LinuxCNC INI file.  The latter is a minor
detail, but I expect it would be useful in a setup with several
machines available.  I am hoping for feedback from the experienced
LinuxCNC developers and users, to make the component even better
before it can go into the mainland LinuxCNC code base.&lt;/p&gt;

&lt;p&gt;Since I started on the MQTT component, I came across
&lt;a href=&quot;https://yewtu.be/watch?v=Bqa2grG0XtA&quot;&gt;another video from Kent
VanderVelden&lt;/a&gt; where he combine LinuxCNC with a set of screen glasses
controlled by a Raspberry Pi, and it occured to me that it would
be useful for such use cases if LinuxCNC also provided a REST API for
querying its status.  I hope to start on such component once the MQTT
component is working well.&lt;/p&gt;
 
&lt;p&gt;As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
&lt;b&gt;&lt;a href=&quot;bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&quot;&gt;15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&lt;/a&gt;&lt;/b&gt;.&lt;/p&gt;
</description>
        </item>
        
        <item>
                <title>ONVIF IP camera management tool finally in Debian</title>
                <link>https://people.skolelinux.org/pere/blog/ONVIF_IP_camera_management_tool_finally_in_Debian.html</link>
                <guid isPermaLink="true">https://people.skolelinux.org/pere/blog/ONVIF_IP_camera_management_tool_finally_in_Debian.html</guid>
                <pubDate>Sat, 24 Dec 2022 08:00:00 +0100</pubDate>
                <description>&lt;p&gt;Merry Christmas to you all.  Here is a small gift to all those with
IP cameras following the &lt;a href=&quot;https://www.onvif.org/&quot;&gt;ONVIF
specification&lt;/a&gt;.  There is finally a nice command line and GUI tool
in Debian to manage ONVIF IP cameras.  After working with upstream for
a few months and sponsoring the upload, I am very happy to report that
the &lt;a href=&quot;https://tracker.debian.org/libonvif&quot;&gt;libonvif package&lt;/a&gt;
entered Debian Sid last night.&lt;/p&gt;

&lt;p&gt;The package provide a C library to communicate with such cameras, a
command line tool to locate and update settings of (like password) the
cameras and a GUI tool to configure and control the units as well as
preview the video from the camera.  Libonvif is available on Both
Linux and Windows and the GUI tool uses the Qt library.  The main
competitors are non-free software, while libonvif is GNU GPL licensed.
I am very glad Debian users in the future can control their cameras
using a free software system provided by Debian.  But the ONVIF world
is full of slightly broken firmware, where the cameras pretend to
follow the ONVIF specification but fail to set some configuration
values or refuse to provide video to more than one recipient at the
time, and the onvif project is quite young and might take a while
before it completely work with your camera.  Upstream seem eager to
improve the library, so handling any broken camera might be just &lt;a
href=&quot;https://github.com/sr99622/libonvif/&quot;&gt;a bug report away&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The package just cleared NEW, and need a new source only upload
before it can enter testing.  This will happen in the next few
days.&lt;/p&gt;

&lt;p&gt;As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
&lt;b&gt;&lt;a href=&quot;bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&quot;&gt;15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&lt;/a&gt;&lt;/b&gt;.&lt;/p&gt;
</description>
        </item>
        
        <item>
                <title>Managing and using ONVIF IP cameras with Linux</title>
                <link>https://people.skolelinux.org/pere/blog/Managing_and_using_ONVIF_IP_cameras_with_Linux.html</link>
                <guid isPermaLink="true">https://people.skolelinux.org/pere/blog/Managing_and_using_ONVIF_IP_cameras_with_Linux.html</guid>
                <pubDate>Wed, 19 Oct 2022 12:30:00 +0200</pubDate>
                <description>&lt;p&gt;Recently I have been looking at how to control and collect data
from a handful IP cameras using Linux.  I both wanted to change their
settings and to make their imagery available via a free software
service under my control.  Here is a summary of the tools I found.&lt;/p&gt;

&lt;p&gt;First I had to identify the cameras and their protocols.  As far as
I could tell, they were using some SOAP looking protocol and their
internal web server seem to only work with Microsoft Internet Explorer
with some proprietary binary plugin, which in these days of course is
a security disaster and also made it impossible for me to use the
camera web interface.  Luckily I discovered that the SOAP looking
protocol is actually following &lt;a href=&quot;https://www.onvif.org/&quot;&gt;the
ONVIF specification&lt;/a&gt;, which seem to be supported by a lot of IP
cameras these days.&lt;/p&gt;

&lt;p&gt;Once the protocol was identified, I was able to find what appear to
be the most popular way to configure ONVIF cameras, the free software
Windows tool named
&lt;a href=&quot;https://sourceforge.net/projects/onvifdm/&quot;&gt;ONVIF Device
Manager&lt;/a&gt;.  Lacking any other options at the time, I tried
unsuccessfully to get it running using Wine, but was missing a dotnet
40 library and I found no way around it to run it on Linux.&lt;/p&gt;

&lt;p&gt;The next tool I found to configure the cameras were a non-free Linux Qt
client &lt;a href=&quot;https://www.lingodigit.com/onvif_nvcdemo.html&quot;&gt;ONVIF
Device Tool&lt;/a&gt;.  I did not like its terms of use, so did not spend
much time on it.&lt;/p&gt;

&lt;p&gt;To collect the video and make it available in a web interface, I
found the Zoneminder tool in Debian.  A recent version was able to
automatically detect and configure ONVIF devices, so I could use it to
set up motion detection in and collection of the camera output.  I had
initial problems getting the ONVIF autodetection to work, as both
Firefox and Chromium &lt;a href=&quot;https://bugs.debian.org/1001188&quot;&gt;refused
the inter-tab communication&lt;/a&gt; being used by the Zoneminder web
pages, but managed to get konqueror to work.  Apparently the &quot;Enhanced
Tracking Protection&quot; in Firefox cause the problem.  I ended up
upgrading to the Bookworm edition of Zoneminder in the process to try
to fix the issue, and believe the problem might be solved now.&lt;/p&gt;

&lt;p&gt;In the process I came across the nice Linux GUI tool
&lt;a href=&quot;https://gitlab.com/caspermeijn/onvifviewer/&quot;&gt;ONVIF Viewer&lt;/a&gt;
allowing me to preview the camera output and validate the login
passwords required.  Sadly its author has grown tired of maintaining
the software, so it might not see any future updates.  Which is sad,
as the viewer is sightly unstable and the picture tend to lock up.
Note, this lockup might be due to limitations in the cameras and not
the viewer implementation.  I suspect the camera is only able to
provide pictures to one client at the time, and the Zoneminder feed
might interfere with the GUI viewer.  I have
&lt;a href=&quot;https://bugs.debian.org/1000820&quot;&gt;asked for the tool to be
included in Debian&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Finally, I found what appear to be very nice Linux free software
replacement for the Windows tool, named
&lt;a href=&quot;https://github.com/sr99622/libonvif/&quot;&gt;libonvif&lt;/a&gt;.  It
provide a C library to talk to ONVIF devices as well as a command line
and GUI tool using the library.  Using the GUI tool I was able to change
the admin passwords and update other settings of the cameras.  I have
&lt;a href=&quot;https://bugs.debian.org/1021980&quot;&gt;asked for the package to be
included in Debian&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
&lt;b&gt;&lt;a href=&quot;bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&quot;&gt;15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&lt;/a&gt;&lt;/b&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Update 2022-10-20&lt;/strong&gt;: Since my initial publication of
this text, I got several suggestions for more free software Linux
tools.  There is &lt;a href=&quot;https://github.com/quatanium/python-onvif&quot;&gt;a
ONVIF python library&lt;/a&gt; (already
&lt;a href=&quot;https://bugs.debian.org/824240&quot;&gt;requested into Debian&lt;/a&gt;) and
&lt;a href=&quot;https://github.com/FalkTannhaeuser/python-onvif-zeep&quot;&gt;a python 3
fork&lt;/a&gt; using a different SOAP dependency.  There is also
&lt;a href=&quot;https://www.home-assistant.io/integrations/onvif/&quot;&gt;support for
ONVIF in Home Assistant&lt;/a&gt;, and there is an alternative to Zoneminder
called &lt;a href=&quot;https://www.shinobi.video/&quot;&gt;Shinobi&lt;/a&gt;.  The latter
two are not included in Debian either.  I have not tested any of these
so far.&lt;/p&gt;
</description>
        </item>
        
        <item>
                <title>Time to translate the Bullseye edition of the Debian Administrator&#39;s Handbook</title>
                <link>https://people.skolelinux.org/pere/blog/Time_to_translate_the_Bullseye_edition_of_the_Debian_Administrator_s_Handbook.html</link>
                <guid isPermaLink="true">https://people.skolelinux.org/pere/blog/Time_to_translate_the_Bullseye_edition_of_the_Debian_Administrator_s_Handbook.html</guid>
                <pubDate>Mon, 12 Sep 2022 15:45:00 +0200</pubDate>
                <description>&lt;p align=&quot;center&quot;&gt;&lt;img align=&quot;center&quot; src=&quot;http://people.skolelinux.org/pere/blog/images/2020-10-20-debian-handbook-nb-testprint.jpeg&quot; width=&quot;60%&quot;/&gt;&lt;/p&gt;

&lt;p&gt;(The picture is of the previous edition.)&lt;/p&gt;

&lt;p&gt;Almost two years after the previous Norwegian Bokmål translation of
the &quot;&lt;a href=&quot;https://debian-handbook.info/&quot;&gt;The Debian Administrator&#39;s
Handbook&lt;/a&gt;&quot; was published, a new edition is finally being prepared.  The
english text is updated, and it is time to start working on the
translations.  Around 37 percent of the strings have been updated, one
way or another, and the translations starting from a complete Debian Buster
edition now need to bring their translation up from 63% to 100%.  The
complete book is licensed using a Creative Commons license, and has
been published in several languages over the years.  The translations
are done by volunteers to bring Linux in their native tongue.  The
last time I checked, it complete text was available in English,
Norwegian Bokmål, German, Indonesian, Brazil Portuguese and Spanish.
In addition, work has been started for Arabic (Morocco), Catalan,
Chinese (Simplified), Chinese (Traditional), Croatian, Czech, Danish,
Dutch, French, Greek, Italian, Japanese, Korean, Persian, Polish,
Romanian, Russian, Swedish, Turkish and Vietnamese.&lt;/p&gt;

&lt;p&gt;The translation is conducted on
&lt;a href=&quot;https://hosted.weblate.org/projects/debian-handbook/&quot;&gt;the
hosted weblate project page&lt;/a&gt;.  Prospective translators are
recommeded to subscribe to
&lt;a href=&quot;http://lists.alioth.debian.org/mailman/listinfo/debian-handbook-translators&quot;&gt;the
translators mailing list&lt;/a&gt; and should also check out
&lt;a href=&quot;https://debian-handbook.info/contribute/&quot;&gt;the instructions for
contributors&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;I am one of the Norwegian Bokmål translators of this book, and we
have just started.  Your contribution is most welcome.&lt;/p&gt;

&lt;p&gt;As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
&lt;b&gt;&lt;a href=&quot;bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&quot;&gt;15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&lt;/a&gt;&lt;/b&gt;.&lt;/p&gt;
</description>
        </item>
        
        <item>
                <title>Automatic LinuxCNC servo PID tuning?</title>
                <link>https://people.skolelinux.org/pere/blog/Automatic_LinuxCNC_servo_PID_tuning_.html</link>
                <guid isPermaLink="true">https://people.skolelinux.org/pere/blog/Automatic_LinuxCNC_servo_PID_tuning_.html</guid>
                <pubDate>Sat, 16 Jul 2022 22:30:00 +0200</pubDate>
                <description>&lt;p&gt;While working on a CNC with servo motors controlled by the
&lt;a href=&quot;https://en.wikipedia.org/wiki/LinuxCNC&quot;&gt;LinuxCNC&lt;/a&gt;
&lt;a href=&quot;https://en.wikipedia.org/wiki/PID_controller&quot;&gt;PID
controller&lt;/a&gt;, I recently had to learn how to tune the collection of values
that control such mathematical machinery that a PID controller is.  It
proved to be a lot harder than I hoped, and I still have not succeeded
in getting the Z PID controller to successfully defy gravity, nor X
and Y to move accurately and reliably.  But while climbing up this
rather steep learning curve, I discovered that some motor control
systems are able to tune their PID controllers.  I got the impression
from the documentation that LinuxCNC were not.  This proved to be not
true.&lt;/p&gt;

&lt;p&gt;The LinuxCNC
&lt;a href=&quot;http://linuxcnc.org/docs/html/man/man9/pid.9.html&quot;&gt;pid
component&lt;/a&gt; is the recommended PID controller to use.  It uses eight
constants &lt;tt&gt;Pgain&lt;/tt&gt;, &lt;tt&gt;Igain&lt;/tt&gt;, &lt;tt&gt;Dgain&lt;/tt&gt;,
&lt;tt&gt;bias&lt;/tt&gt;, &lt;tt&gt;FF0&lt;/tt&gt;, &lt;tt&gt;FF1&lt;/tt&gt;, &lt;tt&gt;FF2&lt;/tt&gt; and
&lt;tt&gt;FF3&lt;/tt&gt; to calculate the output value based on current and wanted
state, and all of these need to have a sensible value for the
controller to behave properly.  Note, there are even more values
involved, theser are just the most important ones.  In my case I need
the X, Y and Z axes to follow the requested path with little error.
This has proved quite a challenge for someone who have never tuned a
PID controller before, but there is at least some help to be found.

&lt;p&gt;I discovered that included in LinuxCNC was this old PID component
at_pid claiming to have auto tuning capabilities.  Sadly it had been
neglected since 2011, and could not be used as a plug in replacement
for the default pid component.  One would have to rewriting the
LinuxCNC HAL setup to test at_pid.  This was rather sad, when I wanted
to quickly test auto tuning to see if it did a better job than me at
figuring out good P, I and D values to use.&lt;/p&gt;

&lt;p&gt;I decided to have a look if the situation could be improved.  This
involved trying to understand the code and history of the pid and
at_pid components.  Apparently they had a common ancestor, as code
structure, comments and variable names were quite close to each other.
Sadly this was not reflected in the git history, making it hard to
figure out what really happened.  My guess is that the author of
&lt;a href=&quot;https://github.com/LinuxCNC/linuxcnc/blob/master/src/hal/components/at_pid.c&quot;&gt;at_pid.c&lt;/a&gt;
took a version of
&lt;a href=&quot;https://github.com/LinuxCNC/linuxcnc/blob/master/src/hal/components/pid.c&quot;&gt;pid.c&lt;/a&gt;,
rewrote it to follow the structure he wished pid.c to have, then added
support for auto tuning and finally got it included into the LinuxCNC
repository.  The restructuring and lack of early history made it
harder to figure out which part of the code were relevant to the auto
tuning, and which part of the code needed to be updated to work the
same way as the current pid.c implementation.  I started by trying to
isolate relevant changes in pid.c, and applying them to at_pid.c.  My
aim was to make sure the at_pid component could replace the pid
component with a simple change in the HAL setup loadrt line, without
having to &quot;rewire&quot; the rest of the HAL configuration.  After a few
hours following this approach, I had learned quite a lot about the
code structure of both components, while concluding I was heading down
the wrong rabbit hole, and should get back to the surface and find a
different path.&lt;/p&gt;

&lt;p&gt;For the second attempt, I decided to throw away all the PID control
related part of the original at_pid.c, and instead isolate and lift
the auto tuning part of the code and inject it into a copy of pid.c.
This ensured compatibility with the current pid component, while
adding auto tuning as a run time option.  To make it easier to identify
the relevant parts in the future, I wrapped all the auto tuning code
with &#39;#ifdef AUTO_TUNER&#39;.  The end result behave just like the current
pid component by default, as that part of the code is identical.  The
&lt;a href=&quot;https://github.com/LinuxCNC/linuxcnc/pull/1820&quot;&gt;end result
entered the LinuxCNC master branch&lt;/a&gt; a few days ago.&lt;/p&gt;

&lt;p&gt;To enable auto tuning, one need to set a few HAL pins in the PID
component.  The most important ones are &lt;tt&gt;tune-effort&lt;/tt&gt;,
&lt;tt&gt;tune-mode&lt;/tt&gt; and &lt;tt&gt;tune-start&lt;/tt&gt;.  But lets take a step
back, and see what the auto tuning code will do.  I do not know the
mathematical foundation of the at_pid algorithm, but from observation
I can tell that the algorithm will, when enabled, produce a square
wave pattern centered around the &lt;tt&gt;bias&lt;/tt&gt; value on the output pin
of the PID controller.  This can be seen using the HAL Scope provided
by LinuxCNC.  In my case, this is translated into voltage (+-10V) sent
to the motor controller, which in turn is translated into motor speed.
So at_pid will ask the motor to move the axis back and forth.  The
number of cycles in the pattern is controlled by the
&lt;tt&gt;tune-cycles&lt;/tt&gt; pin, and the extremes of the wave pattern is
controlled by the &lt;tt&gt;tune-effort&lt;/tt&gt; pin.  Of course, trying to
change the direction of a physical object instantly (as in going
directly from a positive voltage to the equivalent negative voltage)
do not change velocity instantly, and it take some time for the object
to slow down and move in the opposite direction.  This result in a
more smooth movement wave form, as the axis in question were vibrating
back and forth.  When the axis reached the target speed in the
opposing direction, the auto tuner change direction again.  After
several of these changes, the average time delay between the &#39;peaks&#39;
and &#39;valleys&#39; of this movement graph is then used to calculate
proposed values for Pgain, Igain and Dgain, and insert them into the
HAL model to use by the pid controller.  The auto tuned settings are
not great, but htye work a lot better than the values I had been able
to cook up on my own, at least for the horizontal X and Y axis.  But I
had to use very small &lt;tt&gt;tune-effort&lt;tt&gt; values, as my motor
controllers error out if the voltage change too quickly.  I&#39;ve been
less lucky with the Z axis, which is moving a heavy object up and
down, and seem to confuse the algorithm.  The Z axis movement became a
lot better when I introduced a &lt;tt&gt;bias&lt;/tt&gt; value to counter the
gravitational drag, but I will have to work a lot more on the Z axis
PID values.&lt;/p&gt;

&lt;p&gt;Armed with this knowledge, it is time to look at how to do the
tuning.  Lets say the HAL configuration in question load the PID
component for X, Y and Z like this:&lt;/p&gt;
 
&lt;blockquote&gt;&lt;pre&gt;
loadrt pid names=pid.x,pid.y,pid.z
&lt;/pre&gt;&lt;/blockquote&gt;

&lt;p&gt;Armed with the new and improved at_pid component, the new line will
look like this:&lt;/p&gt;

&lt;blockquote&gt;&lt;pre&gt;
loadrt at_pid names=pid.x,pid.y,pid.z
&lt;/pre&gt;&lt;/blockquote&gt;

&lt;p&gt;The rest of the HAL setup can stay the same.  This work because the
components are referenced by name.  If the component had used count=3
instead, all use of pid.# had to be changed to at_pid.#.&lt;/p&gt;

&lt;p&gt;To start tuning the X axis, move the axis to the middle of its
range, to make sure it do not hit anything when it start moving back
and forth.  Next, set the &lt;tt&gt;tune-effort&lt;/tt&gt; to a low number in the
output range.  I used 0.1 as my initial value.  Next, assign 1 to the
&lt;tt&gt;tune-mode&lt;/tt&gt; value. Note, this will disable the pid controlling
part and feed 0 to the output pin, which in my case initially caused a
lot of drift.  In my case it proved to be a good idea with X and Y to
tune the motor driver to make sure 0 voltage stopped the motor
rotation.  On the other hand, for the Z axis this proved to be a bad
idea, so it will depend on your setup.  It might help to set the
&lt;tt&gt;bias&lt;/tt&gt; value to a output value that reduce or eliminate the
axis drift.  Finally, after setting &lt;tt&gt;tune-mode&lt;/tt&gt;, set
&lt;tt&gt;tune-start&lt;/tt&gt; to 1 to activate the auto tuning.  If all go well,
your axis will vibrate for a few seconds and when it is done, new
values for Pgain, Igain and Dgain will be active.  To test them,
change &lt;tt&gt;tune-mode&lt;/tt&gt; back to 0.  Note that this might cause the
machine to suddenly jerk as it bring the axis back to its commanded
position, which it might have drifted away from during tuning.  To
summarize with some halcmd lines:&lt;/p&gt;

&lt;blockquote&gt;&lt;pre&gt;
setp pid.x.tune-effort 0.1
setp pid.x.tune-mode 1
setp pid.x.tune-start 1
# wait for the tuning to complete
setp pid.x.tune-mode 0
&lt;/pre&gt;&lt;/blockquote&gt;

&lt;p&gt;After doing this task quite a few times while trying to figure out
how to properly tune the PID controllers on the machine in, I decided
to figure out if this process could be automated, and wrote a script
to do the entire tuning process from power on.  The end result will
ensure the machine is powered on and ready to run, home all axis if it
is not already done, check that the extra tuning pins are available,
move the axis to its mid point, run the auto tuning and re-enable the
pid controller when it is done.  It can be run several times.  Check
out the
&lt;a href=&quot;https://github.com/SebKuzminsky/MazakVQC1540/blob/bon-dev/scripts/run-auto-pid-tuner&quot;&gt;run-auto-pid-tuner&lt;/a&gt;
script on github if you want to learn how it is done.&lt;/p&gt;

&lt;p&gt;My hope is that this little adventure can inspire someone who know
more about motor PID controller tuning can implement even better
algorithms for automatic PID tuning in LinuxCNC, making life easier
for both me and all the others that want to use LinuxCNC but lack the
in depth knowledge needed to tune PID controllers well.&lt;/p&gt;

&lt;p&gt;As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
&lt;b&gt;&lt;a href=&quot;bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&quot;&gt;15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&lt;/a&gt;&lt;/b&gt;.&lt;/p&gt;
</description>
        </item>
        
        </channel>
</rss>