--- /dev/null
+Title: 45 orphaned Debian packages moved to git, 391 to go
+Tags: english, debian
+Date: 2024-04-25 22:00
+
+<p>Nine days ago, I started migrating orphaned Debian packages with no
+version control system listed in debian/control of the source to git.
+At the time there were 438 such packages. Now there are 391,
+according to the UDD. In reality it is slightly less, as there is a
+delay between uploads and UDD updates. In the nine days since, I have
+thus been able to work my way through ten percent of the packages. I
+am starting to run out of steam, and hope someone else will also help
+brushing some dust of these packages. Here is a recipe how to do it.
+
+I start by picking a random package by querying the UDD for a list of
+10 random packages from the set of remaining packages:
+
+<blockquote><pre>
+PGPASSWORD="udd-mirror" psql --port=5432 --host=udd-mirror.debian.net \
+ --username=udd-mirror udd -c "select source from sources \
+ where release = 'sid' and (vcs_url ilike '%anonscm.debian.org%' \
+ OR vcs_browser ilike '%anonscm.debian.org%' or vcs_url IS NULL \
+ OR vcs_browser IS NULL) AND maintainer ilike '%packages@qa.debian.org%' \
+ order by random() limit 10;"
+</pre></blockquote>
+
+<p>Next, I visit http://salsa.debian.org/debian and search for the
+package name, to ensure no git repository already exist. If it does,
+I clone it and try to get it to an uploadable state, and add the Vcs-*
+entries in d/control to make the repository more widely known. These
+packages are a minority, so I will not cover that use case here.</p>
+
+<p>For packages without an existing git repository, I run the
+following script <tt>debian-snap-to-salsa</tt> to prepare a git
+repository with the existing packaging.</p>
+
+<blockquote><pre>
+#!/bin/sh
+#
+# See also https://bugs.debian.org/804722#31
+
+set -e
+
+# Move to this Standards-Version.
+SV_LATEST=4.7.0
+
+PKG="$1"
+
+if [ -z "$PKG" ]; then
+ echo "usage: $0 <pkgname>"
+ exit 1
+fi
+
+if [ -e "${PKG}-salsa" ]; then
+ echo "error: ${PKG}-salsa already exist, aborting."
+ exit 1
+fi
+
+if [ -z "ALLOWFAILURE" ] ; then
+ ALLOWFAILURE=false
+fi
+
+# Fetch every snapshotted source package. Manually loop until all
+# transfers succeed, as 'gbp import-dscs --debsnap' do not fail on
+# download failures.
+until debsnap --force -v $PKG || $ALLOWFAILURE ; do sleep 1; done
+mkdir ${PKG}-salsa; cd ${PKG}-salsa
+git init
+
+# Specify branches to override any debian/gbp.conf file present in the
+# source package.
+gbp import-dscs --debian-branch=master --upstream-branch=upstream \
+ --pristine-tar ../source-$PKG/*.dsc
+
+# Add Vcs pointing to Salsa Debian project (must be manually created
+# and pushed to).
+if ! grep -q ^Vcs- debian/control ; then
+ awk "BEGIN { s=1 } /^\$/ { if (s==1) { print \"Vcs-Browser: https://salsa.debian.org/debian/$PKG\"; print \"Vcs-Git: https://salsa.debian.org/debian/$PKG.git\" }; s=0 } { print }" < debian/control > debian/control.new && mv debian/control.new debian/control
+ git commit -m "Updated vcs in d/control to Salsa." debian/control
+fi
+
+# Tell gbp to enforce the use of pristine-tar.
+inifile +inifile debian/gbp.conf +create +section DEFAULT +key pristine-tar +value True
+git add debian/gbp.conf
+git commit -m "Added d/gbp.conf to enforce the use of pristine-tar." debian/gbp.conf
+
+# Update to latest Standards-Version.
+SV="$(grep ^Standards-Version: debian/control|awk '{print $2}')"
+if [ $SV_LATEST != $SV ]; then
+ sed -i "s/\(Standards-Version: \)\(.*\)/\1$SV_LATEST/" debian/control
+ git commit -m "Updated Standards-Version from $SV to $SV_LATEST." debian/control
+fi
+
+if grep -q pkg-config debian/control; then
+ sed -i s/pkg-config/pkgconf/ debian/control
+ git commit -m "Replaced obsolete pkg-config build dependency with pkgconf." debian/control
+fi
+
+if grep -q libncurses5-dev debian/control; then
+ sed -i s/libncurses5-dev/libncurses-dev/ debian/control
+ git commit -m "Replaced obsolete libncurses5-dev build dependency with libncurses-dev." debian/control
+fi
+</pre></blockquote>
+
+Some times the debsnap script fail to download some of the versions.
+In those cases I investigate, and if I decide the failing versions
+will not be missed, I call it using ALLOWFAILURE=true to ignore the
+problem and create the git repository anyway.</p>
+
+<p>With the git repository in place, I do a test build (gbp
+buildpackage) to ensure the build is actually working. If it does not
+I pick a different package, or if the build failure is trivial to fix,
+I fix it before continuing. At this stage I revisit
+http://salsa.debian.org/debian and create the project under this group
+for the package. I then follow the instructions to publish the local
+git repository. Here is from a recent example:</p>
+
+<blockquote><pre>
+git remote add origin git@salsa.debian.org:debian/perl-byacc.git
+git push --set-upstream origin master upstream pristine-tar
+git push --tags
+</pre></blockquote>
+
+<p>With a working build, I have a look at the build rules if I want to
+remove some more dust. I normally try to move to debhelper compat
+level 13, which involves removing debian/compat and modifying
+debian/control to build depend on debhelper-compat (=13). I also test
+with 'Rules-Requires-Root: no' in debian/control and verify in
+debian/rules that hardening is enabled, and include all of these if
+the package still build. If it fail to build with level 13, I try
+with 12, 11, 10 and so on until I find a level where it build, as I do
+not want to spend a lot of time fixing build issues.</p>
+
+<p>Some times, when I feel inspired, I make sure debian/copyright is
+converted to the machine readable format, often by starting with
+'debhelper -cc' and then cleaning up the autogenerated content until
+it matches realities. If I feel like it, I might also clean up
+non-dh-based debian/rules files to use the short style dh build
+rules.</p>
+
+<p>Once I have removed all the dust I care to process for the package,
+I run 'gbp dch' to generate a debian/changelog entry based on the
+commits done so far, run 'dch -r' to switch from 'UNRELEASED' to
+'unstable' and get an editor to make sure the 'QA upload' marker is in
+place and that all long commit descriptions are wrapped into sensible
+lengths, run 'debcommit --release -a' to commit and tag the new
+debian/changelog entry, run 'debuild -S' to build a source only
+package, and 'dput ../perl-byacc_2.0-10_source.changes' to do the
+upload. During the entire process, and many times per step, I run
+'debuild' to verify the changes done still work. I also some times
+verify the set of built files using 'find debian' to see if I can spot
+any problems (like no file in usr/bin any more or empty package). I
+also try to fix all lintian issues reported at the end of each
+'debuild' run.</p>
+
+<p>If I find Debian specific patches, I try to ensure their metadata
+is fairly up to date and some times I even try to reach out to
+upstream, to make the upstream project aware of the patches. Most of
+my emails bounce, so the success rate is low. For projects with no
+Homepage entry in debian/control I try to track down one, and for
+packages with no debian/watch file I try to create one. But at least
+for some of the packages I have been unable to find a functioning
+upstream, and must skip both of these.</p>
+
+<p>If I could handle ten percent in nine days, twenty people could
+complete the rest in less then five days. I use approximately twenty
+minutes per package, when I have twenty minutes spare time to spend.
+Perhaps you got twenty minutes to spare too?</p>
+
+<p>As usual, if you use Bitcoin and want to show your support of my
+activities, please send Bitcoin donations to my address
+<b><a href="bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b">15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b</a></b>.</p>
projects / homepage.git / commitdiff