Generated.

[homepage.git] / blog / data / 2024-04-25-debian-vcs-qa.txt

Title: 45 orphaned Debian packages moved to git, 391 to go
Tags: english, debian
Date: 2024-04-25 22:00

<p>Nine days ago, I started migrating orphaned Debian packages with no
version control system listed in debian/control of the source to git.
At the time there were 438 such packages.  Now there are 391,
according to the UDD.  In reality it is slightly less, as there is a
delay between uploads and UDD updates.  In the nine days since, I have
thus been able to work my way through ten percent of the packages.  I
am starting to run out of steam, and hope someone else will also help
brushing some dust of these packages.  Here is a recipe how to do it.

I start by picking a random package by querying the UDD for a list of
10 random packages from the set of remaining packages:

<blockquote><pre>
PGPASSWORD="udd-mirror" psql --port=5432 --host=udd-mirror.debian.net \
  --username=udd-mirror udd -c "select source from sources \
   where release = 'sid' and (vcs_url ilike '%anonscm.debian.org%' \
   OR vcs_browser ilike '%anonscm.debian.org%' or vcs_url IS NULL \
   OR vcs_browser IS NULL) AND maintainer ilike '%packages@qa.debian.org%' \
   order by random() limit 10;"
</pre></blockquote>

<p>Next, I visit http://salsa.debian.org/debian and search for the
package name, to ensure no git repository already exist.  If it does,
I clone it and try to get it to an uploadable state, and add the Vcs-*
entries in d/control to make the repository more widely known.  These
packages are a minority, so I will not cover that use case here.</p>

<p>For packages without an existing git repository, I run the
following script <tt>debian-snap-to-salsa</tt> to prepare a git
repository with the existing packaging.</p>

<blockquote><pre>
#!/bin/sh
#
# See also https://bugs.debian.org/804722#31

set -e

# Move to this Standards-Version.
SV_LATEST=4.7.0

PKG="$1"

if [ -z "$PKG" ]; then
    echo "usage: $0 <pkgname>"
    exit 1
fi

if [ -e "${PKG}-salsa" ]; then
    echo "error: ${PKG}-salsa already exist, aborting."
    exit 1
fi

if [ -z "ALLOWFAILURE" ] ; then
    ALLOWFAILURE=false
fi

# Fetch every snapshotted source package.  Manually loop until all
# transfers succeed, as 'gbp import-dscs --debsnap' do not fail on
# download failures.
until debsnap --force -v $PKG || $ALLOWFAILURE ; do sleep 1; done
mkdir ${PKG}-salsa; cd ${PKG}-salsa
git init

# Specify branches to override any debian/gbp.conf file present in the
# source package.
gbp import-dscs  --debian-branch=master --upstream-branch=upstream \
    --pristine-tar ../source-$PKG/*.dsc

# Add Vcs pointing to Salsa Debian project (must be manually created
# and pushed to).
if ! grep -q ^Vcs- debian/control ; then
    awk "BEGIN { s=1 } /^\$/ { if (s==1) { print \"Vcs-Browser: https://salsa.debian.org/debian/$PKG\"; print \"Vcs-Git: https://salsa.debian.org/debian/$PKG.git\" }; s=0 } { print }" < debian/control > debian/control.new && mv debian/control.new debian/control
    git commit -m "Updated vcs in d/control to Salsa." debian/control
fi

# Tell gbp to enforce the use of pristine-tar.
inifile +inifile debian/gbp.conf +create +section DEFAULT +key pristine-tar +value True
git add debian/gbp.conf
git commit -m "Added d/gbp.conf to enforce the use of pristine-tar." debian/gbp.conf

# Update to latest Standards-Version.
SV="$(grep ^Standards-Version: debian/control|awk '{print $2}')"
if [ $SV_LATEST != $SV ]; then
    sed -i "s/\(Standards-Version: \)\(.*\)/\1$SV_LATEST/" debian/control
    git commit -m "Updated Standards-Version from $SV to $SV_LATEST." debian/control
fi

if grep -q pkg-config debian/control; then
    sed -i s/pkg-config/pkgconf/ debian/control
    git commit -m "Replaced obsolete pkg-config build dependency with pkgconf." debian/control
fi

if grep -q libncurses5-dev debian/control; then
    sed -i s/libncurses5-dev/libncurses-dev/ debian/control
    git commit -m "Replaced obsolete libncurses5-dev build dependency with libncurses-dev." debian/control
fi
</pre></blockquote>

Some times the debsnap script fail to download some of the versions.
In those cases I investigate, and if I decide the failing versions
will not be missed, I call it using ALLOWFAILURE=true to ignore the
problem and create the git repository anyway.</p>

<p>With the git repository in place, I do a test build (gbp
buildpackage) to ensure the build is actually working.  If it does not
I pick a different package, or if the build failure is trivial to fix,
I fix it before continuing.  At this stage I revisit
http://salsa.debian.org/debian and create the project under this group
for the package.  I then follow the instructions to publish the local
git repository.  Here is from a recent example:</p>

<blockquote><pre>
git remote add origin git@salsa.debian.org:debian/perl-byacc.git
git push --set-upstream origin master upstream pristine-tar
git push --tags
</pre></blockquote>

<p>With a working build, I have a look at the build rules if I want to
remove some more dust.  I normally try to move to debhelper compat
level 13, which involves removing debian/compat and modifying
debian/control to build depend on debhelper-compat (=13).  I also test
with 'Rules-Requires-Root: no' in debian/control and verify in
debian/rules that hardening is enabled, and include all of these if
the package still build.  If it fail to build with level 13, I try
with 12, 11, 10 and so on until I find a level where it build, as I do
not want to spend a lot of time fixing build issues.</p>

<p>Some times, when I feel inspired, I make sure debian/copyright is
converted to the machine readable format, often by starting with
'debhelper -cc' and then cleaning up the autogenerated content until
it matches realities.  If I feel like it, I might also clean up
non-dh-based debian/rules files to use the short style dh build
rules.</p>

<p>Once I have removed all the dust I care to process for the package,
I run 'gbp dch' to generate a debian/changelog entry based on the
commits done so far, run 'dch -r' to switch from 'UNRELEASED' to
'unstable' and get an editor to make sure the 'QA upload' marker is in
place and that all long commit descriptions are wrapped into sensible
lengths, run 'debcommit --release -a' to commit and tag the new
debian/changelog entry, run 'debuild -S' to build a source only
package, and 'dput ../perl-byacc_2.0-10_source.changes' to do the
upload.  During the entire process, and many times per step, I run
'debuild' to verify the changes done still work.  I also some times
verify the set of built files using 'find debian' to see if I can spot
any problems (like no file in usr/bin any more or empty package).  I
also try to fix all lintian issues reported at the end of each
'debuild' run.</p>

<p>If I find Debian specific patches, I try to ensure their metadata
is fairly up to date and some times I even try to reach out to
upstream, to make the upstream project aware of the patches.  Most of
my emails bounce, so the success rate is low.  For projects with no
Homepage entry in debian/control I try to track down one, and for
packages with no debian/watch file I try to create one.  But at least
for some of the packages I have been unable to find a functioning
upstream, and must skip both of these.</p>

<p>If I could handle ten percent in nine days, twenty people could
complete the rest in less then five days.  I use approximately twenty
minutes per package, when I have twenty minutes spare time to spend.
Perhaps you got twenty minutes to spare too?</p>

<p>As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
<b><a href="bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b">15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b</a></b>.</p>

<p><strong>Update 2024-05-04:</strong> There is
<a href="http://www.hungry.com/~pere/blog/images/2024-05-04-debian-snap-to-salsa.sh">an
updated edition of my migration script</a>, last updated
2024-05-04.</p>