<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/Release_0_1_1_of_free_software_archive_system_Nikita_announced.html">Release 0.1.1 of free software archive system Nikita announced</a></div>
- <div class="date">10th June 2017</div>
- <div class="body"><p>I am very happy to report that the
-<a href="https://github.com/hiOA-ABI/nikita-noark5-core">Nikita Noark 5
-core project</a> tagged its second release today. The free software
-solution is an implementation of the Norwegian archive standard Noark
-5 used by government offices in Norway. These were the changes in
-version 0.1.1 since version 0.1.0 (from NEWS.md):
-
-<ul>
-
- <li>Continued work on the angularjs GUI, including document upload.</li>
- <li>Implemented correspondencepartPerson, correspondencepartUnit and
- correspondencepartInternal</li>
- <li>Applied for coverity coverage and started submitting code on
- regualr basis.</li>
- <li>Started fixing bugs reported by coverity</li>
- <li>Corrected and completed HATEOAS links to make sure entire API is
- available via URLs in _links.</li>
- <li>Corrected all relation URLs to use trailing slash.</li>
- <li>Add initial support for storing data in ElasticSearch.</li>
- <li>Now able to receive and store uploaded files in the archive.</li>
- <li>Changed JSON output for object lists to have relations in _links.</li>
- <li>Improve JSON output for empty object lists.</li>
- <li>Now uses correct MIME type application/vnd.noark5-v4+json.</li>
- <li>Added support for docker container images.</li>
- <li>Added simple API browser implemented in JavaScript/Angular.</li>
- <li>Started on archive client implemented in JavaScript/Angular.</li>
- <li>Started on prototype to show the public mail journal.</li>
- <li>Improved performance by disabling Sprint FileWatcher.</li>
- <li>Added support for 'arkivskaper', 'saksmappe' and 'journalpost'.</li>
- <li>Added support for some metadata codelists.</li>
- <li>Added support for Cross-origin resource sharing (CORS).</li>
- <li>Changed login method from Basic Auth to JSON Web Token (RFC 7519)
- style.</li>
- <li>Added support for GET-ing ny-* URLs.</li>
- <li>Added support for modifying entities using PUT and eTag.</li>
- <li>Added support for returning XML output on request.</li>
- <li>Removed support for English field and class names, limiting ourself
- to the official names.</li>
- <li>...</li>
-
-</ul>
-
-<p>If this sound interesting to you, please contact us on IRC (#nikita
-on irc.freenode.net) or email
-(<a href="https://lists.nuug.no/mailman/listinfo/nikita-noark">nikita-noark
-mailing list).</p>
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/Legal_to_share_more_than_3000_movies_listed_on_IMDB_.html">Legal to share more than 3000 movies listed on IMDB?</a></div>
+ <div class="date">18th November 2017</div>
+ <div class="body"><p>A month ago, I blogged about my work to
+<a href="http://people.skolelinux.org/pere/blog/Locating_IMDB_IDs_of_movies_in_the_Internet_Archive_using_Wikidata.html">automatically
+check the copyright status of IMDB entries</a>, and try to count the
+number of movies listed in IMDB that is legal to distribute on the
+Internet. I have continued to look for good data sources, and
+identified a few more. The code used to extract information from
+various data sources is available in
+<a href="https://github.com/petterreinholdtsen/public-domain-free-imdb">a
+git repository</a>, currently available from github.</p>
+
+<p>So far I have identified 3186 unique IMDB title IDs. To gain
+better understanding of the structure of the data set, I created a
+histogram of the year associated with each movie (typically release
+year). It is interesting to notice where the peaks and dips in the
+graph are located. I wonder why they are placed there. I suspect
+World War II caused the dip around 1940, but what caused the peak
+around 2010?</p>
+
+<p align="center"><img src="http://people.skolelinux.org/pere/blog/images/2017-11-18-verk-i-det-fri-filmer.png" /></p>
+
+<p>I've so far identified ten sources for IMDB title IDs for movies in
+the public domain or with a free license. This is the statistics
+reported when running 'make stats' in the git repository:</p>
+
+<pre>
+ 249 entries ( 6 unique) with and 288 without IMDB title ID in free-movies-archive-org-butter.json
+ 2301 entries ( 540 unique) with and 0 without IMDB title ID in free-movies-archive-org-wikidata.json
+ 830 entries ( 29 unique) with and 0 without IMDB title ID in free-movies-icheckmovies-archive-mochard.json
+ 2109 entries ( 377 unique) with and 0 without IMDB title ID in free-movies-imdb-pd.json
+ 291 entries ( 122 unique) with and 0 without IMDB title ID in free-movies-letterboxd-pd.json
+ 144 entries ( 135 unique) with and 0 without IMDB title ID in free-movies-manual.json
+ 350 entries ( 1 unique) with and 801 without IMDB title ID in free-movies-publicdomainmovies.json
+ 4 entries ( 0 unique) with and 124 without IMDB title ID in free-movies-publicdomainreview.json
+ 698 entries ( 119 unique) with and 118 without IMDB title ID in free-movies-publicdomaintorrents.json
+ 8 entries ( 8 unique) with and 196 without IMDB title ID in free-movies-vodo.json
+ 3186 unique IMDB title IDs in total
+</pre>
+
+<p>The entries without IMDB title ID are candidates to increase the
+data set, but might equally well be duplicates of entries already
+listed with IMDB title ID in one of the other sources, or represent
+movies that lack a IMDB title ID. I've seen examples of all these
+situations when peeking at the entries without IMDB title ID. Based
+on these data sources, the lower bound for movies listed in IMDB that
+are legal to distribute on the Internet is between 3186 and 4713.
+
+<p>It would be great for improving the accuracy of this measurement,
+if the various sources added IMDB title ID to their metadata. I have
+tried to reach the people behind the various sources to ask if they
+are interested in doing this, without any replies so far. Perhaps you
+can help me get in touch with the people behind VODO, Public Domain
+Torrents, Public Domain Movies and Public Domain Review to try to
+convince them to add more metadata to their movie entries?</p>
+
+<p>Another way you could help is by adding pages to Wikipedia about
+movies that are legal to distribute on the Internet. If such page
+exist and include a link to both IMDB and The Internet Archive, the
+script used to generate free-movies-archive-org-wikidata.json should
+pick up the mapping as soon as wikidata is updates.</p>
</div>
<div class="tags">
- Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug</a>, <a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn</a>, <a href="http://people.skolelinux.org/pere/blog/tags/standard">standard</a>.
+ Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett</a>.
</div>
<div class="padding"></div>
<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/Idea_for_storing_trusted_timestamps_in_a_Noark_5_archive.html">Idea for storing trusted timestamps in a Noark 5 archive</a></div>
- <div class="date"> 7th June 2017</div>
- <div class="body"><p><em>This is a copy of
-<a href="https://lists.nuug.no/pipermail/nikita-noark/2017-June/000297.html">an
-email I posted to the nikita-noark mailing list</a>. Please follow up
-there if you would like to discuss this topic. The background is that
-we are making a free software archive system based on the Norwegian
-<a href="https://www.arkivverket.no/forvaltning-og-utvikling/regelverk-og-standarder/noark-standarden">Noark
-5 standard</a> for government archives.</em></p>
-
-<p>I've been wondering a bit lately how trusted timestamps could be
-stored in Noark 5.
-<a href="https://en.wikipedia.org/wiki/Trusted_timestamping">Trusted
-timestamps</a> can be used to verify that some information
-(document/file/checksum/metadata) have not been changed since a
-specific time in the past. This is useful to verify the integrity of
-the documents in the archive.</p>
-
-<p>Then it occured to me, perhaps the trusted timestamps could be
-stored as dokument variants (ie dokumentobjekt referered to from
-dokumentbeskrivelse) with the filename set to the hash it is
-stamping?</p>
-
-<p>Given a "dokumentbeskrivelse" with an associated "dokumentobjekt",
-a new dokumentobjekt is associated with "dokumentbeskrivelse" with the
-same attributes as the stamped dokumentobjekt except these
-attributes:</p>
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/Some_notes_on_fault_tolerant_storage_systems.html">Some notes on fault tolerant storage systems</a></div>
+ <div class="date"> 1st November 2017</div>
+ <div class="body"><p>If you care about how fault tolerant your storage is, you might
+find these articles and papers interesting. They have formed how I
+think of when designing a storage system.</p>
<ul>
-<li>format -> "RFC3161"
-<li>mimeType -> "application/timestamp-reply"
-<li>formatDetaljer -> "<source URL for timestamp service>"
-<li>filenavn -> "<sjekksum>.tsr"
+<li>USENIX :login; <a
+href="https://www.usenix.org/publications/login/summer2017/ganesan">Redundancy
+Does Not Imply Fault Tolerance. Analysis of Distributed Storage
+Reactions to Single Errors and Corruptions</a> by Aishwarya Ganesan,
+Ramnatthan Alagappan, Andrea C. Arpaci-Dusseau, and Remzi
+H. Arpaci-Dusseau</li>
+
+<li>ZDNet
+<a href="http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/">Why
+RAID 5 stops working in 2009</a> by Robin Harris</li>
+
+<li>ZDNet
+<a href="http://www.zdnet.com/article/why-raid-6-stops-working-in-2019/">Why
+RAID 6 stops working in 2019</a> by Robin Harris</li>
+
+<li>USENIX FAST'07
+<a href="http://research.google.com/archive/disk_failures.pdf">Failure
+Trends in a Large Disk Drive Population</a> by Eduardo Pinheiro,
+Wolf-Dietrich Weber and Luiz André Barroso</li>
+
+<li>USENIX ;login: <a
+href="https://www.usenix.org/system/files/login/articles/hughes12-04.pdf">Data
+Integrity. Finding Truth in a World of Guesses and Lies</a> by Doug
+Hughes</li>
+
+<li>USENIX FAST'08
+<a href="https://www.usenix.org/events/fast08/tech/full_papers/bairavasundaram/bairavasundaram_html/">An
+Analysis of Data Corruption in the Storage Stack</a> by
+L. N. Bairavasundaram, G. R. Goodson, B. Schroeder, A. C.
+Arpaci-Dusseau, and R. H. Arpaci-Dusseau</li>
+
+<li>USENIX FAST'07 <a
+href="https://www.usenix.org/legacy/events/fast07/tech/schroeder/schroeder_html/">Disk
+failures in the real world: what does an MTTF of 1,000,000 hours mean
+to you?</a> by B. Schroeder and G. A. Gibson.</li>
+
+<li>USENIX ;login: <a
+href="https://www.usenix.org/events/fast08/tech/full_papers/jiang/jiang_html/">Are
+Disks the Dominant Contributor for Storage Failures? A Comprehensive
+Study of Storage Subsystem Failure Characteristics</a> by Weihang
+Jiang, Chongfeng Hu, Yuanyuan Zhou, and Arkady Kanevsky</li>
+
+<li>SIGMETRICS 2007
+<a href="http://research.cs.wisc.edu/adsl/Publications/latent-sigmetrics07.pdf">An
+analysis of latent sector errors in disk drives</a> by
+L. N. Bairavasundaram, G. R. Goodson, S. Pasupathy, and J. Schindler</li>
</ul>
-<p>This assume a service following
-<a href="https://tools.ietf.org/html/rfc3161">IETF RFC 3161</a> is
-used, which specifiy the given MIME type for replies and the .tsr file
-ending for the content of such trusted timestamp. As far as I can
-tell from the Noark 5 specifications, it is OK to have several
-variants/renderings of a dokument attached to a given
-dokumentbeskrivelse objekt. It might be stretching it a bit to make
-some of these variants represent crypto-signatures useful for
-verifying the document integrity instead of representing the dokument
-itself.</p>
-
-<p>Using the source of the service in formatDetaljer allow several
-timestamping services to be used. This is useful to spread the risk
-of key compromise over several organisations. It would only be a
-problem to trust the timestamps if all of the organisations are
-compromised.</p>
-
-<p>The following oneliner on Linux can be used to generate the tsr
-file. $input is the path to the file to checksum, and $sha256 is the
-SHA-256 checksum of the file (ie the "<sjekksum>.tsr" value mentioned
-above).</p>
-
-<p><blockquote><pre>
-openssl ts -query -data "$inputfile" -cert -sha256 -no_nonce \
- | curl -s -H "Content-Type: application/timestamp-query" \
- --data-binary "@-" http://zeitstempel.dfn.de > $sha256.tsr
-</pre></blockquote></p>
-
-<p>To verify the timestamp, you first need to download the public key
-of the trusted timestamp service, for example using this command:</p>
-
-<p><blockquote><pre>
-wget -O ca-cert.txt \
- https://pki.pca.dfn.de/global-services-ca/pub/cacert/chain.txt
-</pre></blockquote></p>
-
-<p>Note, the public key should be stored alongside the timestamps in
-the archive to make sure it is also available 100 years from now. It
-is probably a good idea to standardise how and were to store such
-public keys, to make it easier to find for those trying to verify
-documents 100 or 1000 years from now. :)</p>
-
-<p>The verification itself is a simple openssl command:</p>
-
-<p><blockquote><pre>
-openssl ts -verify -data $inputfile -in $sha256.tsr \
- -CAfile ca-cert.txt -text
-</pre></blockquote></p>
-
-<p>Is there any reason this approach would not work? Is it somehow against
-the Noark 5 specification?</p>
+<p>Several of these research papers are based on data collected from
+hundred thousands or millions of disk, and their findings are eye
+opening. The short story is simply do not implicitly trust RAID or
+redundant storage systems. Details matter. And unfortunately there
+are few options on Linux addressing all the identified issues. Both
+ZFS and Btrfs are doing a fairly good job, but have legal and
+practical issues on their own. I wonder how cluster file systems like
+Ceph do in this regard. After all, there is an old saying, you know
+you have a distributed system when the crash of a computer you have
+never heard of stops you from getting any work done. The same holds
+true if fault tolerance do not work.</p>
+
+<p>Just remember, in the end, it do not matter how redundant, or how
+fault tolerant your storage is, if you do not continuously monitor its
+status to detect and replace failed disks.</p>
</div>
<div class="tags">
- Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn</a>, <a href="http://people.skolelinux.org/pere/blog/tags/standard">standard</a>.
+ Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/raid">raid</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
</div>
<div class="padding"></div>
<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/N_r_nynorskoversettelsen_svikter_til_eksamen___.html">Når nynorskoversettelsen svikter til eksamen...</a></div>
- <div class="date"> 3rd June 2017</div>
- <div class="body"><p><a href="http://www.aftenposten.no/norge/Krever-at-elever-ma-fa-annullert-eksamen-etter-rot-med-oppgavetekster-622459b.html">Aftenposten
-melder i dag</a> om feil i eksamensoppgavene for eksamen i politikk og
-menneskerettigheter, der teksten i bokmåls og nynorskutgaven ikke var
-like. Oppgaveteksten er gjengitt i artikkelen, og jeg ble nysgjerring
-på om den fri oversetterløsningen
-<a href="https://www.apertium.org/">Apertium</a> ville gjort en bedre
-jobb enn Utdanningsdirektoratet. Det kan se slik ut.</p>
-
-<p>Her er bokmålsoppgaven fra eksamenen:</p>
-
-<blockquote>
-<p>Drøft utfordringene knyttet til nasjonalstatenes og andre aktørers
-rolle og muligheter til å håndtere internasjonale utfordringer, som
-for eksempel flykningekrisen.</p>
-
-<p>Vedlegge er eksempler på tekster som kan gi relevante perspektiver
-på temaet:</p>
-<ol>
-<li>Flykningeregnskapet 2016, UNHCR og IDMC
-<li>«Grenseløst Europa for fall» A-Magasinet, 26. november 2015
-</ol>
-
-</blockquote>
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/Web_services_for_writing_academic_LaTeX_papers_as_a_team.html">Web services for writing academic LaTeX papers as a team</a></div>
+ <div class="date">31st October 2017</div>
+ <div class="body"><p>I was surprised today to learn that a friend in academia did not
+know there are easily available web services available for writing
+LaTeX documents as a team. I thought it was common knowledge, but to
+make sure at least my readers are aware of it, I would like to mention
+these useful services for writing LaTeX documents. Some of them even
+provide a WYSIWYG editor to ease writing even further.</p>
+
+<p>There are two commercial services available,
+<a href="https://sharelatex.com">ShareLaTeX</a> and
+<a href="https://overleaf.com">Overleaf</a>. They are very easy to
+use. Just start a new document, select which publisher to write for
+(ie which LaTeX style to use), and start writing. Note, these two
+have announced their intention to join forces, so soon it will only be
+one joint service. I've used both for different documents, and they
+work just fine. While
+<a href="https://github.com/sharelatex/sharelatex">ShareLaTeX is free
+software</a>, while the latter is not. According to <a
+href="https://www.overleaf.com/help/17-is-overleaf-open-source">a
+announcement from Overleaf</a>, they plan to keep the ShareLaTeX code
+base maintained as free software.</p>
+
+But these two are not the only alternatives.
+<a href="https://app.fiduswriter.org/">Fidus Writer</a> is another free
+software solution with <a href="https://github.com/fiduswriter">the
+source available on github</a>. I have not used it myself. Several
+others can be found on the nice
+<a href="https://alternativeto.net/software/sharelatex/">alterntiveTo
+web service</a>.
+
+<p>If you like Google Docs or Etherpad, but would like to write
+documents in LaTeX, you should check out these services. You can even
+host your own, if you want to. :)</p>
-<p>Dette oversetter Apertium slik:</p>
-
-<blockquote>
-<p>Drøft utfordringane knytte til nasjonalstatane sine og rolla til
-andre aktørar og høve til å handtera internasjonale utfordringar, som
-til dømes *flykningekrisen.</p>
-
-<p>Vedleggja er døme på tekster som kan gje relevante perspektiv på
-temaet:</p>
-
-<ol>
-<li>*Flykningeregnskapet 2016, *UNHCR og *IDMC</li>
-<li>«*Grenseløst Europa for fall» A-Magasinet, 26. november 2015</li>
-</ol>
-
-</blockquote>
-
-<p>Ord som ikke ble forstått er markert med stjerne (*), og trenger
-ekstra språksjekk. Men ingen ord er forsvunnet, slik det var i
-oppgaven elevene fikk presentert på eksamen. Jeg mistenker dog at
-"andre aktørers rolle og muligheter til ..." burde vært oversatt til
-"rolla til andre aktørar og deira høve til ..." eller noe slikt, men
-det er kanskje flisespikking. Det understreker vel bare at det alltid
-trengs korrekturlesning etter automatisk oversettelse.</p>
</div>
<div class="tags">
- Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk</a>, <a href="http://people.skolelinux.org/pere/blog/tags/stavekontroll">stavekontroll</a>.
+ Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>.
</div>
<div class="padding"></div>
<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/Epost_inn_som_arkivformat_i_Riksarkivarens_forskrift_.html">Epost inn som arkivformat i Riksarkivarens forskrift?</a></div>
- <div class="date">27th April 2017</div>
- <div class="body"><p>I disse dager, med frist 1. mai, har Riksarkivaren ute en høring på
-sin forskrift. Som en kan se er det ikke mye tid igjen før fristen
-som går ut på søndag. Denne forskriften er det som lister opp hvilke
-formater det er greit å arkivere i
-<a href="http://www.arkivverket.no/arkivverket/Offentleg-forvalting/Noark/Noark-5">Noark
-5-løsninger</a> i Norge.</p>
-
-<p>Jeg fant høringsdokumentene hos
-<a href="https://www.arkivrad.no/aktuelt/riksarkivarens-forskrift-pa-horing">Norsk
-Arkivråd</a> etter å ha blitt tipset på epostlisten til
-<a href="https://github.com/hiOA-ABI/nikita-noark5-core">fri
-programvareprosjektet Nikita Noark5-Core</a>, som lager et Noark 5
-Tjenestegresesnitt. Jeg er involvert i Nikita-prosjektet og takket
-være min interesse for tjenestegrensesnittsprosjektet har jeg lest en
-god del Noark 5-relaterte dokumenter, og til min overraskelse oppdaget
-at standard epost ikke er på listen over godkjente formater som kan
-arkiveres. Høringen med frist søndag er en glimrende mulighet til å
-forsøke å gjøre noe med det. Jeg holder på med
-<a href="https://github.com/petterreinholdtsen/noark5-tester/blob/master/docs/hoering-arkivforskrift.tex">egen
-høringsuttalelse</a>, og lurer på om andre er interessert i å støtte
-forslaget om å tillate arkivering av epost som epost i arkivet.</p>
-
-<p>Er du igang med å skrive egen høringsuttalelse allerede? I så fall
-kan du jo vurdere å ta med en formulering om epost-lagring. Jeg tror
-ikke det trengs så mye. Her et kort forslag til tekst:</p>
-
-<p><blockquote>
-
- <p>Viser til høring sendt ut 2017-02-17 (Riksarkivarens referanse
- 2016/9840 HELHJO), og tillater oss å sende inn noen innspill om
- revisjon av Forskrift om utfyllende tekniske og arkivfaglige
- bestemmelser om behandling av offentlige arkiver (Riksarkivarens
- forskrift).</p>
-
- <p>Svært mye av vår kommuikasjon foregår i dag på e-post. Vi
- foreslår derfor at Internett-e-post, slik det er beskrevet i IETF
- RFC 5322,
- <a href="https://tools.ietf.org/html/rfc5322">https://tools.ietf.org/html/rfc5322</a>. bør
- inn som godkjent dokumentformat. Vi foreslår at forskriftens
- oversikt over godkjente dokumentformater ved innlevering i § 5-16
- endres til å ta med Internett-e-post.</p>
-
-</blockquote></p>
-
-<p>Som del av arbeidet med tjenestegrensesnitt har vi testet hvordan
-epost kan lagres i en Noark 5-struktur, og holder på å skrive et
-forslag om hvordan dette kan gjøres som vil bli sendt over til
-arkivverket så snart det er ferdig. De som er interesserte kan
-<a href="https://github.com/petterreinholdtsen/noark5-tester/blob/master/docs/epostlagring.md">følge
-fremdriften på web</a>.</p>
-
-<p>Oppdatering 2017-04-28: I dag ble høringuttalelsen jeg skrev
- <a href="https://www.nuug.no/news/NUUGs_h_ringuttalelse_til_Riksarkivarens_forskrift.shtml">sendt
- inn av foreningen NUUG</a>.</p>
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/Locating_IMDB_IDs_of_movies_in_the_Internet_Archive_using_Wikidata.html">Locating IMDB IDs of movies in the Internet Archive using Wikidata</a></div>
+ <div class="date">25th October 2017</div>
+ <div class="body"><p>Recently, I needed to automatically check the copyright status of a
+set of <a href="http://www.imdb.com/">The Internet Movie database
+(IMDB)</a> entries, to figure out which one of the movies they refer
+to can be freely distributed on the Internet. This proved to be
+harder than it sounds. IMDB for sure list movies without any
+copyright protection, where the copyright protection has expired or
+where the movie is lisenced using a permissive license like one from
+Creative Commons. These are mixed with copyright protected movies,
+and there seem to be no way to separate these classes of movies using
+the information in IMDB.</p>
+
+<p>First I tried to look up entries manually in IMDB,
+<a href="https://www.wikipedia.org/">Wikipedia</a> and
+<a href="https://www.archive.org/">The Internet Archive</a>, to get a
+feel how to do this. It is hard to know for sure using these sources,
+but it should be possible to be reasonable confident a movie is "out
+of copyright" with a few hours work per movie. As I needed to check
+almost 20,000 entries, this approach was not sustainable. I simply
+can not work around the clock for about 6 years to check this data
+set.</p>
+
+<p>I asked the people behind The Internet Archive if they could
+introduce a new metadata field in their metadata XML for IMDB ID, but
+was told that they leave it completely to the uploaders to update the
+metadata. Some of the metadata entries had IMDB links in the
+description, but I found no way to download all metadata files in bulk
+to locate those ones and put that approach aside.</p>
+
+<p>In the process I noticed several Wikipedia articles about movies
+had links to both IMDB and The Internet Archive, and it occured to me
+that I could use the Wikipedia RDF data set to locate entries with
+both, to at least get a lower bound on the number of movies on The
+Internet Archive with a IMDB ID. This is useful based on the
+assumption that movies distributed by The Internet Archive can be
+legally distributed on the Internet. With some help from the RDF
+community (thank you DanC), I was able to come up with this query to
+pass to <a href="https://query.wikidata.org/">the SPARQL interface on
+Wikidata</a>:
+
+<p><pre>
+SELECT ?work ?imdb ?ia ?when ?label
+WHERE
+{
+ ?work wdt:P31/wdt:P279* wd:Q11424.
+ ?work wdt:P345 ?imdb.
+ ?work wdt:P724 ?ia.
+ OPTIONAL {
+ ?work wdt:P577 ?when.
+ ?work rdfs:label ?label.
+ FILTER(LANG(?label) = "en").
+ }
+}
+</pre></p>
+
+<p>If I understand the query right, for every film entry anywhere in
+Wikpedia, it will return the IMDB ID and The Internet Archive ID, and
+when the movie was released and its English title, if either or both
+of the latter two are available. At the moment the result set contain
+2338 entries. Of course, it depend on volunteers including both
+correct IMDB and The Internet Archive IDs in the wikipedia articles
+for the movie. It should be noted that the result will include
+duplicates if the movie have entries in several languages. There are
+some bogus entries, either because The Internet Archive ID contain a
+typo or because the movie is not available from The Internet Archive.
+I did not verify the IMDB IDs, as I am unsure how to do that
+automatically.</p>
+
+<p>I wrote a small python script to extract the data set from Wikidata
+and check if the XML metadata for the movie is available from The
+Internet Archive, and after around 1.5 hour it produced a list of 2097
+free movies and their IMDB ID. In total, 171 entries in Wikidata lack
+the refered Internet Archive entry. I assume the 70 "disappearing"
+entries (ie 2338-2097-171) are duplicate entries.</p>
+
+<p>This is not too bad, given that The Internet Archive report to
+contain <a href="https://archive.org/details/feature_films">5331
+feature films</a> at the moment, but it also mean more than 3000
+movies are missing on Wikipedia or are missing the pair of references
+on Wikipedia.</p>
+
+<p>I was curious about the distribution by release year, and made a
+little graph to show how the amount of free movies is spread over the
+years:<p>
+
+<p><img src="http://people.skolelinux.org/pere/blog/images/2017-10-25-verk-i-det-fri-filmer.png"></p>
+
+<p>I expect the relative distribution of the remaining 3000 movies to
+be similar.</p>
+
+<p>If you want to help, and want to ensure Wikipedia can be used to
+cross reference The Internet Archive and The Internet Movie Database,
+please make sure entries like this are listed under the "External
+links" heading on the Wikipedia article for the movie:</p>
+
+<p><pre>
+* {{Internet Archive film|id=FightingLady}}
+* {{IMDb title|id=0036823|title=The Fighting Lady}}
+</pre></p>
+
+<p>Please verify the links on the final page, to make sure you did not
+introduce a typo.</p>
+
+<p>Here is the complete list, if you want to correct the 171
+identified Wikipedia entries with broken links to The Internet
+Archive: <a href="http://www.wikidata.org/entity/Q1140317">Q1140317</a>,
+<a href="http://www.wikidata.org/entity/Q458656">Q458656</a>,
+<a href="http://www.wikidata.org/entity/Q458656">Q458656</a>,
+<a href="http://www.wikidata.org/entity/Q470560">Q470560</a>,
+<a href="http://www.wikidata.org/entity/Q743340">Q743340</a>,
+<a href="http://www.wikidata.org/entity/Q822580">Q822580</a>,
+<a href="http://www.wikidata.org/entity/Q480696">Q480696</a>,
+<a href="http://www.wikidata.org/entity/Q128761">Q128761</a>,
+<a href="http://www.wikidata.org/entity/Q1307059">Q1307059</a>,
+<a href="http://www.wikidata.org/entity/Q1335091">Q1335091</a>,
+<a href="http://www.wikidata.org/entity/Q1537166">Q1537166</a>,
+<a href="http://www.wikidata.org/entity/Q1438334">Q1438334</a>,
+<a href="http://www.wikidata.org/entity/Q1479751">Q1479751</a>,
+<a href="http://www.wikidata.org/entity/Q1497200">Q1497200</a>,
+<a href="http://www.wikidata.org/entity/Q1498122">Q1498122</a>,
+<a href="http://www.wikidata.org/entity/Q865973">Q865973</a>,
+<a href="http://www.wikidata.org/entity/Q834269">Q834269</a>,
+<a href="http://www.wikidata.org/entity/Q841781">Q841781</a>,
+<a href="http://www.wikidata.org/entity/Q841781">Q841781</a>,
+<a href="http://www.wikidata.org/entity/Q1548193">Q1548193</a>,
+<a href="http://www.wikidata.org/entity/Q499031">Q499031</a>,
+<a href="http://www.wikidata.org/entity/Q1564769">Q1564769</a>,
+<a href="http://www.wikidata.org/entity/Q1585239">Q1585239</a>,
+<a href="http://www.wikidata.org/entity/Q1585569">Q1585569</a>,
+<a href="http://www.wikidata.org/entity/Q1624236">Q1624236</a>,
+<a href="http://www.wikidata.org/entity/Q4796595">Q4796595</a>,
+<a href="http://www.wikidata.org/entity/Q4853469">Q4853469</a>,
+<a href="http://www.wikidata.org/entity/Q4873046">Q4873046</a>,
+<a href="http://www.wikidata.org/entity/Q915016">Q915016</a>,
+<a href="http://www.wikidata.org/entity/Q4660396">Q4660396</a>,
+<a href="http://www.wikidata.org/entity/Q4677708">Q4677708</a>,
+<a href="http://www.wikidata.org/entity/Q4738449">Q4738449</a>,
+<a href="http://www.wikidata.org/entity/Q4756096">Q4756096</a>,
+<a href="http://www.wikidata.org/entity/Q4766785">Q4766785</a>,
+<a href="http://www.wikidata.org/entity/Q880357">Q880357</a>,
+<a href="http://www.wikidata.org/entity/Q882066">Q882066</a>,
+<a href="http://www.wikidata.org/entity/Q882066">Q882066</a>,
+<a href="http://www.wikidata.org/entity/Q204191">Q204191</a>,
+<a href="http://www.wikidata.org/entity/Q204191">Q204191</a>,
+<a href="http://www.wikidata.org/entity/Q1194170">Q1194170</a>,
+<a href="http://www.wikidata.org/entity/Q940014">Q940014</a>,
+<a href="http://www.wikidata.org/entity/Q946863">Q946863</a>,
+<a href="http://www.wikidata.org/entity/Q172837">Q172837</a>,
+<a href="http://www.wikidata.org/entity/Q573077">Q573077</a>,
+<a href="http://www.wikidata.org/entity/Q1219005">Q1219005</a>,
+<a href="http://www.wikidata.org/entity/Q1219599">Q1219599</a>,
+<a href="http://www.wikidata.org/entity/Q1643798">Q1643798</a>,
+<a href="http://www.wikidata.org/entity/Q1656352">Q1656352</a>,
+<a href="http://www.wikidata.org/entity/Q1659549">Q1659549</a>,
+<a href="http://www.wikidata.org/entity/Q1660007">Q1660007</a>,
+<a href="http://www.wikidata.org/entity/Q1698154">Q1698154</a>,
+<a href="http://www.wikidata.org/entity/Q1737980">Q1737980</a>,
+<a href="http://www.wikidata.org/entity/Q1877284">Q1877284</a>,
+<a href="http://www.wikidata.org/entity/Q1199354">Q1199354</a>,
+<a href="http://www.wikidata.org/entity/Q1199354">Q1199354</a>,
+<a href="http://www.wikidata.org/entity/Q1199451">Q1199451</a>,
+<a href="http://www.wikidata.org/entity/Q1211871">Q1211871</a>,
+<a href="http://www.wikidata.org/entity/Q1212179">Q1212179</a>,
+<a href="http://www.wikidata.org/entity/Q1238382">Q1238382</a>,
+<a href="http://www.wikidata.org/entity/Q4906454">Q4906454</a>,
+<a href="http://www.wikidata.org/entity/Q320219">Q320219</a>,
+<a href="http://www.wikidata.org/entity/Q1148649">Q1148649</a>,
+<a href="http://www.wikidata.org/entity/Q645094">Q645094</a>,
+<a href="http://www.wikidata.org/entity/Q5050350">Q5050350</a>,
+<a href="http://www.wikidata.org/entity/Q5166548">Q5166548</a>,
+<a href="http://www.wikidata.org/entity/Q2677926">Q2677926</a>,
+<a href="http://www.wikidata.org/entity/Q2698139">Q2698139</a>,
+<a href="http://www.wikidata.org/entity/Q2707305">Q2707305</a>,
+<a href="http://www.wikidata.org/entity/Q2740725">Q2740725</a>,
+<a href="http://www.wikidata.org/entity/Q2024780">Q2024780</a>,
+<a href="http://www.wikidata.org/entity/Q2117418">Q2117418</a>,
+<a href="http://www.wikidata.org/entity/Q2138984">Q2138984</a>,
+<a href="http://www.wikidata.org/entity/Q1127992">Q1127992</a>,
+<a href="http://www.wikidata.org/entity/Q1058087">Q1058087</a>,
+<a href="http://www.wikidata.org/entity/Q1070484">Q1070484</a>,
+<a href="http://www.wikidata.org/entity/Q1080080">Q1080080</a>,
+<a href="http://www.wikidata.org/entity/Q1090813">Q1090813</a>,
+<a href="http://www.wikidata.org/entity/Q1251918">Q1251918</a>,
+<a href="http://www.wikidata.org/entity/Q1254110">Q1254110</a>,
+<a href="http://www.wikidata.org/entity/Q1257070">Q1257070</a>,
+<a href="http://www.wikidata.org/entity/Q1257079">Q1257079</a>,
+<a href="http://www.wikidata.org/entity/Q1197410">Q1197410</a>,
+<a href="http://www.wikidata.org/entity/Q1198423">Q1198423</a>,
+<a href="http://www.wikidata.org/entity/Q706951">Q706951</a>,
+<a href="http://www.wikidata.org/entity/Q723239">Q723239</a>,
+<a href="http://www.wikidata.org/entity/Q2079261">Q2079261</a>,
+<a href="http://www.wikidata.org/entity/Q1171364">Q1171364</a>,
+<a href="http://www.wikidata.org/entity/Q617858">Q617858</a>,
+<a href="http://www.wikidata.org/entity/Q5166611">Q5166611</a>,
+<a href="http://www.wikidata.org/entity/Q5166611">Q5166611</a>,
+<a href="http://www.wikidata.org/entity/Q324513">Q324513</a>,
+<a href="http://www.wikidata.org/entity/Q374172">Q374172</a>,
+<a href="http://www.wikidata.org/entity/Q7533269">Q7533269</a>,
+<a href="http://www.wikidata.org/entity/Q970386">Q970386</a>,
+<a href="http://www.wikidata.org/entity/Q976849">Q976849</a>,
+<a href="http://www.wikidata.org/entity/Q7458614">Q7458614</a>,
+<a href="http://www.wikidata.org/entity/Q5347416">Q5347416</a>,
+<a href="http://www.wikidata.org/entity/Q5460005">Q5460005</a>,
+<a href="http://www.wikidata.org/entity/Q5463392">Q5463392</a>,
+<a href="http://www.wikidata.org/entity/Q3038555">Q3038555</a>,
+<a href="http://www.wikidata.org/entity/Q5288458">Q5288458</a>,
+<a href="http://www.wikidata.org/entity/Q2346516">Q2346516</a>,
+<a href="http://www.wikidata.org/entity/Q5183645">Q5183645</a>,
+<a href="http://www.wikidata.org/entity/Q5185497">Q5185497</a>,
+<a href="http://www.wikidata.org/entity/Q5216127">Q5216127</a>,
+<a href="http://www.wikidata.org/entity/Q5223127">Q5223127</a>,
+<a href="http://www.wikidata.org/entity/Q5261159">Q5261159</a>,
+<a href="http://www.wikidata.org/entity/Q1300759">Q1300759</a>,
+<a href="http://www.wikidata.org/entity/Q5521241">Q5521241</a>,
+<a href="http://www.wikidata.org/entity/Q7733434">Q7733434</a>,
+<a href="http://www.wikidata.org/entity/Q7736264">Q7736264</a>,
+<a href="http://www.wikidata.org/entity/Q7737032">Q7737032</a>,
+<a href="http://www.wikidata.org/entity/Q7882671">Q7882671</a>,
+<a href="http://www.wikidata.org/entity/Q7719427">Q7719427</a>,
+<a href="http://www.wikidata.org/entity/Q7719444">Q7719444</a>,
+<a href="http://www.wikidata.org/entity/Q7722575">Q7722575</a>,
+<a href="http://www.wikidata.org/entity/Q2629763">Q2629763</a>,
+<a href="http://www.wikidata.org/entity/Q2640346">Q2640346</a>,
+<a href="http://www.wikidata.org/entity/Q2649671">Q2649671</a>,
+<a href="http://www.wikidata.org/entity/Q7703851">Q7703851</a>,
+<a href="http://www.wikidata.org/entity/Q7747041">Q7747041</a>,
+<a href="http://www.wikidata.org/entity/Q6544949">Q6544949</a>,
+<a href="http://www.wikidata.org/entity/Q6672759">Q6672759</a>,
+<a href="http://www.wikidata.org/entity/Q2445896">Q2445896</a>,
+<a href="http://www.wikidata.org/entity/Q12124891">Q12124891</a>,
+<a href="http://www.wikidata.org/entity/Q3127044">Q3127044</a>,
+<a href="http://www.wikidata.org/entity/Q2511262">Q2511262</a>,
+<a href="http://www.wikidata.org/entity/Q2517672">Q2517672</a>,
+<a href="http://www.wikidata.org/entity/Q2543165">Q2543165</a>,
+<a href="http://www.wikidata.org/entity/Q426628">Q426628</a>,
+<a href="http://www.wikidata.org/entity/Q426628">Q426628</a>,
+<a href="http://www.wikidata.org/entity/Q12126890">Q12126890</a>,
+<a href="http://www.wikidata.org/entity/Q13359969">Q13359969</a>,
+<a href="http://www.wikidata.org/entity/Q13359969">Q13359969</a>,
+<a href="http://www.wikidata.org/entity/Q2294295">Q2294295</a>,
+<a href="http://www.wikidata.org/entity/Q2294295">Q2294295</a>,
+<a href="http://www.wikidata.org/entity/Q2559509">Q2559509</a>,
+<a href="http://www.wikidata.org/entity/Q2559912">Q2559912</a>,
+<a href="http://www.wikidata.org/entity/Q7760469">Q7760469</a>,
+<a href="http://www.wikidata.org/entity/Q6703974">Q6703974</a>,
+<a href="http://www.wikidata.org/entity/Q4744">Q4744</a>,
+<a href="http://www.wikidata.org/entity/Q7766962">Q7766962</a>,
+<a href="http://www.wikidata.org/entity/Q7768516">Q7768516</a>,
+<a href="http://www.wikidata.org/entity/Q7769205">Q7769205</a>,
+<a href="http://www.wikidata.org/entity/Q7769988">Q7769988</a>,
+<a href="http://www.wikidata.org/entity/Q2946945">Q2946945</a>,
+<a href="http://www.wikidata.org/entity/Q3212086">Q3212086</a>,
+<a href="http://www.wikidata.org/entity/Q3212086">Q3212086</a>,
+<a href="http://www.wikidata.org/entity/Q18218448">Q18218448</a>,
+<a href="http://www.wikidata.org/entity/Q18218448">Q18218448</a>,
+<a href="http://www.wikidata.org/entity/Q18218448">Q18218448</a>,
+<a href="http://www.wikidata.org/entity/Q6909175">Q6909175</a>,
+<a href="http://www.wikidata.org/entity/Q7405709">Q7405709</a>,
+<a href="http://www.wikidata.org/entity/Q7416149">Q7416149</a>,
+<a href="http://www.wikidata.org/entity/Q7239952">Q7239952</a>,
+<a href="http://www.wikidata.org/entity/Q7317332">Q7317332</a>,
+<a href="http://www.wikidata.org/entity/Q7783674">Q7783674</a>,
+<a href="http://www.wikidata.org/entity/Q7783704">Q7783704</a>,
+<a href="http://www.wikidata.org/entity/Q7857590">Q7857590</a>,
+<a href="http://www.wikidata.org/entity/Q3372526">Q3372526</a>,
+<a href="http://www.wikidata.org/entity/Q3372642">Q3372642</a>,
+<a href="http://www.wikidata.org/entity/Q3372816">Q3372816</a>,
+<a href="http://www.wikidata.org/entity/Q3372909">Q3372909</a>,
+<a href="http://www.wikidata.org/entity/Q7959649">Q7959649</a>,
+<a href="http://www.wikidata.org/entity/Q7977485">Q7977485</a>,
+<a href="http://www.wikidata.org/entity/Q7992684">Q7992684</a>,
+<a href="http://www.wikidata.org/entity/Q3817966">Q3817966</a>,
+<a href="http://www.wikidata.org/entity/Q3821852">Q3821852</a>,
+<a href="http://www.wikidata.org/entity/Q3420907">Q3420907</a>,
+<a href="http://www.wikidata.org/entity/Q3429733">Q3429733</a>,
+<a href="http://www.wikidata.org/entity/Q774474">Q774474</a></p>
</div>
<div class="tags">
- Tags: <a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk</a>, <a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn</a>, <a href="http://people.skolelinux.org/pere/blog/tags/standard">standard</a>.
+ Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett</a>.
</div>
<div class="padding"></div>
<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/Offentlig_elektronisk_postjournal_blokkerer_tilgang_for_utvalgte_webklienter.html">Offentlig elektronisk postjournal blokkerer tilgang for utvalgte webklienter</a></div>
- <div class="date">20th April 2017</div>
- <div class="body"><p>Jeg oppdaget i dag at <a href="https://www.oep.no/">nettstedet som
-publiserer offentlige postjournaler fra statlige etater</a>, OEP, har
-begynt å blokkerer enkelte typer webklienter fra å få tilgang. Vet
-ikke hvor mange det gjelder, men det gjelder i hvert fall libwww-perl
-og curl. For å teste selv, kjør følgende:</p>
-
-<blockquote><pre>
-% curl -v -s https://www.oep.no/pub/report.xhtml?reportId=3 2>&1 |grep '< HTTP'
-< HTTP/1.1 404 Not Found
-% curl -v -s --header 'User-Agent:Opera/12.0' https://www.oep.no/pub/report.xhtml?reportId=3 2>&1 |grep '< HTTP'
-< HTTP/1.1 200 OK
-%
-</pre></blockquote>
-
-<p>Her kan en se at tjenesten gir «404 Not Found» for curl i
-standardoppsettet, mens den gir «200 OK» hvis curl hevder å være Opera
-versjon 12.0. Offentlig elektronisk postjournal startet blokkeringen
-2017-03-02.</p>
-
-<p>Blokkeringen vil gjøre det litt vanskeligere å maskinelt hente
-informasjon fra oep.no. Kan blokkeringen være gjort for å hindre
-automatisert innsamling av informasjon fra OEP, slik Pressens
-Offentlighetsutvalg gjorde for å dokumentere hvordan departementene
-hindrer innsyn i
-<a href="http://presse.no/dette-mener-np/undergraver-offentlighetsloven/">rapporten
-«Slik hindrer departementer innsyn» som ble publiserte i januar
-2017</a>. Det virker usannsynlig, da det jo er trivielt å bytte
-User-Agent til noe nytt.</p>
-
-<p>Finnes det juridisk grunnlag for det offentlige å diskriminere
-webklienter slik det gjøres her? Der tilgang gis eller ikke alt etter
-hva klienten sier at den heter? Da OEP eies av DIFI og driftes av
-Basefarm, finnes det kanskje noen dokumenter sendt mellom disse to
-aktørene man kan be om innsyn i for å forstå hva som har skjedd. Men
-<a href="https://www.oep.no/search/result.html?period=dateRange&fromDate=01.01.2016&toDate=01.04.2017&dateType=documentDate&caseDescription=&descType=both&caseNumber=&documentNumber=&sender=basefarm&senderType=both&documentType=all&legalAuthority=&archiveCode=&list2=196&searchType=advanced&Search=Search+in+records">postjournalen
-til DIFI viser kun to dokumenter</a> det siste året mellom DIFI og
-Basefarm.
-<a href="https://www.mimesbronn.no/request/blokkering_av_tilgang_til_oep_fo">Mimes brønn neste</a>,
-tenker jeg.</p>
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/A_one_way_wall_on_the_border_.html">A one-way wall on the border?</a></div>
+ <div class="date">14th October 2017</div>
+ <div class="body"><p>I find it fascinating how many of the people being locked inside
+the proposed border wall between USA and Mexico support the idea. The
+proposal to keep Mexicans out reminds me of
+<a href="http://www.history.com/news/10-things-you-may-not-know-about-the-berlin-wall">the
+propaganda twist from the East Germany government</a> calling the wall
+the “Antifascist Bulwark” after erecting the Berlin Wall, claiming
+that the wall was erected to keep enemies from creeping into East
+Germany, while it was obvious to the people locked inside it that it
+was erected to keep the people from escaping.</p>
+
+<p>Do the people in USA supporting this wall really believe it is a
+one way wall, only keeping people on the outside from getting in,
+while not keeping people in the inside from getting out?</p>
</div>
<div class="tags">
- Tags: <a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk</a>, <a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn</a>.
+ Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>.
</div>
<div class="padding"></div>
<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/Free_software_archive_system_Nikita_now_able_to_store_documents.html">Free software archive system Nikita now able to store documents</a></div>
- <div class="date">19th March 2017</div>
- <div class="body"><p>The <a href="https://github.com/hiOA-ABI/nikita-noark5-core">Nikita
-Noark 5 core project</a> is implementing the Norwegian standard for
-keeping an electronic archive of government documents.
-<a href="http://www.arkivverket.no/arkivverket/Offentlig-forvaltning/Noark/Noark-5/English-version">The
-Noark 5 standard</a> document the requirement for data systems used by
-the archives in the Norwegian government, and the Noark 5 web interface
-specification document a REST web service for storing, searching and
-retrieving documents and metadata in such archive. I've been involved
-in the project since a few weeks before Christmas, when the Norwegian
-Unix User Group
-<a href="https://www.nuug.no/news/NOARK5_kjerne_som_fri_programvare_f_r_epostliste_hos_NUUG.shtml">announced
-it supported the project</a>. I believe this is an important project,
-and hope it can make it possible for the government archives in the
-future to use free software to keep the archives we citizens depend
-on. But as I do not hold such archive myself, personally my first use
-case is to store and analyse public mail journal metadata published
-from the government. I find it useful to have a clear use case in
-mind when developing, to make sure the system scratches one of my
-itches.</p>
-
-<p>If you would like to help make sure there is a free software
-alternatives for the archives, please join our IRC channel
-(<a href="irc://irc.freenode.net/%23nikita"">#nikita on
-irc.freenode.net</a>) and
-<a href="https://lists.nuug.no/mailman/listinfo/nikita-noark">the
-project mailing list</a>.</p>
-
-<p>When I got involved, the web service could store metadata about
-documents. But a few weeks ago, a new milestone was reached when it
-became possible to store full text documents too. Yesterday, I
-completed an implementation of a command line tool
-<tt>archive-pdf</tt> to upload a PDF file to the archive using this
-API. The tool is very simple at the moment, and find existing
-<a href="https://en.wikipedia.org/wiki/Fonds">fonds</a>, series and
-files while asking the user to select which one to use if more than
-one exist. Once a file is identified, the PDF is associated with the
-file and uploaded, using the title extracted from the PDF itself. The
-process is fairly similar to visiting the archive, opening a cabinet,
-locating a file and storing a piece of paper in the archive. Here is
-a test run directly after populating the database with test data using
-our API tester:</p>
-
-<p><blockquote><pre>
-~/src//noark5-tester$ ./archive-pdf mangelmelding/mangler.pdf
-using arkiv: Title of the test fonds created 2017-03-18T23:49:32.103446
-using arkivdel: Title of the test series created 2017-03-18T23:49:32.103446
-
- 0 - Title of the test case file created 2017-03-18T23:49:32.103446
- 1 - Title of the test file created 2017-03-18T23:49:32.103446
-Select which mappe you want (or search term): 0
-Uploading mangelmelding/mangler.pdf
- PDF title: Mangler i spesifikasjonsdokumentet for NOARK 5 Tjenestegrensesnitt
- File 2017/1: Title of the test case file created 2017-03-18T23:49:32.103446
-~/src//noark5-tester$
-</pre></blockquote></p>
-
-<p>You can see here how the fonds (arkiv) and serie (arkivdel) only had
-one option, while the user need to choose which file (mappe) to use
-among the two created by the API tester. The <tt>archive-pdf</tt>
-tool can be found in the git repository for the API tester.</p>
-
-<p>In the project, I have been mostly working on
-<a href="https://github.com/petterreinholdtsen/noark5-tester">the API
-tester</a> so far, while getting to know the code base. The API
-tester currently use
-<a href="https://en.wikipedia.org/wiki/HATEOAS">the HATEOAS links</a>
-to traverse the entire exposed service API and verify that the exposed
-operations and objects match the specification, as well as trying to
-create objects holding metadata and uploading a simple XML file to
-store. The tester has proved very useful for finding flaws in our
-implementation, as well as flaws in the reference site and the
-specification.</p>
-
-<p>The test document I uploaded is a summary of all the specification
-defects we have collected so far while implementing the web service.
-There are several unclear and conflicting parts of the specification,
-and we have
-<a href="https://github.com/petterreinholdtsen/noark5-tester/tree/master/mangelmelding">started
-writing down</a> the questions we get from implementing it. We use a
-format inspired by how <a href="http://www.opengroup.org/austin/">The
-Austin Group</a> collect defect reports for the POSIX standard with
-<a href="http://www.opengroup.org/austin/mantis.html">their
-instructions for the MANTIS defect tracker system</a>, in lack of an official way to structure defect reports for Noark 5 (our first submitted defect report was a <a href="https://github.com/petterreinholdtsen/noark5-tester/blob/master/mangelmelding/sendt/2017-03-15-mangel-prosess.md">request for a procedure for submitting defect reports</a> :).
-
-<p>The Nikita project is implemented using Java and Spring, and is
-fairly easy to get up and running using Docker containers for those
-that want to test the current code base. The API tester is
-implemented in Python.</p>
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/Generating_3D_prints_in_Debian_using_Cura_and_Slic3r__prusa_.html">Generating 3D prints in Debian using Cura and Slic3r(-prusa)</a></div>
+ <div class="date"> 9th October 2017</div>
+ <div class="body"><p>At my nearby maker space,
+<a href="http://sonen.ifi.uio.no/">Sonen</a>, I heard the story that it
+was easier to generate gcode files for theyr 3D printers (Ultimake 2+)
+on Windows and MacOS X than Linux, because the software involved had
+to be manually compiled and set up on Linux while premade packages
+worked out of the box on Windows and MacOS X. I found this annoying,
+as the software involved,
+<a href="https://github.com/Ultimaker/Cura">Cura</a>, is free software
+and should be trivial to get up and running on Linux if someone took
+the time to package it for the relevant distributions. I even found
+<a href="https://bugs.debian.org/706656">a request for adding into
+Debian</a> from 2013, which had seem some activity over the years but
+never resulted in the software showing up in Debian. So a few days
+ago I offered my help to try to improve the situation.</p>
+
+<p>Now I am very happy to see that all the packages required by a
+working Cura in Debian are uploaded into Debian and waiting in the NEW
+queue for the ftpmasters to have a look. You can track the progress
+on
+<a href="https://qa.debian.org/developer.php?email=3dprinter-general%40lists.alioth.debian.org">the
+status page for the 3D printer team</a>.</p>
+
+<p>The uploaded packages are a bit behind upstream, and was uploaded
+now to get slots in <a href="https://ftp-master.debian.org/new.html">the NEW
+queue</a> while we work up updating the packages to the latest
+upstream version.</p>
+
+<p>On a related note, two competitors for Cura, which I found harder
+to use and was unable to configure correctly for Ultimaker 2+ in the
+short time I spent on it, are already in Debian. If you are looking
+for 3D printer "slicers" and want something already available in
+Debian, check out
+<a href="https://tracker.debian.org/pkg/slic3r">slic3r</a> and
+<a href="https://tracker.debian.org/pkg/slic3r-prusa">slic3r-prusa</a>.
+The latter is a fork of the former.</p>
</div>
<div class="tags">
- Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug</a>, <a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn</a>, <a href="http://people.skolelinux.org/pere/blog/tags/standard">standard</a>.
+ Tags: <a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer</a>, <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>.
</div>
<div class="padding"></div>
<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/Detecting_NFS_hangs_on_Linux_without_hanging_yourself___.html">Detecting NFS hangs on Linux without hanging yourself...</a></div>
- <div class="date"> 9th March 2017</div>
- <div class="body"><p>Over the years, administrating thousand of NFS mounting linux
-computers at the time, I often needed a way to detect if the machine
-was experiencing NFS hang. If you try to use <tt>df</tt> or look at a
-file or directory affected by the hang, the process (and possibly the
-shell) will hang too. So you want to be able to detect this without
-risking the detection process getting stuck too. It has not been
-obvious how to do this. When the hang has lasted a while, it is
-possible to find messages like these in dmesg:</p>
-
-<p><blockquote>
-nfs: server nfsserver not responding, still trying
-<br>nfs: server nfsserver OK
-</blockquote></p>
-
-<p>It is hard to know if the hang is still going on, and it is hard to
-be sure looking in dmesg is going to work. If there are lots of other
-messages in dmesg the lines might have rotated out of site before they
-are noticed.</p>
-
-<p>While reading through the nfs client implementation in linux kernel
-code, I came across some statistics that seem to give a way to detect
-it. The om_timeouts sunrpc value in the kernel will increase every
-time the above log entry is inserted into dmesg. And after digging a
-bit further, I discovered that this value show up in
-/proc/self/mountstats on Linux.</p>
-
-<p>The mountstats content seem to be shared between files using the
-same file system context, so it is enough to check one of the
-mountstats files to get the state of the mount point for the machine.
-I assume this will not show lazy umounted NFS points, nor NFS mount
-points in a different process context (ie with a different filesystem
-view), but that does not worry me.</p>
-
-<p>The content for a NFS mount point look similar to this:</p>
-
-<p><blockquote><pre>
-[...]
-device /dev/mapper/Debian-var mounted on /var with fstype ext3
-device nfsserver:/mnt/nfsserver/home0 mounted on /mnt/nfsserver/home0 with fstype nfs statvers=1.1
- opts: rw,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,soft,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=129.240.3.145,mountvers=3,mountport=4048,mountproto=udp,local_lock=all
- age: 7863311
- caps: caps=0x3fe7,wtmult=4096,dtsize=8192,bsize=0,namlen=255
- sec: flavor=1,pseudoflavor=1
- events: 61063112 732346265 1028140 35486205 16220064 8162542 761447191 71714012 37189 3891185 45561809 110486139 4850138 420353 15449177 296502 52736725 13523379 0 52182 9016896 1231 0 0 0 0 0
- bytes: 166253035039 219519120027 0 0 40783504807 185466229638 11677877 45561809
- RPC iostats version: 1.0 p/v: 100003/3 (nfs)
- xprt: tcp 925 1 6810 0 0 111505412 111480497 109 2672418560317 0 248 53869103 22481820
- per-op statistics
- NULL: 0 0 0 0 0 0 0 0
- GETATTR: 61063106 61063108 0 9621383060 6839064400 453650 77291321 78926132
- SETATTR: 463469 463470 0 92005440 66739536 63787 603235 687943
- LOOKUP: 17021657 17021657 0 3354097764 4013442928 57216 35125459 35566511
- ACCESS: 14281703 14290009 5 2318400592 1713803640 1709282 4865144 7130140
- READLINK: 125 125 0 20472 18620 0 1112 1118
- READ: 4214236 4214237 0 715608524 41328653212 89884 22622768 22806693
- WRITE: 8479010 8494376 22 187695798568 1356087148 178264904 51506907 231671771
- CREATE: 171708 171708 0 38084748 46702272 873 1041833 1050398
- MKDIR: 3680 3680 0 773980 993920 26 23990 24245
- SYMLINK: 903 903 0 233428 245488 6 5865 5917
- MKNOD: 80 80 0 20148 21760 0 299 304
- REMOVE: 429921 429921 0 79796004 61908192 3313 2710416 2741636
- RMDIR: 3367 3367 0 645112 484848 22 5782 6002
- RENAME: 466201 466201 0 130026184 121212260 7075 5935207 5961288
- LINK: 289155 289155 0 72775556 67083960 2199 2565060 2585579
- READDIR: 2933237 2933237 0 516506204 13973833412 10385 3190199 3297917
- READDIRPLUS: 1652839 1652839 0 298640972 6895997744 84735 14307895 14448937
- FSSTAT: 6144 6144 0 1010516 1032192 51 9654 10022
- FSINFO: 2 2 0 232 328 0 1 1
- PATHCONF: 1 1 0 116 140 0 0 0
- COMMIT: 0 0 0 0 0 0 0 0
-
-device binfmt_misc mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc
-[...]
-</pre></blockquote></p>
-
-<p>The key number to look at is the third number in the per-op list.
-It is the number of NFS timeouts experiences per file system
-operation. Here 22 write timeouts and 5 access timeouts. If these
-numbers are increasing, I believe the machine is experiencing NFS
-hang. Unfortunately the timeout value do not start to increase right
-away. The NFS operations need to time out first, and this can take a
-while. The exact timeout value depend on the setup. For example the
-defaults for TCP and UDP mount points are quite different, and the
-timeout value is affected by the soft, hard, timeo and retrans NFS
-mount options.</p>
-
-<p>The only way I have been able to get working on Debian and RedHat
-Enterprise Linux for getting the timeout count is to peek in /proc/.
-But according to
-<ahref="http://docs.oracle.com/cd/E19253-01/816-4555/netmonitor-12/index.html">Solaris
-10 System Administration Guide: Network Services</a>, the 'nfsstat -c'
-command can be used to get these timeout values. But this do not work
-on Linux, as far as I can tell. I
-<ahref="http://bugs.debian.org/857043">asked Debian about this</a>,
-but have not seen any replies yet.</p>
-
-<p>Is there a better way to figure out if a Linux NFS client is
-experiencing NFS hangs? Is there a way to detect which processes are
-affected? Is there a way to get the NFS mount going quickly once the
-network problem causing the NFS hang has been cleared? I would very
-much welcome some clues, as we regularly run into NFS hangs.</p>
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/Mangler_du_en_skrue__eller_har_du_en_skrue_l_s_.html">Mangler du en skrue, eller har du en skrue løs?</a></div>
+ <div class="date"> 4th October 2017</div>
+ <div class="body">Når jeg holder på med ulike prosjekter, så trenger jeg stadig ulike
+skruer. Det siste prosjektet jeg holder på med er å lage
+<a href="https://www.thingiverse.com/thing:676916">en boks til en
+HDMI-touch-skjerm</a> som skal brukes med Raspberry Pi. Boksen settes
+sammen med skruer og bolter, og jeg har vært i tvil om hvor jeg kan
+få tak i de riktige skruene. Clas Ohlson og Jernia i nærheten har
+sjelden hatt det jeg trenger. Men her om dagen fikk jeg et fantastisk
+tips for oss som bor i Oslo.
+<a href="http://www.zachskruer.no/">Zachariassen Jernvare AS</a> i
+<a href="http://www.openstreetmap.org/?mlat=59.93421&mlon=10.76795#map=19/59.93421/10.76795">Hegermannsgate
+23A på Torshov</a> har et fantastisk utvalg, og åpent mellom 09:00 og
+17:00. De selger skruer, muttere, bolter, skiver etc i løs vekt, og
+så langt har jeg fått alt jeg har lett etter. De har i tillegg det
+meste av annen jernvare, som verktøy, lamper, ledninger, etc. Jeg
+håper de har nok kunder til å holde det gående lenge, da dette er en
+butikk jeg kommer til å besøke ofte. Butikken er et funn å ha i
+nabolaget for oss som liker å bygge litt selv. :)</p>
</div>
<div class="tags">
- Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
+ Tags: <a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk</a>.
</div>
<div class="padding"></div>
<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/How_does_it_feel_to_be_wiretapped__when_you_should_be_doing_the_wiretapping___.html">How does it feel to be wiretapped, when you should be doing the wiretapping...</a></div>
- <div class="date"> 8th March 2017</div>
- <div class="body"><p>So the new president in the United States of America claim to be
-surprised to discover that he was wiretapped during the election
-before he was elected president. He even claim this must be illegal.
-Well, doh, if it is one thing the confirmations from Snowden
-documented, it is that the entire population in USA is wiretapped, one
-way or another. Of course the president candidates were wiretapped,
-alongside the senators, judges and the rest of the people in USA.</p>
-
-<p>Next, the Federal Bureau of Investigation ask the Department of
-Justice to go public rejecting the claims that Donald Trump was
-wiretapped illegally. I fail to see the relevance, given that I am
-sure the surveillance industry in USA believe they have all the legal
-backing they need to conduct mass surveillance on the entire
-world.</p>
-
-<p>There is even the director of the FBI stating that he never saw an
-order requesting wiretapping of Donald Trump. That is not very
-surprising, given how the FISA court work, with all its activity being
-secret. Perhaps he only heard about it?</p>
-
-<p>What I find most sad in this story is how Norwegian journalists
-present it. In a news reports the other day in the radio from the
-Norwegian National broadcasting Company (NRK), I heard the journalist
-claim that 'the FBI denies any wiretapping', while the reality is that
-'the FBI denies any illegal wiretapping'. There is a fundamental and
-important difference, and it make me sad that the journalists are
-unable to grasp it.</p>
-
-<p><strong>Update 2017-03-13:</strong> Look like
-<a href="https://theintercept.com/2017/03/13/rand-paul-is-right-nsa-routinely-monitors-americans-communications-without-warrants/">The
-Intercept report that US Senator Rand Paul confirm what I state above</a>.</p>
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/Visualizing_GSM_radio_chatter_using_gr_gsm_and_Hopglass.html">Visualizing GSM radio chatter using gr-gsm and Hopglass</a></div>
+ <div class="date">29th September 2017</div>
+ <div class="body"><p>Every mobile phone announce its existence over radio to the nearby
+mobile cell towers. And this radio chatter is available for anyone
+with a radio receiver capable of receiving them. Details about the
+mobile phones with very good accuracy is of course collected by the
+phone companies, but this is not the topic of this blog post. The
+mobile phone radio chatter make it possible to figure out when a cell
+phone is nearby, as it include the SIM card ID (IMSI). By paying
+attention over time, one can see when a phone arrive and when it leave
+an area. I believe it would be nice to make this information more
+available to the general public, to make more people aware of how
+their phones are announcing their whereabouts to anyone that care to
+listen.</p>
+
+<p>I am very happy to report that we managed to get something
+visualizing this information up and running for
+<a href="http://norwaymakers.org/osf17">Oslo Skaperfestival 2017</a>
+(Oslo Makers Festival) taking place today and tomorrow at Deichmanske
+library. The solution is based on the
+<a href="http://people.skolelinux.org/pere/blog/Easier_recipe_to_observe_the_cell_phones_around_you.html">simple
+recipe for listening to GSM chatter</a> I posted a few days ago, and
+will show up at the stand of <a href="http://sonen.ifi.uio.no/">Åpen
+Sone from the Computer Science department of the University of
+Oslo</a>. The presentation will show the nearby mobile phones (aka
+IMSIs) as dots in a web browser graph, with lines to the dot
+representing mobile base station it is talking to. It was working in
+the lab yesterday, and was moved into place this morning.</p>
+
+<p>We set up a fairly powerful desktop machine using Debian
+Buster/Testing with several (five, I believe) RTL2838 DVB-T receivers
+connected and visualize the visible cell phone towers using an
+<a href="https://github.com/marlow925/hopglass">English version of
+Hopglass</a>. A fairly powerfull machine is needed as the
+grgsm_livemon_headless processes from
+<a href="https://tracker.debian.org/pkg/gr-gsm">gr-gsm</a> converting
+the radio signal to data packages is quite CPU intensive.</p>
+
+<p>The frequencies to listen to, are identified using a slightly
+patched scan-and-livemon (to set the --args values for each receiver),
+and the Hopglass data is generated using the
+<a href="https://github.com/petterreinholdtsen/IMSI-catcher/tree/meshviewer-output">patches
+in my meshviewer-output branch</a>. For some reason we could not get
+more than four SDRs working. There is also a geographical map trying
+to show the location of the base stations, but I believe their
+coordinates are hardcoded to some random location in Germany, I
+believe. The code should be replaced with code to look up location in
+a text file, a sqlite database or one of the online databases
+mentioned in
+<a href="https://github.com/Oros42/IMSI-catcher/issues/14">the github
+issue for the topic</a>.
+
+<p>If this sound interesting, visit the stand at the festival!</p>
</div>
<div class="tags">
- Tags: <a href="http://people.skolelinux.org/pere/blog/tags/
english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance</a>.
+ Tags: <a href="http://people.skolelinux.org/pere/blog/tags/
debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern</a>, <a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance</a>.
</div>
<div class="padding"></div>
<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/Norwegian_Bokm_l_translation_of_The_Debian_Administrator_s_Handbook_complete__proofreading_in_progress.html">Norwegian Bokmål translation of The Debian Administrator's Handbook complete, proofreading in progress</a></div>
- <div class="date"> 3rd March 2017</div>
- <div class="body"><p>For almost a year now, we have been working on making a Norwegian
-Bokmål edition of <a href="https://debian-handbook.info/">The Debian
-Administrator's Handbook</a>. Now, thanks to the tireless effort of
-Ole-Erik, Ingrid and Andreas, the initial translation is complete, and
-we are working on the proof reading to ensure consistent language and
-use of correct computer science terms. The plan is to make the book
-available on paper, as well as in electronic form. For that to
-happen, the proof reading must be completed and all the figures need
-to be translated. If you want to help out, get in touch.</p>
-
-<p><a href="http://people.skolelinux.org/pere/debian-handbook/debian-handbook-nb-NO.pdf">A
-
-fresh PDF edition</a> in A4 format (the final book will have smaller
-pages) of the book created every morning is available for
-proofreading. If you find any errors, please
-<a href="https://hosted.weblate.org/projects/debian-handbook/">visit
-Weblate and correct the error</a>. The
-<a href="http://l.github.io/debian-handbook/stat/nb-NO/index.html">state
-of the translation including figures</a> is a useful source for those
-provide Norwegian bokmål screen shots and figures.</p>
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/Easier_recipe_to_observe_the_cell_phones_around_you.html">Easier recipe to observe the cell phones around you</a></div>
+ <div class="date">24th September 2017</div>
+ <div class="body"><p>A little more than a month ago I wrote
+<a href="http://people.skolelinux.org/pere/blog/Simpler_recipe_on_how_to_make_a_simple__7_IMSI_Catcher_using_Debian.html">how
+to observe the SIM card ID (aka IMSI number) of mobile phones talking
+to nearby mobile phone base stations using Debian GNU/Linux and a
+cheap USB software defined radio</a>, and thus being able to pinpoint
+the location of people and equipment (like cars and trains) with an
+accuracy of a few kilometer. Since then we have worked to make the
+procedure even simpler, and it is now possible to do this without any
+manual frequency tuning and without building your own packages.</p>
+
+<p>The <a href="https://tracker.debian.org/pkg/gr-gsm">gr-gsm</a>
+package is now included in Debian testing and unstable, and the
+IMSI-catcher code no longer require root access to fetch and decode
+the GSM data collected using gr-gsm.</p>
+
+<p>Here is an updated recipe, using packages built by Debian and a git
+clone of two python scripts:</p>
+
+<ol>
+
+<li>Start with a Debian machine running the Buster version (aka
+ testing).</li>
+
+<li>Run '<tt>apt install gr-gsm python-numpy python-scipy
+ python-scapy</tt>' as root to install required packages.</li>
+
+<li>Fetch the code decoding GSM packages using '<tt>git clone
+ github.com/Oros42/IMSI-catcher.git</tt>'.</li>
+
+<li>Insert USB software defined radio supported by GNU Radio.</li>
+
+<li>Enter the IMSI-catcher directory and run '<tt>python
+ scan-and-livemon</tt>' to locate the frequency of nearby base
+ stations and start listening for GSM packages on one of them.</li>
+
+<li>Enter the IMSI-catcher directory and run '<tt>python
+ simple_IMSI-catcher.py</tt>' to display the collected information.</li>
+
+</ol>
+
+<p>Note, due to a bug somewhere the scan-and-livemon program (actually
+<a href="https://github.com/ptrkrysik/gr-gsm/issues/336">its underlying
+program grgsm_scanner</a>) do not work with the HackRF radio. It does
+work with RTL 8232 and other similar USB radio receivers you can get
+very cheaply
+(<a href="https://www.ebay.com/sch/items/?_nkw=rtl+2832">for example
+from ebay</a>), so for now the solution is to scan using the RTL radio
+and only use HackRF for fetching GSM data.</p>
+
+<p>As far as I can tell, a cell phone only show up on one of the
+frequencies at the time, so if you are going to track and count every
+cell phone around you, you need to listen to all the frequencies used.
+To listen to several frequencies, use the --numrecv argument to
+scan-and-livemon to use several receivers. Further, I am not sure if
+phones using 3G or 4G will show as talking GSM to base stations, so
+this approach might not see all phones around you. I typically see
+0-400 IMSI numbers an hour when looking around where I live.</p>
+
+<p>I've tried to run the scanner on a
+<a href="https://wiki.debian.org/RaspberryPi">Raspberry Pi 2 and 3
+running Debian Buster</a>, but the grgsm_livemon_headless process seem
+to be too CPU intensive to keep up. When GNU Radio print 'O' to
+stdout, I am told there it is caused by a buffer overflow between the
+radio and GNU Radio, caused by the program being unable to read the
+GSM data fast enough. If you see a stream of 'O's from the terminal
+where you started scan-and-livemon, you need a give the process more
+CPU power. Perhaps someone are able to optimize the code to a point
+where it become possible to set up RPi3 based GSM sniffers? I tried
+using Raspbian instead of Debian, but there seem to be something wrong
+with GNU Radio on raspbian, causing glibc to abort().</p>
</div>
<div class="tags">
- Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/debian-handbook">debian-handbook</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>.
+ Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern</a>, <a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance</a>.
</div>
<div class="padding"></div>
<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/Unlimited_randomness_with_the_ChaosKey_.html">Unlimited randomness with the ChaosKey?</a></div>
- <div class="date"> 1st March 2017</div>
- <div class="body"><p>A few days ago I ordered a small batch of
-<a href="http://altusmetrum.org/ChaosKey/">the ChaosKey</a>, a small
-USB dongle for generating entropy created by Bdale Garbee and Keith
-Packard. Yesterday it arrived, and I am very happy to report that it
-work great! According to its designers, to get it to work out of the
-box, you need the Linux kernel version 4.1 or later. I tested on a
-Debian Stretch machine (kernel version 4.9), and there it worked just
-fine, increasing the available entropy very quickly. I wrote a small
-test oneliner to test. It first print the current entropy level,
-drain /dev/random, and then print the entropy level for five seconds.
-Here is the situation without the ChaosKey inserted:</p>
-
-<blockquote><pre>
-% cat /proc/sys/kernel/random/entropy_avail; \
- dd bs=1M if=/dev/random of=/dev/null count=1; \
- for n in $(seq 1 5); do \
- cat /proc/sys/kernel/random/entropy_avail; \
- sleep 1; \
- done
-300
-0+1 oppføringer inn
-0+1 oppføringer ut
-28 byte kopiert, 0,000264565 s, 106 kB/s
-4
-8
-12
-17
-21
-%
-</pre></blockquote>
-
-<p>The entropy level increases by 3-4 every second. In such case any
-application requiring random bits (like a HTTPS enabled web server)
-will halt and wait for more entrpy. And here is the situation with
-the ChaosKey inserted:</p>
-
-<blockquote><pre>
-% cat /proc/sys/kernel/random/entropy_avail; \
- dd bs=1M if=/dev/random of=/dev/null count=1; \
- for n in $(seq 1 5); do \
- cat /proc/sys/kernel/random/entropy_avail; \
- sleep 1; \
- done
-1079
-0+1 oppføringer inn
-0+1 oppføringer ut
-104 byte kopiert, 0,000487647 s, 213 kB/s
-433
-1028
-1031
-1035
-1038
-%
-</pre></blockquote>
-
-<p>Quite the difference. :) I bought a few more than I need, in case
-someone want to buy one here in Norway. :)</p>
-
-<p>Update: The dongle was presented at Debconf last year. You might
-find <a href="https://debconf16.debconf.org/talks/94/">the talk
-recording illuminating</a>. It explains exactly what the source of
-randomness is, if you are unable to spot it from the schema drawing
-available from the ChaosKey web site linked at the start of this blog
-post.</p>
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/Datalagringsdirektivet_kaster_skygger_over_H_yre_og_Arbeiderpartiet.html">Datalagringsdirektivet kaster skygger over Høyre og Arbeiderpartiet</a></div>
+ <div class="date"> 7th September 2017</div>
+ <div class="body"><p>For noen dager siden publiserte Jon Wessel-Aas en bloggpost om
+«<a href="http://www.uhuru.biz/?p=1821">Konklusjonen om datalagring som
+EU-kommisjonen ikke ville at vi skulle få se</a>». Det er en
+interessant gjennomgang av EU-domstolens syn på snurpenotovervåkning
+av befolkningen, som er klar på at det er i strid med
+EU-lovgivingen.</p>
+
+<p>Valgkampen går for fullt i Norge, og om noen få dager er siste
+frist for å avgi stemme. En ting er sikkert, Høyre og Arbeiderpartiet
+får ikke min stemme
+<a href="http://people.skolelinux.org/pere/blog/Datalagringsdirektivet_gj_r_at_Oslo_H_yre_og_Arbeiderparti_ikke_f_r_min_stemme_i__r.html">denne
+gangen heller</a>. Jeg har ikke glemt at de tvang igjennom loven som
+skulle pålegge alle data- og teletjenesteleverandører å overvåke alle
+sine kunder. En lov som er vedtatt, og aldri opphevet igjen.</p>
+
+<p>Det er tydelig fra diskusjonen rundt grenseløs digital overvåkning
+(eller "Digital Grenseforsvar" som det kalles i Orvellisk nytale) at
+hverken Høyre og Arbeiderpartiet har noen prinsipielle sperrer mot å
+overvåke hele befolkningen, og diskusjonen så langt tyder på at flere
+av de andre partiene heller ikke har det. Mange av
+<a href="https://data.holderdeord.no/votes/1301946411e">de som stemte
+for Datalagringsdirektivet i Stortinget</a> (64 fra Arbeiderpartiet,
+25 fra Høyre) er fortsatt aktive og argumenterer fortsatt for å radere
+vekk mer av innbyggernes privatsfære.</p>
+
+<p>Når myndighetene demonstrerer sin mistillit til folket, tror jeg
+folket selv bør legge litt innsats i å verne sitt privatliv, ved å ta
+i bruk ende-til-ende-kryptert kommunikasjon med sine kjente og kjære,
+og begrense hvor mye privat informasjon som deles med uvedkommende.
+Det er jo ingenting som tyder på at myndighetene kommer til å være vår
+privatsfære.
+<a href="http://people.skolelinux.org/pere/blog/How_to_talk_with_your_loved_ones_in_private.html">Det
+er mange muligheter</a>. Selv har jeg litt sans for
+<a href="https://ring.cx/">Ring</a>, som er basert på p2p-teknologi
+uten sentral kontroll, er fri programvare, og støtter meldinger, tale
+og video. Systemet er tilgjengelig ut av boksen fra
+<a href="https://tracker.debian.org/pkg/ring">Debian</a> og
+<a href="https://launchpad.net/ubuntu/+source/ring">Ubuntu</a>, og det
+finnes pakker for Android, MacOSX og Windows. Foreløpig er det få
+brukere med Ring, slik at jeg også bruker
+<a href="https://signal.org/">Signal</a> som nettleserutvidelse.</p>
</div>
<div class="tags">
- Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>.
+ Tags: <a href="http://people.skolelinux.org/pere/blog/tags/dld">dld</a>, <a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk</a>, <a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern</a>, <a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget</a>, <a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance</a>, <a href="http://people.skolelinux.org/pere/blog/tags/valg">valg</a>.
</div>
<li><a href="http://people.skolelinux.org/pere/blog/archive/2017/04/">April (2)</a></li>
-<li><a href="http://people.skolelinux.org/pere/blog/archive/2017/06/">June (3)</a></li>
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2017/06/">June (5)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2017/07/">July (1)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2017/08/">August (1)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2017/09/">September (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2017/10/">October (5)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2017/11/">November (2)</a></li>
</ul></li>
<h2>Tags</h2>
<ul>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (13)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (14)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/amiga">amiga (1)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/chrpath">chrpath (2)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/debian">debian (149)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/debian">debian (154)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu (158)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/debian-handbook">debian-handbook (3)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/debian-handbook">debian-handbook (4)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/digistan">digistan (10)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/dld">dld (16)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/dld">dld (17)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/docbook">docbook (23)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/docbook">docbook (24)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/drivstoffpriser">drivstoffpriser (4)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/english">english (348)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/english">english (359)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/fiksgatami">fiksgatami (23)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (12)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (29)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (30)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/freedombox">freedombox (9)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/ldap">ldap (9)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/lego">lego (4)</a></li>
+
<li><a href="http://people.skolelinux.org/pere/blog/tags/lenker">lenker (8)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/lsdvd">lsdvd (2)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/nice free software">nice free software (9)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk (290)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk (293)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug (189)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/open311">open311 (2)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (64)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (66)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern (99)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern (104)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/raid">raid (1)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/raid">raid (2)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/reactos">reactos (1)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/scraperwiki">scraperwiki (2)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/sikkerhet">sikkerhet (52)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/sikkerhet">sikkerhet (53)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/sitesummary">sitesummary (4)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/stavekontroll">stavekontroll (6)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (11)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (12)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (48)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (52)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin (3)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin (4)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/usenix">usenix (2)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/valg">valg (8)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/valg">valg (9)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/video">video (59)</a></li>
projects / homepage.git / blobdiff