<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/Some_notes_on_fault_tolerant_storage_systems.html">Some notes on fault tolerant storage systems</a></div>
- <div class="date"> 1st November 2017</div>
- <div class="body"><p>If you care about how fault tolerant your storage is, you might
-find these articles and papers interesting. They have formed how I
-think of when designing a storage system.</p>
-
-<ul>
-
-<li>USENIX :login; <a
-href="https://www.usenix.org/publications/login/summer2017/ganesan">Redundancy
-Does Not Imply Fault Tolerance. Analysis of Distributed Storage
-Reactions to Single Errors and Corruptions</a> by Aishwarya Ganesan,
-Ramnatthan Alagappan, Andrea C. Arpaci-Dusseau, and Remzi
-H. Arpaci-Dusseau</li>
-
-<li>ZDNet
-<a href="http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/">Why
-RAID 5 stops working in 2009</a> by Robin Harris</li>
-
-<li>ZDNet
-<a href="http://www.zdnet.com/article/why-raid-6-stops-working-in-2019/">Why
-RAID 6 stops working in 2019</a> by Robin Harris</li>
-
-<li>USENIX FAST'07
-<a href="http://research.google.com/archive/disk_failures.pdf">Failure
-Trends in a Large Disk Drive Population</a> by Eduardo Pinheiro,
-Wolf-Dietrich Weber and Luiz André Barroso</li>
-
-<li>USENIX ;login: <a
-href="https://www.usenix.org/system/files/login/articles/hughes12-04.pdf">Data
-Integrity. Finding Truth in a World of Guesses and Lies</a> by Doug
-Hughes</li>
-
-<li>USENIX FAST'08
-<a href="https://www.usenix.org/events/fast08/tech/full_papers/bairavasundaram/bairavasundaram_html/">An
-cAnalysis of Data Corruption in the Storage Stack</a> by
-L. N. Bairavasundaram, G. R. Goodson, B. Schroeder, A. C.
-Arpaci-Dusseau, and R. H. Arpaci-Dusseau</li>
-
-<li>USENIX FAST'07 <a
-href="https://www.usenix.org/legacy/events/fast07/tech/schroeder/schroeder_html/">Disk
-failures in the real world: what does an MTTF of 1,000,000 hours mean
-to you?</a> by B. Schroeder and G. A. Gibson.</li>
-
-<li>USENIX ;login: <a
-href="https://www.usenix.org/events/fast08/tech/full_papers/jiang/jiang_html/">Are
-Disks the Dominant Contributor for Storage Failures? A Comprehensive
-Study of Storage Subsystem Failure Characteristics</a> by Weihang
-Jiang, Chongfeng Hu, Yuanyuan Zhou, and Arkady Kanevsky</li>
-
-<li>SIGMETRICS 2007
-<a href="http://research.cs.wisc.edu/adsl/Publications/latent-sigmetrics07.pdf">An
-analysis of latent sector errors in disk drives</a> by
-L. N. Bairavasundaram, G. R. Goodson, S. Pasupathy, and J. Schindler</li>
-
-</ul>
-
-<p>Several of these research papers are based on data collected from
-hundred thousands or millions of disk, and their findings are eye
-opening. The short story is simply do not implicitly trust RAID or
-redundant storage systems. Details matter. And unfortunately there
-are few options on Linux addressing all the identified issues. Both
-ZFS and Btrfs are doing a fairly good job, but have legal and
-practical issues on their own. I wonder how cluster file systems like
-Ceph do in this regard. After, all the old saying, you know you have
-a distributed system when the crash of a compyter you have never heard
-of stops you from getting any work done. The same holds true if fault
-tolerance do not work.</p>
-
-<p>Just remember, in the end, it do not matter how redundant, or how
-fault tolerant your storage is, if you do not continuously monitor its
-status to detect and replace failed disks.</p>
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/Overv_kning_i_Kina_vs__Norge.html">Overvåkning i Kina vs. Norge</a></div>
+ <div class="date">12th February 2018</div>
+ <div class="body"><p>Jeg lar meg fascinere av en artikkel
+<a href="https://www.dagbladet.no/kultur/terroristene-star-pa-dora/69436116">i
+Dagbladet om Kinas håndtering av Xinjiang</a>, spesielt følgende
+utsnitt:</p>
+
+<p><blockquote>
+
+<p>«I den sørvestlige byen Kashgar nærmere grensa til
+Sentral-Asia meldes det nå at 120.000 uigurer er internert i såkalte
+omskoleringsleirer. Samtidig er det innført et omfattende
+helsesjekk-program med innsamling og lagring av DNA-prøver fra
+absolutt alle innbyggerne. De mest avanserte overvåkingsmetodene
+testes ut her. Programmer for å gjenkjenne ansikter og stemmer er på
+plass i regionen. Der har de lokale myndighetene begynt å installere
+GPS-systemer i alle kjøretøy og egne sporingsapper i
+mobiltelefoner.</p>
+
+<p>Politimetodene griper så dypt inn i folks dagligliv at motstanden
+mot Beijing-regimet øker.»</p>
+
+</blockquote></p>
+
+<p>Beskrivelsen avviker jo desverre ikke så veldig mye fra tilstanden
+her i Norge.</p>
+
+<table>
+<tr>
+<th>Dataregistrering</th>
+<th>Kina</th>
+<th>Norge</th>
+
+<tr>
+<td>Innsamling og lagring av DNA-prøver fra befolkningen</td>
+<td>Ja</td>
+<td>Delvis, planlagt for alle nyfødte.</td>
+</tr>
+
+<tr>
+<td>Ansiktsgjenkjenning</td>
+<td>Ja</td>
+<td>Ja</td>
+</tr>
+
+<tr>
+<td>Stemmegjenkjenning</td>
+<td>Ja</td>
+<td>Nei</td>
+</tr>
+
+<tr>
+<td>Posisjons-sporing av mobiltelefoner</td>
+<td>Ja</td>
+<td>Ja</td>
+</tr>
+
+<tr>
+<td>Posisjons-sporing av biler</td>
+<td>Ja</td>
+<td>Ja</td>
+</tr>
+
+</table>
+
+<p>I Norge har jo situasjonen rundt Folkehelseinstituttets lagring av
+DNA-informasjon på vegne av politiet, der de nektet å slette
+informasjon politiet ikke hadde lov til å ta vare på, gjort det klart
+at DNA tar vare på ganske lenge. I tillegg finnes det utallige
+biobanker som lagres til evig tid, og det er planer om å innføre
+<a href="https://www.aftenposten.no/norge/i/75E9/4-av-10-mener-staten-bor-lagre-DNA-profiler-pa-alle-nyfodte">evig
+lagring av DNA-materiale fra alle spebarn som fødes</a> (med mulighet
+for å be om sletting).</p>
+
+<p>I Norge er det system på plass for ansiktsgjenkjenning, som
+<a href="https://www.nrk.no/norge/kun-gardermoen-har-teknologi-for-ansiktsgjenkjenning-i-norge-1.12719461">en
+NRK-artikkel fra 2015</a> forteller er aktiv på Gardermoen, samt
+<a href="https://www.dagbladet.no/nyheter/inntil-27-000-bor-i-norge-under-falsk-id/60500781">brukes
+til å analysere bilder innsamlet av myndighetene</a>. Brukes det også
+flere plasser? Det er tett med overvåkningskamera kontrollert av
+politi og andre myndigheter i for eksempel Oslo sentrum.</p>
+
+<p>Jeg er ikke kjent med at Norge har noe system for identifisering av
+personer ved hjelp av stemmegjenkjenning.</p>
+
+<p>Posisjons-sporing av mobiltelefoner er ruinemessig tilgjengelig for
+blant annet politi, NAV og Finanstilsynet, i tråd med krav i
+telefonselskapenes konsesjon. I tillegg rapporterer smarttelefoner
+sin posisjon til utviklerne av utallige mobil-apper, der myndigheter
+og andre kan hente ut informasjon ved behov. Det er intet behov for
+noen egen app for dette.</p>
+
+<p>Posisjons-sporing av biler er rutinemessig tilgjengelig via et tett
+nett av målepunkter på veiene (automatiske bomstasjoner,
+køfribrikke-registrering, automatiske fartsmålere og andre veikamera).
+Det er i tillegg vedtatt at alle nye biler skal selges med utstyr for
+GPS-sporing (eCall).</p>
+
+<p>Det er jammen godt vi lever i et liberalt demokrati, og ikke en
+overvåkningsstat, eller?</p>
</div>
<div class="tags">
- Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/raid">raid</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
+ Tags: <a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk</a>, <a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance</a>.
</div>
<div class="padding"></div>
<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/Web_services_for_writing_academic_LaTeX_papers_as_a_team.html">Web services for writing academic LaTeX papers as a team</a></div>
- <div class="date">31st October 2017</div>
- <div class="body"><p>I was surprised today to learn that a friend in academia did not
-know there are easily available web services available for writing
-LaTeX documents as a team. I thought it was common knowledge, but to
-make sure at least my readers are aware of it, I would like to mention
-these useful services for writing LaTeX documents. Some of them even
-provide a WYSIWYG editor to ease writing even further.</p>
-
-<p>There are two commercial services available,
-<a href="https://sharelatex.com">ShareLaTeX</a> and
-<a href="https://overleaf.com">Overleaf</a>. They are very easy to
-use. Just start a new document, select which publisher to write for
-(ie which LaTeX style to use), and start writing. Note, these two
-have announced their intention to join forces, so soon it will only be
-one joint service. I've used both for different documents, and they
-work just fine. While
-<a href="https://github.com/sharelatex/sharelatex">ShareLaTeX is free
-software</a>, while the latter is not. According to <a
-href="https://www.overleaf.com/help/17-is-overleaf-open-source">a
-announcement from Overleaf</a>, they plan to keep the ShareLaTeX code
-base maintained as free software.</p>
-
-But these two are not the only alternatives.
-<a href="https://app.fiduswriter.org/">Fidus Writer</a> is another free
-software solution with <a href="https://github.com/fiduswriter">the
-source available on github</a>. I have not used it myself. Several
-others can be found on the nice
-<a href="https://alternativeto.net/software/sharelatex/">alterntiveTo
-web service</a>.
-
-<p>If you like Google Docs or Etherpad, but would like to write
-documents in LaTeX, you should check out these services. You can even
-host your own, if you want to. :)</p>
-
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/How_hard_can______and___be_.html">How hard can æ, ø and å be?</a></div>
+ <div class="date">11th February 2018</div>
+ <div class="body"><img src="http://people.skolelinux.org/pere/blog/images/2018-02-11-peppes-unicode.jpeg" align="right"/>
+
+<p>We write 2018, and it is 30 years since Unicode was introduced.
+Most of us in Norway have come to expect the use of our alphabet to
+just work with any computer system. But it is apparently beyond reach
+of the computers printing recites at a restaurant. Recently I visited
+a Peppes pizza resturant, and noticed a few details on the recite.
+Notice how 'ø' and 'å' are replaced with strange symbols in
+'Servitør', 'Å BETALE', 'Beløp pr. gjest', 'Takk for besøket.' and 'Vi
+gleder oss til å se deg igjen'.</p>
+
+<p>I would say that this state is passed sad and over in embarrassing.</p>
+
+<p>I removed personal and private information to be nice.</p>
</div>
<div class="tags">
<div class="padding"></div>
<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/Locating_IMDB_IDs_of_movies_in_the_Internet_Archive_using_Wikidata.html">Locating IMDB IDs of movies in the Internet Archive using Wikidata</a></div>
- <div class="date">25th October 2017</div>
- <div class="body"><p>Recently, I needed to automatically check the copyright status of a
-set of <a href="http://www.imdb.com/">The Internet Movie database
-(IMDB)</a> entries, to figure out which one of the movies they refer
-to can be freely distributed on the Internet. This proved to be
-harder than it sounds. IMDB for sure list movies without any
-copyright protection, where the copyright protection has expired or
-where the movie is lisenced using a permissive license like one from
-Creative Commons. These are mixed with copyright protected movies,
-and there seem to be no way to separate these classes of movies using
-the information in IMDB.</p>
-
-<p>First I tried to look up entries manually in IMDB,
-<a href="https://www.wikipedia.org/">Wikipedia</a> and
-<a href="https://www.archive.org/">The Internet Archive</a>, to get a
-feel how to do this. It is hard to know for sure using these sources,
-but it should be possible to be reasonable confident a movie is "out
-of copyright" with a few hours work per movie. As I needed to check
-almost 20,000 entries, this approach was not sustainable. I simply
-can not work around the clock for about 6 years to check this data
-set.</p>
-
-<p>I asked the people behind The Internet Archive if they could
-introduce a new metadata field in their metadata XML for IMDB ID, but
-was told that they leave it completely to the uploaders to update the
-metadata. Some of the metadata entries had IMDB links in the
-description, but I found no way to download all metadata files in bulk
-to locate those ones and put that approach aside.</p>
-
-<p>In the process I noticed several Wikipedia articles about movies
-had links to both IMDB and The Internet Archive, and it occured to me
-that I could use the Wikipedia RDF data set to locate entries with
-both, to at least get a lower bound on the number of movies on The
-Internet Archive with a IMDB ID. This is useful based on the
-assumption that movies distributed by The Internet Archive can be
-legally distributed on the Internet. With some help from the RDF
-community (thank you DanC), I was able to come up with this query to
-pass to <a href="https://query.wikidata.org/">the SPARQL interface on
-Wikidata</a>:
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/Legal_to_share_more_than_11_000_movies_listed_on_IMDB_.html">Legal to share more than 11,000 movies listed on IMDB?</a></div>
+ <div class="date"> 7th January 2018</div>
+ <div class="body"><p>I've continued to track down list of movies that are legal to
+distribute on the Internet, and identified more than 11,000 title IDs
+in The Internet Movie Database (IMDB) so far. Most of them (57%) are
+feature films from USA published before 1923. I've also tracked down
+more than 24,000 movies I have not yet been able to map to IMDB title
+ID, so the real number could be a lot higher. According to the front
+web page for <a href="https://retrofilmvault.com/">Retro Film
+Vault</A>, there are 44,000 public domain films, so I guess there are
+still some left to identify.</p>
+
+<p>The complete data set is available from
+<a href="https://github.com/petterreinholdtsen/public-domain-free-imdb">a
+public git repository</a>, including the scripts used to create it.
+Most of the data is collected using web scraping, for example from the
+"product catalog" of companies selling copies of public domain movies,
+but any source I find believable is used. I've so far had to throw
+out three sources because I did not trust the public domain status of
+the movies listed.</p>
+
+<p>Anyway, this is the summary of the 28 collected data sources so
+far:</p>
<p><pre>
-SELECT ?work ?imdb ?ia ?when ?label
-WHERE
-{
- ?work wdt:P31/wdt:P279* wd:Q11424.
- ?work wdt:P345 ?imdb.
- ?work wdt:P724 ?ia.
- OPTIONAL {
- ?work wdt:P577 ?when.
- ?work rdfs:label ?label.
- FILTER(LANG(?label) = "en").
- }
-}
+ 2352 entries ( 66 unique) with and 15983 without IMDB title ID in free-movies-archive-org-search.json
+ 2302 entries ( 120 unique) with and 0 without IMDB title ID in free-movies-archive-org-wikidata.json
+ 195 entries ( 63 unique) with and 200 without IMDB title ID in free-movies-cinemovies.json
+ 89 entries ( 52 unique) with and 38 without IMDB title ID in free-movies-creative-commons.json
+ 344 entries ( 28 unique) with and 655 without IMDB title ID in free-movies-fesfilm.json
+ 668 entries ( 209 unique) with and 1064 without IMDB title ID in free-movies-filmchest-com.json
+ 830 entries ( 21 unique) with and 0 without IMDB title ID in free-movies-icheckmovies-archive-mochard.json
+ 19 entries ( 19 unique) with and 0 without IMDB title ID in free-movies-imdb-c-expired-gb.json
+ 6822 entries ( 6669 unique) with and 0 without IMDB title ID in free-movies-imdb-c-expired-us.json
+ 137 entries ( 0 unique) with and 0 without IMDB title ID in free-movies-imdb-externlist.json
+ 1205 entries ( 57 unique) with and 0 without IMDB title ID in free-movies-imdb-pd.json
+ 84 entries ( 20 unique) with and 167 without IMDB title ID in free-movies-infodigi-pd.json
+ 158 entries ( 135 unique) with and 0 without IMDB title ID in free-movies-letterboxd-looney-tunes.json
+ 113 entries ( 4 unique) with and 0 without IMDB title ID in free-movies-letterboxd-pd.json
+ 182 entries ( 100 unique) with and 0 without IMDB title ID in free-movies-letterboxd-silent.json
+ 229 entries ( 87 unique) with and 1 without IMDB title ID in free-movies-manual.json
+ 44 entries ( 2 unique) with and 64 without IMDB title ID in free-movies-openflix.json
+ 291 entries ( 33 unique) with and 474 without IMDB title ID in free-movies-profilms-pd.json
+ 211 entries ( 7 unique) with and 0 without IMDB title ID in free-movies-publicdomainmovies-info.json
+ 1232 entries ( 57 unique) with and 1875 without IMDB title ID in free-movies-publicdomainmovies-net.json
+ 46 entries ( 13 unique) with and 81 without IMDB title ID in free-movies-publicdomainreview.json
+ 698 entries ( 64 unique) with and 118 without IMDB title ID in free-movies-publicdomaintorrents.json
+ 1758 entries ( 882 unique) with and 3786 without IMDB title ID in free-movies-retrofilmvault.json
+ 16 entries ( 0 unique) with and 0 without IMDB title ID in free-movies-thehillproductions.json
+ 63 entries ( 16 unique) with and 141 without IMDB title ID in free-movies-vodo.json
+11583 unique IMDB title IDs in total, 8724 only in one list, 24647 without IMDB title ID
</pre></p>
-<p>If I understand the query right, for every film entry anywhere in
-Wikpedia, it will return the IMDB ID and The Internet Archive ID, and
-when the movie was released and its English title, if either or both
-of the latter two are available. At the moment the result set contain
-2338 entries. Of course, it depend on volunteers including both
-correct IMDB and The Internet Archive IDs in the wikipedia articles
-for the movie. It should be noted that the result will include
-duplicates if the movie have entries in several languages. There are
-some bogus entries, either because The Internet Archive ID contain a
-typo or because the movie is not available from The Internet Archive.
-I did not verify the IMDB IDs, as I am unsure how to do that
-automatically.</p>
-
-<p>I wrote a small python script to extract the data set from Wikidata
-and check if the XML metadata for the movie is available from The
-Internet Archive, and after around 1.5 hour it produced a list of 2097
-free movies and their IMDB ID. In total, 171 entries in Wikidata lack
-the refered Internet Archive entry. I assume the 70 "disappearing"
-entries (ie 2338-2097-171) are duplicate entries.</p>
-
-<p>This is not too bad, given that The Internet Archive report to
-contain <a href="https://archive.org/details/feature_films">5331
-feature films</a> at the moment, but it also mean more than 3000
-movies are missing on Wikipedia or are missing the pair of references
-on Wikipedia.</p>
-
-<p>I was curious about the distribution by release year, and made a
-little graph to show how the amount of free movies is spread over the
-years:<p>
-
-<p><img src="http://people.skolelinux.org/pere/blog/images/2017-10-25-verk-i-det-fri-filmer.png"></p>
-
-<p>I expect the relative distribution of the remaining 3000 movies to
-be similar.</p>
-
-<p>If you want to help, and want to ensure Wikipedia can be used to
-cross reference The Internet Archive and The Internet Movie Database,
-please make sure entries like this are listed under the "External
-links" heading on the Wikipedia article for the movie:</p>
-
-<p><pre>
-* {{Internet Archive film|id=FightingLady}}
-* {{IMDb title|id=0036823|title=The Fighting Lady}}
-</pre></p>
-
-<p>Please verify the links on the final page, to make sure you did not
-introduce a typo.</p>
-
-<p>Here is the complete list, if you want to correct the 171
-identified Wikipedia entries with broken links to The Internet
-Archive: <a href="http://www.wikidata.org/entity/Q1140317">Q1140317</a>,
-<a href="http://www.wikidata.org/entity/Q458656">Q458656</a>,
-<a href="http://www.wikidata.org/entity/Q458656">Q458656</a>,
-<a href="http://www.wikidata.org/entity/Q470560">Q470560</a>,
-<a href="http://www.wikidata.org/entity/Q743340">Q743340</a>,
-<a href="http://www.wikidata.org/entity/Q822580">Q822580</a>,
-<a href="http://www.wikidata.org/entity/Q480696">Q480696</a>,
-<a href="http://www.wikidata.org/entity/Q128761">Q128761</a>,
-<a href="http://www.wikidata.org/entity/Q1307059">Q1307059</a>,
-<a href="http://www.wikidata.org/entity/Q1335091">Q1335091</a>,
-<a href="http://www.wikidata.org/entity/Q1537166">Q1537166</a>,
-<a href="http://www.wikidata.org/entity/Q1438334">Q1438334</a>,
-<a href="http://www.wikidata.org/entity/Q1479751">Q1479751</a>,
-<a href="http://www.wikidata.org/entity/Q1497200">Q1497200</a>,
-<a href="http://www.wikidata.org/entity/Q1498122">Q1498122</a>,
-<a href="http://www.wikidata.org/entity/Q865973">Q865973</a>,
-<a href="http://www.wikidata.org/entity/Q834269">Q834269</a>,
-<a href="http://www.wikidata.org/entity/Q841781">Q841781</a>,
-<a href="http://www.wikidata.org/entity/Q841781">Q841781</a>,
-<a href="http://www.wikidata.org/entity/Q1548193">Q1548193</a>,
-<a href="http://www.wikidata.org/entity/Q499031">Q499031</a>,
-<a href="http://www.wikidata.org/entity/Q1564769">Q1564769</a>,
-<a href="http://www.wikidata.org/entity/Q1585239">Q1585239</a>,
-<a href="http://www.wikidata.org/entity/Q1585569">Q1585569</a>,
-<a href="http://www.wikidata.org/entity/Q1624236">Q1624236</a>,
-<a href="http://www.wikidata.org/entity/Q4796595">Q4796595</a>,
-<a href="http://www.wikidata.org/entity/Q4853469">Q4853469</a>,
-<a href="http://www.wikidata.org/entity/Q4873046">Q4873046</a>,
-<a href="http://www.wikidata.org/entity/Q915016">Q915016</a>,
-<a href="http://www.wikidata.org/entity/Q4660396">Q4660396</a>,
-<a href="http://www.wikidata.org/entity/Q4677708">Q4677708</a>,
-<a href="http://www.wikidata.org/entity/Q4738449">Q4738449</a>,
-<a href="http://www.wikidata.org/entity/Q4756096">Q4756096</a>,
-<a href="http://www.wikidata.org/entity/Q4766785">Q4766785</a>,
-<a href="http://www.wikidata.org/entity/Q880357">Q880357</a>,
-<a href="http://www.wikidata.org/entity/Q882066">Q882066</a>,
-<a href="http://www.wikidata.org/entity/Q882066">Q882066</a>,
-<a href="http://www.wikidata.org/entity/Q204191">Q204191</a>,
-<a href="http://www.wikidata.org/entity/Q204191">Q204191</a>,
-<a href="http://www.wikidata.org/entity/Q1194170">Q1194170</a>,
-<a href="http://www.wikidata.org/entity/Q940014">Q940014</a>,
-<a href="http://www.wikidata.org/entity/Q946863">Q946863</a>,
-<a href="http://www.wikidata.org/entity/Q172837">Q172837</a>,
-<a href="http://www.wikidata.org/entity/Q573077">Q573077</a>,
-<a href="http://www.wikidata.org/entity/Q1219005">Q1219005</a>,
-<a href="http://www.wikidata.org/entity/Q1219599">Q1219599</a>,
-<a href="http://www.wikidata.org/entity/Q1643798">Q1643798</a>,
-<a href="http://www.wikidata.org/entity/Q1656352">Q1656352</a>,
-<a href="http://www.wikidata.org/entity/Q1659549">Q1659549</a>,
-<a href="http://www.wikidata.org/entity/Q1660007">Q1660007</a>,
-<a href="http://www.wikidata.org/entity/Q1698154">Q1698154</a>,
-<a href="http://www.wikidata.org/entity/Q1737980">Q1737980</a>,
-<a href="http://www.wikidata.org/entity/Q1877284">Q1877284</a>,
-<a href="http://www.wikidata.org/entity/Q1199354">Q1199354</a>,
-<a href="http://www.wikidata.org/entity/Q1199354">Q1199354</a>,
-<a href="http://www.wikidata.org/entity/Q1199451">Q1199451</a>,
-<a href="http://www.wikidata.org/entity/Q1211871">Q1211871</a>,
-<a href="http://www.wikidata.org/entity/Q1212179">Q1212179</a>,
-<a href="http://www.wikidata.org/entity/Q1238382">Q1238382</a>,
-<a href="http://www.wikidata.org/entity/Q4906454">Q4906454</a>,
-<a href="http://www.wikidata.org/entity/Q320219">Q320219</a>,
-<a href="http://www.wikidata.org/entity/Q1148649">Q1148649</a>,
-<a href="http://www.wikidata.org/entity/Q645094">Q645094</a>,
-<a href="http://www.wikidata.org/entity/Q5050350">Q5050350</a>,
-<a href="http://www.wikidata.org/entity/Q5166548">Q5166548</a>,
-<a href="http://www.wikidata.org/entity/Q2677926">Q2677926</a>,
-<a href="http://www.wikidata.org/entity/Q2698139">Q2698139</a>,
-<a href="http://www.wikidata.org/entity/Q2707305">Q2707305</a>,
-<a href="http://www.wikidata.org/entity/Q2740725">Q2740725</a>,
-<a href="http://www.wikidata.org/entity/Q2024780">Q2024780</a>,
-<a href="http://www.wikidata.org/entity/Q2117418">Q2117418</a>,
-<a href="http://www.wikidata.org/entity/Q2138984">Q2138984</a>,
-<a href="http://www.wikidata.org/entity/Q1127992">Q1127992</a>,
-<a href="http://www.wikidata.org/entity/Q1058087">Q1058087</a>,
-<a href="http://www.wikidata.org/entity/Q1070484">Q1070484</a>,
-<a href="http://www.wikidata.org/entity/Q1080080">Q1080080</a>,
-<a href="http://www.wikidata.org/entity/Q1090813">Q1090813</a>,
-<a href="http://www.wikidata.org/entity/Q1251918">Q1251918</a>,
-<a href="http://www.wikidata.org/entity/Q1254110">Q1254110</a>,
-<a href="http://www.wikidata.org/entity/Q1257070">Q1257070</a>,
-<a href="http://www.wikidata.org/entity/Q1257079">Q1257079</a>,
-<a href="http://www.wikidata.org/entity/Q1197410">Q1197410</a>,
-<a href="http://www.wikidata.org/entity/Q1198423">Q1198423</a>,
-<a href="http://www.wikidata.org/entity/Q706951">Q706951</a>,
-<a href="http://www.wikidata.org/entity/Q723239">Q723239</a>,
-<a href="http://www.wikidata.org/entity/Q2079261">Q2079261</a>,
-<a href="http://www.wikidata.org/entity/Q1171364">Q1171364</a>,
-<a href="http://www.wikidata.org/entity/Q617858">Q617858</a>,
-<a href="http://www.wikidata.org/entity/Q5166611">Q5166611</a>,
-<a href="http://www.wikidata.org/entity/Q5166611">Q5166611</a>,
-<a href="http://www.wikidata.org/entity/Q324513">Q324513</a>,
-<a href="http://www.wikidata.org/entity/Q374172">Q374172</a>,
-<a href="http://www.wikidata.org/entity/Q7533269">Q7533269</a>,
-<a href="http://www.wikidata.org/entity/Q970386">Q970386</a>,
-<a href="http://www.wikidata.org/entity/Q976849">Q976849</a>,
-<a href="http://www.wikidata.org/entity/Q7458614">Q7458614</a>,
-<a href="http://www.wikidata.org/entity/Q5347416">Q5347416</a>,
-<a href="http://www.wikidata.org/entity/Q5460005">Q5460005</a>,
-<a href="http://www.wikidata.org/entity/Q5463392">Q5463392</a>,
-<a href="http://www.wikidata.org/entity/Q3038555">Q3038555</a>,
-<a href="http://www.wikidata.org/entity/Q5288458">Q5288458</a>,
-<a href="http://www.wikidata.org/entity/Q2346516">Q2346516</a>,
-<a href="http://www.wikidata.org/entity/Q5183645">Q5183645</a>,
-<a href="http://www.wikidata.org/entity/Q5185497">Q5185497</a>,
-<a href="http://www.wikidata.org/entity/Q5216127">Q5216127</a>,
-<a href="http://www.wikidata.org/entity/Q5223127">Q5223127</a>,
-<a href="http://www.wikidata.org/entity/Q5261159">Q5261159</a>,
-<a href="http://www.wikidata.org/entity/Q1300759">Q1300759</a>,
-<a href="http://www.wikidata.org/entity/Q5521241">Q5521241</a>,
-<a href="http://www.wikidata.org/entity/Q7733434">Q7733434</a>,
-<a href="http://www.wikidata.org/entity/Q7736264">Q7736264</a>,
-<a href="http://www.wikidata.org/entity/Q7737032">Q7737032</a>,
-<a href="http://www.wikidata.org/entity/Q7882671">Q7882671</a>,
-<a href="http://www.wikidata.org/entity/Q7719427">Q7719427</a>,
-<a href="http://www.wikidata.org/entity/Q7719444">Q7719444</a>,
-<a href="http://www.wikidata.org/entity/Q7722575">Q7722575</a>,
-<a href="http://www.wikidata.org/entity/Q2629763">Q2629763</a>,
-<a href="http://www.wikidata.org/entity/Q2640346">Q2640346</a>,
-<a href="http://www.wikidata.org/entity/Q2649671">Q2649671</a>,
-<a href="http://www.wikidata.org/entity/Q7703851">Q7703851</a>,
-<a href="http://www.wikidata.org/entity/Q7747041">Q7747041</a>,
-<a href="http://www.wikidata.org/entity/Q6544949">Q6544949</a>,
-<a href="http://www.wikidata.org/entity/Q6672759">Q6672759</a>,
-<a href="http://www.wikidata.org/entity/Q2445896">Q2445896</a>,
-<a href="http://www.wikidata.org/entity/Q12124891">Q12124891</a>,
-<a href="http://www.wikidata.org/entity/Q3127044">Q3127044</a>,
-<a href="http://www.wikidata.org/entity/Q2511262">Q2511262</a>,
-<a href="http://www.wikidata.org/entity/Q2517672">Q2517672</a>,
-<a href="http://www.wikidata.org/entity/Q2543165">Q2543165</a>,
-<a href="http://www.wikidata.org/entity/Q426628">Q426628</a>,
-<a href="http://www.wikidata.org/entity/Q426628">Q426628</a>,
-<a href="http://www.wikidata.org/entity/Q12126890">Q12126890</a>,
-<a href="http://www.wikidata.org/entity/Q13359969">Q13359969</a>,
-<a href="http://www.wikidata.org/entity/Q13359969">Q13359969</a>,
-<a href="http://www.wikidata.org/entity/Q2294295">Q2294295</a>,
-<a href="http://www.wikidata.org/entity/Q2294295">Q2294295</a>,
-<a href="http://www.wikidata.org/entity/Q2559509">Q2559509</a>,
-<a href="http://www.wikidata.org/entity/Q2559912">Q2559912</a>,
-<a href="http://www.wikidata.org/entity/Q7760469">Q7760469</a>,
-<a href="http://www.wikidata.org/entity/Q6703974">Q6703974</a>,
-<a href="http://www.wikidata.org/entity/Q4744">Q4744</a>,
-<a href="http://www.wikidata.org/entity/Q7766962">Q7766962</a>,
-<a href="http://www.wikidata.org/entity/Q7768516">Q7768516</a>,
-<a href="http://www.wikidata.org/entity/Q7769205">Q7769205</a>,
-<a href="http://www.wikidata.org/entity/Q7769988">Q7769988</a>,
-<a href="http://www.wikidata.org/entity/Q2946945">Q2946945</a>,
-<a href="http://www.wikidata.org/entity/Q3212086">Q3212086</a>,
-<a href="http://www.wikidata.org/entity/Q3212086">Q3212086</a>,
-<a href="http://www.wikidata.org/entity/Q18218448">Q18218448</a>,
-<a href="http://www.wikidata.org/entity/Q18218448">Q18218448</a>,
-<a href="http://www.wikidata.org/entity/Q18218448">Q18218448</a>,
-<a href="http://www.wikidata.org/entity/Q6909175">Q6909175</a>,
-<a href="http://www.wikidata.org/entity/Q7405709">Q7405709</a>,
-<a href="http://www.wikidata.org/entity/Q7416149">Q7416149</a>,
-<a href="http://www.wikidata.org/entity/Q7239952">Q7239952</a>,
-<a href="http://www.wikidata.org/entity/Q7317332">Q7317332</a>,
-<a href="http://www.wikidata.org/entity/Q7783674">Q7783674</a>,
-<a href="http://www.wikidata.org/entity/Q7783704">Q7783704</a>,
-<a href="http://www.wikidata.org/entity/Q7857590">Q7857590</a>,
-<a href="http://www.wikidata.org/entity/Q3372526">Q3372526</a>,
-<a href="http://www.wikidata.org/entity/Q3372642">Q3372642</a>,
-<a href="http://www.wikidata.org/entity/Q3372816">Q3372816</a>,
-<a href="http://www.wikidata.org/entity/Q3372909">Q3372909</a>,
-<a href="http://www.wikidata.org/entity/Q7959649">Q7959649</a>,
-<a href="http://www.wikidata.org/entity/Q7977485">Q7977485</a>,
-<a href="http://www.wikidata.org/entity/Q7992684">Q7992684</a>,
-<a href="http://www.wikidata.org/entity/Q3817966">Q3817966</a>,
-<a href="http://www.wikidata.org/entity/Q3821852">Q3821852</a>,
-<a href="http://www.wikidata.org/entity/Q3420907">Q3420907</a>,
-<a href="http://www.wikidata.org/entity/Q3429733">Q3429733</a>,
-<a href="http://www.wikidata.org/entity/Q774474">Q774474</a></p>
+<p> I keep finding more data sources. I found the cinemovies source
+just a few days ago, and as you can see from the summary, it extended
+my list with 63 movies. Check out the mklist-* scripts in the git
+repository if you are curious how the lists are created. Many of the
+titles are extracted using searches on IMDB, where I look for the
+title and year, and accept search results with only one movie listed
+if the year matches. This allow me to automatically use many lists of
+movies without IMDB title ID references at the cost of increasing the
+risk of wrongly identify a IMDB title ID as public domain. So far my
+random manual checks have indicated that the method is solid, but I
+really wish all lists of public domain movies would include unique
+movie identifier like the IMDB title ID. It would make the job of
+counting movies in the public domain a lot easier.</p>
+
+<p>As usual, if you use Bitcoin and want to show your support of my
+activities, please send Bitcoin donations to my address
+<b><a href="bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b">15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b</a></b>.</p>
</div>
<div class="tags">
- Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett</a>.
+ Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett</a>, <a href="http://people.skolelinux.org/pere/blog/tags/verkidetfri">verkidetfri</a>.
</div>
<div class="padding"></div>
<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/A_one_way_wall_on_the_border_.html">A one-way wall on the border?</a></div>
- <div class="date">14th October 2017</div>
- <div class="body"><p>I find it fascinating how many of the people being locked inside
-the proposed border wall between USA and Mexico support the idea. The
-proposal to keep Mexicans out reminds me of
-<a href="http://www.history.com/news/10-things-you-may-not-know-about-the-berlin-wall">the
-propaganda twist from the East Germany government</a> calling the wall
-the “Antifascist Bulwark” after erecting the Berlin Wall, claiming
-that the wall was erected to keep enemies from creeping into East
-Germany, while it was obvious to the people locked inside it that it
-was erected to keep the people from escaping.</p>
-
-<p>Do the people in USA supporting this wall really believe it is a
-one way wall, only keeping people on the outside from getting in,
-while not keeping people in the inside from getting out?</p>
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/Kommentarer_til__Evaluation_of__il_legality__for_Popcorn_Time.html">Kommentarer til «Evaluation of (il)legality» for Popcorn Time</a></div>
+ <div class="date">20th December 2017</div>
+ <div class="body"><p>I går var jeg i Follo tingrett som sakkyndig vitne og presenterte
+ mine undersøkelser rundt
+ <a href="https://github.com/petterreinholdtsen/public-domain-free-imdb">telling
+ av filmverk i det fri</a>, relatert til
+ <a href="https://www.nuug.no/">foreningen NUUG</a>s involvering i
+ <a href="https://www.nuug.no/news/tags/dns-domenebeslag/">saken om
+ Økokrims beslag og senere inndragning av DNS-domenet
+ popcorn-time.no</a>. Jeg snakket om flere ting, men mest om min
+ vurdering av hvordan filmbransjen har målt hvor ulovlig Popcorn Time
+ er. Filmbransjens måling er så vidt jeg kan se videreformidlet uten
+ endringer av norsk politi, og domstolene har lagt målingen til grunn
+ når de har vurdert Popcorn Time både i Norge og i utlandet (tallet
+ 99% er referert også i utenlandske domsavgjørelser).</p>
+
+<p>I forkant av mitt vitnemål skrev jeg et notat, mest til meg selv,
+ med de punktene jeg ønsket å få frem. Her er en kopi av notatet jeg
+ skrev og ga til aktoratet. Merkelig nok ville ikke dommerene ha
+ notatet, så hvis jeg forsto rettsprosessen riktig ble kun
+ histogram-grafen lagt inn i dokumentasjonen i saken. Dommerne var
+ visst bare interessert i å forholde seg til det jeg sa i retten,
+ ikke det jeg hadde skrevet i forkant. Uansett så antar jeg at flere
+ enn meg kan ha glede av teksten, og publiserer den derfor her.
+ Legger ved avskrift av dokument 09,13, som er det sentrale
+ dokumentet jeg kommenterer.</p>
+
+<p><strong>Kommentarer til «Evaluation of (il)legality» for Popcorn
+ Time</strong></p>
+
+<p><strong>Oppsummering</strong></p>
+
+<p>Målemetoden som Økokrim har lagt til grunn når de påstår at 99% av
+ filmene tilgjengelig fra Popcorn Time deles ulovlig har
+ svakheter.</p>
+
+<p>De eller den som har vurdert hvorvidt filmer kan lovlig deles har
+ ikke lyktes med å identifisere filmer som kan deles lovlig og har
+ tilsynelatende antatt at kun veldig gamle filmer kan deles lovlig.
+ Økokrim legger til grunn at det bare finnes èn film, Charlie
+ Chaplin-filmen «The Circus» fra 1928, som kan deles fritt blant de
+ som ble observert tilgjengelig via ulike Popcorn Time-varianter.
+ Jeg finner tre flere blant de observerte filmene: «The Brain That
+ Wouldn't Die» fra 1962, «God’s Little Acre» fra 1958 og «She Wore a
+ Yellow Ribbon» fra 1949. Det er godt mulig det finnes flere. Det
+ finnes dermed minst fire ganger så mange filmer som lovlig kan deles
+ på Internett i datasettet Økokrim har lagt til grunn når det påstås
+ at mindre enn 1 % kan deles lovlig.</p>
+
+<p>Dernest, utplukket som gjøres ved søk på tilfeldige ord hentet fra
+ ordlisten til Dale-Chall avviker fra årsfordelingen til de brukte
+ filmkatalogene som helhet, hvilket påvirker fordelingen mellom
+ filmer som kan lovlig deles og filmer som ikke kan lovlig deles. I
+ tillegg gir valg av øvre del (de fem første) av søkeresultatene et
+ avvik fra riktig årsfordeling, hvilket påvirker fordelingen av verk
+ i det fri i søkeresultatet.</p>
+
+<p>Det som måles er ikke (u)lovligheten knyttet til bruken av Popcorn
+ Time, men (u)lovligheten til innholdet i bittorrent-filmkataloger
+ som vedlikeholdes uavhengig av Popcorn Time.</p>
+
+<p>Omtalte dokumenter: 09,12, <a href="#dok-09-13">09,13</a>, 09,14,
+09,18, 09,19, 09,20.</p>
+
+<p><strong>Utfyllende kommentarer</strong></p>
+
+<p>Økokrim har forklart domstolene at minst 99% av alt som er
+ tilgjengelig fra ulike Popcorn Time-varianter deles ulovlig på
+ Internet. Jeg ble nysgjerrig på hvordan de er kommet frem til dette
+ tallet, og dette notatet er en samling kommentarer rundt målingen
+ Økokrim henviser til. Litt av bakgrunnen for at jeg valgte å se på
+ saken er at jeg er interessert i å identifisere og telle hvor mange
+ kunstneriske verk som er falt i det fri eller av andre grunner kan
+ lovlig deles på Internett, og dermed var interessert i hvordan en
+ hadde funnet den ene prosenten som kanskje deles lovlig.</p>
+
+<p>Andelen på 99% kommer fra et ukreditert og udatert notatet som tar
+ mål av seg å dokumentere en metode for å måle hvor (u)lovlig ulike
+ Popcorn Time-varianter er.</p>
+
+<p>Raskt oppsummert, så forteller metodedokumentet at på grunn av at
+ det ikke er mulig å få tak i komplett liste over alle filmtitler
+ tilgjengelig via Popcorn Time, så lages noe som skal være et
+ representativt utvalg ved å velge 50 søkeord større enn tre tegn fra
+ ordlisten kjent som Dale-Chall. For hvert søkeord gjøres et søk og
+ de første fem filmene i søkeresultatet samles inn inntil 100 unike
+ filmtitler er funnet. Hvis 50 søkeord ikke var tilstrekkelig for å
+ nå 100 unike filmtitler ble flere filmer fra hvert søkeresultat lagt
+ til. Hvis dette heller ikke var tilstrekkelig, så ble det hentet ut
+ og søkt på flere tilfeldig valgte søkeord inntil 100 unike
+ filmtitler var identifisert.</p>
+
+<p>Deretter ble for hver av filmtitlene «vurdert hvorvidt det var
+ rimelig å forvente om at verket var vernet av copyright, ved å se på
+ om filmen var tilgjengelig i IMDB, samt se på regissør,
+ utgivelsesår, når det var utgitt for bestemte markedsområder samt
+ hvilke produksjons- og distribusjonsselskap som var registrert» (min
+ oversettelse).</p>
+
+<p>Metoden er gjengitt både i de ukrediterte dokumentene 09,13 og
+ 09,19, samt beskrevet fra side 47 i dokument 09,20, lysark datert
+ 2017-02-01. Sistnevnte er kreditert Geerart Bourlon fra Motion
+ Picture Association EMEA. Metoden virker å ha flere svakheter som
+ gir resultatene en slagside. Den starter med å slå fast at det ikke
+ er mulig å hente ut en komplett liste over alle filmtitler som er
+ tilgjengelig, og at dette er bakgrunnen for metodevalget. Denne
+ forutsetningen er ikke i tråd med det som står i dokument 09,12, som
+ ikke heller har oppgitt forfatter og dato. Dokument 09,12 forteller
+ hvordan hele kataloginnholdet ble lasted ned og talt opp. Dokument
+ 09,12 er muligens samme rapport som ble referert til i dom fra Oslo
+ Tingrett 2017-11-03
+ (<a href="https://www.domstol.no/no/Enkelt-domstol/Oslo--tingrett/Nyheter/ma-sperre-for-popcorn-time/">sak
+ 17-093347TVI-OTIR/05</a>) som rapport av 1. juni 2017 av Alexander
+ Kind Petersen, men jeg har ikke sammenlignet dokumentene ord for ord
+ for å kontrollere dette.</p>
+
+<p>IMDB er en forkortelse for The Internet Movie Database, en
+ anerkjent kommersiell nettjeneste som brukes aktivt av både
+ filmbransjen og andre til å holde rede på hvilke spillefilmer (og
+ endel andre filmer) som finnes eller er under produksjon, og
+ informasjon om disse filmene. Datakvaliteten er høy, med få feil og
+ få filmer som mangler. IMDB viser ikke informasjon om
+ opphavsrettslig status for filmene på infosiden for hver film. Som
+ del av IMDB-tjenesten finnes det lister med filmer laget av
+ frivillige som lister opp det som antas å være verk i det fri.</p>
+
+<p>Det finnes flere kilder som kan brukes til å finne filmer som er
+ allemannseie (public domain) eller har bruksvilkår som gjør det
+ lovlig for alleå dele dem på Internett. Jeg har de siste ukene
+ forsøkt å samle og krysskoble disse listene for å forsøke å telle
+ antall filmer i det fri. Ved å ta utgangspunkt i slike lister (og
+ publiserte filmer for Internett-arkivets del), har jeg så langt
+ klart å identifisere over 11 000 filmer, hovedsaklig spillefilmer.
+
+<p>De aller fleste oppføringene er hentet fra IMDB selv, basert på det
+ faktum at alle filmer laget i USA før 1923 er falt i det fri.
+ Tilsvarende tidsgrense for Storbritannia er 1912-07-01, men dette
+ utgjør bare veldig liten del av spillefilmene i IMDB (19 totalt).
+ En annen stor andel kommer fra Internett-arkivet, der jeg har
+ identifisert filmer med referanse til IMDB. Internett-arkivet, som
+ holder til i USA, har som
+ <a href="https://archive.org/about/terms.php">policy å kun publisere
+ filmer som det er lovlig å distribuere</a>. Jeg har under arbeidet
+ kommet over flere filmer som har blitt fjernet fra
+ Internett-arkivet, hvilket gjør at jeg konkluderer med at folkene
+ som kontrollerer Internett-arkivet har et aktivt forhold til å kun
+ ha lovlig innhold der, selv om det i stor grad er drevet av
+ frivillige. En annen stor liste med filmer kommer fra det
+ kommersielle selskapet Retro Film Vault, som selger allemannseide
+ filmer til TV- og filmbransjen, Jeg har også benyttet meg av lister
+ over filmer som hevdes å være allemannseie, det være seg Public
+ Domain Review, Public Domain Torrents og Public Domain Movies (.net
+ og .info), samt lister over filmer med Creative Commons-lisensiering
+ fra Wikipedia, VODO og The Hill Productions. Jeg har gjort endel
+ stikkontroll ved å vurdere filmer som kun omtales på en liste. Der
+ jeg har funnet feil som har gjort meg i tvil om vurderingen til de
+ som har laget listen har jeg forkastet listen fullstendig (gjelder
+ en liste fra IMDB).</p>
+
+<p>Ved å ta utgangspunkt i verk som kan antas å være lovlig delt på
+ Internett (fra blant annet Internett-arkivet, Public Domain
+ Torrents, Public Domain Reivew og Public Domain Movies), og knytte
+ dem til oppføringer i IMDB, så har jeg så langt klart å identifisere
+ over 11 000 filmer (hovedsaklig spillefilmer) det er grunn til å tro
+ kan lovlig distribueres av alle på Internett. Som ekstra kilder er
+ det brukt lister over filmer som antas/påstås å være allemannseie.
+ Disse kildene kommer fra miljøer som jobber for å gjøre tilgjengelig
+ for almennheten alle verk som er falt i det fri eller har
+ bruksvilkår som tillater deling.
+
+<p>I tillegg til de over 11 000 filmene der tittel-ID i IMDB er
+ identifisert, har jeg funnet mer enn 20 000 oppføringer der jeg ennå
+ ikke har hatt kapasitet til å spore opp tittel-ID i IMDB. Noen av
+ disse er nok duplikater av de IMDB-oppføringene som er identifisert
+ så langt, men neppe alle. Retro Film Vault hevder å ha 44 000
+ filmverk i det fri i sin katalog, så det er mulig at det reelle
+ tallet er betydelig høyere enn de jeg har klart å identifisere så
+ langt. Konklusjonen er at tallet 11 000 er nedre grense for hvor
+ mange filmer i IMDB som kan lovlig deles på Internett. I følge <a
+ href="http://www.imdb.com/stats">statistikk fra IMDB</a> er det 4.6
+ millioner titler registrert, hvorav 3 millioner er TV-serieepisoder.
+ Jeg har ikke funnet ut hvordan de fordeler seg per år.</p>
+
+<p>Hvis en fordeler på år alle tittel-IDene i IMDB som hevdes å lovlig
+ kunne deles på Internett, får en følgende histogram:</p>
+
+<p align="center"><img width="80%" src="http://people.skolelinux.org/pere/blog/images/2017-12-20-histogram-year.png"></p>
+
+<p>En kan i histogrammet se at effekten av manglende registrering
+ eller fornying av registrering er at mange filmer gitt ut i USA før
+ 1978 er allemannseie i dag. I tillegg kan en se at det finnes flere
+ filmer gitt ut de siste årene med bruksvilkår som tillater deling,
+ muligens på grunn av fremveksten av
+ <a href="https://creativecommons.org/">Creative
+ Commons</a>-bevegelsen..</p>
+
+<p>For maskinell analyse av katalogene har jeg laget et lite program
+ som kobler seg til bittorrent-katalogene som brukes av ulike Popcorn
+ Time-varianter og laster ned komplett liste over filmer i
+ katalogene, noe som bekrefter at det er mulig å hente ned komplett
+ liste med alle filmtitler som er tilgjengelig. Jeg har sett på fire
+ bittorrent-kataloger. Den ene brukes av klienten tilgjengelig fra
+ www.popcorntime.sh og er navngitt 'sh' i dette dokumentet. Den
+ andre brukes i følge dokument 09,12 av klienten tilgjengelig fra
+ popcorntime.ag og popcorntime.sh og er navngitt 'yts' i dette
+ dokumentet. Den tredje brukes av websidene tilgjengelig fra
+ popcorntime-online.tv og er navngitt 'apidomain' i dette dokumentet.
+ Den fjerde brukes av klienten tilgjenglig fra popcorn-time.to i
+ følge dokument 09,12, og er navngitt 'ukrfnlge' i dette
+ dokumentet.</p>
+
+<p>Metoden Økokrim legger til grunn skriver i sitt punkt fire at
+ skjønn er en egnet metode for å finne ut om en film kan lovlig deles
+ på Internett eller ikke, og sier at det ble «vurdert hvorvidt det
+ var rimelig å forvente om at verket var vernet av copyright». For
+ det første er det ikke nok å slå fast om en film er «vernet av
+ copyright» for å vite om det er lovlig å dele den på Internett eller
+ ikke, da det finnes flere filmer med opphavsrettslige bruksvilkår
+ som tillater deling på Internett. Eksempler på dette er Creative
+ Commons-lisensierte filmer som Citizenfour fra 2014 og Sintel fra
+ 2010. I tillegg til slike finnes det flere filmer som nå er
+ allemannseie (public domain) på grunn av manglende registrering
+ eller fornying av registrering selv om både regisør,
+ produksjonsselskap og distributør ønsker seg vern. Eksempler på
+ dette er Plan 9 from Outer Space fra 1959 og Night of the Living
+ Dead fra 1968. Alle filmer fra USA som var allemannseie før
+ 1989-03-01 forble i det fri da Bern-konvensjonen, som tok effekt i
+ USA på det tidspunktet, ikke ble gitt tilbakevirkende kraft. Hvis
+ det er noe
+ <a href="http://www.latimes.com/local/lanow/la-me-ln-happy-birthday-song-lawsuit-decision-20150922-story.html">historien
+ om sangen «Happy birthday»</a> forteller oss, der betaling for bruk
+ har vært krevd inn i flere tiår selv om sangen ikke egentlig var
+ vernet av åndsverksloven, så er det at hvert enkelt verk må vurderes
+ nøye og i detalj før en kan slå fast om verket er allemannseie eller
+ ikke, det holder ikke å tro på selverklærte rettighetshavere. Flere
+ eksempel på verk i det fri som feilklassifiseres som vernet er fra
+ dokument 09,18, som lister opp søkeresultater for klienten omtalt
+ som popcorntime.sh og i følge notatet kun inneholder en film (The
+ Circus fra 1928) som under tvil kan antas å være allemannseie.</p>
+
+<p>Ved rask gjennomlesning av dokument 09,18, som inneholder
+ skjermbilder fra bruk av en Popcorn Time-variant, fant jeg omtalt
+ både filmen «The Brain That Wouldn't Die» fra 1962 som er
+ <a href="https://archive.org/details/brain_that_wouldnt_die">tilgjengelig
+ fra Internett-arkivet</a> og som
+ <a href="https://en.wikipedia.org/wiki/List_of_films_in_the_public_domain_in_the_United_States">i
+ følge Wikipedia er allemannseie i USA</a> da den ble gitt ut i
+ 1962 uten 'copyright'-merking, og filmen «God’s Little Acre» fra
+ 1958 <a href="https://en.wikipedia.org/wiki/God%27s_Little_Acre_%28film%29">som
+ er lagt ut på Wikipedia</a>, der det fortelles at
+ sort/hvit-utgaven er allemannseie. Det fremgår ikke fra dokument
+ 09,18 om filmen omtalt der er sort/hvit-utgaven. Av
+ kapasitetsårsaker og på grunn av at filmoversikten i dokument 09,18
+ ikke er maskinlesbart har jeg ikke forsøkt å sjekke alle filmene som
+ listes opp der om mot liste med filmer som er antatt lovlig kan
+ distribueres på Internet.</p>
+
+<p>Ved maskinell gjennomgang av listen med IMDB-referanser under
+ regnearkfanen 'Unique titles' i dokument 09.14, fant jeg i tillegg
+ filmen «She Wore a Yellow Ribbon» fra 1949) som nok også er
+ feilklassifisert. Filmen «She Wore a Yellow Ribbon» er tilgjengelig
+ fra Internett-arkivet og markert som allemannseie der. Det virker
+ dermed å være minst fire ganger så mange filmer som kan lovlig deles
+ på Internett enn det som er lagt til grunn når en påstår at minst
+ 99% av innholdet er ulovlig. Jeg ser ikke bort fra at nærmere
+ undersøkelser kan avdekke flere. Poenget er uansett at metodens
+ punkt om «rimelig å forvente om at verket var vernet av copyright»
+ gjør metoden upålitelig.</p>
+
+<p>Den omtalte målemetoden velger ut tilfeldige søketermer fra
+ ordlisten Dale-Chall. Den ordlisten inneholder 3000 enkle engelske
+ som fjerdeklassinger i USA er forventet å forstå. Det fremgår ikke
+ hvorfor akkurat denne ordlisten er valgt, og det er uklart for meg
+ om den er egnet til å få et representativt utvalg av filmer. Mange
+ av ordene gir tomt søkeresultat. Ved å simulerte tilsvarende søk
+ ser jeg store avvik fra fordelingen i katalogen for enkeltmålinger.
+ Dette antyder at enkeltmålinger av 100 filmer slik målemetoden
+ beskriver er gjort, ikke er velegnet til å finne andel ulovlig
+ innhold i bittorrent-katalogene.</p>
+
+<p>En kan motvirke dette store avviket for enkeltmålinger ved å gjøre
+ mange søk og slå sammen resultatet. Jeg har testet ved å
+ gjennomføre 100 enkeltmålinger (dvs. måling av (100x100=) 10 000
+ tilfeldig valgte filmer) som gir mindre, men fortsatt betydelig
+ avvik, i forhold til telling av filmer pr år i hele katalogen.</p>
+
+<p>Målemetoden henter ut de fem øverste i søkeresultatet.
+ Søkeresultatene er sortert på antall bittorrent-klienter registrert
+ som delere i katalogene, hvilket kan gi en slagside mot hvilke
+ filmer som er populære blant de som bruker bittorrent-katalogene,
+ uten at det forteller noe om hvilket innhold som er tilgjengelig
+ eller hvilket innhold som deles med Popcorn Time-klienter. Jeg har
+ forsøkt å måle hvor stor en slik slagside eventuelt er ved å
+ sammenligne fordelingen hvis en tar de 5 nederste i søkeresultatet i
+ stedet. Avviket for disse to metodene for endel kataloger er godt
+ synlig på histogramet. Her er histogram over filmer funnet i den
+ komplette katalogen (grønn strek), og filmer funnet ved søk etter
+ ord i Dale-Chall. Grafer merket 'top' henter fra de 5 første i
+ søkeresultatet, mens de merket 'bottom' henter fra de 5 siste. En
+ kan her se at resultatene påvirkes betydelig av hvorvidt en ser på
+ de første eller de siste filmene i et søketreff.</p>
+
+<p align="center">
+ <img width="40%" src="http://people.skolelinux.org/pere/blog/images/2017-12-20-histogram-year-sh-top.png"/>
+ <img width="40%" src="http://people.skolelinux.org/pere/blog/images/2017-12-20-histogram-year-sh-bottom.png"/>
+ <br>
+ <img width="40%" src="http://people.skolelinux.org/pere/blog/images/2017-12-20-histogram-year-yts-top.png"/>
+ <img width="40%" src="http://people.skolelinux.org/pere/blog/images/2017-12-20-histogram-year-yts-bottom.png"/>
+ <br>
+ <img width="40%" src="http://people.skolelinux.org/pere/blog/images/2017-12-20-histogram-year-ukrfnlge-top.png"/>
+ <img width="40%" src="http://people.skolelinux.org/pere/blog/images/2017-12-20-histogram-year-ukrfnlge-bottom.png"/>
+ <br>
+ <img width="40%" src="http://people.skolelinux.org/pere/blog/images/2017-12-20-histogram-year-apidomain-top.png"/>
+ <img width="40%" src="http://people.skolelinux.org/pere/blog/images/2017-12-20-histogram-year-apidomain-bottom.png"/>
+</p>
+
+<p>Det er verdt å bemerke at de omtalte bittorrent-katalogene ikke er
+ laget for bruk med Popcorn Time. Eksempelvis tilhører katalogen
+ YTS, som brukes av klientet som ble lastes ned fra popcorntime.sh,
+ et selvstendig fildelings-relatert nettsted YTS.AG med et separat
+ brukermiljø. Målemetoden foreslått av Økokrim måler dermed ikke
+ (u)lovligheten rundt bruken av Popcorn Time, men (u)lovligheten til
+ innholdet i disse katalogene.</p>
+
+<hr>
+
+<p id="dok-09-13">Metoden fra Økokrims dokument 09,13 i straffesaken
+om DNS-beslag.</p>
+
+<p><strong>1. Evaluation of (il)legality</strong></p>
+
+<p><strong>1.1. Methodology</strong>
+
+<p>Due to its technical configuration, Popcorn Time applications don't
+allow to make a full list of all titles made available. In order to
+evaluate the level of illegal operation of PCT, the following
+methodology was applied:</p>
+
+<ol>
+
+ <li>A random selection of 50 keywords, greater than 3 letters, was
+ made from the Dale-Chall list that contains 3000 simple English
+ words1. The selection was made by using a Random Number
+ Generator2.</li>
+
+ <li>For each keyword, starting with the first randomly selected
+ keyword, a search query was conducted in the movie section of the
+ respective Popcorn Time application. For each keyword, the first
+ five results were added to the title list until the number of 100
+ unique titles was reached (duplicates were removed).</li>
+
+ <li>For one fork, .CH, insufficient titles were generated via this
+ approach to reach 100 titles. This was solved by adding any
+ additional query results above five for each of the 50 keywords.
+ Since this still was not enough, another 42 random keywords were
+ selected to finally reach 100 titles.</li>
+
+ <li>It was verified whether or not there is a reasonable expectation
+ that the work is copyrighted by checking if they are available on
+ IMDb, also verifying the director, the year when the title was
+ released, the release date for a certain market, the production
+ company/ies of the title and the distribution company/ies.</li>
+
+</ol>
+
+<p><strong>1.2. Results</strong></p>
+
+<p>Between 6 and 9 June 2016, four forks of Popcorn Time were
+investigated: popcorn-time.to, popcorntime.ag, popcorntime.sh and
+popcorntime.ch. An excel sheet with the results is included in
+Appendix 1. Screenshots were secured in separate Appendixes for each
+respective fork, see Appendix 2-5.</p>
+
+<p>For each fork, out of 100, de-duplicated titles it was possible to
+retrieve data according to the parameters set out above that indicate
+that the title is commercially available. Per fork, there was 1 title
+that presumably falls within the public domain, i.e. the 1928 movie
+"The Circus" by and with Charles Chaplin.</p>
+
+<p>Based on the above it is reasonable to assume that 99% of the movie
+content of each fork is copyright protected and is made available
+illegally.</p>
+
+<p>This exercise was not repeated for TV series, but considering that
+besides production companies and distribution companies also
+broadcasters may have relevant rights, it is reasonable to assume that
+at least a similar level of infringement will be established.</p>
+
+<p>Based on the above it is reasonable to assume that 99% of all the
+content of each fork is copyright protected and are made available
+illegally.</p>
</div>
<div class="tags">
- Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>.
+ Tags: <a href="http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling</a>, <a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture</a>, <a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk</a>, <a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug</a>, <a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett</a>, <a href="http://people.skolelinux.org/pere/blog/tags/verkidetfri">verkidetfri</a>, <a href="http://people.skolelinux.org/pere/blog/tags/video">video</a>.
</div>
<div class="padding"></div>
<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/Generating_3D_prints_in_Debian_using_Cura_and_Slic3r__prusa_.html">Generating 3D prints in Debian using Cura and Slic3r(-prusa)</a></div>
- <div class="date"> 9th October 2017</div>
- <div class="body"><p>At my nearby maker space,
-<a href="http://sonen.ifi.uio.no/">Sonen</a>, I heard the story that it
-was easier to generate gcode files for theyr 3D printers (Ultimake 2+)
-on Windows and MacOS X than Linux, because the software involved had
-to be manually compiled and set up on Linux while premade packages
-worked out of the box on Windows and MacOS X. I found this annoying,
-as the software involved,
-<a href="https://github.com/Ultimaker/Cura">Cura</a>, is free software
-and should be trivial to get up and running on Linux if someone took
-the time to package it for the relevant distributions. I even found
-<a href="https://bugs.debian.org/706656">a request for adding into
-Debian</a> from 2013, which had seem some activity over the years but
-never resulted in the software showing up in Debian. So a few days
-ago I offered my help to try to improve the situation.</p>
-
-<p>Now I am very happy to see that all the packages required by a
-working Cura in Debian are uploaded into Debian and waiting in the NEW
-queue for the ftpmasters to have a look. You can track the progress
-on
-<a href="https://qa.debian.org/developer.php?email=3dprinter-general%40lists.alioth.debian.org">the
-status page for the 3D printer team</a>.</p>
-
-<p>The uploaded packages are a bit behind upstream, and was uploaded
-now to get slots in <a href="https://ftp-master.debian.org/new.html">the NEW
-queue</a> while we work up updating the packages to the latest
-upstream version.</p>
-
-<p>On a related note, two competitors for Cura, which I found harder
-to use and was unable to configure correctly for Ultimaker 2+ in the
-short time I spent on it, are already in Debian. If you are looking
-for 3D printer "slicers" and want something already available in
-Debian, check out
-<a href="https://tracker.debian.org/pkg/slic3r">slic3r</a> and
-<a href="https://tracker.debian.org/pkg/slic3r-prusa">slic3r-prusa</a>.
-The latter is a fork of the former.</p>
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/Cura__the_nice_3D_print_slicer__is_now_in_Debian_Unstable.html">Cura, the nice 3D print slicer, is now in Debian Unstable</a></div>
+ <div class="date">17th December 2017</div>
+ <div class="body"><p>After several months of working and waiting, I am happy to report
+that the nice and user friendly 3D printer slicer software Cura just
+entered Debian Unstable. It consist of five packages,
+<a href="https://tracker.debian.org/pkg/cura">cura</a>,
+<a href="https://tracker.debian.org/pkg/cura-engine">cura-engine</a>,
+<a href="https://tracker.debian.org/pkg/libarcus">libarcus</a>,
+<a href="https://tracker.debian.org/pkg/fdm-materials">fdm-materials</a>,
+<a href="https://tracker.debian.org/pkg/libsavitar">libsavitar</a> and
+<a href="https://tracker.debian.org/pkg/uranium">uranium</a>. The last
+two, uranium and cura, entered Unstable yesterday. This should make
+it easier for Debian users to print on at least the Ultimaker class of
+3D printers. My nearest 3D printer is an Ultimaker 2+, so it will
+make life easier for at least me. :)</p>
+
+<p>The work to make this happen was done by Gregor Riepl, and I was
+happy to assist him in sponsoring the packages. With the introduction
+of Cura, Debian is up to three 3D printer slicers at your service,
+Cura, Slic3r and Slic3r Prusa. If you own or have access to a 3D
+printer, give it a go. :)</p>
+
+<p>The 3D printer software is maintained by the 3D printer Debian
+team, flocking together on the
+<a href="http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/3dprinter-general">3dprinter-general</a>
+mailing list and the
+<a href="irc://irc.debian.org/#debian-3dprinting">#debian-3dprinting</a>
+IRC channel.</p>
+
+<p>The next step for Cura in Debian is to update the cura package to
+version 3.0.3 and then update the entire set of packages to version
+3.1.0 which showed up the last few days.</p>
</div>
<div class="tags">
<div class="padding"></div>
<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/Mangler_du_en_skrue__eller_har_du_en_skrue_l_s_.html">Mangler du en skrue, eller har du en skrue løs?</a></div>
- <div class="date"> 4th October 2017</div>
- <div class="body">Når jeg holder på med ulike prosjekter, så trenger jeg stadig ulike
-skruer. Det siste prosjektet jeg holder på med er å lage
-<a href="https://www.thingiverse.com/thing:676916">en boks til en
-HDMI-touch-skjerm</a> som skal brukes med Raspberry Pi. Boksen settes
-sammen med skruer og bolter, og jeg har vært i tvil om hvor jeg kan
-få tak i de riktige skruene. Clas Ohlson og Jernia i nærheten har
-sjelden hatt det jeg trenger. Men her om dagen fikk jeg et fantastisk
-tips for oss som bor i Oslo.
-<a href="http://www.zachskruer.no/">Zachariassen Jernvare AS</a> i
-<a href="http://www.openstreetmap.org/?mlat=59.93421&mlon=10.76795#map=19/59.93421/10.76795">Hegermannsgate
-23A på Torshov</a> har et fantastisk utvalg, og åpent mellom 09:00 og
-17:00. De selger skruer, muttere, bolter, skiver etc i løs vekt, og
-så langt har jeg fått alt jeg har lett etter. De har i tillegg det
-meste av annen jernvare, som verktøy, lamper, ledninger, etc. Jeg
-håper de har nok kunder til å holde det gående lenge, da dette er en
-butikk jeg kommer til å besøke ofte. Butikken er et funn å ha i
-nabolaget for oss som liker å bygge litt selv. :)</p>
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/Idea_for_finding_all_public_domain_movies_in_the_USA.html">Idea for finding all public domain movies in the USA</a></div>
+ <div class="date">13th December 2017</div>
+ <div class="body"><p>While looking at
+<a href="http://onlinebooks.library.upenn.edu/cce/">the scanned copies
+for the copyright renewal entries for movies published in the USA</a>,
+an idea occurred to me. The number of renewals are so few per year, it
+should be fairly quick to transcribe them all and add references to
+the corresponding IMDB title ID. This would give the (presumably)
+complete list of movies published 28 years earlier that did _not_
+enter the public domain for the transcribed year. By fetching the
+list of USA movies published 28 years earlier and subtract the movies
+with renewals, we should be left with movies registered in IMDB that
+are now in the public domain. For the year 1955 (which is the one I
+have looked at the most), the total number of pages to transcribe is
+21. For the 28 years from 1950 to 1978, it should be in the range
+500-600 pages. It is just a few days of work, and spread among a
+small group of people it should be doable in a few weeks of spare
+time.</p>
+
+<p>A typical copyright renewal entry look like this (the first one
+listed for 1955):</p>
+
+<p><blockquote>
+ ADAM AND EVIL, a photoplay in seven reels by Metro-Goldwyn-Mayer
+ Distribution Corp. (c) 17Aug27; L24293. Loew's Incorporated (PWH);
+ 10Jun55; R151558.
+</blockquote></p>
+
+<p>The movie title as well as registration and renewal dates are easy
+enough to locate by a program (split on first comma and look for
+DDmmmYY). The rest of the text is not required to find the movie in
+IMDB, but is useful to confirm the correct movie is found. I am not
+quite sure what the L and R numbers mean, but suspect they are
+reference numbers into the archive of the US Copyright Office.</p>
+
+<p>Tracking down the equivalent IMDB title ID is probably going to be
+a manual task, but given the year it is fairly easy to search for the
+movie title using for example
+<a href="http://www.imdb.com/find?q=adam+and+evil+1927&s=all">http://www.imdb.com/find?q=adam+and+evil+1927&s=all</a>.
+Using this search, I find that the equivalent IMDB title ID for the
+first renewal entry from 1955 is
+<a href="http://www.imdb.com/title/tt0017588/">http://www.imdb.com/title/tt0017588/</a>.</p>
+
+<p>I suspect the best way to do this would be to make a specialised
+web service to make it easy for contributors to transcribe and track
+down IMDB title IDs. In the web service, once a entry is transcribed,
+the title and year could be extracted from the text, a search in IMDB
+conducted for the user to pick the equivalent IMDB title ID right
+away. By spreading out the work among volunteers, it would also be
+possible to make at least two persons transcribe the same entries to
+be able to discover any typos introduced. But I will need help to
+make this happen, as I lack the spare time to do all of this on my
+own. If you would like to help, please get in touch. Perhaps you can
+draft a web service for crowd sourcing the task?</p>
+
+<p>Note, Project Gutenberg already have some
+<a href="http://www.gutenberg.org/ebooks/search/?query=copyright+office+renewals">transcribed
+copies of the US Copyright Office renewal protocols</a>, but I have
+not been able to find any film renewals there, so I suspect they only
+have copies of renewal for written works. I have not been able to find
+any transcribed versions of movie renewals so far. Perhaps they exist
+somewhere?</p>
+
+<p>I would love to figure out methods for finding all the public
+domain works in other countries too, but it is a lot harder. At least
+for Norway and Great Britain, such work involve tracking down the
+people involved in making the movie and figuring out when they died.
+It is hard enough to figure out who was part of making a movie, but I
+do not know how to automate such procedure without a registry of every
+person involved in making movies and their death year.</p>
+
+<p>As usual, if you use Bitcoin and want to show your support of my
+activities, please send Bitcoin donations to my address
+<b><a href="bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b">15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b</a></b>.</p>
</div>
<div class="tags">
- Tags: <a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk</a>.
+ Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett</a>, <a href="http://people.skolelinux.org/pere/blog/tags/verkidetfri">verkidetfri</a>.
</div>
<div class="padding"></div>
<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/Visualizing_GSM_radio_chatter_using_gr_gsm_and_Hopglass.html">Visualizing GSM radio chatter using gr-gsm and Hopglass</a></div>
- <div class="date">29th September 2017</div>
- <div class="body"><p>Every mobile phone announce its existence over radio to the nearby
-mobile cell towers. And this radio chatter is available for anyone
-with a radio receiver capable of receiving them. Details about the
-mobile phones with very good accuracy is of course collected by the
-phone companies, but this is not the topic of this blog post. The
-mobile phone radio chatter make it possible to figure out when a cell
-phone is nearby, as it include the SIM card ID (IMSI). By paying
-attention over time, one can see when a phone arrive and when it leave
-an area. I believe it would be nice to make this information more
-available to the general public, to make more people aware of how
-their phones are announcing their whereabouts to anyone that care to
-listen.</p>
-
-<p>I am very happy to report that we managed to get something
-visualizing this information up and running for
-<a href="http://norwaymakers.org/osf17">Oslo Skaperfestival 2017</a>
-(Oslo Makers Festival) taking place today and tomorrow at Deichmanske
-library. The solution is based on the
-<a href="http://people.skolelinux.org/pere/blog/Easier_recipe_to_observe_the_cell_phones_around_you.html">simple
-recipe for listening to GSM chatter</a> I posted a few days ago, and
-will show up at the stand of <a href="http://sonen.ifi.uio.no/">Åpen
-Sone from the Computer Science department of the University of
-Oslo</a>. The presentation will show the nearby mobile phones (aka
-IMSIs) as dots in a web browser graph, with lines to the dot
-representing mobile base station it is talking to. It was working in
-the lab yesterday, and was moved into place this morning.</p>
-
-<p>We set up a fairly powerful desktop machine using Debian
-Buster/Testing with several (five, I believe) RTL2838 DVB-T receivers
-connected and visualize the visible cell phone towers using an
-<a href="https://github.com/marlow925/hopglass">English version of
-Hopglass</a>. A fairly powerfull machine is needed as the
-grgsm_livemon_headless processes from
-<a href="https://tracker.debian.org/pkg/gr-gsm">gr-gsm</a> converting
-the radio signal to data packages is quite CPU intensive.</p>
-
-<p>The frequencies to listen to, are identified using a slightly
-patched scan-and-livemon (to set the --args values for each receiver),
-and the Hopglass data is generated using the
-<a href="https://github.com/petterreinholdtsen/IMSI-catcher/tree/meshviewer-output">patches
-in my meshviewer-output branch</a>. For some reason we could not get
-more than four SDRs working. There is also a geographical map trying
-to show the location of the base stations, but I believe their
-coordinates are hardcoded to some random location in Germany, I
-believe. The code should be replaced with code to look up location in
-a text file, a sqlite database or one of the online databases
-mentioned in
-<a href="https://github.com/Oros42/IMSI-catcher/issues/14">the github
-issue for the topic</a>.
-
-<p>If this sound interesting, visit the stand at the festival!</p>
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/Is_the_short_movie__Empty_Socks__from_1927_in_the_public_domain_or_not_.html">Is the short movie «Empty Socks» from 1927 in the public domain or not?</a></div>
+ <div class="date"> 5th December 2017</div>
+ <div class="body"><p>Three years ago, a presumed lost animation film,
+<a href="https://en.wikipedia.org/wiki/Empty_Socks">Empty Socks from
+1927</a>, was discovered in the Norwegian National Library. At the
+time it was discovered, it was generally assumed to be copyrighted by
+The Walt Disney Company, and I blogged about
+<a href="http://people.skolelinux.org/pere/blog/Opphavsretts_status_for__Empty_Socks__fra_1927_.html">my
+reasoning to conclude</a> that it would would enter the Norwegian
+equivalent of the public domain in 2053, based on my understanding of
+Norwegian Copyright Law. But a few days ago, I came across
+<a href="http://www.toonzone.net/forums/threads/exposed-disneys-repurchase-of-oswald-the-rabbit-a-sham.4792291/">a
+blog post claiming the movie was already in the public domain</a>, at
+least in USA. The reasoning is as follows: The film was released in
+November or Desember 1927 (sources disagree), and presumably
+registered its copyright that year. At that time, right holders of
+movies registered by the copyright office received government
+protection for there work for 28 years. After 28 years, the copyright
+had to be renewed if the wanted the government to protect it further.
+The blog post I found claim such renewal did not happen for this
+movie, and thus it entered the public domain in 1956. Yet someone
+claim the copyright was renewed and the movie is still copyright
+protected. Can anyone help me to figure out which claim is correct?
+I have not been able to find Empty Socks in Catalog of copyright
+entries. Ser.3 pt.12-13 v.9-12 1955-1958 Motion Pictures
+<a href="http://onlinebooks.library.upenn.edu/cce/1955r.html#film">available
+from the University of Pennsylvania</a>, neither in
+<a href="https://babel.hathitrust.org/cgi/pt?id=mdp.39015084451130;page=root;view=image;size=100;seq=83;num=45">page
+45 for the first half of 1955</a>, nor in
+<a href="https://babel.hathitrust.org/cgi/pt?id=mdp.39015084451130;page=root;view=image;size=100;seq=175;num=119">page
+119 for the second half of 1955</a>. It is of course possible that
+the renewal entry was left out of the printed catalog by mistake. Is
+there some way to rule out this possibility? Please help, and update
+the wikipedia page with your findings.
+
+<p>As usual, if you use Bitcoin and want to show your support of my
+activities, please send Bitcoin donations to my address
+<b><a href="bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b">15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b</a></b>.</p>
</div>
<div class="tags">
- Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern</a>, <a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance</a>.
+ Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture</a>, <a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett</a>, <a href="http://people.skolelinux.org/pere/blog/tags/verkidetfri">verkidetfri</a>, <a href="http://people.skolelinux.org/pere/blog/tags/video">video</a>.
</div>
<div class="padding"></div>
<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/Easier_recipe_to_observe_the_cell_phones_around_you.html">Easier recipe to observe the cell phones around you</a></div>
- <div class="date">24th September 2017</div>
- <div class="body"><p>A little more than a month ago I wrote
-<a href="http://people.skolelinux.org/pere/blog/Simpler_recipe_on_how_to_make_a_simple__7_IMSI_Catcher_using_Debian.html">how
-to observe the SIM card ID (aka IMSI number) of mobile phones talking
-to nearby mobile phone base stations using Debian GNU/Linux and a
-cheap USB software defined radio</a>, and thus being able to pinpoint
-the location of people and equipment (like cars and trains) with an
-accuracy of a few kilometer. Since then we have worked to make the
-procedure even simpler, and it is now possible to do this without any
-manual frequency tuning and without building your own packages.</p>
-
-<p>The <a href="https://tracker.debian.org/pkg/gr-gsm">gr-gsm</a>
-package is now included in Debian testing and unstable, and the
-IMSI-catcher code no longer require root access to fetch and decode
-the GSM data collected using gr-gsm.</p>
-
-<p>Here is an updated recipe, using packages built by Debian and a git
-clone of two python scripts:</p>
-
-<ol>
-
-<li>Start with a Debian machine running the Buster version (aka
- testing).</li>
-
-<li>Run '<tt>apt install gr-gsm python-numpy python-scipy
- python-scapy</tt>' as root to install required packages.</li>
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/Metadata_proposal_for_movies_on_the_Internet_Archive.html">Metadata proposal for movies on the Internet Archive</a></div>
+ <div class="date">28th November 2017</div>
+ <div class="body"><p>It would be easier to locate the movie you want to watch in
+<a href="https://www.archive.org/">the Internet Archive</a>, if the
+metadata about each movie was more complete and accurate. In the
+archiving community, a well known saying state that good metadata is a
+love letter to the future. The metadata in the Internet Archive could
+use a face lift for the future to love us back. Here is a proposal
+for a small improvement that would make the metadata more useful
+today. I've been unable to find any document describing the various
+standard fields available when uploading videos to the archive, so
+this proposal is based on my best quess and searching through several
+of the existing movies.</p>
+
+<p>I have a few use cases in mind. First of all, I would like to be
+able to count the number of distinct movies in the Internet Archive,
+without duplicates. I would further like to identify the IMDB title
+ID of the movies in the Internet Archive, to be able to look up a IMDB
+title ID and know if I can fetch the video from there and share it
+with my friends.</p>
+
+<p>Second, I would like the Butter data provider for The Internet
+archive
+(<a href="https://github.com/butterproviders/butter-provider-archive">available
+from github</a>), to list as many of the good movies as possible. The
+plugin currently do a search in the archive with the following
+parameters:</p>
-<li>Fetch the code decoding GSM packages using '<tt>git clone
- github.com/Oros42/IMSI-catcher.git</tt>'.</li>
-
-<li>Insert USB software defined radio supported by GNU Radio.</li>
-
-<li>Enter the IMSI-catcher directory and run '<tt>python
- scan-and-livemon</tt>' to locate the frequency of nearby base
- stations and start listening for GSM packages on one of them.</li>
-
-<li>Enter the IMSI-catcher directory and run '<tt>python
- simple_IMSI-catcher.py</tt>' to display the collected information.</li>
-
-</ol>
+<p><pre>
+collection:moviesandfilms
+AND NOT collection:movie_trailers
+AND -mediatype:collection
+AND format:"Archive BitTorrent"
+AND year
+</pre></p>
-<p>Note, due to a bug somewhere the scan-and-livemon program (actually
-<a href="https://github.com/ptrkrysik/gr-gsm/issues/336">its underlying
-program grgsm_scanner</a>) do not work with the HackRF radio. It does
-work with RTL 8232 and other similar USB radio receivers you can get
-very cheaply
-(<a href="https://www.ebay.com/sch/items/?_nkw=rtl+2832">for example
-from ebay</a>), so for now the solution is to scan using the RTL radio
-and only use HackRF for fetching GSM data.</p>
-
-<p>As far as I can tell, a cell phone only show up on one of the
-frequencies at the time, so if you are going to track and count every
-cell phone around you, you need to listen to all the frequencies used.
-To listen to several frequencies, use the --numrecv argument to
-scan-and-livemon to use several receivers. Further, I am not sure if
-phones using 3G or 4G will show as talking GSM to base stations, so
-this approach might not see all phones around you. I typically see
-0-400 IMSI numbers an hour when looking around where I live.</p>
-
-<p>I've tried to run the scanner on a
-<a href="https://wiki.debian.org/RaspberryPi">Raspberry Pi 2 and 3
-running Debian Buster</a>, but the grgsm_livemon_headless process seem
-to be too CPU intensive to keep up. When GNU Radio print 'O' to
-stdout, I am told there it is caused by a buffer overflow between the
-radio and GNU Radio, caused by the program being unable to read the
-GSM data fast enough. If you see a stream of 'O's from the terminal
-where you started scan-and-livemon, you need a give the process more
-CPU power. Perhaps someone are able to optimize the code to a point
-where it become possible to set up RPi3 based GSM sniffers? I tried
-using Raspbian instead of Debian, but there seem to be something wrong
-with GNU Radio on raspbian, causing glibc to abort().</p>
+<p>Most of the cool movies that fail to show up in Butter do so
+because the 'year' field is missing. The 'year' field is populated by
+the year part from the 'date' field, and should be when the movie was
+released (date or year). Two such examples are
+<a href="https://archive.org/details/SidneyOlcottsBen-hur1905">Ben Hur
+from 1905</a> and
+<a href="https://archive.org/details/Caminandes2GranDillama">Caminandes
+2: Gran Dillama from 2013</a>, where the year metadata field is
+missing.</p>
+
+So, my proposal is simply, for every movie in The Internet Archive
+where an IMDB title ID exist, please fill in these metadata fields
+(note, they can be updated also long after the video was uploaded, but
+as far as I can tell, only by the uploader):
+
+<dl>
+
+<dt>mediatype</dt>
+<dd>Should be 'movie' for movies.</dd>
+
+<dt>collection</dt>
+<dd>Should contain 'moviesandfilms'.</dd>
+
+<dt>title</dt>
+<dd>The title of the movie, without the publication year.</dd>
+
+<dt>date</dt>
+<dd>The data or year the movie was released. This make the movie show
+up in Butter, as well as make it possible to know the age of the
+movie and is useful to figure out copyright status.</dd>
+
+<dt>director</dt>
+<dd>The director of the movie. This make it easier to know if the
+correct movie is found in movie databases.</dd>
+
+<dt>publisher</dt>
+<dd>The production company making the movie. Also useful for
+identifying the correct movie.</dd>
+
+<dt>links</dt>
+
+<dd>Add a link to the IMDB title page, for example like this: <a
+href="http://www.imdb.com/title/tt0028496/">Movie in
+IMDB</a>. This make it easier to find duplicates and allow for
+counting of number of unique movies in the Archive. Other external
+references, like to TMDB, could be added like this too.</dd>
+
+</dl>
+
+<p>I did consider proposing a Custom field for the IMDB title ID (for
+example 'imdb_title_url', 'imdb_code' or simply 'imdb', but suspect it
+will be easier to simply place it in the links free text field.</p>
+
+<p>I created
+<a href="https://github.com/petterreinholdtsen/public-domain-free-imdb">a
+list of IMDB title IDs for several thousand movies in the Internet
+Archive</a>, but I also got a list of several thousand movies without
+such IMDB title ID (and quite a few duplicates). It would be great if
+this data set could be integrated into the Internet Archive metadata
+to be available for everyone in the future, but with the current
+policy of leaving metadata editing to the uploaders, it will take a
+while before this happen. If you have uploaded movies into the
+Internet Archive, you can help. Please consider following my proposal
+above for your movies, to ensure that movie is properly
+counted. :)</p>
+
+<p>The list is mostly generated using wikidata, which based on
+Wikipedia articles make it possible to link between IMDB and movies in
+the Internet Archive. But there are lots of movies without a
+Wikipedia article, and some movies where only a collection page exist
+(like for <a href="https://en.wikipedia.org/wiki/Caminandes">the
+Caminandes example above</a>, where there are three movies but only
+one Wikidata entry).</p>
+
+<p>As usual, if you use Bitcoin and want to show your support of my
+activities, please send Bitcoin donations to my address
+<b><a href="bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b">15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b</a></b>.</p>
</div>
<div class="tags">
- Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern</a>, <a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance</a>.
+ Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett</a>, <a href="http://people.skolelinux.org/pere/blog/tags/verkidetfri">verkidetfri</a>.
</div>
<div class="padding"></div>
<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/Datalagringsdirektivet_kaster_skygger_over_H_yre_og_Arbeiderpartiet.html">Datalagringsdirektivet kaster skygger over Høyre og Arbeiderpartiet</a></div>
- <div class="date"> 7th September 2017</div>
- <div class="body"><p>For noen dager siden publiserte Jon Wessel-Aas en bloggpost om
-«<a href="http://www.uhuru.biz/?p=1821">Konklusjonen om datalagring som
-EU-kommisjonen ikke ville at vi skulle få se</a>». Det er en
-interessant gjennomgang av EU-domstolens syn på snurpenotovervåkning
-av befolkningen, som er klar på at det er i strid med
-EU-lovgivingen.</p>
-
-<p>Valgkampen går for fullt i Norge, og om noen få dager er siste
-frist for å avgi stemme. En ting er sikkert, Høyre og Arbeiderpartiet
-får ikke min stemme
-<a href="http://people.skolelinux.org/pere/blog/Datalagringsdirektivet_gj_r_at_Oslo_H_yre_og_Arbeiderparti_ikke_f_r_min_stemme_i__r.html">denne
-gangen heller</a>. Jeg har ikke glemt at de tvang igjennom loven som
-skulle pålegge alle data- og teletjenesteleverandører å overvåke alle
-sine kunder. En lov som er vedtatt, og aldri opphevet igjen.</p>
-
-<p>Det er tydelig fra diskusjonen rundt grenseløs digital overvåkning
-(eller "Digital Grenseforsvar" som det kalles i Orvellisk nytale) at
-hverken Høyre og Arbeiderpartiet har noen prinsipielle sperrer mot å
-overvåke hele befolkningen, og diskusjonen så langt tyder på at flere
-av de andre partiene heller ikke har det. Mange av
-<a href="https://data.holderdeord.no/votes/1301946411e">de som stemte
-for Datalagringsdirektivet i Stortinget</a> (64 fra Arbeiderpartiet,
-25 fra Høyre) er fortsatt aktive og argumenterer fortsatt for å radere
-vekk mer av innbyggernes privatsfære.</p>
-
-<p>Når myndighetene demonstrerer sin mistillit til folket, tror jeg
-folket selv bør legge litt innsats i å verne sitt privatliv, ved å ta
-i bruk ende-til-ende-kryptert kommunikasjon med sine kjente og kjære,
-og begrense hvor mye privat informasjon som deles med uvedkommende.
-Det er jo ingenting som tyder på at myndighetene kommer til å være vår
-privatsfære.
-<a href="http://people.skolelinux.org/pere/blog/How_to_talk_with_your_loved_ones_in_private.html">Det
-er mange muligheter</a>. Selv har jeg litt sans for
-<a href="https://ring.cx/">Ring</a>, som er basert på p2p-teknologi
-uten sentral kontroll, er fri programvare, og støtter meldinger, tale
-og video. Systemet er tilgjengelig ut av boksen fra
-<a href="https://tracker.debian.org/pkg/ring">Debian</a> og
-<a href="https://launchpad.net/ubuntu/+source/ring">Ubuntu</a>, og det
-finnes pakker for Android, MacOSX og Windows. Foreløpig er det få
-brukere med Ring, slik at jeg også bruker
-<a href="https://signal.org/">Signal</a> som nettleserutvidelse.</p>
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/Legal_to_share_more_than_3000_movies_listed_on_IMDB_.html">Legal to share more than 3000 movies listed on IMDB?</a></div>
+ <div class="date">18th November 2017</div>
+ <div class="body"><p>A month ago, I blogged about my work to
+<a href="http://people.skolelinux.org/pere/blog/Locating_IMDB_IDs_of_movies_in_the_Internet_Archive_using_Wikidata.html">automatically
+check the copyright status of IMDB entries</a>, and try to count the
+number of movies listed in IMDB that is legal to distribute on the
+Internet. I have continued to look for good data sources, and
+identified a few more. The code used to extract information from
+various data sources is available in
+<a href="https://github.com/petterreinholdtsen/public-domain-free-imdb">a
+git repository</a>, currently available from github.</p>
+
+<p>So far I have identified 3186 unique IMDB title IDs. To gain
+better understanding of the structure of the data set, I created a
+histogram of the year associated with each movie (typically release
+year). It is interesting to notice where the peaks and dips in the
+graph are located. I wonder why they are placed there. I suspect
+World War II caused the dip around 1940, but what caused the peak
+around 2010?</p>
+
+<p align="center"><img src="http://people.skolelinux.org/pere/blog/images/2017-11-18-verk-i-det-fri-filmer.png" /></p>
+
+<p>I've so far identified ten sources for IMDB title IDs for movies in
+the public domain or with a free license. This is the statistics
+reported when running 'make stats' in the git repository:</p>
+
+<pre>
+ 249 entries ( 6 unique) with and 288 without IMDB title ID in free-movies-archive-org-butter.json
+ 2301 entries ( 540 unique) with and 0 without IMDB title ID in free-movies-archive-org-wikidata.json
+ 830 entries ( 29 unique) with and 0 without IMDB title ID in free-movies-icheckmovies-archive-mochard.json
+ 2109 entries ( 377 unique) with and 0 without IMDB title ID in free-movies-imdb-pd.json
+ 291 entries ( 122 unique) with and 0 without IMDB title ID in free-movies-letterboxd-pd.json
+ 144 entries ( 135 unique) with and 0 without IMDB title ID in free-movies-manual.json
+ 350 entries ( 1 unique) with and 801 without IMDB title ID in free-movies-publicdomainmovies.json
+ 4 entries ( 0 unique) with and 124 without IMDB title ID in free-movies-publicdomainreview.json
+ 698 entries ( 119 unique) with and 118 without IMDB title ID in free-movies-publicdomaintorrents.json
+ 8 entries ( 8 unique) with and 196 without IMDB title ID in free-movies-vodo.json
+ 3186 unique IMDB title IDs in total
+</pre>
+
+<p>The entries without IMDB title ID are candidates to increase the
+data set, but might equally well be duplicates of entries already
+listed with IMDB title ID in one of the other sources, or represent
+movies that lack a IMDB title ID. I've seen examples of all these
+situations when peeking at the entries without IMDB title ID. Based
+on these data sources, the lower bound for movies listed in IMDB that
+are legal to distribute on the Internet is between 3186 and 4713.
+
+<p>It would be great for improving the accuracy of this measurement,
+if the various sources added IMDB title ID to their metadata. I have
+tried to reach the people behind the various sources to ask if they
+are interested in doing this, without any replies so far. Perhaps you
+can help me get in touch with the people behind VODO, Public Domain
+Torrents, Public Domain Movies and Public Domain Review to try to
+convince them to add more metadata to their movie entries?</p>
+
+<p>Another way you could help is by adding pages to Wikipedia about
+movies that are legal to distribute on the Internet. If such page
+exist and include a link to both IMDB and The Internet Archive, the
+script used to generate free-movies-archive-org-wikidata.json should
+pick up the mapping as soon as wikidata is updates.</p>
+
+<p>As usual, if you use Bitcoin and want to show your support of my
+activities, please send Bitcoin donations to my address
+<b><a href="bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b">15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b</a></b>.</p>
</div>
<div class="tags">
- Tags: <a href="http://people.skolelinux.org/pere/blog/tags/dld">dld</a>, <a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk</a>, <a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern</a>, <a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget</a>, <a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance</a>, <a href="http://people.skolelinux.org/pere/blog/tags/valg">valg</a>.
+ Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett</a>, <a href="http://people.skolelinux.org/pere/blog/tags/verkidetfri">verkidetfri</a>.
</div>
<div class="padding"></div>
<div class="entry">
- <div class="title"><a href="http://people.skolelinux.org/pere/blog/Simpler_recipe_on_how_to_make_a_simple__7_IMSI_Catcher_using_Debian.html">Simpler recipe on how to make a simple $7 IMSI Catcher using Debian</a></div>
- <div class="date"> 9th August 2017</div>
- <div class="body"><p>On friday, I came across an interesting article in the Norwegian
-web based ICT news magazine digi.no on
-<a href="https://www.digi.no/artikler/sikkerhetsforsker-lagde-enkel-imsi-catcher-for-60-kroner-na-kan-mobiler-kartlegges-av-alle/398588">how
-to collect the IMSI numbers of nearby cell phones</a> using the cheap
-DVB-T software defined radios. The article refered to instructions
-and <a href="https://www.youtube.com/watch?v=UjwgNd_as30">a recipe by
-Keld Norman on Youtube on how to make a simple $7 IMSI Catcher</a>, and I decided to test them out.</p>
-
-<p>The instructions said to use Ubuntu, install pip using apt (to
-bypass apt), use pip to install pybombs (to bypass both apt and pip),
-and the ask pybombs to fetch and build everything you need from
-scratch. I wanted to see if I could do the same on the most recent
-Debian packages, but this did not work because pybombs tried to build
-stuff that no longer build with the most recent openssl library or
-some other version skew problem. While trying to get this recipe
-working, I learned that the apt->pip->pybombs route was a long detour,
-and the only piece of software dependency missing in Debian was the
-gr-gsm package. I also found out that the lead upstream developer of
-gr-gsm (the name stand for GNU Radio GSM) project already had a set of
-Debian packages provided in an Ubuntu PPA repository. All I needed to
-do was to dget the Debian source package and built it.</p>
-
-<p>The IMSI collector is a python script listening for packages on the
-loopback network device and printing to the terminal some specific GSM
-packages with IMSI numbers in them. The code is fairly short and easy
-to understand. The reason this work is because gr-gsm include a tool
-to read GSM data from a software defined radio like a DVB-T USB stick
-and other software defined radios, decode them and inject them into a
-network device on your Linux machine (using the loopback device by
-default). This proved to work just fine, and I've been testing the
-collector for a few days now.</p>
-
-<p>The updated and simpler recipe is thus to</p>
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/Some_notes_on_fault_tolerant_storage_systems.html">Some notes on fault tolerant storage systems</a></div>
+ <div class="date"> 1st November 2017</div>
+ <div class="body"><p>If you care about how fault tolerant your storage is, you might
+find these articles and papers interesting. They have formed how I
+think of when designing a storage system.</p>
-<ol>
+<ul>
-<li>start with a Debian machine running Stretch or newer,</li>
+<li>USENIX :login; <a
+href="https://www.usenix.org/publications/login/summer2017/ganesan">Redundancy
+Does Not Imply Fault Tolerance. Analysis of Distributed Storage
+Reactions to Single Errors and Corruptions</a> by Aishwarya Ganesan,
+Ramnatthan Alagappan, Andrea C. Arpaci-Dusseau, and Remzi
+H. Arpaci-Dusseau</li>
-<li>build and install the gr-gsm package available from
-<a href="http://ppa.launchpad.net/ptrkrysik/gr-gsm/ubuntu/pool/main/g/gr-gsm/">http://ppa.launchpad.net/ptrkrysik/gr-gsm/ubuntu/pool/main/g/gr-gsm/</a>,</li>
+<li>ZDNet
+<a href="http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/">Why
+RAID 5 stops working in 2009</a> by Robin Harris</li>
-<li>clone the git repostory from <a href="https://github.com/Oros42/IMSI-catcher">https://github.com/Oros42/IMSI-catcher</a>,</li>
+<li>ZDNet
+<a href="http://www.zdnet.com/article/why-raid-6-stops-working-in-2019/">Why
+RAID 6 stops working in 2019</a> by Robin Harris</li>
-<li>run grgsm_livemon and adjust the frequency until the terminal
-where it was started is filled with a stream of text (meaning you
-found a GSM station).</li>
+<li>USENIX FAST'07
+<a href="http://research.google.com/archive/disk_failures.pdf">Failure
+Trends in a Large Disk Drive Population</a> by Eduardo Pinheiro,
+Wolf-Dietrich Weber and Luiz André Barroso</li>
-<li>go into the IMSI-catcher directory and run 'sudo python simple_IMSI-catcher.py' to extract the IMSI numbers.</li>
+<li>USENIX ;login: <a
+href="https://www.usenix.org/system/files/login/articles/hughes12-04.pdf">Data
+Integrity. Finding Truth in a World of Guesses and Lies</a> by Doug
+Hughes</li>
-</ol>
+<li>USENIX FAST'08
+<a href="https://www.usenix.org/events/fast08/tech/full_papers/bairavasundaram/bairavasundaram_html/">An
+Analysis of Data Corruption in the Storage Stack</a> by
+L. N. Bairavasundaram, G. R. Goodson, B. Schroeder, A. C.
+Arpaci-Dusseau, and R. H. Arpaci-Dusseau</li>
+
+<li>USENIX FAST'07 <a
+href="https://www.usenix.org/legacy/events/fast07/tech/schroeder/schroeder_html/">Disk
+failures in the real world: what does an MTTF of 1,000,000 hours mean
+to you?</a> by B. Schroeder and G. A. Gibson.</li>
+
+<li>USENIX ;login: <a
+href="https://www.usenix.org/events/fast08/tech/full_papers/jiang/jiang_html/">Are
+Disks the Dominant Contributor for Storage Failures? A Comprehensive
+Study of Storage Subsystem Failure Characteristics</a> by Weihang
+Jiang, Chongfeng Hu, Yuanyuan Zhou, and Arkady Kanevsky</li>
+
+<li>SIGMETRICS 2007
+<a href="http://research.cs.wisc.edu/adsl/Publications/latent-sigmetrics07.pdf">An
+analysis of latent sector errors in disk drives</a> by
+L. N. Bairavasundaram, G. R. Goodson, S. Pasupathy, and J. Schindler</li>
+
+</ul>
+
+<p>Several of these research papers are based on data collected from
+hundred thousands or millions of disk, and their findings are eye
+opening. The short story is simply do not implicitly trust RAID or
+redundant storage systems. Details matter. And unfortunately there
+are few options on Linux addressing all the identified issues. Both
+ZFS and Btrfs are doing a fairly good job, but have legal and
+practical issues on their own. I wonder how cluster file systems like
+Ceph do in this regard. After all, there is an old saying, you know
+you have a distributed system when the crash of a computer you have
+never heard of stops you from getting any work done. The same holds
+true if fault tolerance do not work.</p>
+
+<p>Just remember, in the end, it do not matter how redundant, or how
+fault tolerant your storage is, if you do not continuously monitor its
+status to detect and replace failed disks.</p>
-<p>To make it even easier in the future to get this sniffer up and
-running, I decided to package
-<a href="https://github.com/ptrkrysik/gr-gsm/">the gr-gsm project</a>
-for Debian (<a href="https://bugs.debian.org/871055">WNPP
-#871055</a>), and the package was uploaded into the NEW queue today.
-Luckily the gnuradio maintainer has promised to help me, as I do not
-know much about gnuradio stuff yet.</p>
-
-<p>I doubt this "IMSI cacher" is anywhere near as powerfull as
-commercial tools like
-<a href="https://www.thespyphone.com/portable-imsi-imei-catcher/">The
-Spy Phone Portable IMSI / IMEI Catcher</a> or the
-<a href="https://en.wikipedia.org/wiki/Stingray_phone_tracker">Harris
-Stingray</a>, but I hope the existance of cheap alternatives can make
-more people realise how their whereabouts when carrying a cell phone
-is easily tracked. Seeing the data flow on the screen, realizing that
-I live close to a police station and knowing that the police is also
-wearing cell phones, I wonder how hard it would be for criminals to
-track the position of the police officers to discover when there are
-police near by, or for foreign military forces to track the location
-of the Norwegian military forces, or for anyone to track the location
-of government officials...</p>
-
-<p>It is worth noting that the data reported by the IMSI-catcher
-script mentioned above is only a fraction of the data broadcasted on
-the GSM network. It will only collect one frequency at the time,
-while a typical phone will be using several frequencies, and not all
-phones will be using the frequencies tracked by the grgsm_livemod
-program. Also, there is a lot of radio chatter being ignored by the
-simple_IMSI-catcher script, which would be collected by extending the
-parser code. I wonder if gr-gsm can be set up to listen to more than
-one frequency?</p>
+<p>As usual, if you use Bitcoin and want to show your support of my
+activities, please send Bitcoin donations to my address
+<b><a href="bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b">15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b</a></b>.</p>
</div>
<div class="tags">
- Tags: <a href="http://people.skolelinux.org/pere/blog/tags/debian">debian</a>, <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern</a>, <a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance</a>.
+ Tags: <a href="http://people.skolelinux.org/pere/blog/tags/english">english</a>, <a href="http://people.skolelinux.org/pere/blog/tags/raid">raid</a>, <a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin</a>.
</div>
<h2>Archive</h2>
<ul>
+<li>2018
+<ul>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2018/01/">January (1)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2018/02/">February (2)</a></li>
+
+</ul></li>
+
<li>2017
<ul>
<li><a href="http://people.skolelinux.org/pere/blog/archive/2017/10/">October (5)</a></li>
-<li><a href="http://people.skolelinux.org/pere/blog/archive/2017/11/">November (1)</a></li>
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2017/11/">November (3)</a></li>
+
+<li><a href="http://people.skolelinux.org/pere/blog/archive/2017/12/">December (4)</a></li>
</ul></li>
<h2>Tags</h2>
<ul>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (14)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/3d-printer">3d-printer (15)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/amiga">amiga (1)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/chrpath">chrpath (2)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/debian">debian (154)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/debian">debian (155)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/debian edu">debian edu (158)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/drivstoffpriser">drivstoffpriser (4)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/english">english (358)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/english">english (365)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/fiksgatami">fiksgatami (23)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (12)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/fildeling">fildeling (13)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (30)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/freeculture">freeculture (32)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/freedombox">freedombox (9)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/ldap">ldap (9)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/lego">lego (4)</a></li>
+
<li><a href="http://people.skolelinux.org/pere/blog/tags/lenker">lenker (8)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/lsdvd">lsdvd (2)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/nice free software">nice free software (9)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk (293)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/norsk">norsk (295)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug (189)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/nuug">nuug (190)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/offentlig innsyn">offentlig innsyn (33)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/open311">open311 (2)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (65)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/opphavsrett">opphavsrett (71)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/personvern">personvern (104)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/stortinget">stortinget (12)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (52)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/surveillance">surveillance (53)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/sysadmin">sysadmin (4)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/valg">valg (9)</a></li>
- <li><a href="http://people.skolelinux.org/pere/blog/tags/video">video (59)</a></li>
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/verkidetfri">verkidetfri (10)</a></li>
+
+ <li><a href="http://people.skolelinux.org/pere/blog/tags/video">video (61)</a></li>
<li><a href="http://people.skolelinux.org/pere/blog/tags/vitenskap">vitenskap (4)</a></li>
projects / homepage.git / blobdiff