blog/data/2017-10-25-verk-i-det-fri-filmer.txt

   1 Title: Locating IMDB IDs of movies in the Internet Archive using Wikidata
   2 Tags: english, opphavsrett, verkidetfri
   3 Date: 2017-10-25 12:20
   4
   5 <p>Recently, I needed to automatically check the copyright status of a
   6 set of <a href="http://www.imdb.com/">The Internet Movie database
   7 (IMDB)</a> entries, to figure out which one of the movies they refer
   8 to can be freely distributed on the Internet.  This proved to be
   9 harder than it sounds.  IMDB for sure list movies without any
  10 copyright protection, where the copyright protection has expired or
  11 where the movie is lisenced using a permissive license like one from
  12 Creative Commons.  These are mixed with copyright protected movies,
  13 and there seem to be no way to separate these classes of movies using
  14 the information in IMDB.</p>
  15
  16 <p>First I tried to look up entries manually in IMDB,
  17 <a href="https://www.wikipedia.org/">Wikipedia</a> and
  18 <a href="https://www.archive.org/">The Internet Archive</a>, to get a
  19 feel how to do this.  It is hard to know for sure using these sources,
  20 but it should be possible to be reasonable confident a movie is "out
  21 of copyright" with a few hours work per movie.  As I needed to check
  22 almost 20,000 entries, this approach was not sustainable.  I simply
  23 can not work around the clock for about 6 years to check this data
  24 set.</p>
  25
  26 <p>I asked the people behind The Internet Archive if they could
  27 introduce a new metadata field in their metadata XML for IMDB ID, but
  28 was told that they leave it completely to the uploaders to update the
  29 metadata.  Some of the metadata entries had IMDB links in the
  30 description, but I found no way to download all metadata files in bulk
  31 to locate those ones and put that approach aside.</p>
  32
  33 <p>In the process I noticed several Wikipedia articles about movies
  34 had links to both IMDB and The Internet Archive, and it occured to me
  35 that I could use the Wikipedia RDF data set to locate entries with
  36 both, to at least get a lower bound on the number of movies on The
  37 Internet Archive with a IMDB ID.  This is useful based on the
  38 assumption that movies distributed by The Internet Archive can be
  39 legally distributed on the Internet.  With some help from the RDF
  40 community (thank you DanC), I was able to come up with this query to
  41 pass to <a href="https://query.wikidata.org/">the SPARQL interface on
  42 Wikidata</a>:
  43
  44 <p><pre>
  45 SELECT ?work ?imdb ?ia ?when ?label
  46 WHERE
  47 {
  48   ?work wdt:P31/wdt:P279* wd:Q11424.
  49   ?work wdt:P345 ?imdb.
  50   ?work wdt:P724 ?ia.
  51   OPTIONAL {
  52         ?work wdt:P577 ?when.
  53         ?work rdfs:label ?label.
  54         FILTER(LANG(?label) = "en").
  55   }
  56 }
  57 </pre></p>
  58
  59 <p>If I understand the query right, for every film entry anywhere in
  60 Wikpedia, it will return the IMDB ID and The Internet Archive ID, and
  61 when the movie was released and its English title, if either or both
  62 of the latter two are available.  At the moment the result set contain
  63 2338 entries.  Of course, it depend on volunteers including both
  64 correct IMDB and The Internet Archive IDs in the wikipedia articles
  65 for the movie.  It should be noted that the result will include
  66 duplicates if the movie have entries in several languages.  There are
  67 some bogus entries, either because The Internet Archive ID contain a
  68 typo or because the movie is not available from The Internet Archive.
  69 I did not verify the IMDB IDs, as I am unsure how to do that
  70 automatically.</p>
  71
  72 <p>I wrote a small python script to extract the data set from Wikidata
  73 and check if the XML metadata for the movie is available from The
  74 Internet Archive, and after around 1.5 hour it produced a list of 2097
  75 free movies and their IMDB ID.  In total, 171 entries in Wikidata lack
  76 the refered Internet Archive entry.  I assume the 70 "disappearing"
  77 entries (ie 2338-2097-171) are duplicate entries.</p>
  78
  79 <p>This is not too bad, given that The Internet Archive report to
  80 contain <a href="https://archive.org/details/feature_films">5331
  81 feature films</a> at the moment, but it also mean more than 3000
  82 movies are missing on Wikipedia or are missing the pair of references
  83 on Wikipedia.</p>
  84
  85 <p>I was curious about the distribution by release year, and made a
  86 little graph to show how the amount of free movies is spread over the
  87 years:<p>
  88
  89 <p><img src="http://people.skolelinux.org/pere/blog/images/2017-10-25-verk-i-det-fri-filmer.png"></p>
  90
  91 <p>I expect the relative distribution of the remaining 3000 movies to
  92 be similar.</p>
  93
  94 <p>If you want to help, and want to ensure Wikipedia can be used to
  95 cross reference The Internet Archive and The Internet Movie Database,
  96 please make sure entries like this are listed under the "External
  97 links" heading on the Wikipedia article for the movie:</p>
  98
  99 <p><pre>
 100 * {{Internet Archive film|id=FightingLady}}
 101 * {{IMDb title|id=0036823|title=The Fighting Lady}}
 102 </pre></p>
 103
 104 <p>Please verify the links on the final page, to make sure you did not
 105 introduce a typo.</p>
 106
 107 <p>Here is the complete list, if you want to correct the 171
 108 identified Wikipedia entries with broken links to The Internet
 109 Archive: <a href="http://www.wikidata.org/entity/Q1140317">Q1140317</a>,
 110 <a href="http://www.wikidata.org/entity/Q458656">Q458656</a>,
 111 <a href="http://www.wikidata.org/entity/Q458656">Q458656</a>,
 112 <a href="http://www.wikidata.org/entity/Q470560">Q470560</a>,
 113 <a href="http://www.wikidata.org/entity/Q743340">Q743340</a>,
 114 <a href="http://www.wikidata.org/entity/Q822580">Q822580</a>,
 115 <a href="http://www.wikidata.org/entity/Q480696">Q480696</a>,
 116 <a href="http://www.wikidata.org/entity/Q128761">Q128761</a>,
 117 <a href="http://www.wikidata.org/entity/Q1307059">Q1307059</a>,
 118 <a href="http://www.wikidata.org/entity/Q1335091">Q1335091</a>,
 119 <a href="http://www.wikidata.org/entity/Q1537166">Q1537166</a>,
 120 <a href="http://www.wikidata.org/entity/Q1438334">Q1438334</a>,
 121 <a href="http://www.wikidata.org/entity/Q1479751">Q1479751</a>,
 122 <a href="http://www.wikidata.org/entity/Q1497200">Q1497200</a>,
 123 <a href="http://www.wikidata.org/entity/Q1498122">Q1498122</a>,
 124 <a href="http://www.wikidata.org/entity/Q865973">Q865973</a>,
 125 <a href="http://www.wikidata.org/entity/Q834269">Q834269</a>,
 126 <a href="http://www.wikidata.org/entity/Q841781">Q841781</a>,
 127 <a href="http://www.wikidata.org/entity/Q841781">Q841781</a>,
 128 <a href="http://www.wikidata.org/entity/Q1548193">Q1548193</a>,
 129 <a href="http://www.wikidata.org/entity/Q499031">Q499031</a>,
 130 <a href="http://www.wikidata.org/entity/Q1564769">Q1564769</a>,
 131 <a href="http://www.wikidata.org/entity/Q1585239">Q1585239</a>,
 132 <a href="http://www.wikidata.org/entity/Q1585569">Q1585569</a>,
 133 <a href="http://www.wikidata.org/entity/Q1624236">Q1624236</a>,
 134 <a href="http://www.wikidata.org/entity/Q4796595">Q4796595</a>,
 135 <a href="http://www.wikidata.org/entity/Q4853469">Q4853469</a>,
 136 <a href="http://www.wikidata.org/entity/Q4873046">Q4873046</a>,
 137 <a href="http://www.wikidata.org/entity/Q915016">Q915016</a>,
 138 <a href="http://www.wikidata.org/entity/Q4660396">Q4660396</a>,
 139 <a href="http://www.wikidata.org/entity/Q4677708">Q4677708</a>,
 140 <a href="http://www.wikidata.org/entity/Q4738449">Q4738449</a>,
 141 <a href="http://www.wikidata.org/entity/Q4756096">Q4756096</a>,
 142 <a href="http://www.wikidata.org/entity/Q4766785">Q4766785</a>,
 143 <a href="http://www.wikidata.org/entity/Q880357">Q880357</a>,
 144 <a href="http://www.wikidata.org/entity/Q882066">Q882066</a>,
 145 <a href="http://www.wikidata.org/entity/Q882066">Q882066</a>,
 146 <a href="http://www.wikidata.org/entity/Q204191">Q204191</a>,
 147 <a href="http://www.wikidata.org/entity/Q204191">Q204191</a>,
 148 <a href="http://www.wikidata.org/entity/Q1194170">Q1194170</a>,
 149 <a href="http://www.wikidata.org/entity/Q940014">Q940014</a>,
 150 <a href="http://www.wikidata.org/entity/Q946863">Q946863</a>,
 151 <a href="http://www.wikidata.org/entity/Q172837">Q172837</a>,
 152 <a href="http://www.wikidata.org/entity/Q573077">Q573077</a>,
 153 <a href="http://www.wikidata.org/entity/Q1219005">Q1219005</a>,
 154 <a href="http://www.wikidata.org/entity/Q1219599">Q1219599</a>,
 155 <a href="http://www.wikidata.org/entity/Q1643798">Q1643798</a>,
 156 <a href="http://www.wikidata.org/entity/Q1656352">Q1656352</a>,
 157 <a href="http://www.wikidata.org/entity/Q1659549">Q1659549</a>,
 158 <a href="http://www.wikidata.org/entity/Q1660007">Q1660007</a>,
 159 <a href="http://www.wikidata.org/entity/Q1698154">Q1698154</a>,
 160 <a href="http://www.wikidata.org/entity/Q1737980">Q1737980</a>,
 161 <a href="http://www.wikidata.org/entity/Q1877284">Q1877284</a>,
 162 <a href="http://www.wikidata.org/entity/Q1199354">Q1199354</a>,
 163 <a href="http://www.wikidata.org/entity/Q1199354">Q1199354</a>,
 164 <a href="http://www.wikidata.org/entity/Q1199451">Q1199451</a>,
 165 <a href="http://www.wikidata.org/entity/Q1211871">Q1211871</a>,
 166 <a href="http://www.wikidata.org/entity/Q1212179">Q1212179</a>,
 167 <a href="http://www.wikidata.org/entity/Q1238382">Q1238382</a>,
 168 <a href="http://www.wikidata.org/entity/Q4906454">Q4906454</a>,
 169 <a href="http://www.wikidata.org/entity/Q320219">Q320219</a>,
 170 <a href="http://www.wikidata.org/entity/Q1148649">Q1148649</a>,
 171 <a href="http://www.wikidata.org/entity/Q645094">Q645094</a>,
 172 <a href="http://www.wikidata.org/entity/Q5050350">Q5050350</a>,
 173 <a href="http://www.wikidata.org/entity/Q5166548">Q5166548</a>,
 174 <a href="http://www.wikidata.org/entity/Q2677926">Q2677926</a>,
 175 <a href="http://www.wikidata.org/entity/Q2698139">Q2698139</a>,
 176 <a href="http://www.wikidata.org/entity/Q2707305">Q2707305</a>,
 177 <a href="http://www.wikidata.org/entity/Q2740725">Q2740725</a>,
 178 <a href="http://www.wikidata.org/entity/Q2024780">Q2024780</a>,
 179 <a href="http://www.wikidata.org/entity/Q2117418">Q2117418</a>,
 180 <a href="http://www.wikidata.org/entity/Q2138984">Q2138984</a>,
 181 <a href="http://www.wikidata.org/entity/Q1127992">Q1127992</a>,
 182 <a href="http://www.wikidata.org/entity/Q1058087">Q1058087</a>,
 183 <a href="http://www.wikidata.org/entity/Q1070484">Q1070484</a>,
 184 <a href="http://www.wikidata.org/entity/Q1080080">Q1080080</a>,
 185 <a href="http://www.wikidata.org/entity/Q1090813">Q1090813</a>,
 186 <a href="http://www.wikidata.org/entity/Q1251918">Q1251918</a>,
 187 <a href="http://www.wikidata.org/entity/Q1254110">Q1254110</a>,
 188 <a href="http://www.wikidata.org/entity/Q1257070">Q1257070</a>,
 189 <a href="http://www.wikidata.org/entity/Q1257079">Q1257079</a>,
 190 <a href="http://www.wikidata.org/entity/Q1197410">Q1197410</a>,
 191 <a href="http://www.wikidata.org/entity/Q1198423">Q1198423</a>,
 192 <a href="http://www.wikidata.org/entity/Q706951">Q706951</a>,
 193 <a href="http://www.wikidata.org/entity/Q723239">Q723239</a>,
 194 <a href="http://www.wikidata.org/entity/Q2079261">Q2079261</a>,
 195 <a href="http://www.wikidata.org/entity/Q1171364">Q1171364</a>,
 196 <a href="http://www.wikidata.org/entity/Q617858">Q617858</a>,
 197 <a href="http://www.wikidata.org/entity/Q5166611">Q5166611</a>,
 198 <a href="http://www.wikidata.org/entity/Q5166611">Q5166611</a>,
 199 <a href="http://www.wikidata.org/entity/Q324513">Q324513</a>,
 200 <a href="http://www.wikidata.org/entity/Q374172">Q374172</a>,
 201 <a href="http://www.wikidata.org/entity/Q7533269">Q7533269</a>,
 202 <a href="http://www.wikidata.org/entity/Q970386">Q970386</a>,
 203 <a href="http://www.wikidata.org/entity/Q976849">Q976849</a>,
 204 <a href="http://www.wikidata.org/entity/Q7458614">Q7458614</a>,
 205 <a href="http://www.wikidata.org/entity/Q5347416">Q5347416</a>,
 206 <a href="http://www.wikidata.org/entity/Q5460005">Q5460005</a>,
 207 <a href="http://www.wikidata.org/entity/Q5463392">Q5463392</a>,
 208 <a href="http://www.wikidata.org/entity/Q3038555">Q3038555</a>,
 209 <a href="http://www.wikidata.org/entity/Q5288458">Q5288458</a>,
 210 <a href="http://www.wikidata.org/entity/Q2346516">Q2346516</a>,
 211 <a href="http://www.wikidata.org/entity/Q5183645">Q5183645</a>,
 212 <a href="http://www.wikidata.org/entity/Q5185497">Q5185497</a>,
 213 <a href="http://www.wikidata.org/entity/Q5216127">Q5216127</a>,
 214 <a href="http://www.wikidata.org/entity/Q5223127">Q5223127</a>,
 215 <a href="http://www.wikidata.org/entity/Q5261159">Q5261159</a>,
 216 <a href="http://www.wikidata.org/entity/Q1300759">Q1300759</a>,
 217 <a href="http://www.wikidata.org/entity/Q5521241">Q5521241</a>,
 218 <a href="http://www.wikidata.org/entity/Q7733434">Q7733434</a>,
 219 <a href="http://www.wikidata.org/entity/Q7736264">Q7736264</a>,
 220 <a href="http://www.wikidata.org/entity/Q7737032">Q7737032</a>,
 221 <a href="http://www.wikidata.org/entity/Q7882671">Q7882671</a>,
 222 <a href="http://www.wikidata.org/entity/Q7719427">Q7719427</a>,
 223 <a href="http://www.wikidata.org/entity/Q7719444">Q7719444</a>,
 224 <a href="http://www.wikidata.org/entity/Q7722575">Q7722575</a>,
 225 <a href="http://www.wikidata.org/entity/Q2629763">Q2629763</a>,
 226 <a href="http://www.wikidata.org/entity/Q2640346">Q2640346</a>,
 227 <a href="http://www.wikidata.org/entity/Q2649671">Q2649671</a>,
 228 <a href="http://www.wikidata.org/entity/Q7703851">Q7703851</a>,
 229 <a href="http://www.wikidata.org/entity/Q7747041">Q7747041</a>,
 230 <a href="http://www.wikidata.org/entity/Q6544949">Q6544949</a>,
 231 <a href="http://www.wikidata.org/entity/Q6672759">Q6672759</a>,
 232 <a href="http://www.wikidata.org/entity/Q2445896">Q2445896</a>,
 233 <a href="http://www.wikidata.org/entity/Q12124891">Q12124891</a>,
 234 <a href="http://www.wikidata.org/entity/Q3127044">Q3127044</a>,
 235 <a href="http://www.wikidata.org/entity/Q2511262">Q2511262</a>,
 236 <a href="http://www.wikidata.org/entity/Q2517672">Q2517672</a>,
 237 <a href="http://www.wikidata.org/entity/Q2543165">Q2543165</a>,
 238 <a href="http://www.wikidata.org/entity/Q426628">Q426628</a>,
 239 <a href="http://www.wikidata.org/entity/Q426628">Q426628</a>,
 240 <a href="http://www.wikidata.org/entity/Q12126890">Q12126890</a>,
 241 <a href="http://www.wikidata.org/entity/Q13359969">Q13359969</a>,
 242 <a href="http://www.wikidata.org/entity/Q13359969">Q13359969</a>,
 243 <a href="http://www.wikidata.org/entity/Q2294295">Q2294295</a>,
 244 <a href="http://www.wikidata.org/entity/Q2294295">Q2294295</a>,
 245 <a href="http://www.wikidata.org/entity/Q2559509">Q2559509</a>,
 246 <a href="http://www.wikidata.org/entity/Q2559912">Q2559912</a>,
 247 <a href="http://www.wikidata.org/entity/Q7760469">Q7760469</a>,
 248 <a href="http://www.wikidata.org/entity/Q6703974">Q6703974</a>,
 249 <a href="http://www.wikidata.org/entity/Q4744">Q4744</a>,
 250 <a href="http://www.wikidata.org/entity/Q7766962">Q7766962</a>,
 251 <a href="http://www.wikidata.org/entity/Q7768516">Q7768516</a>,
 252 <a href="http://www.wikidata.org/entity/Q7769205">Q7769205</a>,
 253 <a href="http://www.wikidata.org/entity/Q7769988">Q7769988</a>,
 254 <a href="http://www.wikidata.org/entity/Q2946945">Q2946945</a>,
 255 <a href="http://www.wikidata.org/entity/Q3212086">Q3212086</a>,
 256 <a href="http://www.wikidata.org/entity/Q3212086">Q3212086</a>,
 257 <a href="http://www.wikidata.org/entity/Q18218448">Q18218448</a>,
 258 <a href="http://www.wikidata.org/entity/Q18218448">Q18218448</a>,
 259 <a href="http://www.wikidata.org/entity/Q18218448">Q18218448</a>,
 260 <a href="http://www.wikidata.org/entity/Q6909175">Q6909175</a>,
 261 <a href="http://www.wikidata.org/entity/Q7405709">Q7405709</a>,
 262 <a href="http://www.wikidata.org/entity/Q7416149">Q7416149</a>,
 263 <a href="http://www.wikidata.org/entity/Q7239952">Q7239952</a>,
 264 <a href="http://www.wikidata.org/entity/Q7317332">Q7317332</a>,
 265 <a href="http://www.wikidata.org/entity/Q7783674">Q7783674</a>,
 266 <a href="http://www.wikidata.org/entity/Q7783704">Q7783704</a>,
 267 <a href="http://www.wikidata.org/entity/Q7857590">Q7857590</a>,
 268 <a href="http://www.wikidata.org/entity/Q3372526">Q3372526</a>,
 269 <a href="http://www.wikidata.org/entity/Q3372642">Q3372642</a>,
 270 <a href="http://www.wikidata.org/entity/Q3372816">Q3372816</a>,
 271 <a href="http://www.wikidata.org/entity/Q3372909">Q3372909</a>,
 272 <a href="http://www.wikidata.org/entity/Q7959649">Q7959649</a>,
 273 <a href="http://www.wikidata.org/entity/Q7977485">Q7977485</a>,
 274 <a href="http://www.wikidata.org/entity/Q7992684">Q7992684</a>,
 275 <a href="http://www.wikidata.org/entity/Q3817966">Q3817966</a>,
 276 <a href="http://www.wikidata.org/entity/Q3821852">Q3821852</a>,
 277 <a href="http://www.wikidata.org/entity/Q3420907">Q3420907</a>,
 278 <a href="http://www.wikidata.org/entity/Q3429733">Q3429733</a>,
 279 <a href="http://www.wikidata.org/entity/Q774474">Q774474</a></p>
 280
 281 <p>As usual, if you use Bitcoin and want to show your support of my
 282 activities, please send Bitcoin donations to my address
 283 <b><a href="bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b">15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b</a></b>.</p>