blog/archive/2017/11/11.rss

   1 <?xml version="1.0" encoding="ISO-8859-1"?>
   2 <rss version='2.0' xmlns:lj='http://www.livejournal.org/rss/lj/1.0/'>
   3         <channel>
   4                 <title>Petter Reinholdtsen - Entries from November 2017</title>
   5                 <description>Entries from November 2017</description>
   6                 <link>http://people.skolelinux.org/pere/blog/</link>
   7
   8
   9         <item>
  10                 <title>Legal to share more than 3000 movies listed on IMDB?</title>
  11                 <link>http://people.skolelinux.org/pere/blog/Legal_to_share_more_than_3000_movies_listed_on_IMDB_.html</link>
  12                 <guid isPermaLink="true">http://people.skolelinux.org/pere/blog/Legal_to_share_more_than_3000_movies_listed_on_IMDB_.html</guid>
  13                 <pubDate>Sat, 18 Nov 2017 21:20:00 +0100</pubDate>
  14                 <description>&lt;p&gt;A month ago, I blogged about my work to
  15 &lt;a href=&quot;http://people.skolelinux.org/pere/blog/Locating_IMDB_IDs_of_movies_in_the_Internet_Archive_using_Wikidata.html&quot;&gt;automatically
  16 check the copyright status of IMDB entries&lt;/a&gt;, and try to count the
  17 number of movies listed in IMDB that is legal to distribute on the
  18 Internet.  I have continued to look for good data sources, and
  19 identified a few more.  The code used to extract information from
  20 various data sources is available in
  21 &lt;a href=&quot;https://github.com/petterreinholdtsen/public-domain-free-imdb&quot;&gt;a
  22 git repository&lt;/a&gt;, currently available from github.&lt;/p&gt;
  23
  24 &lt;p&gt;So far I have identified 3186 unique IMDB title IDs.  To gain
  25 better understanding of the structure of the data set, I created a
  26 histogram of the year associated with each movie (typically release
  27 year).  It is interesting to notice where the peaks and dips in the
  28 graph are located.  I wonder why they are placed there.  I suspect
  29 World Word II caused the dip around 1940, but what caused the peak
  30 around 2010?&lt;/p&gt;
  31
  32 &lt;p align=&quot;center&quot;&gt;&lt;img src=&quot;http://people.skolelinux.org/pere/blog/images/2017-11-18-verk-i-det-fri-filmer.png&quot; /&gt;&lt;/p&gt;
  33
  34 &lt;p&gt;I&#39;ve so far identified ten sources for IMDB title IDs for movies in
  35 the public domain or with a free license.  This is the statistics
  36 reported when running &#39;make stats&#39; in the git repository:&lt;/p&gt;
  37
  38 &lt;pre&gt;
  39   249 entries (    6 unique) with and   288 without IMDB title ID in free-movies-archive-org-butter.json
  40  2301 entries (  540 unique) with and     0 without IMDB title ID in free-movies-archive-org-wikidata.json
  41   830 entries (   29 unique) with and     0 without IMDB title ID in free-movies-icheckmovies-archive-mochard.json
  42  2109 entries (  377 unique) with and     0 without IMDB title ID in free-movies-imdb-pd.json
  43   291 entries (  122 unique) with and     0 without IMDB title ID in free-movies-letterboxd-pd.json
  44   144 entries (  135 unique) with and     0 without IMDB title ID in free-movies-manual.json
  45   350 entries (    1 unique) with and   801 without IMDB title ID in free-movies-publicdomainmovies.json
  46     4 entries (    0 unique) with and   124 without IMDB title ID in free-movies-publicdomainreview.json
  47   698 entries (  119 unique) with and   118 without IMDB title ID in free-movies-publicdomaintorrents.json
  48     8 entries (    8 unique) with and   196 without IMDB title ID in free-movies-vodo.json
  49  3186 unique IMDB title IDs in total
  50 &lt;/pre&gt;
  51
  52 &lt;p&gt;The entries without IMDB title ID are candidates to increase the
  53 data set, but might equally well be duplicates of entries already
  54 listed with IMDB title ID in one of the other sources, or represent
  55 movies that lack a IMDB title ID.  I&#39;ve seen examples of all these
  56 situations when peeking at the entries without IMDB title ID.  Based
  57 on these data sources, the lower bound for movies listed in IMDB that
  58 are legal to distribute on the Internet is between 3186 and 4713.
  59
  60 &lt;p&gt;It would be great for improving the accuracy of this measurement,
  61 if the various sources added IMDB title ID to their metadata.  I have
  62 tried to reach the people behind the various sources to ask if they
  63 are interested in doing this, without any replies so far.  Perhaps you
  64 can help me get in touch with the people behind VODO, Public Domain
  65 Torrents, Public Domain Movies and Public Domain Review to try to
  66 convince them to add more metadata to their movie entries?&lt;/p&gt;
  67
  68 &lt;p&gt;Another way you could help is by adding pages to Wikipedia about
  69 movies that are legal to distribute on the Internet.  If such page
  70 exist and include a link to both IMDB and The Internet Archive, the
  71 script used to generate free-movies-archive-org-wikidata.json should
  72 pick up the mapping as soon as wikidata is updates.&lt;/p&gt;
  73 </description>
  74         </item>
  75
  76         <item>
  77                 <title>Some notes on fault tolerant storage systems</title>
  78                 <link>http://people.skolelinux.org/pere/blog/Some_notes_on_fault_tolerant_storage_systems.html</link>
  79                 <guid isPermaLink="true">http://people.skolelinux.org/pere/blog/Some_notes_on_fault_tolerant_storage_systems.html</guid>
  80                 <pubDate>Wed, 1 Nov 2017 15:35:00 +0100</pubDate>
  81                 <description>&lt;p&gt;If you care about how fault tolerant your storage is, you might
  82 find these articles and papers interesting.  They have formed how I
  83 think of when designing a storage system.&lt;/p&gt;
  84
  85 &lt;ul&gt;
  86
  87 &lt;li&gt;USENIX :login; &lt;a
  88 href=&quot;https://www.usenix.org/publications/login/summer2017/ganesan&quot;&gt;Redundancy
  89 Does Not Imply Fault Tolerance.  Analysis of Distributed Storage
  90 Reactions to Single Errors and Corruptions&lt;/a&gt; by Aishwarya Ganesan,
  91 Ramnatthan Alagappan, Andrea C. Arpaci-Dusseau, and Remzi
  92 H. Arpaci-Dusseau&lt;/li&gt;
  93
  94 &lt;li&gt;ZDNet
  95 &lt;a href=&quot;http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/&quot;&gt;Why
  96 RAID 5 stops working in 2009&lt;/a&gt; by Robin Harris&lt;/li&gt;
  97
  98 &lt;li&gt;ZDNet
  99 &lt;a href=&quot;http://www.zdnet.com/article/why-raid-6-stops-working-in-2019/&quot;&gt;Why
 100 RAID 6 stops working in 2019&lt;/a&gt; by Robin Harris&lt;/li&gt;
 101
 102 &lt;li&gt;USENIX FAST&#39;07
 103 &lt;a href=&quot;http://research.google.com/archive/disk_failures.pdf&quot;&gt;Failure
 104 Trends in a Large Disk Drive Population&lt;/a&gt; by Eduardo Pinheiro,
 105 Wolf-Dietrich Weber and Luiz André Barroso&lt;/li&gt;
 106
 107 &lt;li&gt;USENIX ;login: &lt;a
 108 href=&quot;https://www.usenix.org/system/files/login/articles/hughes12-04.pdf&quot;&gt;Data
 109 Integrity.  Finding Truth in a World of Guesses and Lies&lt;/a&gt; by Doug
 110 Hughes&lt;/li&gt;
 111
 112 &lt;li&gt;USENIX FAST&#39;08
 113 &lt;a href=&quot;https://www.usenix.org/events/fast08/tech/full_papers/bairavasundaram/bairavasundaram_html/&quot;&gt;An
 114 Analysis of Data Corruption in the Storage Stack&lt;/a&gt; by
 115 L. N. Bairavasundaram, G. R. Goodson, B. Schroeder, A. C.
 116 Arpaci-Dusseau, and R. H. Arpaci-Dusseau&lt;/li&gt;
 117
 118 &lt;li&gt;USENIX FAST&#39;07 &lt;a
 119 href=&quot;https://www.usenix.org/legacy/events/fast07/tech/schroeder/schroeder_html/&quot;&gt;Disk
 120 failures in the real world: what does an MTTF of 1,000,000 hours mean
 121 to you?&lt;/a&gt; by B. Schroeder and G. A. Gibson.&lt;/li&gt;
 122
 123 &lt;li&gt;USENIX ;login: &lt;a
 124 href=&quot;https://www.usenix.org/events/fast08/tech/full_papers/jiang/jiang_html/&quot;&gt;Are
 125 Disks the Dominant Contributor for Storage Failures?  A Comprehensive
 126 Study of Storage Subsystem Failure Characteristics&lt;/a&gt; by Weihang
 127 Jiang, Chongfeng Hu, Yuanyuan Zhou, and Arkady Kanevsky&lt;/li&gt;
 128
 129 &lt;li&gt;SIGMETRICS 2007
 130 &lt;a href=&quot;http://research.cs.wisc.edu/adsl/Publications/latent-sigmetrics07.pdf&quot;&gt;An
 131 analysis of latent sector errors in disk drives&lt;/a&gt; by
 132 L. N. Bairavasundaram, G. R. Goodson, S. Pasupathy, and J. Schindler&lt;/li&gt;
 133
 134 &lt;/ul&gt;
 135
 136 &lt;p&gt;Several of these research papers are based on data collected from
 137 hundred thousands or millions of disk, and their findings are eye
 138 opening.  The short story is simply do not implicitly trust RAID or
 139 redundant storage systems.  Details matter.  And unfortunately there
 140 are few options on Linux addressing all the identified issues.  Both
 141 ZFS and Btrfs are doing a fairly good job, but have legal and
 142 practical issues on their own.  I wonder how cluster file systems like
 143 Ceph do in this regard.  After all, there is an old saying, you know
 144 you have a distributed system when the crash of a computer you have
 145 never heard of stops you from getting any work done.  The same holds
 146 true if fault tolerance do not work.&lt;/p&gt;
 147
 148 &lt;p&gt;Just remember, in the end, it do not matter how redundant, or how
 149 fault tolerant your storage is, if you do not continuously monitor its
 150 status to detect and replace failed disks.&lt;/p&gt;
 151 </description>
 152         </item>
 153
 154         </channel>
 155 </rss>