Converted pages to temp site.

[homepage.git] / blog / archive / 2017 / 11 / 11.rss

<?xml version="1.0" encoding="ISO-8859-1"?>
<rss version='2.0' xmlns:lj='http://www.livejournal.org/rss/lj/1.0/'>
        <channel>
                <title>Petter Reinholdtsen - Entries from November 2017</title>
                <description>Entries from November 2017</description>
                <link>https://www.hungry.com/~pere/blog/</link>

        
        <item>
                <title>Metadata proposal for movies on the Internet Archive</title>
                <link>https://www.hungry.com/~pere/blog/Metadata_proposal_for_movies_on_the_Internet_Archive.html</link>        
                <guid isPermaLink="true">https://www.hungry.com/~pere/blog/Metadata_proposal_for_movies_on_the_Internet_Archive.html</guid>
                <pubDate>Tue, 28 Nov 2017 12:00:00 +0100</pubDate>
                <description>&lt;p&gt;It would be easier to locate the movie you want to watch in
&lt;a href=&quot;https://www.archive.org/&quot;&gt;the Internet Archive&lt;/a&gt;, if the
metadata about each movie was more complete and accurate.  In the
archiving community, a well known saying state that good metadata is a
love letter to the future.  The metadata in the Internet Archive could
use a face lift for the future to love us back.  Here is a proposal
for a small improvement that would make the metadata more useful
today.  I&#39;ve been unable to find any document describing the various
standard fields available when uploading videos to the archive, so
this proposal is based on my best quess and searching through several
of the existing movies.&lt;/p&gt;

&lt;p&gt;I have a few use cases in mind.  First of all, I would like to be
able to count the number of distinct movies in the Internet Archive,
without duplicates.  I would further like to identify the IMDB title
ID of the movies in the Internet Archive, to be able to look up a IMDB
title ID and know if I can fetch the video from there and share it
with my friends.&lt;/p&gt;

&lt;p&gt;Second, I would like the Butter data provider for The Internet
archive
(&lt;a href=&quot;https://github.com/butterproviders/butter-provider-archive&quot;&gt;available
from github&lt;/a&gt;), to list as many of the good movies as possible.  The
plugin currently do a search in the archive with the following
parameters:&lt;/p&gt;

&lt;p&gt;&lt;pre&gt;
collection:moviesandfilms
AND NOT collection:movie_trailers
AND -mediatype:collection
AND format:&quot;Archive BitTorrent&quot;
AND year
&lt;/pre&gt;&lt;/p&gt;

&lt;p&gt;Most of the cool movies that fail to show up in Butter do so
because the &#39;year&#39; field is missing.  The &#39;year&#39; field is populated by
the year part from the &#39;date&#39; field, and should be when the movie was
released (date or year).  Two such examples are
&lt;a href=&quot;https://archive.org/details/SidneyOlcottsBen-hur1905&quot;&gt;Ben Hur
from 1905&lt;/a&gt; and
&lt;a href=&quot;https://archive.org/details/Caminandes2GranDillama&quot;&gt;Caminandes
2: Gran Dillama from 2013&lt;/a&gt;, where the year metadata field is
missing.&lt;/p&gt;

So, my proposal is simply, for every movie in The Internet Archive
where an IMDB title ID exist, please fill in these metadata fields
(note, they can be updated also long after the video was uploaded, but
as far as I can tell, only by the uploader):

&lt;dl&gt;

&lt;dt&gt;mediatype&lt;/dt&gt;
&lt;dd&gt;Should be &#39;movie&#39; for movies.&lt;/dd&gt;

&lt;dt&gt;collection&lt;/dt&gt;
&lt;dd&gt;Should contain &#39;moviesandfilms&#39;.&lt;/dd&gt;

&lt;dt&gt;title&lt;/dt&gt;
&lt;dd&gt;The title of the movie, without the publication year.&lt;/dd&gt;

&lt;dt&gt;date&lt;/dt&gt;
&lt;dd&gt;The data or year the movie was released.  This make the movie show
up in Butter, as well as make it possible to know the age of the
movie and is useful to figure out copyright status.&lt;/dd&gt;

&lt;dt&gt;director&lt;/dt&gt;
&lt;dd&gt;The director of the movie.  This make it easier to know if the
correct movie is found in movie databases.&lt;/dd&gt;

&lt;dt&gt;publisher&lt;/dt&gt;
&lt;dd&gt;The production company making the movie.  Also useful for
identifying the correct movie.&lt;/dd&gt;

&lt;dt&gt;links&lt;/dt&gt;

&lt;dd&gt;Add a link to the IMDB title page, for example like this: &amp;lt;a
href=&quot;http://www.imdb.com/title/tt0028496/&quot;&amp;gt;Movie in
IMDB&amp;lt;/a&amp;gt;.  This make it easier to find duplicates and allow for
counting of number of unique movies in the Archive.  Other external
references, like to TMDB, could be added like this too.&lt;/dd&gt;

&lt;/dl&gt;

&lt;p&gt;I did consider proposing a Custom field for the IMDB title ID (for
example &#39;imdb_title_url&#39;, &#39;imdb_code&#39; or simply &#39;imdb&#39;, but suspect it
will be easier to simply place it in the links free text field.&lt;/p&gt;

&lt;p&gt;I created
&lt;a href=&quot;https://github.com/petterreinholdtsen/public-domain-free-imdb&quot;&gt;a
list of IMDB title IDs for several thousand movies in the Internet
Archive&lt;/a&gt;, but I also got a list of several thousand movies without
such IMDB title ID (and quite a few duplicates).  It would be great if
this data set could be integrated into the Internet Archive metadata
to be available for everyone in the future, but with the current
policy of leaving metadata editing to the uploaders, it will take a
while before this happen.  If you have uploaded movies into the
Internet Archive, you can help.  Please consider following my proposal
above for your movies, to ensure that movie is properly
counted. :)&lt;/p&gt;

&lt;p&gt;The list is mostly generated using wikidata, which based on
Wikipedia articles make it possible to link between IMDB and movies in
the Internet Archive.  But there are lots of movies without a
Wikipedia article, and some movies where only a collection page exist
(like for &lt;a href=&quot;https://en.wikipedia.org/wiki/Caminandes&quot;&gt;the
Caminandes example above&lt;/a&gt;, where there are three movies but only
one Wikidata entry).&lt;/p&gt;

&lt;p&gt;As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
&lt;b&gt;&lt;a href=&quot;bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&quot;&gt;15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&lt;/a&gt;&lt;/b&gt;.&lt;/p&gt;
</description>
        </item>
        
        <item>
                <title>Legal to share more than 3000 movies listed on IMDB?</title>
                <link>https://www.hungry.com/~pere/blog/Legal_to_share_more_than_3000_movies_listed_on_IMDB_.html</link>        
                <guid isPermaLink="true">https://www.hungry.com/~pere/blog/Legal_to_share_more_than_3000_movies_listed_on_IMDB_.html</guid>
                <pubDate>Sat, 18 Nov 2017 21:20:00 +0100</pubDate>
                <description>&lt;p&gt;A month ago, I blogged about my work to
&lt;a href=&quot;https://people.skolelinux.org/pere/blog/Locating_IMDB_IDs_of_movies_in_the_Internet_Archive_using_Wikidata.html&quot;&gt;automatically
check the copyright status of IMDB entries&lt;/a&gt;, and try to count the
number of movies listed in IMDB that is legal to distribute on the
Internet.  I have continued to look for good data sources, and
identified a few more.  The code used to extract information from
various data sources is available in
&lt;a href=&quot;https://github.com/petterreinholdtsen/public-domain-free-imdb&quot;&gt;a
git repository&lt;/a&gt;, currently available from github.&lt;/p&gt;

&lt;p&gt;So far I have identified 3186 unique IMDB title IDs.  To gain
better understanding of the structure of the data set, I created a
histogram of the year associated with each movie (typically release
year).  It is interesting to notice where the peaks and dips in the
graph are located.  I wonder why they are placed there.  I suspect
World War II caused the dip around 1940, but what caused the peak
around 2010?&lt;/p&gt;

&lt;p align=&quot;center&quot;&gt;&lt;img src=&quot;https://people.skolelinux.org/pere/blog/images/2017-11-18-verk-i-det-fri-filmer.png&quot; /&gt;&lt;/p&gt;

&lt;p&gt;I&#39;ve so far identified ten sources for IMDB title IDs for movies in
the public domain or with a free license.  This is the statistics
reported when running &#39;make stats&#39; in the git repository:&lt;/p&gt;

&lt;pre&gt;
  249 entries (    6 unique) with and   288 without IMDB title ID in free-movies-archive-org-butter.json
 2301 entries (  540 unique) with and     0 without IMDB title ID in free-movies-archive-org-wikidata.json
  830 entries (   29 unique) with and     0 without IMDB title ID in free-movies-icheckmovies-archive-mochard.json
 2109 entries (  377 unique) with and     0 without IMDB title ID in free-movies-imdb-pd.json
  291 entries (  122 unique) with and     0 without IMDB title ID in free-movies-letterboxd-pd.json
  144 entries (  135 unique) with and     0 without IMDB title ID in free-movies-manual.json
  350 entries (    1 unique) with and   801 without IMDB title ID in free-movies-publicdomainmovies.json
    4 entries (    0 unique) with and   124 without IMDB title ID in free-movies-publicdomainreview.json
  698 entries (  119 unique) with and   118 without IMDB title ID in free-movies-publicdomaintorrents.json
    8 entries (    8 unique) with and   196 without IMDB title ID in free-movies-vodo.json
 3186 unique IMDB title IDs in total
&lt;/pre&gt;

&lt;p&gt;The entries without IMDB title ID are candidates to increase the
data set, but might equally well be duplicates of entries already
listed with IMDB title ID in one of the other sources, or represent
movies that lack a IMDB title ID.  I&#39;ve seen examples of all these
situations when peeking at the entries without IMDB title ID.  Based
on these data sources, the lower bound for movies listed in IMDB that
are legal to distribute on the Internet is between 3186 and 4713.

&lt;p&gt;It would be great for improving the accuracy of this measurement,
if the various sources added IMDB title ID to their metadata.  I have
tried to reach the people behind the various sources to ask if they
are interested in doing this, without any replies so far.  Perhaps you
can help me get in touch with the people behind VODO, Public Domain
Torrents, Public Domain Movies and Public Domain Review to try to
convince them to add more metadata to their movie entries?&lt;/p&gt;

&lt;p&gt;Another way you could help is by adding pages to Wikipedia about
movies that are legal to distribute on the Internet.  If such page
exist and include a link to both IMDB and The Internet Archive, the
script used to generate free-movies-archive-org-wikidata.json should
pick up the mapping as soon as wikidata is updates.&lt;/p&gt;

&lt;p&gt;As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
&lt;b&gt;&lt;a href=&quot;bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&quot;&gt;15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&lt;/a&gt;&lt;/b&gt;.&lt;/p&gt;
</description>
        </item>
        
        <item>
                <title>Some notes on fault tolerant storage systems</title>
                <link>https://www.hungry.com/~pere/blog/Some_notes_on_fault_tolerant_storage_systems.html</link>        
                <guid isPermaLink="true">https://www.hungry.com/~pere/blog/Some_notes_on_fault_tolerant_storage_systems.html</guid>
                <pubDate>Wed, 1 Nov 2017 15:35:00 +0100</pubDate>
                <description>&lt;p&gt;If you care about how fault tolerant your storage is, you might
find these articles and papers interesting.  They have formed how I
think of when designing a storage system.&lt;/p&gt;

&lt;ul&gt;

&lt;li&gt;USENIX :login; &lt;a
href=&quot;https://www.usenix.org/publications/login/summer2017/ganesan&quot;&gt;Redundancy
Does Not Imply Fault Tolerance.  Analysis of Distributed Storage
Reactions to Single Errors and Corruptions&lt;/a&gt; by Aishwarya Ganesan,
Ramnatthan Alagappan, Andrea C. Arpaci-Dusseau, and Remzi
H. Arpaci-Dusseau&lt;/li&gt;

&lt;li&gt;ZDNet
&lt;a href=&quot;http://www.zdnet.com/article/why-raid-5-stops-working-in-2009/&quot;&gt;Why
RAID 5 stops working in 2009&lt;/a&gt; by Robin Harris&lt;/li&gt;

&lt;li&gt;ZDNet
&lt;a href=&quot;http://www.zdnet.com/article/why-raid-6-stops-working-in-2019/&quot;&gt;Why
RAID 6 stops working in 2019&lt;/a&gt; by Robin Harris&lt;/li&gt;

&lt;li&gt;USENIX FAST&#39;07
&lt;a href=&quot;http://research.google.com/archive/disk_failures.pdf&quot;&gt;Failure
Trends in a Large Disk Drive Population&lt;/a&gt; by Eduardo Pinheiro,
Wolf-Dietrich Weber and Luiz André Barroso&lt;/li&gt;

&lt;li&gt;USENIX ;login: &lt;a
href=&quot;https://www.usenix.org/system/files/login/articles/hughes12-04.pdf&quot;&gt;Data
Integrity.  Finding Truth in a World of Guesses and Lies&lt;/a&gt; by Doug
Hughes&lt;/li&gt;

&lt;li&gt;USENIX FAST&#39;08
&lt;a href=&quot;https://www.usenix.org/events/fast08/tech/full_papers/bairavasundaram/bairavasundaram_html/&quot;&gt;An
Analysis of Data Corruption in the Storage Stack&lt;/a&gt; by
L. N. Bairavasundaram, G. R. Goodson, B. Schroeder, A. C.
Arpaci-Dusseau, and R. H. Arpaci-Dusseau&lt;/li&gt;

&lt;li&gt;USENIX FAST&#39;07 &lt;a
href=&quot;https://www.usenix.org/legacy/events/fast07/tech/schroeder/schroeder_html/&quot;&gt;Disk
failures in the real world: what does an MTTF of 1,000,000 hours mean
to you?&lt;/a&gt; by B. Schroeder and G. A. Gibson.&lt;/li&gt;

&lt;li&gt;USENIX ;login: &lt;a
href=&quot;https://www.usenix.org/events/fast08/tech/full_papers/jiang/jiang_html/&quot;&gt;Are
Disks the Dominant Contributor for Storage Failures?  A Comprehensive
Study of Storage Subsystem Failure Characteristics&lt;/a&gt; by Weihang
Jiang, Chongfeng Hu, Yuanyuan Zhou, and Arkady Kanevsky&lt;/li&gt;

&lt;li&gt;SIGMETRICS 2007
&lt;a href=&quot;http://research.cs.wisc.edu/adsl/Publications/latent-sigmetrics07.pdf&quot;&gt;An
analysis of latent sector errors in disk drives&lt;/a&gt; by
L. N. Bairavasundaram, G. R. Goodson, S. Pasupathy, and J. Schindler&lt;/li&gt;

&lt;/ul&gt;

&lt;p&gt;Several of these research papers are based on data collected from
hundred thousands or millions of disk, and their findings are eye
opening.  The short story is simply do not implicitly trust RAID or
redundant storage systems.  Details matter.  And unfortunately there
are few options on Linux addressing all the identified issues.  Both
ZFS and Btrfs are doing a fairly good job, but have legal and
practical issues on their own.  I wonder how cluster file systems like
Ceph do in this regard.  After all, there is an old saying, you know
you have a distributed system when the crash of a computer you have
never heard of stops you from getting any work done.  The same holds
true if fault tolerance do not work.&lt;/p&gt;

&lt;p&gt;Just remember, in the end, it do not matter how redundant, or how
fault tolerant your storage is, if you do not continuously monitor its
status to detect and replace failed disks.&lt;/p&gt;

&lt;p&gt;As usual, if you use Bitcoin and want to show your support of my
activities, please send Bitcoin donations to my address
&lt;b&gt;&lt;a href=&quot;bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&quot;&gt;15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&lt;/a&gt;&lt;/b&gt;.&lt;/p&gt;
</description>
        </item>
        
        </channel>
</rss>