Publish.

diff --git a/blog/data/2014-04-09-nice-s3ql.txt b/blog/data/2014-04-09-nice-s3ql.txt
index 59bf76d0a42ccc2067aa9c3edfb329462bc0764e..87e9f1d787e761cf6d5a1e9e94389ab920e2a6e8 100644 (file)
--- a/blog/data/2014-04-09-nice-s3ql.txt
+++ b/blog/data/2014-04-09-nice-s3ql.txt
@@ -1,34 +1,35 @@
 Title: s3ql, a locally mounted cloud file system - nice free software
 Tags: english, debian, personvern, sikkerhet
 Title: s3ql, a locally mounted cloud file system - nice free software
 Tags: english, debian, personvern, sikkerhet

-Date: 2014-04-09 11:20
+Date: 2014-04-09 11:30

 
 

-<p>For a while now, I have been looking for a sensible off site backup
-solution to use at home.  My requirements are cheap and locally
-encrypted (in other words, I keep the keys, the storage provider do
-not have access to my private files).  One idea me and my friends have
-had over the years have been to use Google mail as storage, writing a
-Linux block device storing blocks as emails in the mail service
-provided by Google, and thus get heaps of free space.  On top of this
-one can add encryption, RAID and volume management to have lots of
-(fairly slow, I admit that) cheap and encrypted storage.  But I never
-found time to implement such system.  But the last few weeks I have
-looked at a system called
+<p>For a while now, I have been looking for a sensible offsite backup
+solution for use at home.  My requirements are simple, it must be
+cheap and locally encrypted (in other words, I keep the encryption
+keys, the storage provider do not have access to my private files).
+One idea me and my friends have had many years ago, before the cloud
+storage providers showed up, have been to use Google mail as storage,
+writing a Linux block device storing blocks as emails in the mail
+service provided by Google, and thus get heaps of free space.  On top
+of this one can add encryption, RAID and volume management to have
+lots of (fairly slow, I admit that) cheap and encrypted storage.  But
+I never found time to implement such system.  But the last few weeks I
+have looked at a system called

 <a href="https://bitbucket.org/nikratio/s3ql/">S3QL</a>, a locally
 mounted network backed file system with the features I need.</p>
 
 <p>S3QL is a fuse file system with a local cache and cloud storage,
 handling several different storage providers, any with Amazon S3,
 <a href="https://bitbucket.org/nikratio/s3ql/">S3QL</a>, a locally
 mounted network backed file system with the features I need.</p>
 
 <p>S3QL is a fuse file system with a local cache and cloud storage,
 handling several different storage providers, any with Amazon S3,

-Google Drive or OpenStack API.  There are heaps of such providers.  It
-can also use a local directory as storage, which combined with sshfs
-allow for file storage on any ssh server.  S3QL include support for
-encryption, compression, de-duplication, snapshots and immutable file
-systems, allowing me to mount the remote storage as a local mount
-point, look at and use the files as if they were local, while the
-content is stored in the cloud as well.  This allow me to have a
-backup that should survive fire.  The file system can not be shared
-between several machines at the same time, as only one can mount it at
-the time, but any machine with the encryption key and access to the
-storage service can mount it if it is unmounted.</p>
+Google Drive or OpenStack API.  There are heaps of such storage
+providers.  S3QL can also use a local directory as storage, which
+combined with sshfs allow for file storage on any ssh server.  S3QL
+include support for encryption, compression, de-duplication, snapshots
+and immutable file systems, allowing me to mount the remote storage as
+a local mount point, look at and use the files as if they were local,
+while the content is stored in the cloud as well.  This allow me to
+have a backup that should survive fire.  The file system can not be
+shared between several machines at the same time, as only one can
+mount it at the time, but any machine with the encryption key and
+access to the storage service can mount it if it is unmounted.</p>

 
 <p>It is simple to use.  I'm using it on Debian Wheezy, where the
 package is included already.  So to get started, run <tt>apt-get
 
 <p>It is simple to use.  I'm using it on Debian Wheezy, where the
 package is included already.  So to get started, run <tt>apt-get


@@ -36,9 +37,9 @@ install s3ql</tt>.  Next, pick a storage provider.  I ended up picking
 Greenqloud, after reading their nice recipe on
 <a href="https://greenqloud.zendesk.com/entries/44611757-How-To-Use-S3QL-to-mount-a-StorageQloud-bucket-on-Debian-Wheezy">how
 to use s3ql with their Amazon S3 service</a>, because I trust the laws
 Greenqloud, after reading their nice recipe on
 <a href="https://greenqloud.zendesk.com/entries/44611757-How-To-Use-S3QL-to-mount-a-StorageQloud-bucket-on-Debian-Wheezy">how
 to use s3ql with their Amazon S3 service</a>, because I trust the laws

-in Iceland more than those in USA when it come to keeping my data safe
-and private, and thus would rather spend money on a company in
-Iceland.  Another nice recipe is available from the article
+in Iceland more than those in USA when it come to keeping my personal
+data safe and private, and thus would rather spend money on a company
+in Iceland.  Another nice recipe is available from the article

 <a href="http://www.admin-magazine.com/HPC/Articles/HPC-Cloud-Storage">S3QL
 Filesystem for HPC Storage</a> by Jeff Layton in the HPC section of
 Admin magazine.  When the provider is picked, figure out how to get
 <a href="http://www.admin-magazine.com/HPC/Articles/HPC-Cloud-Storage">S3QL
 Filesystem for HPC Storage</a> by Jeff Layton in the HPC section of
 Admin magazine.  When the provider is picked, figure out how to get


@@ -69,7 +70,8 @@ details and password to create it:</p>
 
 <p><blockquote><pre>
 # mkdir -m 700 /var/lib/s3ql-cache
 
 <p><blockquote><pre>
 # mkdir -m 700 /var/lib/s3ql-cache

-# mkfs.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2   --ssl s3c://s.greenqloud.com:443/bucket-name
+# mkfs.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl s3c://s.greenqloud.com:443/bucket-name

 Enter backend login: 
 Enter backend password: 
 Before using S3QL, make sure to read the user's guide, especially
 Enter backend login: 
 Enter backend password: 
 Before using S3QL, make sure to read the user's guide, especially


@@ -94,7 +96,8 @@ Wrote 0.00 MB of compressed metadata.
 <p>The next step is mounting the file system to make the storage available.
 
 <p><blockquote><pre>
 <p>The next step is mounting the file system to make the storage available.
 
 <p><blockquote><pre>

-# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql

 Using 4 upload threads.
 Downloading and decompressing metadata...
 Reading metadata...
 Using 4 upload threads.
 Downloading and decompressing metadata...
 Reading metadata...


@@ -184,14 +187,17 @@ house.  Uploading 685 MiB with a 100 MiB cache gave me 305 kiB/s,
 which is very close to my upload speed, and downloading the same
 Debian installation ISO gave me 610 kiB/s, close to my download speed.
 Both were measured using <tt>dd</tt>.  So for me, the bottleneck is my
 which is very close to my upload speed, and downloading the same
 Debian installation ISO gave me 610 kiB/s, close to my download speed.
 Both were measured using <tt>dd</tt>.  So for me, the bottleneck is my

-network, not the file system code.</p>
+network, not the file system code.  I do not know what a good cache
+size would be, but suspect that the cache should e larger than your
+working set.</p>

 
 

-I mentioned that only one machine can mount the file system at the
+<p>I mentioned that only one machine can mount the file system at the

 time.  If another machine try, it is told that the file system is
 time.  If another machine try, it is told that the file system is

-busy:
+busy:</p>

 
 <p><blockquote><pre>
 
 <p><blockquote><pre>

-# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql

 Using 8 upload threads.
 Backend reports that fs is still mounted elsewhere, aborting.
 #
 Using 8 upload threads.
 Backend reports that fs is still mounted elsewhere, aborting.
 #


@@ -250,6 +256,15 @@ Store and Transformative Parallel I/O Approach</a>" by Hsing-Bung
 Chen, Benjamin McClelland, David Sherrill, Alfred Torrez, Parks Fields
 and Pamela Smith.  Please  have a look.</p>
 
 Chen, Benjamin McClelland, David Sherrill, Alfred Torrez, Parks Fields
 and Pamela Smith.  Please  have a look.</p>
 

+<p>Given my problems with different file systems earlier, I decided to
+check out the mounted S3QL file system to see if it would be usable as
+a home directory (in other word, that it provided POSIX semantics when
+it come to locking and umask handling etc).  Running
+<a href="http://people.skolelinux.org/pere/blog/Testing_if_a_file_system_can_be_used_for_home_directories___.html">my
+test code to check file system semantics, I was happy to discover that
+no error was found.  So the file system can be used for home
+directories, if one chooses to do so.</p>
+

 <p>If you do not want a locally file system, and want something that
 work without the Linux fuse file system, I would like to mention the
 <a href="http://www.tarsnap.com/">Tarsnap service</a>, which also
 <p>If you do not want a locally file system, and want something that
 work without the Linux fuse file system, I would like to mention the
 <a href="http://www.tarsnap.com/">Tarsnap service</a>, which also