+ Petter Reinholdtsen + +

s3ql, a locally mounted cloud file system - nice free software

@@ -381,7 +675,7 @@ image.

@@ -391,7 +685,7 @@ image.

@@ -429,7 +723,7 @@ image.

@@ -447,7 +741,7 @@ image.

diff --git a/blog/index.html b/blog/index.html index 105d5bc96a..1f30861d93 100644 --- a/blog/index.html +++ b/blog/index.html @@ -19,6 +19,294 @@ +

9th April 2014

For a while now, I have been looking for a sensible offsite backup +solution for use at home. My requirements are simple, it must be +cheap and locally encrypted (in other words, I keep the encryption +keys, the storage provider do not have access to my private files). +One idea me and my friends have had many years ago, before the cloud +storage providers showed up, have been to use Google mail as storage, +writing a Linux block device storing blocks as emails in the mail +service provided by Google, and thus get heaps of free space. On top +of this one can add encryption, RAID and volume management to have +lots of (fairly slow, I admit that) cheap and encrypted storage. But +I never found time to implement such system. But the last few weeks I +have looked at a system called +S3QL, a locally +mounted network backed file system with the features I need.

+ +

S3QL is a fuse file system with a local cache and cloud storage, +handling several different storage providers, any with Amazon S3, +Google Drive or OpenStack API. There are heaps of such storage +providers. S3QL can also use a local directory as storage, which +combined with sshfs allow for file storage on any ssh server. S3QL +include support for encryption, compression, de-duplication, snapshots +and immutable file systems, allowing me to mount the remote storage as +a local mount point, look at and use the files as if they were local, +while the content is stored in the cloud as well. This allow me to +have a backup that should survive fire. The file system can not be +shared between several machines at the same time, as only one can +mount it at the time, but any machine with the encryption key and +access to the storage service can mount it if it is unmounted.

+ +

It is simple to use. I'm using it on Debian Wheezy, where the +package is included already. So to get started, run apt-get +install s3ql. Next, pick a storage provider. I ended up picking +Greenqloud, after reading their nice recipe on +how +to use s3ql with their Amazon S3 service, because I trust the laws +in Iceland more than those in USA when it come to keeping my personal +data safe and private, and thus would rather spend money on a company +in Iceland. Another nice recipe is available from the article +S3QL +Filesystem for HPC Storage by Jeff Layton in the HPC section of +Admin magazine. When the provider is picked, figure out how to get +the API key needed to connect to the storage API. With Greencloud, +the key did not show up until I had added payment details to my +account.

+ +

Armed with the API access details, it is time to create the file +system. First, create a new bucket in the cloud. This bucket is the +file system storage area. I picked a bucket name reflecting the +machine that was going to store data there, but any name will do. +I'll refer to it as bucket-name below. In addition, one need +the API login and password, and a locally created password. Store it +all in ~root/.s3ql/authinfo2 like this: + +

+[s3c]
+storage-url: s3c://s.greenqloud.com:443/bucket-name
+backend-login: API-login
+backend-password: API-password
+fs-passphrase: local-password
+

+ +

I create my local passphrase using pwget 50 or similar, +but any sensible way to create a fairly random password should do it. +Armed with these details, it is now time to run mkfs, entering the API +details and password to create it:

+ +

+# mkdir -m 700 /var/lib/s3ql-cache
+# mkfs.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl s3c://s.greenqloud.com:443/bucket-name
+Enter backend login: 
+Enter backend password: 
+Before using S3QL, make sure to read the user's guide, especially
+the 'Important Rules to Avoid Loosing Data' section.
+Enter encryption password: 
+Confirm encryption password: 
+Generating random encryption key...
+Creating metadata tables...
+Dumping metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Compressing and uploading metadata...
+Wrote 0.00 MB of compressed metadata.
+#

+ +

The next step is mounting the file system to make the storage available. + +

+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+Using 4 upload threads.
+Downloading and decompressing metadata...
+Reading metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Mounting filesystem...
+# df -h /mnt
+Filesystem                              Size  Used Avail Use% Mounted on
+s3c://s.greenqloud.com:443/bucket-name  1.0T     0  1.0T   0% /s3ql
+#
+

+ +

The file system is now ready for use. I use rsync to store my +backups in it, and as the metadata used by rsync is downloaded at +mount time, no network traffic (and storage cost) is triggered by +running rsync. To unmount, one should not use the normal umount +command, as this will not flush the cache to the cloud storage, but +instead running the umount.s3ql command like this: + +

+# umount.s3ql /s3ql
+# 
+

+ +

There is a fsck command available to check the file system and +correct any problems detected. This can be used if the local server +crashes while the file system is mounted, to reset the "already +mounted" flag. This is what it look like when processing a working +file system:

+ +

+# fsck.s3ql --force --ssl s3c://s.greenqloud.com:443/bucket-name
+Using cached metadata.
+File system seems clean, checking anyway.
+Checking DB integrity...
+Creating temporary extra indices...
+Checking lost+found...
+Checking cached objects...
+Checking names (refcounts)...
+Checking contents (names)...
+Checking contents (inodes)...
+Checking contents (parent inodes)...
+Checking objects (reference counts)...
+Checking objects (backend)...
+..processed 5000 objects so far..
+..processed 10000 objects so far..
+..processed 15000 objects so far..
+Checking objects (sizes)...
+Checking blocks (referenced objects)...
+Checking blocks (refcounts)...
+Checking inode-block mapping (blocks)...
+Checking inode-block mapping (inodes)...
+Checking inodes (refcounts)...
+Checking inodes (sizes)...
+Checking extended attributes (names)...
+Checking extended attributes (inodes)...
+Checking symlinks (inodes)...
+Checking directory reachability...
+Checking unix conventions...
+Checking referential integrity...
+Dropping temporary indices...
+Backing up old metadata...
+Dumping metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Compressing and uploading metadata...
+Wrote 0.89 MB of compressed metadata.
+# 
+

+ +

Thanks to the cache, working on files that fit in the cache is very +quick, about the same speed as local file access. Uploading large +amount of data is to me limited by the bandwidth out of and into my +house. Uploading 685 MiB with a 100 MiB cache gave me 305 kiB/s, +which is very close to my upload speed, and downloading the same +Debian installation ISO gave me 610 kiB/s, close to my download speed. +Both were measured using dd. So for me, the bottleneck is my +network, not the file system code. I do not know what a good cache +size would be, but suspect that the cache should e larger than your +working set.

+ +

I mentioned that only one machine can mount the file system at the +time. If another machine try, it is told that the file system is +busy:

+ +

+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+Using 8 upload threads.
+Backend reports that fs is still mounted elsewhere, aborting.
+#
+

+ +

The file content is uploaded when the cache is full, while the +metadata is uploaded once every 24 hour by default. To ensure the +file system content is flushed to the cloud, one can either umount the +file system, or ask s3ql to flush the cache and metadata using +s3qlctrl: + +

+# s3qlctrl upload-meta /s3ql
+# s3qlctrl flushcache /s3ql
+# 
+

+ +

If you are curious about how much space your data uses in the +cloud, and how much compression and deduplication cut down on the +storage usage, you can use s3qlstat on the mounted file system to get +a report:

+ +

+# s3qlstat /s3ql
+Directory entries:    9141
+Inodes:               9143
+Data blocks:          8851
+Total data size:      22049.38 MB
+After de-duplication: 21955.46 MB (99.57% of total)
+After compression:    21877.28 MB (99.22% of total, 99.64% of de-duplicated)
+Database size:        2.39 MB (uncompressed)
+(some values do not take into account not-yet-uploaded dirty blocks in cache)
+#
+

+ +

I mentioned earlier that there are several possible suppliers of +storage. I did not try to locate them all, but am aware of at least +Greenqloud, +Google Drive, +Amazon S3 web serivces, +Rackspace and +Crowncloud. The latter even +accept payment in Bitcoin. Pick one that suit your need. Some of +them provide several GiB of free storage, but the prize models are +quire different and you will have to figure out what suit you +best.

+ +

While researching this blog post, I had a look at research papers +and posters discussing the S3QL file system. There are several, which +told me that the file system is getting a critical check by the +science community and increased my confidence in using it. One nice +poster is titled +"An +Innovative Parallel Cloud Storage System using OpenStackâs SwiftObject +Store and Transformative Parallel I/O Approach" by Hsing-Bung +Chen, Benjamin McClelland, David Sherrill, Alfred Torrez, Parks Fields +and Pamela Smith. Please have a look.

+ +

Given my problems with different file systems earlier, I decided to +check out the mounted S3QL file system to see if it would be usable as +a home directory (in other word, that it provided POSIX semantics when +it come to locking and umask handling etc). Running +my +test code to check file system semantics, I was happy to discover that +no error was found. So the file system can be used for home +directories, if one chooses to do so.

+ +

If you do not want a locally file system, and want something that +work without the Linux fuse file system, I would like to mention the +Tarsnap service, which also +provide locally encrypted backup using a command line client. It have +a nicer access control system, where one can split out read and write +access, allowing some systems to write to the backup and others to +only read from it.

+ +

As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

+ + + Tags: debian, english, personvern, sikkerhet. + + +

EU-domstolen bekreftet i dag at datalagringsdirektivet er ulovlig

8th April 2014

@@ -683,104 +971,6 @@ workstation, LTSP client or LTSP server.

Hvordan bÃ¸r RFC 822-formattert epost lagres i en NOARK5-database?

7th March 2014

For noen uker siden ble NXCs fri programvarelisenserte -NOARK5-lÃ¸sning -presentert hos -NUUG (video -pÃ¥ youtube -forelÃ¸big), og det fikk meg til Ã¥ titte litt mer pÃ¥ NOARK5, -standarden for arkivhÃ¥ndtering i det offentlige Norge. Jeg lurer pÃ¥ -om denne kjernen kan vÃ¦re nyttig i et par av mine prosjekter, og for ett -av dem er det mest aktuelt Ã¥ lagre epost. Jeg klarte ikke finne noen -anbefaling om hvordan RFC 822-formattert epost (aka Internett-epost) -burde lagres i NOARK5, selv om jeg vet at noen arkiver tar -PDF-utskrift av eposten med sitt epostprogram og sÃ¥ arkiverer PDF-en -(eller enda vÃ¦rre, tar papirutskrift og lagrer bildet av eposten som -PDF i arkivet).

- -

Det er ikke sÃ¥ mange formater som er akseptert av riksarkivet til -langtidsoppbevaring av offentlige arkiver, og PDF og XML er de mest -aktuelle i sÃ¥ mÃ¥te. Det slo meg at det mÃ¥tte da finnes en eller annen -egnet XML-representasjon og at det kanskje var enighet om hvilken som -burde brukes, sÃ¥ jeg tok mot til meg og spurte -SAMDOK, en gruppe tilknyttet -arkivverket som ser ut til Ã¥ jobbe med NOARK-samhandling, om de hadde -noen anbefalinger: - -

-
Hei.
- -
Usikker pÃ¥ om dette er riktig forum Ã¥ ta opp mitt spÃ¸rsmÃ¥l, men jeg -lurer pÃ¥ om det er definert en anbefaling om hvordan RFC -822-formatterte epost (aka vanlig Internet-epost) bÃ¸r lages hÃ¥ndteres -i NOARK5, slik at en bevarer all informasjon i eposten -(f.eks. Received-linjer). Finnes det en anbefalt XML-mapping ala den -som beskrives pÃ¥ -<URL: https://www.informit.com/articles/article.aspx?p=32074 >? Mitt -mÃ¥l er at det skal vÃ¦re mulig Ã¥ lagre eposten i en NOARK5-kjerne og -kunne fÃ¥ ut en identisk formattert kopi av opprinnelig epost ved -behov.
-

- -

Postmottaker hos SAMDOK mente spÃ¸rsmÃ¥let heller burde stilles -direkte til riksarkivet, og jeg fikk i dag svar derfra formulert av -seniorrÃ¥dgiver Geir Ivar Tungesvik:

- -

-
Riksarkivet har ingen anbefalinger nÃ¥r det gjelder konvertering fra -e-post til XML. Det stÃ¥r arkivskaper fritt Ã¥ eventuelt definere/bruke -eget format. Inklusive da - som det spÃ¸rres om - et format der det er -mulig Ã¥ re-etablere e-post format ut fra XML-en. XML (e-post) -dokumenter mÃ¥ vÃ¦re referert i arkivstrukturen, og det mÃ¥ vedlegges et -gyldig XML skjema (.xsd) for XML-filene. Arkivskaper stÃ¥r altsÃ¥ fritt -til Ã¥ gjÃ¸re hva de vil, bare det dokumenteres og det kan dannes et -utrekk ved avlevering til depot.
- -
De obligatoriske kravene i Noark 5 standarden mÃ¥ altsÃ¥ oppfylles - -etter dialog med Riksarkivet i forbindelse med godkjenning. For -offentlige arkiv er det sÃ¦rlig viktig med filene loependeJournal.xml -og offentligJournal.xml. Private arkiv som vil forholde seg til Noark -5 standarden er selvsagt frie til Ã¥ bruke det som er relevant for dem -av obligatoriske krav.
-

- -

Det ser dermed ut for meg som om det er et lite behov for Ã¥ -standardisere XML-lagring av RFC-822-formatterte meldinger. Noen som -vet om god spesifikasjon i sÃ¥ mÃ¥te? I tillegg til den omtalt over, -har jeg kommet over flere aktuelle beskrivelser (sÃ¸k pÃ¥ "rfc 822 -xml", sÃ¥ finner du aktuelle alternativer).

- -

XML MIME Transformation -protocol (XMTP) fra OpenHealth, sist oppdatert 2001.
An -XML format for mail and other messages utkast fra IETF datert -2001.
xMail: -E-mail as XML en artikkel fra 2003 som beskriver python-modulen -rfc822 som gir ut XML-representasjon av en RFC 822-formattert epost.

- -

Finnes det andre og bedre spesifikasjoner for slik lagring? Send -meg en epost hvis du har innspill.

- - - Tags: norsk, offentlig innsyn. - - -

@@ -798,7 +988,7 @@ meg en epost hvis du har innspill.

@@ -979,7 +1169,7 @@ meg en epost hvis du har innspill.

@@ -989,7 +1179,7 @@ meg en epost hvis du har innspill.

@@ -1027,7 +1217,7 @@ meg en epost hvis du har innspill.

@@ -1045,7 +1235,7 @@ meg en epost hvis du har innspill.

diff --git a/blog/index.rss b/blog/index.rss index 70f6ecb37a..c5fcc608cb 100644 --- a/blog/index.rss +++ b/blog/index.rss @@ -6,6 +6,288 @@ http://people.skolelinux.org/pere/blog/ + + s3ql, a locally mounted cloud file system - nice free software + http://people.skolelinux.org/pere/blog/s3ql__a_locally_mounted_cloud_file_system___nice_free_software.html + http://people.skolelinux.org/pere/blog/s3ql__a_locally_mounted_cloud_file_system___nice_free_software.html + Wed, 9 Apr 2014 11:30:00 +0200 + For a while now, I have been looking for a sensible offsite backup +solution for use at home. My requirements are simple, it must be +cheap and locally encrypted (in other words, I keep the encryption +keys, the storage provider do not have access to my private files). +One idea me and my friends have had many years ago, before the cloud +storage providers showed up, have been to use Google mail as storage, +writing a Linux block device storing blocks as emails in the mail +service provided by Google, and thus get heaps of free space. On top +of this one can add encryption, RAID and volume management to have +lots of (fairly slow, I admit that) cheap and encrypted storage. But +I never found time to implement such system. But the last few weeks I +have looked at a system called +<a href="https://bitbucket.org/nikratio/s3ql/">S3QL</a>, a locally +mounted network backed file system with the features I need. + +S3QL is a fuse file system with a local cache and cloud storage, +handling several different storage providers, any with Amazon S3, +Google Drive or OpenStack API. There are heaps of such storage +providers. S3QL can also use a local directory as storage, which +combined with sshfs allow for file storage on any ssh server. S3QL +include support for encryption, compression, de-duplication, snapshots +and immutable file systems, allowing me to mount the remote storage as +a local mount point, look at and use the files as if they were local, +while the content is stored in the cloud as well. This allow me to +have a backup that should survive fire. The file system can not be +shared between several machines at the same time, as only one can +mount it at the time, but any machine with the encryption key and +access to the storage service can mount it if it is unmounted. + +It is simple to use. I'm using it on Debian Wheezy, where the +package is included already. So to get started, run <tt>apt-get +install s3ql</tt>. Next, pick a storage provider. I ended up picking +Greenqloud, after reading their nice recipe on +<a href="https://greenqloud.zendesk.com/entries/44611757-How-To-Use-S3QL-to-mount-a-StorageQloud-bucket-on-Debian-Wheezy">how +to use s3ql with their Amazon S3 service</a>, because I trust the laws +in Iceland more than those in USA when it come to keeping my personal +data safe and private, and thus would rather spend money on a company +in Iceland. Another nice recipe is available from the article +<a href="http://www.admin-magazine.com/HPC/Articles/HPC-Cloud-Storage">S3QL +Filesystem for HPC Storage</a> by Jeff Layton in the HPC section of +Admin magazine. When the provider is picked, figure out how to get +the API key needed to connect to the storage API. With Greencloud, +the key did not show up until I had added payment details to my +account. + +Armed with the API access details, it is time to create the file +system. First, create a new bucket in the cloud. This bucket is the +file system storage area. I picked a bucket name reflecting the +machine that was going to store data there, but any name will do. +I'll refer to it as <tt>bucket-name</tt> below. In addition, one need +the API login and password, and a locally created password. Store it +all in ~root/.s3ql/authinfo2 like this: + +<blockquote><pre> +[s3c] +storage-url: s3c://s.greenqloud.com:443/bucket-name +backend-login: API-login +backend-password: API-password +fs-passphrase: local-password +</pre></blockquote> + +I create my local passphrase using <tt>pwget 50</tt> or similar, +but any sensible way to create a fairly random password should do it. +Armed with these details, it is now time to run mkfs, entering the API +details and password to create it: + +<blockquote><pre> +# mkdir -m 700 /var/lib/s3ql-cache +# mkfs.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \ + --ssl s3c://s.greenqloud.com:443/bucket-name +Enter backend login: +Enter backend password: +Before using S3QL, make sure to read the user's guide, especially +the 'Important Rules to Avoid Loosing Data' section. +Enter encryption password: +Confirm encryption password: +Generating random encryption key... +Creating metadata tables... +Dumping metadata... +..objects.. +..blocks.. +..inodes.. +..inode_blocks.. +..symlink_targets.. +..names.. +..contents.. +..ext_attributes.. +Compressing and uploading metadata... +Wrote 0.00 MB of compressed metadata. +# </pre></blockquote> + +The next step is mounting the file system to make the storage available. + +<blockquote><pre> +# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \ + --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql +Using 4 upload threads. +Downloading and decompressing metadata... +Reading metadata... +..objects.. +..blocks.. +..inodes.. +..inode_blocks.. +..symlink_targets.. +..names.. +..contents.. +..ext_attributes.. +Mounting filesystem... +# df -h /mnt +Filesystem Size Used Avail Use% Mounted on +s3c://s.greenqloud.com:443/bucket-name 1.0T 0 1.0T 0% /s3ql +# +</pre></blockquote> + +The file system is now ready for use. I use rsync to store my +backups in it, and as the metadata used by rsync is downloaded at +mount time, no network traffic (and storage cost) is triggered by +running rsync. To unmount, one should not use the normal umount +command, as this will not flush the cache to the cloud storage, but +instead running the umount.s3ql command like this: + +<blockquote><pre> +# umount.s3ql /s3ql +# +</pre></blockquote> + +There is a fsck command available to check the file system and +correct any problems detected. This can be used if the local server +crashes while the file system is mounted, to reset the "already +mounted" flag. This is what it look like when processing a working +file system: + +<blockquote><pre> +# fsck.s3ql --force --ssl s3c://s.greenqloud.com:443/bucket-name +Using cached metadata. +File system seems clean, checking anyway. +Checking DB integrity... +Creating temporary extra indices... +Checking lost+found... +Checking cached objects... +Checking names (refcounts)... +Checking contents (names)... +Checking contents (inodes)... +Checking contents (parent inodes)... +Checking objects (reference counts)... +Checking objects (backend)... +..processed 5000 objects so far.. +..processed 10000 objects so far.. +..processed 15000 objects so far.. +Checking objects (sizes)... +Checking blocks (referenced objects)... +Checking blocks (refcounts)... +Checking inode-block mapping (blocks)... +Checking inode-block mapping (inodes)... +Checking inodes (refcounts)... +Checking inodes (sizes)... +Checking extended attributes (names)... +Checking extended attributes (inodes)... +Checking symlinks (inodes)... +Checking directory reachability... +Checking unix conventions... +Checking referential integrity... +Dropping temporary indices... +Backing up old metadata... +Dumping metadata... +..objects.. +..blocks.. +..inodes.. +..inode_blocks.. +..symlink_targets.. +..names.. +..contents.. +..ext_attributes.. +Compressing and uploading metadata... +Wrote 0.89 MB of compressed metadata. +# +</pre></blockquote> + +Thanks to the cache, working on files that fit in the cache is very +quick, about the same speed as local file access. Uploading large +amount of data is to me limited by the bandwidth out of and into my +house. Uploading 685 MiB with a 100 MiB cache gave me 305 kiB/s, +which is very close to my upload speed, and downloading the same +Debian installation ISO gave me 610 kiB/s, close to my download speed. +Both were measured using <tt>dd</tt>. So for me, the bottleneck is my +network, not the file system code. I do not know what a good cache +size would be, but suspect that the cache should e larger than your +working set. + +I mentioned that only one machine can mount the file system at the +time. If another machine try, it is told that the file system is +busy: + +<blockquote><pre> +# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \ + --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql +Using 8 upload threads. +Backend reports that fs is still mounted elsewhere, aborting. +# +</pre></blockquote> + +The file content is uploaded when the cache is full, while the +metadata is uploaded once every 24 hour by default. To ensure the +file system content is flushed to the cloud, one can either umount the +file system, or ask s3ql to flush the cache and metadata using +s3qlctrl: + +<blockquote><pre> +# s3qlctrl upload-meta /s3ql +# s3qlctrl flushcache /s3ql +# +</pre></blockquote> + +If you are curious about how much space your data uses in the +cloud, and how much compression and deduplication cut down on the +storage usage, you can use s3qlstat on the mounted file system to get +a report: + +<blockquote><pre> +# s3qlstat /s3ql +Directory entries: 9141 +Inodes: 9143 +Data blocks: 8851 +Total data size: 22049.38 MB +After de-duplication: 21955.46 MB (99.57% of total) +After compression: 21877.28 MB (99.22% of total, 99.64% of de-duplicated) +Database size: 2.39 MB (uncompressed) +(some values do not take into account not-yet-uploaded dirty blocks in cache) +# +</pre></blockquote> + +I mentioned earlier that there are several possible suppliers of +storage. I did not try to locate them all, but am aware of at least +<a href="https://www.greenqloud.com/">Greenqloud</a>, +<a href="http://drive.google.com/">Google Drive</a>, +<a href="http://aws.amazon.com/s3/">Amazon S3 web serivces</a>, +<a href="http://www.rackspace.com/">Rackspace</a> and +<a href="http://crowncloud.net/">Crowncloud</A>. The latter even +accept payment in Bitcoin. Pick one that suit your need. Some of +them provide several GiB of free storage, but the prize models are +quire different and you will have to figure out what suit you +best. + +While researching this blog post, I had a look at research papers +and posters discussing the S3QL file system. There are several, which +told me that the file system is getting a critical check by the +science community and increased my confidence in using it. One nice +poster is titled +"<a href="http://www.lanl.gov/orgs/adtsc/publications/science_highlights_2013/docs/pg68_69.pdf">An +Innovative Parallel Cloud Storage System using OpenStackâs SwiftObject +Store and Transformative Parallel I/O Approach</a>" by Hsing-Bung +Chen, Benjamin McClelland, David Sherrill, Alfred Torrez, Parks Fields +and Pamela Smith. Please have a look. + +Given my problems with different file systems earlier, I decided to +check out the mounted S3QL file system to see if it would be usable as +a home directory (in other word, that it provided POSIX semantics when +it come to locking and umask handling etc). Running +<a href="http://people.skolelinux.org/pere/blog/Testing_if_a_file_system_can_be_used_for_home_directories___.html">my +test code to check file system semantics, I was happy to discover that +no error was found. So the file system can be used for home +directories, if one chooses to do so. + +If you do not want a locally file system, and want something that +work without the Linux fuse file system, I would like to mention the +<a href="http://www.tarsnap.com/">Tarsnap service</a>, which also +provide locally encrypted backup using a command line client. It have +a nicer access control system, where one can split out read and write +access, allowing some systems to write to the backup and others to +only read from it. + +As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +<a href="bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&label=PetterReinholdtsenBlog">15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b</a>. + + + EU-domstolen bekreftet i dag at datalagringsdirektivet er ulovlig http://people.skolelinux.org/pere/blog/EU_domstolen_bekreftet_i_dag_at_datalagringsdirektivet_er_ulovlig.html @@ -616,97 +898,5 @@ workstation, LTSP client or LTSP server. - - Hvordan bÃ¸r RFC 822-formattert epost lagres i en NOARK5-database? - http://people.skolelinux.org/pere/blog/Hvordan_b_r_RFC_822_formattert_epost_lagres_i_en_NOARK5_database_.html - http://people.skolelinux.org/pere/blog/Hvordan_b_r_RFC_822_formattert_epost_lagres_i_en_NOARK5_database_.html - Fri, 7 Mar 2014 15:20:00 +0100 - For noen uker siden ble NXCs fri programvarelisenserte -NOARK5-lÃ¸sning -<a href="http://www.nuug.no/aktiviteter/20140211-noark/">presentert hos -NUUG</a> (video -<a href="https://www.youtube.com/watch?v=JCb_dNS3MHQ">pÃ¥ youtube -forelÃ¸big</a>), og det fikk meg til Ã¥ titte litt mer pÃ¥ NOARK5, -standarden for arkivhÃ¥ndtering i det offentlige Norge. Jeg lurer pÃ¥ -om denne kjernen kan vÃ¦re nyttig i et par av mine prosjekter, og for ett -av dem er det mest aktuelt Ã¥ lagre epost. Jeg klarte ikke finne noen -anbefaling om hvordan RFC 822-formattert epost (aka Internett-epost) -burde lagres i NOARK5, selv om jeg vet at noen arkiver tar -PDF-utskrift av eposten med sitt epostprogram og sÃ¥ arkiverer PDF-en -(eller enda vÃ¦rre, tar papirutskrift og lagrer bildet av eposten som -PDF i arkivet). - -Det er ikke sÃ¥ mange formater som er akseptert av riksarkivet til -langtidsoppbevaring av offentlige arkiver, og PDF og XML er de mest -aktuelle i sÃ¥ mÃ¥te. Det slo meg at det mÃ¥tte da finnes en eller annen -egnet XML-representasjon og at det kanskje var enighet om hvilken som -burde brukes, sÃ¥ jeg tok mot til meg og spurte -<a href="http://samdok.com/">SAMDOK</a>, en gruppe tilknyttet -arkivverket som ser ut til Ã¥ jobbe med NOARK-samhandling, om de hadde -noen anbefalinger: - -<blockquote> -Hei. - -Usikker pÃ¥ om dette er riktig forum Ã¥ ta opp mitt spÃ¸rsmÃ¥l, men jeg -lurer pÃ¥ om det er definert en anbefaling om hvordan RFC -822-formatterte epost (aka vanlig Internet-epost) bÃ¸r lages hÃ¥ndteres -i NOARK5, slik at en bevarer all informasjon i eposten -(f.eks. Received-linjer). Finnes det en anbefalt XML-mapping ala den -som beskrives pÃ¥ -<URL: <a href="https://www.informit.com/articles/article.aspx?p=32074">https://www.informit.com/articles/article.aspx?p=32074</a> >? Mitt -mÃ¥l er at det skal vÃ¦re mulig Ã¥ lagre eposten i en NOARK5-kjerne og -kunne fÃ¥ ut en identisk formattert kopi av opprinnelig epost ved -behov. -</blockquote> - -Postmottaker hos SAMDOK mente spÃ¸rsmÃ¥let heller burde stilles -direkte til riksarkivet, og jeg fikk i dag svar derfra formulert av -seniorrÃ¥dgiver Geir Ivar Tungesvik: - -<blockquote> -Riksarkivet har ingen anbefalinger nÃ¥r det gjelder konvertering fra -e-post til XML. Det stÃ¥r arkivskaper fritt Ã¥ eventuelt definere/bruke -eget format. Inklusive da - som det spÃ¸rres om - et format der det er -mulig Ã¥ re-etablere e-post format ut fra XML-en. XML (e-post) -dokumenter mÃ¥ vÃ¦re referert i arkivstrukturen, og det mÃ¥ vedlegges et -gyldig XML skjema (.xsd) for XML-filene. Arkivskaper stÃ¥r altsÃ¥ fritt -til Ã¥ gjÃ¸re hva de vil, bare det dokumenteres og det kan dannes et -utrekk ved avlevering til depot. - -De obligatoriske kravene i Noark 5 standarden mÃ¥ altsÃ¥ oppfylles - -etter dialog med Riksarkivet i forbindelse med godkjenning. For -offentlige arkiv er det sÃ¦rlig viktig med filene loependeJournal.xml -og offentligJournal.xml. Private arkiv som vil forholde seg til Noark -5 standarden er selvsagt frie til Ã¥ bruke det som er relevant for dem -av obligatoriske krav. -</blockquote> - -Det ser dermed ut for meg som om det er et lite behov for Ã¥ -standardisere XML-lagring av RFC-822-formatterte meldinger. Noen som -vet om god spesifikasjon i sÃ¥ mÃ¥te? I tillegg til den omtalt over, -har jeg kommet over flere aktuelle beskrivelser (sÃ¸k pÃ¥ "rfc 822 -xml", sÃ¥ finner du aktuelle alternativer). - -<ul> - -<li><a href="http://www.openhealth.org/xmtp/">XML MIME Transformation -protocol (XMTP)</a> fra OpenHealth, sist oppdatert 2001.</li> - -<li><a href="https://tools.ietf.org/html/draft-klyne-message-rfc822-xml-03">An -XML format for mail and other messages</a> utkast fra IETF datert -2001.</li> - -<li><a href="http://www.informit.com/articles/article.aspx?p=32074">xMail: -E-mail as XML</a> en artikkel fra 2003 som beskriver python-modulen -rfc822 som gir ut XML-representasjon av en RFC 822-formattert epost.</li> - -</ul> - -Finnes det andre og bedre spesifikasjoner for slik lagring? Send -meg en epost hvis du har innspill. - - - diff --git a/blog/jXplorer__a_very_nice_LDAP_GUI.html b/blog/jXplorer__a_very_nice_LDAP_GUI.html index b771fa71cb..fd727bd1fc 100644 --- a/blog/jXplorer__a_very_nice_LDAP_GUI.html +++ b/blog/jXplorer__a_very_nice_LDAP_GUI.html @@ -62,7 +62,7 @@ and remove the failing query. Nothing big, but very annoying.

@@ -243,7 +243,7 @@ and remove the failing query. Nothing big, but very annoying.

@@ -253,7 +253,7 @@ and remove the failing query. Nothing big, but very annoying.

@@ -291,7 +291,7 @@ and remove the failing query. Nothing big, but very annoying.

@@ -309,7 +309,7 @@ and remove the failing query. Nothing big, but very annoying.

diff --git a/blog/s3ql__a_locally_mounted_cloud_file_system___nice_free_software.html b/blog/s3ql__a_locally_mounted_cloud_file_system___nice_free_software.html new file mode 100644 index 0000000000..4c23297e02 --- /dev/null +++ b/blog/s3ql__a_locally_mounted_cloud_file_system___nice_free_software.html @@ -0,0 +1,605 @@ + + + + + Petter Reinholdtsen: s3ql, a locally mounted cloud file system - nice free software + + + + + + +

+ Petter Reinholdtsen + +

+ +

+ + +

s3ql, a locally mounted cloud file system - nice free software

9th April 2014

+ +

+[s3c]
+storage-url: s3c://s.greenqloud.com:443/bucket-name
+backend-login: API-login
+backend-password: API-password
+fs-passphrase: local-password
+

+ +

+# mkdir -m 700 /var/lib/s3ql-cache
+# mkfs.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl s3c://s.greenqloud.com:443/bucket-name
+Enter backend login: 
+Enter backend password: 
+Before using S3QL, make sure to read the user's guide, especially
+the 'Important Rules to Avoid Loosing Data' section.
+Enter encryption password: 
+Confirm encryption password: 
+Generating random encryption key...
+Creating metadata tables...
+Dumping metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Compressing and uploading metadata...
+Wrote 0.00 MB of compressed metadata.
+#

+ +

The next step is mounting the file system to make the storage available. + +

+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+Using 4 upload threads.
+Downloading and decompressing metadata...
+Reading metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Mounting filesystem...
+# df -h /mnt
+Filesystem                              Size  Used Avail Use% Mounted on
+s3c://s.greenqloud.com:443/bucket-name  1.0T     0  1.0T   0% /s3ql
+#
+

+ +

+# umount.s3ql /s3ql
+# 
+

+ +

+# fsck.s3ql --force --ssl s3c://s.greenqloud.com:443/bucket-name
+Using cached metadata.
+File system seems clean, checking anyway.
+Checking DB integrity...
+Creating temporary extra indices...
+Checking lost+found...
+Checking cached objects...
+Checking names (refcounts)...
+Checking contents (names)...
+Checking contents (inodes)...
+Checking contents (parent inodes)...
+Checking objects (reference counts)...
+Checking objects (backend)...
+..processed 5000 objects so far..
+..processed 10000 objects so far..
+..processed 15000 objects so far..
+Checking objects (sizes)...
+Checking blocks (referenced objects)...
+Checking blocks (refcounts)...
+Checking inode-block mapping (blocks)...
+Checking inode-block mapping (inodes)...
+Checking inodes (refcounts)...
+Checking inodes (sizes)...
+Checking extended attributes (names)...
+Checking extended attributes (inodes)...
+Checking symlinks (inodes)...
+Checking directory reachability...
+Checking unix conventions...
+Checking referential integrity...
+Dropping temporary indices...
+Backing up old metadata...
+Dumping metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Compressing and uploading metadata...
+Wrote 0.89 MB of compressed metadata.
+# 
+

+ +

I mentioned that only one machine can mount the file system at the +time. If another machine try, it is told that the file system is +busy:

+ +

+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+Using 8 upload threads.
+Backend reports that fs is still mounted elsewhere, aborting.
+#
+

+ +

+# s3qlctrl upload-meta /s3ql
+# s3qlctrl flushcache /s3ql
+# 
+

+ +

+# s3qlstat /s3ql
+Directory entries:    9141
+Inodes:               9143
+Data blocks:          8851
+Total data size:      22049.38 MB
+After de-duplication: 21955.46 MB (99.57% of total)
+After compression:    21877.28 MB (99.22% of total, 99.64% of de-duplicated)
+Database size:        2.39 MB (uncompressed)
+(some values do not take into account not-yet-uploaded dirty blocks in cache)
+#
+

+ +

As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

+ +

Tags: debian, english, personvern, sikkerhet.

+ + +

+ + + + + +

+ Created by Chronicle v4.6 +

+ + + diff --git a/blog/sitemap.xml b/blog/sitemap.xml index 6c5c48627b..0a301ff2ec 100644 --- a/blog/sitemap.xml +++ b/blog/sitemap.xml @@ -1815,6 +1815,11 @@ 0.50 weekly + + http://people.skolelinux.org/pere/blog/s3ql__a_locally_mounted_cloud_file_system___nice_free_software.html + 0.50 + weekly + http://people.skolelinux.org/pere/blog/Sandy_Island____ya_som_er_synlig_hver_tirsdag_og_fredag_.html 0.50 diff --git a/blog/systemd__an_interesting_alternative_to_upstart.html b/blog/systemd__an_interesting_alternative_to_upstart.html index b79ad60917..1c394866a0 100644 --- a/blog/systemd__an_interesting_alternative_to_upstart.html +++ b/blog/systemd__an_interesting_alternative_to_upstart.html @@ -80,7 +80,7 @@ with parallel booting enabled by default.

@@ -261,7 +261,7 @@ with parallel booting enabled by default.

@@ -271,7 +271,7 @@ with parallel booting enabled by default.

@@ -309,7 +309,7 @@ with parallel booting enabled by default.

@@ -327,7 +327,7 @@ with parallel booting enabled by default.

diff --git a/blog/tags/3d-printer/index.html b/blog/tags/3d-printer/index.html index 9f6d84d9cc..c4a4526683 100644 --- a/blog/tags/3d-printer/index.html +++ b/blog/tags/3d-printer/index.html @@ -612,7 +612,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

@@ -793,7 +793,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

@@ -803,7 +803,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

@@ -841,7 +841,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

@@ -859,7 +859,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

diff --git a/blog/tags/amiga/index.html b/blog/tags/amiga/index.html index ba1ae91067..c85bb337b4 100644 --- a/blog/tags/amiga/index.html +++ b/blog/tags/amiga/index.html @@ -66,7 +66,7 @@ pakke. Kanskje Aros kunne vÃ¦rt interessant for et NUUG-foredrag?

@@ -247,7 +247,7 @@ pakke. Kanskje Aros kunne vÃ¦rt interessant for et NUUG-foredrag?

@@ -257,7 +257,7 @@ pakke. Kanskje Aros kunne vÃ¦rt interessant for et NUUG-foredrag?

@@ -295,7 +295,7 @@ pakke. Kanskje Aros kunne vÃ¦rt interessant for et NUUG-foredrag?

@@ -313,7 +313,7 @@ pakke. Kanskje Aros kunne vÃ¦rt interessant for et NUUG-foredrag?

diff --git a/blog/tags/aros/index.html b/blog/tags/aros/index.html index b7c340e5b9..d5e8c2a81a 100644 --- a/blog/tags/aros/index.html +++ b/blog/tags/aros/index.html @@ -66,7 +66,7 @@ pakke. Kanskje Aros kunne vÃ¦rt interessant for et NUUG-foredrag?

@@ -247,7 +247,7 @@ pakke. Kanskje Aros kunne vÃ¦rt interessant for et NUUG-foredrag?

@@ -257,7 +257,7 @@ pakke. Kanskje Aros kunne vÃ¦rt interessant for et NUUG-foredrag?

@@ -295,7 +295,7 @@ pakke. Kanskje Aros kunne vÃ¦rt interessant for et NUUG-foredrag?

@@ -313,7 +313,7 @@ pakke. Kanskje Aros kunne vÃ¦rt interessant for et NUUG-foredrag?

diff --git a/blog/tags/bankid/index.html b/blog/tags/bankid/index.html index e0acffc4ec..f819289001 100644 --- a/blog/tags/bankid/index.html +++ b/blog/tags/bankid/index.html @@ -472,7 +472,7 @@ til BankID-modellen.

@@ -653,7 +653,7 @@ til BankID-modellen.

@@ -663,7 +663,7 @@ til BankID-modellen.

@@ -701,7 +701,7 @@ til BankID-modellen.

@@ -719,7 +719,7 @@ til BankID-modellen.

diff --git a/blog/tags/bitcoin/index.html b/blog/tags/bitcoin/index.html index aa398423c4..0ec368f9b7 100644 --- a/blog/tags/bitcoin/index.html +++ b/blog/tags/bitcoin/index.html @@ -593,7 +593,7 @@ donations to the address

@@ -774,7 +774,7 @@ donations to the address

@@ -784,7 +784,7 @@ donations to the address

@@ -822,7 +822,7 @@ donations to the address

@@ -840,7 +840,7 @@ donations to the address

diff --git a/blog/tags/bootsystem/index.html b/blog/tags/bootsystem/index.html index 85b5eb2534..1977fbbe4e 100644 --- a/blog/tags/bootsystem/index.html +++ b/blog/tags/bootsystem/index.html @@ -1104,7 +1104,7 @@ insserv'. Will need to test if that work. :)

@@ -1285,7 +1285,7 @@ insserv'. Will need to test if that work. :)

@@ -1295,7 +1295,7 @@ insserv'. Will need to test if that work. :)

@@ -1333,7 +1333,7 @@ insserv'. Will need to test if that work. :)

@@ -1351,7 +1351,7 @@ insserv'. Will need to test if that work. :)

diff --git a/blog/tags/bsa/index.html b/blog/tags/bsa/index.html index 923e2f69ad..36564939b9 100644 --- a/blog/tags/bsa/index.html +++ b/blog/tags/bsa/index.html @@ -154,7 +154,7 @@ pÃ¥ Slashdot.

@@ -335,7 +335,7 @@ pÃ¥ Slashdot.

@@ -345,7 +345,7 @@ pÃ¥ Slashdot.

@@ -383,7 +383,7 @@ pÃ¥ Slashdot.

@@ -401,7 +401,7 @@ pÃ¥ Slashdot.

diff --git a/blog/tags/chrpath/index.html b/blog/tags/chrpath/index.html index 47fb0efaa0..369e84460a 100644 --- a/blog/tags/chrpath/index.html +++ b/blog/tags/chrpath/index.html @@ -161,7 +161,7 @@ include a testsuite check.

@@ -342,7 +342,7 @@ include a testsuite check.

@@ -352,7 +352,7 @@ include a testsuite check.

@@ -390,7 +390,7 @@ include a testsuite check.

@@ -408,7 +408,7 @@ include a testsuite check.

diff --git a/blog/tags/debian edu/index.html b/blog/tags/debian edu/index.html index 1f37a6d81f..8ed3f48e64 100644 --- a/blog/tags/debian edu/index.html +++ b/blog/tags/debian edu/index.html @@ -13466,7 +13466,7 @@ be the only one fitting our needs. :/

@@ -13647,7 +13647,7 @@ be the only one fitting our needs. :/

@@ -13657,7 +13657,7 @@ be the only one fitting our needs. :/

@@ -13695,7 +13695,7 @@ be the only one fitting our needs. :/

@@ -13713,7 +13713,7 @@ be the only one fitting our needs. :/

diff --git a/blog/tags/debian/debian.rss b/blog/tags/debian/debian.rss index fbdefa8740..d4a03d401b 100644 --- a/blog/tags/debian/debian.rss +++ b/blog/tags/debian/debian.rss @@ -6,6 +6,288 @@ http://people.skolelinux.org/pere/blog/ + + s3ql, a locally mounted cloud file system - nice free software + http://people.skolelinux.org/pere/blog/s3ql__a_locally_mounted_cloud_file_system___nice_free_software.html + http://people.skolelinux.org/pere/blog/s3ql__a_locally_mounted_cloud_file_system___nice_free_software.html + Wed, 9 Apr 2014 11:30:00 +0200 + For a while now, I have been looking for a sensible offsite backup +solution for use at home. My requirements are simple, it must be +cheap and locally encrypted (in other words, I keep the encryption +keys, the storage provider do not have access to my private files). +One idea me and my friends have had many years ago, before the cloud +storage providers showed up, have been to use Google mail as storage, +writing a Linux block device storing blocks as emails in the mail +service provided by Google, and thus get heaps of free space. On top +of this one can add encryption, RAID and volume management to have +lots of (fairly slow, I admit that) cheap and encrypted storage. But +I never found time to implement such system. But the last few weeks I +have looked at a system called +<a href="https://bitbucket.org/nikratio/s3ql/">S3QL</a>, a locally +mounted network backed file system with the features I need. + +S3QL is a fuse file system with a local cache and cloud storage, +handling several different storage providers, any with Amazon S3, +Google Drive or OpenStack API. There are heaps of such storage +providers. S3QL can also use a local directory as storage, which +combined with sshfs allow for file storage on any ssh server. S3QL +include support for encryption, compression, de-duplication, snapshots +and immutable file systems, allowing me to mount the remote storage as +a local mount point, look at and use the files as if they were local, +while the content is stored in the cloud as well. This allow me to +have a backup that should survive fire. The file system can not be +shared between several machines at the same time, as only one can +mount it at the time, but any machine with the encryption key and +access to the storage service can mount it if it is unmounted. + +It is simple to use. I'm using it on Debian Wheezy, where the +package is included already. So to get started, run <tt>apt-get +install s3ql</tt>. Next, pick a storage provider. I ended up picking +Greenqloud, after reading their nice recipe on +<a href="https://greenqloud.zendesk.com/entries/44611757-How-To-Use-S3QL-to-mount-a-StorageQloud-bucket-on-Debian-Wheezy">how +to use s3ql with their Amazon S3 service</a>, because I trust the laws +in Iceland more than those in USA when it come to keeping my personal +data safe and private, and thus would rather spend money on a company +in Iceland. Another nice recipe is available from the article +<a href="http://www.admin-magazine.com/HPC/Articles/HPC-Cloud-Storage">S3QL +Filesystem for HPC Storage</a> by Jeff Layton in the HPC section of +Admin magazine. When the provider is picked, figure out how to get +the API key needed to connect to the storage API. With Greencloud, +the key did not show up until I had added payment details to my +account. + +Armed with the API access details, it is time to create the file +system. First, create a new bucket in the cloud. This bucket is the +file system storage area. I picked a bucket name reflecting the +machine that was going to store data there, but any name will do. +I'll refer to it as <tt>bucket-name</tt> below. In addition, one need +the API login and password, and a locally created password. Store it +all in ~root/.s3ql/authinfo2 like this: + +<blockquote><pre> +[s3c] +storage-url: s3c://s.greenqloud.com:443/bucket-name +backend-login: API-login +backend-password: API-password +fs-passphrase: local-password +</pre></blockquote> + +I create my local passphrase using <tt>pwget 50</tt> or similar, +but any sensible way to create a fairly random password should do it. +Armed with these details, it is now time to run mkfs, entering the API +details and password to create it: + +<blockquote><pre> +# mkdir -m 700 /var/lib/s3ql-cache +# mkfs.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \ + --ssl s3c://s.greenqloud.com:443/bucket-name +Enter backend login: +Enter backend password: +Before using S3QL, make sure to read the user's guide, especially +the 'Important Rules to Avoid Loosing Data' section. +Enter encryption password: +Confirm encryption password: +Generating random encryption key... +Creating metadata tables... +Dumping metadata... +..objects.. +..blocks.. +..inodes.. +..inode_blocks.. +..symlink_targets.. +..names.. +..contents.. +..ext_attributes.. +Compressing and uploading metadata... +Wrote 0.00 MB of compressed metadata. +# </pre></blockquote> + +The next step is mounting the file system to make the storage available. + +<blockquote><pre> +# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \ + --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql +Using 4 upload threads. +Downloading and decompressing metadata... +Reading metadata... +..objects.. +..blocks.. +..inodes.. +..inode_blocks.. +..symlink_targets.. +..names.. +..contents.. +..ext_attributes.. +Mounting filesystem... +# df -h /mnt +Filesystem Size Used Avail Use% Mounted on +s3c://s.greenqloud.com:443/bucket-name 1.0T 0 1.0T 0% /s3ql +# +</pre></blockquote> + +The file system is now ready for use. I use rsync to store my +backups in it, and as the metadata used by rsync is downloaded at +mount time, no network traffic (and storage cost) is triggered by +running rsync. To unmount, one should not use the normal umount +command, as this will not flush the cache to the cloud storage, but +instead running the umount.s3ql command like this: + +<blockquote><pre> +# umount.s3ql /s3ql +# +</pre></blockquote> + +There is a fsck command available to check the file system and +correct any problems detected. This can be used if the local server +crashes while the file system is mounted, to reset the "already +mounted" flag. This is what it look like when processing a working +file system: + +<blockquote><pre> +# fsck.s3ql --force --ssl s3c://s.greenqloud.com:443/bucket-name +Using cached metadata. +File system seems clean, checking anyway. +Checking DB integrity... +Creating temporary extra indices... +Checking lost+found... +Checking cached objects... +Checking names (refcounts)... +Checking contents (names)... +Checking contents (inodes)... +Checking contents (parent inodes)... +Checking objects (reference counts)... +Checking objects (backend)... +..processed 5000 objects so far.. +..processed 10000 objects so far.. +..processed 15000 objects so far.. +Checking objects (sizes)... +Checking blocks (referenced objects)... +Checking blocks (refcounts)... +Checking inode-block mapping (blocks)... +Checking inode-block mapping (inodes)... +Checking inodes (refcounts)... +Checking inodes (sizes)... +Checking extended attributes (names)... +Checking extended attributes (inodes)... +Checking symlinks (inodes)... +Checking directory reachability... +Checking unix conventions... +Checking referential integrity... +Dropping temporary indices... +Backing up old metadata... +Dumping metadata... +..objects.. +..blocks.. +..inodes.. +..inode_blocks.. +..symlink_targets.. +..names.. +..contents.. +..ext_attributes.. +Compressing and uploading metadata... +Wrote 0.89 MB of compressed metadata. +# +</pre></blockquote> + +Thanks to the cache, working on files that fit in the cache is very +quick, about the same speed as local file access. Uploading large +amount of data is to me limited by the bandwidth out of and into my +house. Uploading 685 MiB with a 100 MiB cache gave me 305 kiB/s, +which is very close to my upload speed, and downloading the same +Debian installation ISO gave me 610 kiB/s, close to my download speed. +Both were measured using <tt>dd</tt>. So for me, the bottleneck is my +network, not the file system code. I do not know what a good cache +size would be, but suspect that the cache should e larger than your +working set. + +I mentioned that only one machine can mount the file system at the +time. If another machine try, it is told that the file system is +busy: + +<blockquote><pre> +# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \ + --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql +Using 8 upload threads. +Backend reports that fs is still mounted elsewhere, aborting. +# +</pre></blockquote> + +The file content is uploaded when the cache is full, while the +metadata is uploaded once every 24 hour by default. To ensure the +file system content is flushed to the cloud, one can either umount the +file system, or ask s3ql to flush the cache and metadata using +s3qlctrl: + +<blockquote><pre> +# s3qlctrl upload-meta /s3ql +# s3qlctrl flushcache /s3ql +# +</pre></blockquote> + +If you are curious about how much space your data uses in the +cloud, and how much compression and deduplication cut down on the +storage usage, you can use s3qlstat on the mounted file system to get +a report: + +<blockquote><pre> +# s3qlstat /s3ql +Directory entries: 9141 +Inodes: 9143 +Data blocks: 8851 +Total data size: 22049.38 MB +After de-duplication: 21955.46 MB (99.57% of total) +After compression: 21877.28 MB (99.22% of total, 99.64% of de-duplicated) +Database size: 2.39 MB (uncompressed) +(some values do not take into account not-yet-uploaded dirty blocks in cache) +# +</pre></blockquote> + +I mentioned earlier that there are several possible suppliers of +storage. I did not try to locate them all, but am aware of at least +<a href="https://www.greenqloud.com/">Greenqloud</a>, +<a href="http://drive.google.com/">Google Drive</a>, +<a href="http://aws.amazon.com/s3/">Amazon S3 web serivces</a>, +<a href="http://www.rackspace.com/">Rackspace</a> and +<a href="http://crowncloud.net/">Crowncloud</A>. The latter even +accept payment in Bitcoin. Pick one that suit your need. Some of +them provide several GiB of free storage, but the prize models are +quire different and you will have to figure out what suit you +best. + +While researching this blog post, I had a look at research papers +and posters discussing the S3QL file system. There are several, which +told me that the file system is getting a critical check by the +science community and increased my confidence in using it. One nice +poster is titled +"<a href="http://www.lanl.gov/orgs/adtsc/publications/science_highlights_2013/docs/pg68_69.pdf">An +Innovative Parallel Cloud Storage System using OpenStackâs SwiftObject +Store and Transformative Parallel I/O Approach</a>" by Hsing-Bung +Chen, Benjamin McClelland, David Sherrill, Alfred Torrez, Parks Fields +and Pamela Smith. Please have a look. + +Given my problems with different file systems earlier, I decided to +check out the mounted S3QL file system to see if it would be usable as +a home directory (in other word, that it provided POSIX semantics when +it come to locking and umask handling etc). Running +<a href="http://people.skolelinux.org/pere/blog/Testing_if_a_file_system_can_be_used_for_home_directories___.html">my +test code to check file system semantics, I was happy to discover that +no error was found. So the file system can be used for home +directories, if one chooses to do so. + +If you do not want a locally file system, and want something that +work without the Linux fuse file system, I would like to mention the +<a href="http://www.tarsnap.com/">Tarsnap service</a>, which also +provide locally encrypted backup using a command line client. It have +a nicer access control system, where one can split out read and write +access, allowing some systems to write to the backup and others to +only read from it. + +As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +<a href="bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&label=PetterReinholdtsenBlog">15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b</a>. + + + Freedombox on Dreamplug, Raspberry Pi and virtual x86 machine http://people.skolelinux.org/pere/blog/Freedombox_on_Dreamplug__Raspberry_Pi_and_virtual_x86_machine.html diff --git a/blog/tags/debian/index.html b/blog/tags/debian/index.html index 6a5c8ecb11..608ecddb2e 100644 --- a/blog/tags/debian/index.html +++ b/blog/tags/debian/index.html @@ -20,6 +20,300 @@

Entries tagged "debian".

+ s3ql, a locally mounted cloud file system - nice free software +

+ 9th April 2014 +

+ +

+[s3c]
+storage-url: s3c://s.greenqloud.com:443/bucket-name
+backend-login: API-login
+backend-password: API-password
+fs-passphrase: local-password
+

+ +

+# mkdir -m 700 /var/lib/s3ql-cache
+# mkfs.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl s3c://s.greenqloud.com:443/bucket-name
+Enter backend login: 
+Enter backend password: 
+Before using S3QL, make sure to read the user's guide, especially
+the 'Important Rules to Avoid Loosing Data' section.
+Enter encryption password: 
+Confirm encryption password: 
+Generating random encryption key...
+Creating metadata tables...
+Dumping metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Compressing and uploading metadata...
+Wrote 0.00 MB of compressed metadata.
+#

+ +

The next step is mounting the file system to make the storage available. + +

+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+Using 4 upload threads.
+Downloading and decompressing metadata...
+Reading metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Mounting filesystem...
+# df -h /mnt
+Filesystem                              Size  Used Avail Use% Mounted on
+s3c://s.greenqloud.com:443/bucket-name  1.0T     0  1.0T   0% /s3ql
+#
+

+ +

+# umount.s3ql /s3ql
+# 
+

+ +

+# fsck.s3ql --force --ssl s3c://s.greenqloud.com:443/bucket-name
+Using cached metadata.
+File system seems clean, checking anyway.
+Checking DB integrity...
+Creating temporary extra indices...
+Checking lost+found...
+Checking cached objects...
+Checking names (refcounts)...
+Checking contents (names)...
+Checking contents (inodes)...
+Checking contents (parent inodes)...
+Checking objects (reference counts)...
+Checking objects (backend)...
+..processed 5000 objects so far..
+..processed 10000 objects so far..
+..processed 15000 objects so far..
+Checking objects (sizes)...
+Checking blocks (referenced objects)...
+Checking blocks (refcounts)...
+Checking inode-block mapping (blocks)...
+Checking inode-block mapping (inodes)...
+Checking inodes (refcounts)...
+Checking inodes (sizes)...
+Checking extended attributes (names)...
+Checking extended attributes (inodes)...
+Checking symlinks (inodes)...
+Checking directory reachability...
+Checking unix conventions...
+Checking referential integrity...
+Dropping temporary indices...
+Backing up old metadata...
+Dumping metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Compressing and uploading metadata...
+Wrote 0.89 MB of compressed metadata.
+# 
+

+ +

I mentioned that only one machine can mount the file system at the +time. If another machine try, it is told that the file system is +busy:

+ +

+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+Using 8 upload threads.
+Backend reports that fs is still mounted elsewhere, aborting.
+#
+

+ +

+# s3qlctrl upload-meta /s3ql
+# s3qlctrl flushcache /s3ql
+# 
+

+ +

+# s3qlstat /s3ql
+Directory entries:    9141
+Inodes:               9143
+Data blocks:          8851
+Total data size:      22049.38 MB
+After de-duplication: 21955.46 MB (99.57% of total)
+After compression:    21877.28 MB (99.22% of total, 99.64% of de-duplicated)
+Database size:        2.39 MB (uncompressed)
+(some values do not take into account not-yet-uploaded dirty blocks in cache)
+#
+

+ +

As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

+ +

+ + + Tags: debian, english, personvern, sikkerhet. + + +

Freedombox on Dreamplug, Raspberry Pi and virtual x86 machine @@ -7502,7 +7796,7 @@ be the only one fitting our needs. :/

@@ -7683,7 +7977,7 @@ be the only one fitting our needs. :/

@@ -7693,7 +7987,7 @@ be the only one fitting our needs. :/

@@ -7731,7 +8025,7 @@ be the only one fitting our needs. :/

@@ -7749,7 +8043,7 @@ be the only one fitting our needs. :/

diff --git a/blog/tags/digistan/index.html b/blog/tags/digistan/index.html index a4f4125a70..b7a7f5d846 100644 --- a/blog/tags/digistan/index.html +++ b/blog/tags/digistan/index.html @@ -1253,7 +1253,7 @@ produkter basert pÃ¥ standarden.

@@ -1434,7 +1434,7 @@ produkter basert pÃ¥ standarden.

@@ -1444,7 +1444,7 @@ produkter basert pÃ¥ standarden.

@@ -1482,7 +1482,7 @@ produkter basert pÃ¥ standarden.

@@ -1500,7 +1500,7 @@ produkter basert pÃ¥ standarden.

diff --git a/blog/tags/docbook/index.html b/blog/tags/docbook/index.html index 3406651fd5..d3a3a4500d 100644 --- a/blog/tags/docbook/index.html +++ b/blog/tags/docbook/index.html @@ -778,7 +778,7 @@ slik at du kan oppdatere direkte.

@@ -959,7 +959,7 @@ slik at du kan oppdatere direkte.

@@ -969,7 +969,7 @@ slik at du kan oppdatere direkte.

@@ -1007,7 +1007,7 @@ slik at du kan oppdatere direkte.

@@ -1025,7 +1025,7 @@ slik at du kan oppdatere direkte.

diff --git a/blog/tags/drivstoffpriser/index.html b/blog/tags/drivstoffpriser/index.html index 653a2b0232..33b8cbe90b 100644 --- a/blog/tags/drivstoffpriser/index.html +++ b/blog/tags/drivstoffpriser/index.html @@ -495,7 +495,7 @@ en liste med stasjoner pÃ¥ samme format som PriserVedStasjoner.

@@ -676,7 +676,7 @@ en liste med stasjoner pÃ¥ samme format som PriserVedStasjoner.

@@ -686,7 +686,7 @@ en liste med stasjoner pÃ¥ samme format som PriserVedStasjoner.

@@ -724,7 +724,7 @@ en liste med stasjoner pÃ¥ samme format som PriserVedStasjoner.

@@ -742,7 +742,7 @@ en liste med stasjoner pÃ¥ samme format som PriserVedStasjoner.

diff --git a/blog/tags/english/english.rss b/blog/tags/english/english.rss index 82c2e22ccd..0de36d3283 100644 --- a/blog/tags/english/english.rss +++ b/blog/tags/english/english.rss @@ -6,6 +6,288 @@ http://people.skolelinux.org/pere/blog/ + + s3ql, a locally mounted cloud file system - nice free software + http://people.skolelinux.org/pere/blog/s3ql__a_locally_mounted_cloud_file_system___nice_free_software.html + http://people.skolelinux.org/pere/blog/s3ql__a_locally_mounted_cloud_file_system___nice_free_software.html + Wed, 9 Apr 2014 11:30:00 +0200 + For a while now, I have been looking for a sensible offsite backup +solution for use at home. My requirements are simple, it must be +cheap and locally encrypted (in other words, I keep the encryption +keys, the storage provider do not have access to my private files). +One idea me and my friends have had many years ago, before the cloud +storage providers showed up, have been to use Google mail as storage, +writing a Linux block device storing blocks as emails in the mail +service provided by Google, and thus get heaps of free space. On top +of this one can add encryption, RAID and volume management to have +lots of (fairly slow, I admit that) cheap and encrypted storage. But +I never found time to implement such system. But the last few weeks I +have looked at a system called +<a href="https://bitbucket.org/nikratio/s3ql/">S3QL</a>, a locally +mounted network backed file system with the features I need. + +S3QL is a fuse file system with a local cache and cloud storage, +handling several different storage providers, any with Amazon S3, +Google Drive or OpenStack API. There are heaps of such storage +providers. S3QL can also use a local directory as storage, which +combined with sshfs allow for file storage on any ssh server. S3QL +include support for encryption, compression, de-duplication, snapshots +and immutable file systems, allowing me to mount the remote storage as +a local mount point, look at and use the files as if they were local, +while the content is stored in the cloud as well. This allow me to +have a backup that should survive fire. The file system can not be +shared between several machines at the same time, as only one can +mount it at the time, but any machine with the encryption key and +access to the storage service can mount it if it is unmounted. + +It is simple to use. I'm using it on Debian Wheezy, where the +package is included already. So to get started, run <tt>apt-get +install s3ql</tt>. Next, pick a storage provider. I ended up picking +Greenqloud, after reading their nice recipe on +<a href="https://greenqloud.zendesk.com/entries/44611757-How-To-Use-S3QL-to-mount-a-StorageQloud-bucket-on-Debian-Wheezy">how +to use s3ql with their Amazon S3 service</a>, because I trust the laws +in Iceland more than those in USA when it come to keeping my personal +data safe and private, and thus would rather spend money on a company +in Iceland. Another nice recipe is available from the article +<a href="http://www.admin-magazine.com/HPC/Articles/HPC-Cloud-Storage">S3QL +Filesystem for HPC Storage</a> by Jeff Layton in the HPC section of +Admin magazine. When the provider is picked, figure out how to get +the API key needed to connect to the storage API. With Greencloud, +the key did not show up until I had added payment details to my +account. + +Armed with the API access details, it is time to create the file +system. First, create a new bucket in the cloud. This bucket is the +file system storage area. I picked a bucket name reflecting the +machine that was going to store data there, but any name will do. +I'll refer to it as <tt>bucket-name</tt> below. In addition, one need +the API login and password, and a locally created password. Store it +all in ~root/.s3ql/authinfo2 like this: + +<blockquote><pre> +[s3c] +storage-url: s3c://s.greenqloud.com:443/bucket-name +backend-login: API-login +backend-password: API-password +fs-passphrase: local-password +</pre></blockquote> + +I create my local passphrase using <tt>pwget 50</tt> or similar, +but any sensible way to create a fairly random password should do it. +Armed with these details, it is now time to run mkfs, entering the API +details and password to create it: + +<blockquote><pre> +# mkdir -m 700 /var/lib/s3ql-cache +# mkfs.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \ + --ssl s3c://s.greenqloud.com:443/bucket-name +Enter backend login: +Enter backend password: +Before using S3QL, make sure to read the user's guide, especially +the 'Important Rules to Avoid Loosing Data' section. +Enter encryption password: +Confirm encryption password: +Generating random encryption key... +Creating metadata tables... +Dumping metadata... +..objects.. +..blocks.. +..inodes.. +..inode_blocks.. +..symlink_targets.. +..names.. +..contents.. +..ext_attributes.. +Compressing and uploading metadata... +Wrote 0.00 MB of compressed metadata. +# </pre></blockquote> + +The next step is mounting the file system to make the storage available. + +<blockquote><pre> +# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \ + --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql +Using 4 upload threads. +Downloading and decompressing metadata... +Reading metadata... +..objects.. +..blocks.. +..inodes.. +..inode_blocks.. +..symlink_targets.. +..names.. +..contents.. +..ext_attributes.. +Mounting filesystem... +# df -h /mnt +Filesystem Size Used Avail Use% Mounted on +s3c://s.greenqloud.com:443/bucket-name 1.0T 0 1.0T 0% /s3ql +# +</pre></blockquote> + +The file system is now ready for use. I use rsync to store my +backups in it, and as the metadata used by rsync is downloaded at +mount time, no network traffic (and storage cost) is triggered by +running rsync. To unmount, one should not use the normal umount +command, as this will not flush the cache to the cloud storage, but +instead running the umount.s3ql command like this: + +<blockquote><pre> +# umount.s3ql /s3ql +# +</pre></blockquote> + +There is a fsck command available to check the file system and +correct any problems detected. This can be used if the local server +crashes while the file system is mounted, to reset the "already +mounted" flag. This is what it look like when processing a working +file system: + +<blockquote><pre> +# fsck.s3ql --force --ssl s3c://s.greenqloud.com:443/bucket-name +Using cached metadata. +File system seems clean, checking anyway. +Checking DB integrity... +Creating temporary extra indices... +Checking lost+found... +Checking cached objects... +Checking names (refcounts)... +Checking contents (names)... +Checking contents (inodes)... +Checking contents (parent inodes)... +Checking objects (reference counts)... +Checking objects (backend)... +..processed 5000 objects so far.. +..processed 10000 objects so far.. +..processed 15000 objects so far.. +Checking objects (sizes)... +Checking blocks (referenced objects)... +Checking blocks (refcounts)... +Checking inode-block mapping (blocks)... +Checking inode-block mapping (inodes)... +Checking inodes (refcounts)... +Checking inodes (sizes)... +Checking extended attributes (names)... +Checking extended attributes (inodes)... +Checking symlinks (inodes)... +Checking directory reachability... +Checking unix conventions... +Checking referential integrity... +Dropping temporary indices... +Backing up old metadata... +Dumping metadata... +..objects.. +..blocks.. +..inodes.. +..inode_blocks.. +..symlink_targets.. +..names.. +..contents.. +..ext_attributes.. +Compressing and uploading metadata... +Wrote 0.89 MB of compressed metadata. +# +</pre></blockquote> + +Thanks to the cache, working on files that fit in the cache is very +quick, about the same speed as local file access. Uploading large +amount of data is to me limited by the bandwidth out of and into my +house. Uploading 685 MiB with a 100 MiB cache gave me 305 kiB/s, +which is very close to my upload speed, and downloading the same +Debian installation ISO gave me 610 kiB/s, close to my download speed. +Both were measured using <tt>dd</tt>. So for me, the bottleneck is my +network, not the file system code. I do not know what a good cache +size would be, but suspect that the cache should e larger than your +working set. + +I mentioned that only one machine can mount the file system at the +time. If another machine try, it is told that the file system is +busy: + +<blockquote><pre> +# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \ + --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql +Using 8 upload threads. +Backend reports that fs is still mounted elsewhere, aborting. +# +</pre></blockquote> + +The file content is uploaded when the cache is full, while the +metadata is uploaded once every 24 hour by default. To ensure the +file system content is flushed to the cloud, one can either umount the +file system, or ask s3ql to flush the cache and metadata using +s3qlctrl: + +<blockquote><pre> +# s3qlctrl upload-meta /s3ql +# s3qlctrl flushcache /s3ql +# +</pre></blockquote> + +If you are curious about how much space your data uses in the +cloud, and how much compression and deduplication cut down on the +storage usage, you can use s3qlstat on the mounted file system to get +a report: + +<blockquote><pre> +# s3qlstat /s3ql +Directory entries: 9141 +Inodes: 9143 +Data blocks: 8851 +Total data size: 22049.38 MB +After de-duplication: 21955.46 MB (99.57% of total) +After compression: 21877.28 MB (99.22% of total, 99.64% of de-duplicated) +Database size: 2.39 MB (uncompressed) +(some values do not take into account not-yet-uploaded dirty blocks in cache) +# +</pre></blockquote> + +I mentioned earlier that there are several possible suppliers of +storage. I did not try to locate them all, but am aware of at least +<a href="https://www.greenqloud.com/">Greenqloud</a>, +<a href="http://drive.google.com/">Google Drive</a>, +<a href="http://aws.amazon.com/s3/">Amazon S3 web serivces</a>, +<a href="http://www.rackspace.com/">Rackspace</a> and +<a href="http://crowncloud.net/">Crowncloud</A>. The latter even +accept payment in Bitcoin. Pick one that suit your need. Some of +them provide several GiB of free storage, but the prize models are +quire different and you will have to figure out what suit you +best. + +While researching this blog post, I had a look at research papers +and posters discussing the S3QL file system. There are several, which +told me that the file system is getting a critical check by the +science community and increased my confidence in using it. One nice +poster is titled +"<a href="http://www.lanl.gov/orgs/adtsc/publications/science_highlights_2013/docs/pg68_69.pdf">An +Innovative Parallel Cloud Storage System using OpenStackâs SwiftObject +Store and Transformative Parallel I/O Approach</a>" by Hsing-Bung +Chen, Benjamin McClelland, David Sherrill, Alfred Torrez, Parks Fields +and Pamela Smith. Please have a look. + +Given my problems with different file systems earlier, I decided to +check out the mounted S3QL file system to see if it would be usable as +a home directory (in other word, that it provided POSIX semantics when +it come to locking and umask handling etc). Running +<a href="http://people.skolelinux.org/pere/blog/Testing_if_a_file_system_can_be_used_for_home_directories___.html">my +test code to check file system semantics, I was happy to discover that +no error was found. So the file system can be used for home +directories, if one chooses to do so. + +If you do not want a locally file system, and want something that +work without the Linux fuse file system, I would like to mention the +<a href="http://www.tarsnap.com/">Tarsnap service</a>, which also +provide locally encrypted backup using a command line client. It have +a nicer access control system, where one can split out read and write +access, allowing some systems to write to the backup and others to +only read from it. + +As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +<a href="bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&label=PetterReinholdtsenBlog">15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b</a>. + + + ReactOS Windows clone - nice free software http://people.skolelinux.org/pere/blog/ReactOS_Windows_clone___nice_free_software.html diff --git a/blog/tags/english/index.html b/blog/tags/english/index.html index 1a4e3a4da7..8607cff0ac 100644 --- a/blog/tags/english/index.html +++ b/blog/tags/english/index.html @@ -20,6 +20,300 @@

Entries tagged "english".

+ s3ql, a locally mounted cloud file system - nice free software +

+ 9th April 2014 +

+ +

+[s3c]
+storage-url: s3c://s.greenqloud.com:443/bucket-name
+backend-login: API-login
+backend-password: API-password
+fs-passphrase: local-password
+

+ +

+# mkdir -m 700 /var/lib/s3ql-cache
+# mkfs.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl s3c://s.greenqloud.com:443/bucket-name
+Enter backend login: 
+Enter backend password: 
+Before using S3QL, make sure to read the user's guide, especially
+the 'Important Rules to Avoid Loosing Data' section.
+Enter encryption password: 
+Confirm encryption password: 
+Generating random encryption key...
+Creating metadata tables...
+Dumping metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Compressing and uploading metadata...
+Wrote 0.00 MB of compressed metadata.
+#

+ +

The next step is mounting the file system to make the storage available. + +

+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+Using 4 upload threads.
+Downloading and decompressing metadata...
+Reading metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Mounting filesystem...
+# df -h /mnt
+Filesystem                              Size  Used Avail Use% Mounted on
+s3c://s.greenqloud.com:443/bucket-name  1.0T     0  1.0T   0% /s3ql
+#
+

+ +

+# umount.s3ql /s3ql
+# 
+

+ +

+# fsck.s3ql --force --ssl s3c://s.greenqloud.com:443/bucket-name
+Using cached metadata.
+File system seems clean, checking anyway.
+Checking DB integrity...
+Creating temporary extra indices...
+Checking lost+found...
+Checking cached objects...
+Checking names (refcounts)...
+Checking contents (names)...
+Checking contents (inodes)...
+Checking contents (parent inodes)...
+Checking objects (reference counts)...
+Checking objects (backend)...
+..processed 5000 objects so far..
+..processed 10000 objects so far..
+..processed 15000 objects so far..
+Checking objects (sizes)...
+Checking blocks (referenced objects)...
+Checking blocks (refcounts)...
+Checking inode-block mapping (blocks)...
+Checking inode-block mapping (inodes)...
+Checking inodes (refcounts)...
+Checking inodes (sizes)...
+Checking extended attributes (names)...
+Checking extended attributes (inodes)...
+Checking symlinks (inodes)...
+Checking directory reachability...
+Checking unix conventions...
+Checking referential integrity...
+Dropping temporary indices...
+Backing up old metadata...
+Dumping metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Compressing and uploading metadata...
+Wrote 0.89 MB of compressed metadata.
+# 
+

+ +

I mentioned that only one machine can mount the file system at the +time. If another machine try, it is told that the file system is +busy:

+ +

+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+Using 8 upload threads.
+Backend reports that fs is still mounted elsewhere, aborting.
+#
+

+ +

+# s3qlctrl upload-meta /s3ql
+# s3qlctrl flushcache /s3ql
+# 
+

+ +

+# s3qlstat /s3ql
+Directory entries:    9141
+Inodes:               9143
+Data blocks:          8851
+Total data size:      22049.38 MB
+After de-duplication: 21955.46 MB (99.57% of total)
+After compression:    21877.28 MB (99.22% of total, 99.64% of de-duplicated)
+Database size:        2.39 MB (uncompressed)
+(some values do not take into account not-yet-uploaded dirty blocks in cache)
+#
+

+ +

As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

+ +

+ + + Tags: debian, english, personvern, sikkerhet. + + +

ReactOS Windows clone - nice free software @@ -19391,7 +19685,7 @@ be the only one fitting our needs. :/

@@ -19572,7 +19866,7 @@ be the only one fitting our needs. :/

@@ -19582,7 +19876,7 @@ be the only one fitting our needs. :/

@@ -19620,7 +19914,7 @@ be the only one fitting our needs. :/

@@ -19638,7 +19932,7 @@ be the only one fitting our needs. :/

diff --git a/blog/tags/fiksgatami/index.html b/blog/tags/fiksgatami/index.html index de257e0164..a65817fecf 100644 --- a/blog/tags/fiksgatami/index.html +++ b/blog/tags/fiksgatami/index.html @@ -1269,7 +1269,7 @@ med dem. Dette blir bra.

@@ -1450,7 +1450,7 @@ med dem. Dette blir bra.

@@ -1460,7 +1460,7 @@ med dem. Dette blir bra.

@@ -1498,7 +1498,7 @@ med dem. Dette blir bra.

@@ -1516,7 +1516,7 @@ med dem. Dette blir bra.

diff --git a/blog/tags/fildeling/index.html b/blog/tags/fildeling/index.html index 8bbe37a2cf..0d64952da3 100644 --- a/blog/tags/fildeling/index.html +++ b/blog/tags/fildeling/index.html @@ -639,7 +639,7 @@ og fildeling av slike filer er fullt ut lovlig.

@@ -820,7 +820,7 @@ og fildeling av slike filer er fullt ut lovlig.

@@ -830,7 +830,7 @@ og fildeling av slike filer er fullt ut lovlig.

@@ -868,7 +868,7 @@ og fildeling av slike filer er fullt ut lovlig.

@@ -886,7 +886,7 @@ og fildeling av slike filer er fullt ut lovlig.

diff --git a/blog/tags/freeculture/index.html b/blog/tags/freeculture/index.html index 5ff0a4e154..d3e8376802 100644 --- a/blog/tags/freeculture/index.html +++ b/blog/tags/freeculture/index.html @@ -893,7 +893,7 @@ slik at du kan oppdatere direkte.

@@ -1074,7 +1074,7 @@ slik at du kan oppdatere direkte.

@@ -1084,7 +1084,7 @@ slik at du kan oppdatere direkte.

@@ -1122,7 +1122,7 @@ slik at du kan oppdatere direkte.

@@ -1140,7 +1140,7 @@ slik at du kan oppdatere direkte.

diff --git a/blog/tags/freedombox/index.html b/blog/tags/freedombox/index.html index 70820899b6..1ab4d82294 100644 --- a/blog/tags/freedombox/index.html +++ b/blog/tags/freedombox/index.html @@ -757,7 +757,7 @@ default password is 'secret'.

@@ -938,7 +938,7 @@ default password is 'secret'.

@@ -948,7 +948,7 @@ default password is 'secret'.

@@ -986,7 +986,7 @@ default password is 'secret'.

@@ -1004,7 +1004,7 @@ default password is 'secret'.

diff --git a/blog/tags/frikanalen/index.html b/blog/tags/frikanalen/index.html index 15fe965008..3d3252c0fc 100644 --- a/blog/tags/frikanalen/index.html +++ b/blog/tags/frikanalen/index.html @@ -977,7 +977,7 @@ NUUG lykkes med Ã¥ fÃ¥ ut sine opptak med like stor suksess.

@@ -1158,7 +1158,7 @@ NUUG lykkes med Ã¥ fÃ¥ ut sine opptak med like stor suksess.

@@ -1168,7 +1168,7 @@ NUUG lykkes med Ã¥ fÃ¥ ut sine opptak med like stor suksess.

@@ -1206,7 +1206,7 @@ NUUG lykkes med Ã¥ fÃ¥ ut sine opptak med like stor suksess.

@@ -1224,7 +1224,7 @@ NUUG lykkes med Ã¥ fÃ¥ ut sine opptak med like stor suksess.

diff --git a/blog/tags/intervju/index.html b/blog/tags/intervju/index.html index 905fa09f03..84c36eae56 100644 --- a/blog/tags/intervju/index.html +++ b/blog/tags/intervju/index.html @@ -4833,7 +4833,7 @@ veldig bra utvalg av gratis spill som er av hÃ¸y kvalitet. Veldig lett

@@ -5014,7 +5014,7 @@ veldig bra utvalg av gratis spill som er av hÃ¸y kvalitet. Veldig lett

@@ -5024,7 +5024,7 @@ veldig bra utvalg av gratis spill som er av hÃ¸y kvalitet. Veldig lett

@@ -5062,7 +5062,7 @@ veldig bra utvalg av gratis spill som er av hÃ¸y kvalitet. Veldig lett

@@ -5080,7 +5080,7 @@ veldig bra utvalg av gratis spill som er av hÃ¸y kvalitet. Veldig lett

diff --git a/blog/tags/isenkram/index.html b/blog/tags/isenkram/index.html index e37c40ba89..c923b2e596 100644 --- a/blog/tags/isenkram/index.html +++ b/blog/tags/isenkram/index.html @@ -791,7 +791,7 @@ please send me an email. :)

@@ -972,7 +972,7 @@ please send me an email. :)

@@ -982,7 +982,7 @@ please send me an email. :)

@@ -1020,7 +1020,7 @@ please send me an email. :)

@@ -1038,7 +1038,7 @@ please send me an email. :)

diff --git a/blog/tags/kart/index.html b/blog/tags/kart/index.html index 408aa2453f..13e72c1872 100644 --- a/blog/tags/kart/index.html +++ b/blog/tags/kart/index.html @@ -1138,7 +1138,7 @@ det viser at behovet for fribruks-sjÃ¸kart er til stedet.

@@ -1319,7 +1319,7 @@ det viser at behovet for fribruks-sjÃ¸kart er til stedet.

@@ -1329,7 +1329,7 @@ det viser at behovet for fribruks-sjÃ¸kart er til stedet.

@@ -1367,7 +1367,7 @@ det viser at behovet for fribruks-sjÃ¸kart er til stedet.

@@ -1385,7 +1385,7 @@ det viser at behovet for fribruks-sjÃ¸kart er til stedet.

diff --git a/blog/tags/ldap/index.html b/blog/tags/ldap/index.html index e470c9885c..108c53c34e 100644 --- a/blog/tags/ldap/index.html +++ b/blog/tags/ldap/index.html @@ -1056,7 +1056,7 @@ new IETF work group?

@@ -1237,7 +1237,7 @@ new IETF work group?

@@ -1247,7 +1247,7 @@ new IETF work group?

@@ -1285,7 +1285,7 @@ new IETF work group?

@@ -1303,7 +1303,7 @@ new IETF work group?

diff --git a/blog/tags/lenker/index.html b/blog/tags/lenker/index.html index 66ed7380b4..43a79ee6cf 100644 --- a/blog/tags/lenker/index.html +++ b/blog/tags/lenker/index.html @@ -477,7 +477,7 @@ Word 2007 hÃ¥ndterer ODF dÃ¥rlig

@@ -658,7 +658,7 @@ Word 2007 hÃ¥ndterer ODF dÃ¥rlig

@@ -668,7 +668,7 @@ Word 2007 hÃ¥ndterer ODF dÃ¥rlig

@@ -706,7 +706,7 @@ Word 2007 hÃ¥ndterer ODF dÃ¥rlig

@@ -724,7 +724,7 @@ Word 2007 hÃ¥ndterer ODF dÃ¥rlig

diff --git a/blog/tags/ltsp/index.html b/blog/tags/ltsp/index.html index 4e51c6e688..d0bb138fc5 100644 --- a/blog/tags/ltsp/index.html +++ b/blog/tags/ltsp/index.html @@ -71,7 +71,7 @@ of these cards.

@@ -252,7 +252,7 @@ of these cards.

@@ -262,7 +262,7 @@ of these cards.

@@ -300,7 +300,7 @@ of these cards.

@@ -318,7 +318,7 @@ of these cards.

diff --git a/blog/tags/mesh network/index.html b/blog/tags/mesh network/index.html index ccbcd738c2..8dc2725af5 100644 --- a/blog/tags/mesh network/index.html +++ b/blog/tags/mesh network/index.html @@ -721,7 +721,7 @@ mesh system.

@@ -902,7 +902,7 @@ mesh system.

@@ -912,7 +912,7 @@ mesh system.

@@ -950,7 +950,7 @@ mesh system.

@@ -968,7 +968,7 @@ mesh system.

diff --git a/blog/tags/multimedia/index.html b/blog/tags/multimedia/index.html index 9abb2d89a2..590ba97a96 100644 --- a/blog/tags/multimedia/index.html +++ b/blog/tags/multimedia/index.html @@ -1949,7 +1949,7 @@ be the only one fitting our needs. :/

@@ -2130,7 +2130,7 @@ be the only one fitting our needs. :/

@@ -2140,7 +2140,7 @@ be the only one fitting our needs. :/

@@ -2178,7 +2178,7 @@ be the only one fitting our needs. :/

@@ -2196,7 +2196,7 @@ be the only one fitting our needs. :/

diff --git a/blog/tags/norsk/index.html b/blog/tags/norsk/index.html index e352ce6a4e..395cb0aa7f 100644 --- a/blog/tags/norsk/index.html +++ b/blog/tags/norsk/index.html @@ -18421,7 +18421,7 @@ forsÃ¸k.

@@ -18602,7 +18602,7 @@ forsÃ¸k.

@@ -18612,7 +18612,7 @@ forsÃ¸k.

@@ -18650,7 +18650,7 @@ forsÃ¸k.

@@ -18668,7 +18668,7 @@ forsÃ¸k.

diff --git a/blog/tags/nuug/index.html b/blog/tags/nuug/index.html index f6d57ce95c..754458af22 100644 --- a/blog/tags/nuug/index.html +++ b/blog/tags/nuug/index.html @@ -10694,7 +10694,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

@@ -10875,7 +10875,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

@@ -10885,7 +10885,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

@@ -10923,7 +10923,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

@@ -10941,7 +10941,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

diff --git a/blog/tags/offentlig innsyn/index.html b/blog/tags/offentlig innsyn/index.html index cf61667801..7ee2cb9eeb 100644 --- a/blog/tags/offentlig innsyn/index.html +++ b/blog/tags/offentlig innsyn/index.html @@ -1095,7 +1095,7 @@ til Ã¥ levere hver uke. Har ikke undersÃ¸kt noen av de andre.

@@ -1276,7 +1276,7 @@ til Ã¥ levere hver uke. Har ikke undersÃ¸kt noen av de andre.

@@ -1286,7 +1286,7 @@ til Ã¥ levere hver uke. Har ikke undersÃ¸kt noen av de andre.

@@ -1324,7 +1324,7 @@ til Ã¥ levere hver uke. Har ikke undersÃ¸kt noen av de andre.

@@ -1342,7 +1342,7 @@ til Ã¥ levere hver uke. Har ikke undersÃ¸kt noen av de andre.

diff --git a/blog/tags/open311/index.html b/blog/tags/open311/index.html index 873c89fb7c..6905c4ffdb 100644 --- a/blog/tags/open311/index.html +++ b/blog/tags/open311/index.html @@ -166,7 +166,7 @@ work like the free software project communities I am used to.

@@ -347,7 +347,7 @@ work like the free software project communities I am used to.

@@ -357,7 +357,7 @@ work like the free software project communities I am used to.

@@ -395,7 +395,7 @@ work like the free software project communities I am used to.

@@ -413,7 +413,7 @@ work like the free software project communities I am used to.

diff --git a/blog/tags/opphavsrett/index.html b/blog/tags/opphavsrett/index.html index 7bac985780..04ebbbf2df 100644 --- a/blog/tags/opphavsrett/index.html +++ b/blog/tags/opphavsrett/index.html @@ -3451,7 +3451,7 @@ og endrer pÃ¥ betingelsene.

@@ -3632,7 +3632,7 @@ og endrer pÃ¥ betingelsene.

@@ -3642,7 +3642,7 @@ og endrer pÃ¥ betingelsene.

@@ -3680,7 +3680,7 @@ og endrer pÃ¥ betingelsene.

@@ -3698,7 +3698,7 @@ og endrer pÃ¥ betingelsene.

diff --git a/blog/tags/personvern/index.html b/blog/tags/personvern/index.html index 417bc842fe..1a424ae39f 100644 --- a/blog/tags/personvern/index.html +++ b/blog/tags/personvern/index.html @@ -20,6 +20,300 @@

Entries tagged "personvern".

+ s3ql, a locally mounted cloud file system - nice free software +

+ 9th April 2014 +

+ +

+[s3c]
+storage-url: s3c://s.greenqloud.com:443/bucket-name
+backend-login: API-login
+backend-password: API-password
+fs-passphrase: local-password
+

+ +

+# mkdir -m 700 /var/lib/s3ql-cache
+# mkfs.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl s3c://s.greenqloud.com:443/bucket-name
+Enter backend login: 
+Enter backend password: 
+Before using S3QL, make sure to read the user's guide, especially
+the 'Important Rules to Avoid Loosing Data' section.
+Enter encryption password: 
+Confirm encryption password: 
+Generating random encryption key...
+Creating metadata tables...
+Dumping metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Compressing and uploading metadata...
+Wrote 0.00 MB of compressed metadata.
+#

+ +

The next step is mounting the file system to make the storage available. + +

+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+Using 4 upload threads.
+Downloading and decompressing metadata...
+Reading metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Mounting filesystem...
+# df -h /mnt
+Filesystem                              Size  Used Avail Use% Mounted on
+s3c://s.greenqloud.com:443/bucket-name  1.0T     0  1.0T   0% /s3ql
+#
+

+ +

+# umount.s3ql /s3ql
+# 
+

+ +

+# fsck.s3ql --force --ssl s3c://s.greenqloud.com:443/bucket-name
+Using cached metadata.
+File system seems clean, checking anyway.
+Checking DB integrity...
+Creating temporary extra indices...
+Checking lost+found...
+Checking cached objects...
+Checking names (refcounts)...
+Checking contents (names)...
+Checking contents (inodes)...
+Checking contents (parent inodes)...
+Checking objects (reference counts)...
+Checking objects (backend)...
+..processed 5000 objects so far..
+..processed 10000 objects so far..
+..processed 15000 objects so far..
+Checking objects (sizes)...
+Checking blocks (referenced objects)...
+Checking blocks (refcounts)...
+Checking inode-block mapping (blocks)...
+Checking inode-block mapping (inodes)...
+Checking inodes (refcounts)...
+Checking inodes (sizes)...
+Checking extended attributes (names)...
+Checking extended attributes (inodes)...
+Checking symlinks (inodes)...
+Checking directory reachability...
+Checking unix conventions...
+Checking referential integrity...
+Dropping temporary indices...
+Backing up old metadata...
+Dumping metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Compressing and uploading metadata...
+Wrote 0.89 MB of compressed metadata.
+# 
+

+ +

I mentioned that only one machine can mount the file system at the +time. If another machine try, it is told that the file system is +busy:

+ +

+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+Using 8 upload threads.
+Backend reports that fs is still mounted elsewhere, aborting.
+#
+

+ +

+# s3qlctrl upload-meta /s3ql
+# s3qlctrl flushcache /s3ql
+# 
+

+ +

+# s3qlstat /s3ql
+Directory entries:    9141
+Inodes:               9143
+Data blocks:          8851
+Total data size:      22049.38 MB
+After de-duplication: 21955.46 MB (99.57% of total)
+After compression:    21877.28 MB (99.22% of total, 99.64% of de-duplicated)
+Database size:        2.39 MB (uncompressed)
+(some values do not take into account not-yet-uploaded dirty blocks in cache)
+#
+

+ +

As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

+ +

+ + + Tags: debian, english, personvern, sikkerhet. + + +

EU-domstolen bekreftet i dag at datalagringsdirektivet er ulovlig @@ -5895,7 +6189,7 @@ kontanter for noen dager siden.

@@ -6076,7 +6370,7 @@ kontanter for noen dager siden.

@@ -6086,7 +6380,7 @@ kontanter for noen dager siden.

@@ -6124,7 +6418,7 @@ kontanter for noen dager siden.

@@ -6142,7 +6436,7 @@ kontanter for noen dager siden.

diff --git a/blog/tags/personvern/personvern.rss b/blog/tags/personvern/personvern.rss index 183ceaeda8..c8cf4d68f9 100644 --- a/blog/tags/personvern/personvern.rss +++ b/blog/tags/personvern/personvern.rss @@ -6,6 +6,288 @@ http://people.skolelinux.org/pere/blog/ + + s3ql, a locally mounted cloud file system - nice free software + http://people.skolelinux.org/pere/blog/s3ql__a_locally_mounted_cloud_file_system___nice_free_software.html + http://people.skolelinux.org/pere/blog/s3ql__a_locally_mounted_cloud_file_system___nice_free_software.html + Wed, 9 Apr 2014 11:30:00 +0200 + For a while now, I have been looking for a sensible offsite backup +solution for use at home. My requirements are simple, it must be +cheap and locally encrypted (in other words, I keep the encryption +keys, the storage provider do not have access to my private files). +One idea me and my friends have had many years ago, before the cloud +storage providers showed up, have been to use Google mail as storage, +writing a Linux block device storing blocks as emails in the mail +service provided by Google, and thus get heaps of free space. On top +of this one can add encryption, RAID and volume management to have +lots of (fairly slow, I admit that) cheap and encrypted storage. But +I never found time to implement such system. But the last few weeks I +have looked at a system called +<a href="https://bitbucket.org/nikratio/s3ql/">S3QL</a>, a locally +mounted network backed file system with the features I need. + +S3QL is a fuse file system with a local cache and cloud storage, +handling several different storage providers, any with Amazon S3, +Google Drive or OpenStack API. There are heaps of such storage +providers. S3QL can also use a local directory as storage, which +combined with sshfs allow for file storage on any ssh server. S3QL +include support for encryption, compression, de-duplication, snapshots +and immutable file systems, allowing me to mount the remote storage as +a local mount point, look at and use the files as if they were local, +while the content is stored in the cloud as well. This allow me to +have a backup that should survive fire. The file system can not be +shared between several machines at the same time, as only one can +mount it at the time, but any machine with the encryption key and +access to the storage service can mount it if it is unmounted. + +It is simple to use. I'm using it on Debian Wheezy, where the +package is included already. So to get started, run <tt>apt-get +install s3ql</tt>. Next, pick a storage provider. I ended up picking +Greenqloud, after reading their nice recipe on +<a href="https://greenqloud.zendesk.com/entries/44611757-How-To-Use-S3QL-to-mount-a-StorageQloud-bucket-on-Debian-Wheezy">how +to use s3ql with their Amazon S3 service</a>, because I trust the laws +in Iceland more than those in USA when it come to keeping my personal +data safe and private, and thus would rather spend money on a company +in Iceland. Another nice recipe is available from the article +<a href="http://www.admin-magazine.com/HPC/Articles/HPC-Cloud-Storage">S3QL +Filesystem for HPC Storage</a> by Jeff Layton in the HPC section of +Admin magazine. When the provider is picked, figure out how to get +the API key needed to connect to the storage API. With Greencloud, +the key did not show up until I had added payment details to my +account. + +Armed with the API access details, it is time to create the file +system. First, create a new bucket in the cloud. This bucket is the +file system storage area. I picked a bucket name reflecting the +machine that was going to store data there, but any name will do. +I'll refer to it as <tt>bucket-name</tt> below. In addition, one need +the API login and password, and a locally created password. Store it +all in ~root/.s3ql/authinfo2 like this: + +<blockquote><pre> +[s3c] +storage-url: s3c://s.greenqloud.com:443/bucket-name +backend-login: API-login +backend-password: API-password +fs-passphrase: local-password +</pre></blockquote> + +I create my local passphrase using <tt>pwget 50</tt> or similar, +but any sensible way to create a fairly random password should do it. +Armed with these details, it is now time to run mkfs, entering the API +details and password to create it: + +<blockquote><pre> +# mkdir -m 700 /var/lib/s3ql-cache +# mkfs.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \ + --ssl s3c://s.greenqloud.com:443/bucket-name +Enter backend login: +Enter backend password: +Before using S3QL, make sure to read the user's guide, especially +the 'Important Rules to Avoid Loosing Data' section. +Enter encryption password: +Confirm encryption password: +Generating random encryption key... +Creating metadata tables... +Dumping metadata... +..objects.. +..blocks.. +..inodes.. +..inode_blocks.. +..symlink_targets.. +..names.. +..contents.. +..ext_attributes.. +Compressing and uploading metadata... +Wrote 0.00 MB of compressed metadata. +# </pre></blockquote> + +The next step is mounting the file system to make the storage available. + +<blockquote><pre> +# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \ + --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql +Using 4 upload threads. +Downloading and decompressing metadata... +Reading metadata... +..objects.. +..blocks.. +..inodes.. +..inode_blocks.. +..symlink_targets.. +..names.. +..contents.. +..ext_attributes.. +Mounting filesystem... +# df -h /mnt +Filesystem Size Used Avail Use% Mounted on +s3c://s.greenqloud.com:443/bucket-name 1.0T 0 1.0T 0% /s3ql +# +</pre></blockquote> + +The file system is now ready for use. I use rsync to store my +backups in it, and as the metadata used by rsync is downloaded at +mount time, no network traffic (and storage cost) is triggered by +running rsync. To unmount, one should not use the normal umount +command, as this will not flush the cache to the cloud storage, but +instead running the umount.s3ql command like this: + +<blockquote><pre> +# umount.s3ql /s3ql +# +</pre></blockquote> + +There is a fsck command available to check the file system and +correct any problems detected. This can be used if the local server +crashes while the file system is mounted, to reset the "already +mounted" flag. This is what it look like when processing a working +file system: + +<blockquote><pre> +# fsck.s3ql --force --ssl s3c://s.greenqloud.com:443/bucket-name +Using cached metadata. +File system seems clean, checking anyway. +Checking DB integrity... +Creating temporary extra indices... +Checking lost+found... +Checking cached objects... +Checking names (refcounts)... +Checking contents (names)... +Checking contents (inodes)... +Checking contents (parent inodes)... +Checking objects (reference counts)... +Checking objects (backend)... +..processed 5000 objects so far.. +..processed 10000 objects so far.. +..processed 15000 objects so far.. +Checking objects (sizes)... +Checking blocks (referenced objects)... +Checking blocks (refcounts)... +Checking inode-block mapping (blocks)... +Checking inode-block mapping (inodes)... +Checking inodes (refcounts)... +Checking inodes (sizes)... +Checking extended attributes (names)... +Checking extended attributes (inodes)... +Checking symlinks (inodes)... +Checking directory reachability... +Checking unix conventions... +Checking referential integrity... +Dropping temporary indices... +Backing up old metadata... +Dumping metadata... +..objects.. +..blocks.. +..inodes.. +..inode_blocks.. +..symlink_targets.. +..names.. +..contents.. +..ext_attributes.. +Compressing and uploading metadata... +Wrote 0.89 MB of compressed metadata. +# +</pre></blockquote> + +Thanks to the cache, working on files that fit in the cache is very +quick, about the same speed as local file access. Uploading large +amount of data is to me limited by the bandwidth out of and into my +house. Uploading 685 MiB with a 100 MiB cache gave me 305 kiB/s, +which is very close to my upload speed, and downloading the same +Debian installation ISO gave me 610 kiB/s, close to my download speed. +Both were measured using <tt>dd</tt>. So for me, the bottleneck is my +network, not the file system code. I do not know what a good cache +size would be, but suspect that the cache should e larger than your +working set. + +I mentioned that only one machine can mount the file system at the +time. If another machine try, it is told that the file system is +busy: + +<blockquote><pre> +# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \ + --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql +Using 8 upload threads. +Backend reports that fs is still mounted elsewhere, aborting. +# +</pre></blockquote> + +The file content is uploaded when the cache is full, while the +metadata is uploaded once every 24 hour by default. To ensure the +file system content is flushed to the cloud, one can either umount the +file system, or ask s3ql to flush the cache and metadata using +s3qlctrl: + +<blockquote><pre> +# s3qlctrl upload-meta /s3ql +# s3qlctrl flushcache /s3ql +# +</pre></blockquote> + +If you are curious about how much space your data uses in the +cloud, and how much compression and deduplication cut down on the +storage usage, you can use s3qlstat on the mounted file system to get +a report: + +<blockquote><pre> +# s3qlstat /s3ql +Directory entries: 9141 +Inodes: 9143 +Data blocks: 8851 +Total data size: 22049.38 MB +After de-duplication: 21955.46 MB (99.57% of total) +After compression: 21877.28 MB (99.22% of total, 99.64% of de-duplicated) +Database size: 2.39 MB (uncompressed) +(some values do not take into account not-yet-uploaded dirty blocks in cache) +# +</pre></blockquote> + +I mentioned earlier that there are several possible suppliers of +storage. I did not try to locate them all, but am aware of at least +<a href="https://www.greenqloud.com/">Greenqloud</a>, +<a href="http://drive.google.com/">Google Drive</a>, +<a href="http://aws.amazon.com/s3/">Amazon S3 web serivces</a>, +<a href="http://www.rackspace.com/">Rackspace</a> and +<a href="http://crowncloud.net/">Crowncloud</A>. The latter even +accept payment in Bitcoin. Pick one that suit your need. Some of +them provide several GiB of free storage, but the prize models are +quire different and you will have to figure out what suit you +best. + +While researching this blog post, I had a look at research papers +and posters discussing the S3QL file system. There are several, which +told me that the file system is getting a critical check by the +science community and increased my confidence in using it. One nice +poster is titled +"<a href="http://www.lanl.gov/orgs/adtsc/publications/science_highlights_2013/docs/pg68_69.pdf">An +Innovative Parallel Cloud Storage System using OpenStackâs SwiftObject +Store and Transformative Parallel I/O Approach</a>" by Hsing-Bung +Chen, Benjamin McClelland, David Sherrill, Alfred Torrez, Parks Fields +and Pamela Smith. Please have a look. + +Given my problems with different file systems earlier, I decided to +check out the mounted S3QL file system to see if it would be usable as +a home directory (in other word, that it provided POSIX semantics when +it come to locking and umask handling etc). Running +<a href="http://people.skolelinux.org/pere/blog/Testing_if_a_file_system_can_be_used_for_home_directories___.html">my +test code to check file system semantics, I was happy to discover that +no error was found. So the file system can be used for home +directories, if one chooses to do so. + +If you do not want a locally file system, and want something that +work without the Linux fuse file system, I would like to mention the +<a href="http://www.tarsnap.com/">Tarsnap service</a>, which also +provide locally encrypted backup using a command line client. It have +a nicer access control system, where one can split out read and write +access, allowing some systems to write to the backup and others to +only read from it. + +As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +<a href="bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&label=PetterReinholdtsenBlog">15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b</a>. + + + EU-domstolen bekreftet i dag at datalagringsdirektivet er ulovlig http://people.skolelinux.org/pere/blog/EU_domstolen_bekreftet_i_dag_at_datalagringsdirektivet_er_ulovlig.html diff --git a/blog/tags/raid/index.html b/blog/tags/raid/index.html index 2e91654429..f497a3b72d 100644 --- a/blog/tags/raid/index.html +++ b/blog/tags/raid/index.html @@ -107,7 +107,7 @@ disk(s) is failing when the RAID is running short on disks.

@@ -288,7 +288,7 @@ disk(s) is failing when the RAID is running short on disks.

@@ -298,7 +298,7 @@ disk(s) is failing when the RAID is running short on disks.

@@ -336,7 +336,7 @@ disk(s) is failing when the RAID is running short on disks.

@@ -354,7 +354,7 @@ disk(s) is failing when the RAID is running short on disks.

diff --git a/blog/tags/reactos/index.html b/blog/tags/reactos/index.html index f96e239587..ee36919b43 100644 --- a/blog/tags/reactos/index.html +++ b/blog/tags/reactos/index.html @@ -106,7 +106,7 @@ image.

@@ -287,7 +287,7 @@ image.

@@ -297,7 +297,7 @@ image.

@@ -335,7 +335,7 @@ image.

@@ -353,7 +353,7 @@ image.

diff --git a/blog/tags/reprap/index.html b/blog/tags/reprap/index.html index 90707f570e..29d78fdc7f 100644 --- a/blog/tags/reprap/index.html +++ b/blog/tags/reprap/index.html @@ -533,7 +533,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

@@ -714,7 +714,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

@@ -724,7 +724,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

@@ -762,7 +762,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

@@ -780,7 +780,7 @@ hÃ¥per det ikke gÃ¥r tapt pÃ¥ samme vis.

diff --git a/blog/tags/rfid/index.html b/blog/tags/rfid/index.html index 7cf08359fa..c8d122a2ea 100644 --- a/blog/tags/rfid/index.html +++ b/blog/tags/rfid/index.html @@ -156,7 +156,7 @@ mer om de nye biometriske passene.

@@ -337,7 +337,7 @@ mer om de nye biometriske passene.

@@ -347,7 +347,7 @@ mer om de nye biometriske passene.

@@ -385,7 +385,7 @@ mer om de nye biometriske passene.

@@ -403,7 +403,7 @@ mer om de nye biometriske passene.

diff --git a/blog/tags/robot/index.html b/blog/tags/robot/index.html index 7dedc66a7e..c7c757095a 100644 --- a/blog/tags/robot/index.html +++ b/blog/tags/robot/index.html @@ -471,7 +471,7 @@ firmwaren. :)

@@ -652,7 +652,7 @@ firmwaren. :)

@@ -662,7 +662,7 @@ firmwaren. :)

@@ -700,7 +700,7 @@ firmwaren. :)

@@ -718,7 +718,7 @@ firmwaren. :)

diff --git a/blog/tags/rss/index.html b/blog/tags/rss/index.html index fb7992afc7..075edc29ac 100644 --- a/blog/tags/rss/index.html +++ b/blog/tags/rss/index.html @@ -60,7 +60,7 @@ forsÃ¸k.

@@ -241,7 +241,7 @@ forsÃ¸k.

@@ -251,7 +251,7 @@ forsÃ¸k.

@@ -289,7 +289,7 @@ forsÃ¸k.

@@ -307,7 +307,7 @@ forsÃ¸k.

diff --git a/blog/tags/ruter/index.html b/blog/tags/ruter/index.html index 69c8515a65..1b19c8d76e 100644 --- a/blog/tags/ruter/index.html +++ b/blog/tags/ruter/index.html @@ -214,7 +214,7 @@ OsloomrÃ¥det.

@@ -395,7 +395,7 @@ OsloomrÃ¥det.

@@ -405,7 +405,7 @@ OsloomrÃ¥det.

@@ -443,7 +443,7 @@ OsloomrÃ¥det.

@@ -461,7 +461,7 @@ OsloomrÃ¥det.

diff --git a/blog/tags/scraperwiki/index.html b/blog/tags/scraperwiki/index.html index e658dd4dbd..f18aeb5d10 100644 --- a/blog/tags/scraperwiki/index.html +++ b/blog/tags/scraperwiki/index.html @@ -149,7 +149,7 @@ av Ã¥ dele informasjon med andre uten bruksbegresninger.

@@ -330,7 +330,7 @@ av Ã¥ dele informasjon med andre uten bruksbegresninger.

@@ -340,7 +340,7 @@ av Ã¥ dele informasjon med andre uten bruksbegresninger.

@@ -378,7 +378,7 @@ av Ã¥ dele informasjon med andre uten bruksbegresninger.

@@ -396,7 +396,7 @@ av Ã¥ dele informasjon med andre uten bruksbegresninger.

diff --git a/blog/tags/sikkerhet/index.html b/blog/tags/sikkerhet/index.html index 511f1cec5f..abda6f4a84 100644 --- a/blog/tags/sikkerhet/index.html +++ b/blog/tags/sikkerhet/index.html @@ -20,6 +20,300 @@

Entries tagged "sikkerhet".

+ s3ql, a locally mounted cloud file system - nice free software +

+ 9th April 2014 +

+ +

+[s3c]
+storage-url: s3c://s.greenqloud.com:443/bucket-name
+backend-login: API-login
+backend-password: API-password
+fs-passphrase: local-password
+

+ +

+# mkdir -m 700 /var/lib/s3ql-cache
+# mkfs.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl s3c://s.greenqloud.com:443/bucket-name
+Enter backend login: 
+Enter backend password: 
+Before using S3QL, make sure to read the user's guide, especially
+the 'Important Rules to Avoid Loosing Data' section.
+Enter encryption password: 
+Confirm encryption password: 
+Generating random encryption key...
+Creating metadata tables...
+Dumping metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Compressing and uploading metadata...
+Wrote 0.00 MB of compressed metadata.
+#

+ +

The next step is mounting the file system to make the storage available. + +

+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+Using 4 upload threads.
+Downloading and decompressing metadata...
+Reading metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Mounting filesystem...
+# df -h /mnt
+Filesystem                              Size  Used Avail Use% Mounted on
+s3c://s.greenqloud.com:443/bucket-name  1.0T     0  1.0T   0% /s3ql
+#
+

+ +

+# umount.s3ql /s3ql
+# 
+

+ +

+# fsck.s3ql --force --ssl s3c://s.greenqloud.com:443/bucket-name
+Using cached metadata.
+File system seems clean, checking anyway.
+Checking DB integrity...
+Creating temporary extra indices...
+Checking lost+found...
+Checking cached objects...
+Checking names (refcounts)...
+Checking contents (names)...
+Checking contents (inodes)...
+Checking contents (parent inodes)...
+Checking objects (reference counts)...
+Checking objects (backend)...
+..processed 5000 objects so far..
+..processed 10000 objects so far..
+..processed 15000 objects so far..
+Checking objects (sizes)...
+Checking blocks (referenced objects)...
+Checking blocks (refcounts)...
+Checking inode-block mapping (blocks)...
+Checking inode-block mapping (inodes)...
+Checking inodes (refcounts)...
+Checking inodes (sizes)...
+Checking extended attributes (names)...
+Checking extended attributes (inodes)...
+Checking symlinks (inodes)...
+Checking directory reachability...
+Checking unix conventions...
+Checking referential integrity...
+Dropping temporary indices...
+Backing up old metadata...
+Dumping metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Compressing and uploading metadata...
+Wrote 0.89 MB of compressed metadata.
+# 
+

+ +

I mentioned that only one machine can mount the file system at the +time. If another machine try, it is told that the file system is +busy:

+ +

+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+  --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+Using 8 upload threads.
+Backend reports that fs is still mounted elsewhere, aborting.
+#
+

+ +

+# s3qlctrl upload-meta /s3ql
+# s3qlctrl flushcache /s3ql
+# 
+

+ +

+# s3qlstat /s3ql
+Directory entries:    9141
+Inodes:               9143
+Data blocks:          8851
+Total data size:      22049.38 MB
+After de-duplication: 21955.46 MB (99.57% of total)
+After compression:    21877.28 MB (99.22% of total, 99.64% of de-duplicated)
+Database size:        2.39 MB (uncompressed)
+(some values do not take into account not-yet-uploaded dirty blocks in cache)
+#
+

+ +

As usual, if you use Bitcoin and want to show your support of my +activities, please send Bitcoin donations to my address +15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

+ +

+ + + Tags: debian, english, personvern, sikkerhet. + + +

EU-domstolen bekreftet i dag at datalagringsdirektivet er ulovlig @@ -2672,7 +2966,7 @@ betydelige.