+ <item>
+ <title>S3QL, a locally mounted cloud file system - nice free software</title>
+ <link>http://people.skolelinux.org/pere/blog/S3QL__a_locally_mounted_cloud_file_system___nice_free_software.html</link>
+ <guid isPermaLink="true">http://people.skolelinux.org/pere/blog/S3QL__a_locally_mounted_cloud_file_system___nice_free_software.html</guid>
+ <pubDate>Wed, 9 Apr 2014 11:30:00 +0200</pubDate>
+ <description><p>For a while now, I have been looking for a sensible offsite backup
+solution for use at home. My requirements are simple, it must be
+cheap and locally encrypted (in other words, I keep the encryption
+keys, the storage provider do not have access to my private files).
+One idea me and my friends had many years ago, before the cloud
+storage providers showed up, was to use Google mail as storage,
+writing a Linux block device storing blocks as emails in the mail
+service provided by Google, and thus get heaps of free space. On top
+of this one can add encryption, RAID and volume management to have
+lots of (fairly slow, I admit that) cheap and encrypted storage. But
+I never found time to implement such system. But the last few weeks I
+have looked at a system called
+<a href="https://bitbucket.org/nikratio/s3ql/">S3QL</a>, a locally
+mounted network backed file system with the features I need.</p>
+
+<p>S3QL is a fuse file system with a local cache and cloud storage,
+handling several different storage providers, any with Amazon S3,
+Google Drive or OpenStack API. There are heaps of such storage
+providers. S3QL can also use a local directory as storage, which
+combined with sshfs allow for file storage on any ssh server. S3QL
+include support for encryption, compression, de-duplication, snapshots
+and immutable file systems, allowing me to mount the remote storage as
+a local mount point, look at and use the files as if they were local,
+while the content is stored in the cloud as well. This allow me to
+have a backup that should survive fire. The file system can not be
+shared between several machines at the same time, as only one can
+mount it at the time, but any machine with the encryption key and
+access to the storage service can mount it if it is unmounted.</p>
+
+<p>It is simple to use. I'm using it on Debian Wheezy, where the
+package is included already. So to get started, run <tt>apt-get
+install s3ql</tt>. Next, pick a storage provider. I ended up picking
+Greenqloud, after reading their nice recipe on
+<a href="https://greenqloud.zendesk.com/entries/44611757-How-To-Use-S3QL-to-mount-a-StorageQloud-bucket-on-Debian-Wheezy">how
+to use S3QL with their Amazon S3 service</a>, because I trust the laws
+in Iceland more than those in USA when it come to keeping my personal
+data safe and private, and thus would rather spend money on a company
+in Iceland. Another nice recipe is available from the article
+<a href="http://www.admin-magazine.com/HPC/Articles/HPC-Cloud-Storage">S3QL
+Filesystem for HPC Storage</a> by Jeff Layton in the HPC section of
+Admin magazine. When the provider is picked, figure out how to get
+the API key needed to connect to the storage API. With Greencloud,
+the key did not show up until I had added payment details to my
+account.</p>
+
+<p>Armed with the API access details, it is time to create the file
+system. First, create a new bucket in the cloud. This bucket is the
+file system storage area. I picked a bucket name reflecting the
+machine that was going to store data there, but any name will do.
+I'll refer to it as <tt>bucket-name</tt> below. In addition, one need
+the API login and password, and a locally created password. Store it
+all in ~root/.s3ql/authinfo2 like this:
+
+<p><blockquote><pre>
+[s3c]
+storage-url: s3c://s.greenqloud.com:443/bucket-name
+backend-login: API-login
+backend-password: API-password
+fs-passphrase: local-password
+</pre></blockquote></p>
+
+<p>I create my local passphrase using <tt>pwget 50</tt> or similar,
+but any sensible way to create a fairly random password should do it.
+Armed with these details, it is now time to run mkfs, entering the API
+details and password to create it:</p>
+
+<p><blockquote><pre>
+# mkdir -m 700 /var/lib/s3ql-cache
+# mkfs.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+ --ssl s3c://s.greenqloud.com:443/bucket-name
+Enter backend login:
+Enter backend password:
+Before using S3QL, make sure to read the user's guide, especially
+the 'Important Rules to Avoid Loosing Data' section.
+Enter encryption password:
+Confirm encryption password:
+Generating random encryption key...
+Creating metadata tables...
+Dumping metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Compressing and uploading metadata...
+Wrote 0.00 MB of compressed metadata.
+# </pre></blockquote></p>
+
+<p>The next step is mounting the file system to make the storage available.
+
+<p><blockquote><pre>
+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+ --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+Using 4 upload threads.
+Downloading and decompressing metadata...
+Reading metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Mounting filesystem...
+# df -h /s3ql
+Filesystem Size Used Avail Use% Mounted on
+s3c://s.greenqloud.com:443/bucket-name 1.0T 0 1.0T 0% /s3ql
+#
+</pre></blockquote></p>
+
+<p>The file system is now ready for use. I use rsync to store my
+backups in it, and as the metadata used by rsync is downloaded at
+mount time, no network traffic (and storage cost) is triggered by
+running rsync. To unmount, one should not use the normal umount
+command, as this will not flush the cache to the cloud storage, but
+instead running the umount.s3ql command like this:
+
+<p><blockquote><pre>
+# umount.s3ql /s3ql
+#
+</pre></blockquote></p>
+
+<p>There is a fsck command available to check the file system and
+correct any problems detected. This can be used if the local server
+crashes while the file system is mounted, to reset the "already
+mounted" flag. This is what it look like when processing a working
+file system:</p>
+
+<p><blockquote><pre>
+# fsck.s3ql --force --ssl s3c://s.greenqloud.com:443/bucket-name
+Using cached metadata.
+File system seems clean, checking anyway.
+Checking DB integrity...
+Creating temporary extra indices...
+Checking lost+found...
+Checking cached objects...
+Checking names (refcounts)...
+Checking contents (names)...
+Checking contents (inodes)...
+Checking contents (parent inodes)...
+Checking objects (reference counts)...
+Checking objects (backend)...
+..processed 5000 objects so far..
+..processed 10000 objects so far..
+..processed 15000 objects so far..
+Checking objects (sizes)...
+Checking blocks (referenced objects)...
+Checking blocks (refcounts)...
+Checking inode-block mapping (blocks)...
+Checking inode-block mapping (inodes)...
+Checking inodes (refcounts)...
+Checking inodes (sizes)...
+Checking extended attributes (names)...
+Checking extended attributes (inodes)...
+Checking symlinks (inodes)...
+Checking directory reachability...
+Checking unix conventions...
+Checking referential integrity...
+Dropping temporary indices...
+Backing up old metadata...
+Dumping metadata...
+..objects..
+..blocks..
+..inodes..
+..inode_blocks..
+..symlink_targets..
+..names..
+..contents..
+..ext_attributes..
+Compressing and uploading metadata...
+Wrote 0.89 MB of compressed metadata.
+#
+</pre></blockquote></p>
+
+<p>Thanks to the cache, working on files that fit in the cache is very
+quick, about the same speed as local file access. Uploading large
+amount of data is to me limited by the bandwidth out of and into my
+house. Uploading 685 MiB with a 100 MiB cache gave me 305 kiB/s,
+which is very close to my upload speed, and downloading the same
+Debian installation ISO gave me 610 kiB/s, close to my download speed.
+Both were measured using <tt>dd</tt>. So for me, the bottleneck is my
+network, not the file system code. I do not know what a good cache
+size would be, but suspect that the cache should e larger than your
+working set.</p>
+
+<p>I mentioned that only one machine can mount the file system at the
+time. If another machine try, it is told that the file system is
+busy:</p>
+
+<p><blockquote><pre>
+# mount.s3ql --cachedir /var/lib/s3ql-cache --authfile /root/.s3ql/authinfo2 \
+ --ssl --allow-root s3c://s.greenqloud.com:443/bucket-name /s3ql
+Using 8 upload threads.
+Backend reports that fs is still mounted elsewhere, aborting.
+#
+</pre></blockquote></p>
+
+<p>The file content is uploaded when the cache is full, while the
+metadata is uploaded once every 24 hour by default. To ensure the
+file system content is flushed to the cloud, one can either umount the
+file system, or ask S3QL to flush the cache and metadata using
+s3qlctrl:
+
+<p><blockquote><pre>
+# s3qlctrl upload-meta /s3ql
+# s3qlctrl flushcache /s3ql
+#
+</pre></blockquote></p>
+
+<p>If you are curious about how much space your data uses in the
+cloud, and how much compression and deduplication cut down on the
+storage usage, you can use s3qlstat on the mounted file system to get
+a report:</p>
+
+<p><blockquote><pre>
+# s3qlstat /s3ql
+Directory entries: 9141
+Inodes: 9143
+Data blocks: 8851
+Total data size: 22049.38 MB
+After de-duplication: 21955.46 MB (99.57% of total)
+After compression: 21877.28 MB (99.22% of total, 99.64% of de-duplicated)
+Database size: 2.39 MB (uncompressed)
+(some values do not take into account not-yet-uploaded dirty blocks in cache)
+#
+</pre></blockquote></p>
+
+<p>I mentioned earlier that there are several possible suppliers of
+storage. I did not try to locate them all, but am aware of at least
+<a href="https://www.greenqloud.com/">Greenqloud</a>,
+<a href="http://drive.google.com/">Google Drive</a>,
+<a href="http://aws.amazon.com/s3/">Amazon S3 web serivces</a>,
+<a href="http://www.rackspace.com/">Rackspace</a> and
+<a href="http://crowncloud.net/">Crowncloud</A>. The latter even
+accept payment in Bitcoin. Pick one that suit your need. Some of
+them provide several GiB of free storage, but the prize models are
+quite different and you will have to figure out what suits you
+best.</p>
+
+<p>While researching this blog post, I had a look at research papers
+and posters discussing the S3QL file system. There are several, which
+told me that the file system is getting a critical check by the
+science community and increased my confidence in using it. One nice
+poster is titled
+"<a href="http://www.lanl.gov/orgs/adtsc/publications/science_highlights_2013/docs/pg68_69.pdf">An
+Innovative Parallel Cloud Storage System using OpenStack’s SwiftObject
+Store and Transformative Parallel I/O Approach</a>" by Hsing-Bung
+Chen, Benjamin McClelland, David Sherrill, Alfred Torrez, Parks Fields
+and Pamela Smith. Please have a look.</p>
+
+<p>Given my problems with different file systems earlier, I decided to
+check out the mounted S3QL file system to see if it would be usable as
+a home directory (in other word, that it provided POSIX semantics when
+it come to locking and umask handling etc). Running
+<a href="http://people.skolelinux.org/pere/blog/Testing_if_a_file_system_can_be_used_for_home_directories___.html">my
+test code to check file system semantics</a>, I was happy to discover that
+no error was found. So the file system can be used for home
+directories, if one chooses to do so.</p>
+
+<p>If you do not want a locally file system, and want something that
+work without the Linux fuse file system, I would like to mention the
+<a href="http://www.tarsnap.com/">Tarsnap service</a>, which also
+provide locally encrypted backup using a command line client. It have
+a nicer access control system, where one can split out read and write
+access, allowing some systems to write to the backup and others to
+only read from it.</p>
+
+<p>As usual, if you use Bitcoin and want to show your support of my
+activities, please send Bitcoin donations to my address
+<b><a href="bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&label=PetterReinholdtsenBlog">15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b</a></b>.</p>
+</description>
+ </item>
+
+ <item>
+ <title>EU-domstolen bekreftet i dag at datalagringsdirektivet er ulovlig</title>
+ <link>http://people.skolelinux.org/pere/blog/EU_domstolen_bekreftet_i_dag_at_datalagringsdirektivet_er_ulovlig.html</link>
+ <guid isPermaLink="true">http://people.skolelinux.org/pere/blog/EU_domstolen_bekreftet_i_dag_at_datalagringsdirektivet_er_ulovlig.html</guid>
+ <pubDate>Tue, 8 Apr 2014 11:30:00 +0200</pubDate>
+ <description><p>I dag kom endelig avgjørelsen fra EU-domstolen om
+datalagringsdirektivet, som ikke overraskende ble dømt ulovlig og i
+strid med borgernes grunnleggende rettigheter. Hvis du lurer på hva
+datalagringsdirektivet er for noe, så er det
+<a href="http://tv.nrk.no/program/koid75005313/tema-dine-digitale-spor-datalagringsdirektivet">en
+flott dokumentar tilgjengelig hos NRK</a> som jeg tidligere
+<a href="http://people.skolelinux.org/pere/blog/Dokumentaren_om_Datalagringsdirektivet_sendes_endelig_p__NRK.html">har
+anbefalt</a> alle å se.</p>
+
+<p>Her er et liten knippe nyhetsoppslag om saken, og jeg regner med at
+det kommer flere ut over dagen. Flere kan finnes
+<a href="http://www.mylder.no/?drill=datalagringsdirektivet&intern=1">via
+mylder</a>.</p>
+
+<p><ul>
+
+<li><a href="http://e24.no/digital/eu-domstolen-datalagringsdirektivet-er-ugyldig/22879592">EU-domstolen:
+Datalagringsdirektivet er ugyldig</a> - e24.no 2014-04-08
+
+<li><a href="http://www.aftenposten.no/nyheter/iriks/EU-domstolen-Datalagringsdirektivet-er-ulovlig-7529032.html">EU-domstolen:
+Datalagringsdirektivet er ulovlig</a> - aftenposten.no 2014-04-08
+
+<li><a href="http://www.aftenposten.no/nyheter/iriks/politikk/Krever-DLD-stopp-i-Norge-7530086.html">Krever
+DLD-stopp i Norge</a> - aftenposten.no 2014-04-08
+
+<li><a href="http://www.p4.no/story.aspx?id=566431">Apenes: - En
+gledens dag</a> - p4.no 2014-04-08
+
+<li><a href="http://www.nrk.no/norge/_-datalagringsdirektivet-er-ugyldig-1.11655929">EU-domstolen:
+– Datalagringsdirektivet er ugyldig</a> - nrk.no 2014-04-08</li>
+
+<li><a href="http://www.vg.no/nyheter/utenriks/data-og-nett/eu-domstolen-datalagringsdirektivet-er-ugyldig/a/10130280/">EU-domstolen:
+Datalagringsdirektivet er ugyldig</a> - vg.no 2014-04-08</li>
+
+<li><a href="http://www.dagbladet.no/2014/04/08/nyheter/innenriks/datalagringsdirektivet/personvern/32711646/">-
+Vi bør skrote hele datalagringsdirektivet</a> - dagbladet.no
+2014-04-08</li>
+
+<li><a href="http://www.digi.no/928137/eu-domstolen-dld-er-ugyldig">EU-domstolen:
+DLD er ugyldig</a> - digi.no 2014-04-08</li>
+
+<li><a href="http://www.irishtimes.com/business/sectors/technology/european-court-declares-data-retention-directive-invalid-1.1754150">European
+court declares data retention directive invalid</a> - irishtimes.com
+2014-04-08</li>
+
+<li><a href="http://www.reuters.com/article/2014/04/08/us-eu-data-ruling-idUSBREA370F020140408?feedType=RSS">EU
+court rules against requirement to keep data of telecom users</a> -
+reuters.com 2014-04-08</li>
+
+</ul>
+</p>
+
+<p>Jeg synes det er veldig fint at nok en stemme slår fast at
+totalitær overvåkning av befolkningen er uakseptabelt, men det er
+fortsatt like viktig å beskytte privatsfæren som før, da de
+teknologiske mulighetene fortsatt finnes og utnyttes, og jeg tror
+innsats i prosjekter som
+<a href="https://wiki.debian.org/FreedomBox">Freedombox</a> og
+<a href="http://www.dugnadsnett.no/">Dugnadsnett</a> er viktigere enn
+noen gang.</p>
+
+<p><strong>Update 2014-04-08 12:10</strong>: Kronerullingen for å
+stoppe datalagringsdirektivet i Norge gjøres hos foreningen
+<a href="http://www.digitaltpersonvern.no/">Digitalt Personvern</a>,
+som har samlet inn 843 215,- så langt men trenger nok mye mer hvis
+
+ikke Høyre og Arbeiderpartiet bytter mening i saken. Det var
+<a href="http://www.holderdeord.no/parliament-issues/48650">kun
+partinene Høyre og Arbeiderpartiet</a> som stemte for
+Datalagringsdirektivet, og en av dem må bytte mening for at det skal
+bli flertall mot i Stortinget. Se mer om saken
+<a href="http://www.holderdeord.no/issues/69-innfore-datalagringsdirektivet">Holder
+de ord</a>.</p>
+</description>
+ </item>
+
+ <item>
+ <title>ReactOS Windows clone - nice free software</title>
+ <link>http://people.skolelinux.org/pere/blog/ReactOS_Windows_clone___nice_free_software.html</link>
+ <guid isPermaLink="true">http://people.skolelinux.org/pere/blog/ReactOS_Windows_clone___nice_free_software.html</guid>
+ <pubDate>Tue, 1 Apr 2014 12:10:00 +0200</pubDate>
+ <description><p>Microsoft have announced that Windows XP reaches its end of life
+2014-04-08, in 7 days. But there are heaps of machines still running
+Windows XP, and depending on Windows XP to run their applications, and
+upgrading will be expensive, both when it comes to money and when it
+comes to the amount of effort needed to migrate from Windows XP to a
+new operating system. Some obvious options (buy new a Windows
+machine, buy a MacOSX machine, install Linux on the existing machine)
+are already well known and covered elsewhere. Most of them involve
+leaving the user applications installed on Windows XP behind and
+trying out replacements or updated versions. In this blog post I want
+to mention one strange bird that allow people to keep the hardware and
+the existing Windows XP applications and run them on a free software
+operating system that is Windows XP compatible.</p>
+
+<p><a href="http://www.reactos.org/">ReactOS</a> is a free software
+operating system (GNU GPL licensed) working on providing a operating
+system that is binary compatible with Windows, able to run windows
+programs directly and to use Windows drivers for hardware directly.
+The project goal is for Windows user to keep their existing machines,
+drivers and software, and gain the advantages from user a operating
+system without usage limitations caused by non-free licensing. It is
+a Windows clone running directly on the hardware, so quite different
+from the approach taken by <a href="http://www.winehq.org/">the Wine
+project</a>, which make it possible to run Windows binaries on
+Linux.</p>
+
+<p>The ReactOS project share code with the Wine project, so most
+shared libraries available on Windows are already implemented already.
+There is also a software manager like the one we are used to on Linux,
+allowing the user to install free software applications with a simple
+click directly from the Internet. Check out the
+<a href="http://www.reactos.org/screenshots">screen shots on the
+project web site</a> for an idea what it look like (it looks just like
+Windows before metro).</p>
+
+<p>I do not use ReactOS myself, preferring Linux and Unix like
+operating systems. I've tested it, and it work fine in a virt-manager
+virtual machine. The browser, minesweeper, notepad etc is working
+fine as far as I can tell. Unfortunately, my main test application
+is the software included on a CD with the Lego Mindstorms NXT, which
+seem to install just fine from CD but fail to leave any binaries on
+the disk after the installation. So no luck with that test software.
+No idea why, but hope someone else figure out and fix the problem.
+I've tried the ReactOS Live ISO on a physical machine, and it seemed
+to work just fine. If you like Windows and want to keep running your
+old Windows binaries, check it out by
+<a href="http://www.reactos.org/download">downloading</a> the
+installation CD, the live CD or the preinstalled virtual machine
+image.</p>
+</description>
+ </item>
+
+ <item>
+ <title>Debian Edu interview: Roger Marsal</title>
+ <link>http://people.skolelinux.org/pere/blog/Debian_Edu_interview__Roger_Marsal.html</link>
+ <guid isPermaLink="true">http://people.skolelinux.org/pere/blog/Debian_Edu_interview__Roger_Marsal.html</guid>
+ <pubDate>Sun, 30 Mar 2014 11:40:00 +0200</pubDate>
+ <description><p><a href="http://www.skolelinux.org/">Debian Edu / Skolelinux</a>
+keep gaining new users. Some weeks ago, a person showed up on IRC,
+<a href="irc://irc.debian.org/#debian-edu">#debian-edu</a>, with a
+wish to contribute, and I managed to get a interview with this great
+contributor Roger Marsal to learn more about his background.</p>
+
+<p><strong>Who are you, and how do you spend your days?</strong></p>
+
+<p>My name is Roger Marsal, I'm 27 years old (1986 generation) and I
+live in Barcelona, Spain. I've got a strong business background and I
+work as a patrimony manager and as a real estate agent. Additionally,
+I've co-founded a British based tech company that is nowadays on the
+last development phase of a new social networking concept.</p>
+
+<p>I'm a Linux enthusiast that started its journey with Ubuntu four years
+ago and have recently switched to Debian seeking rock solid stability
+and as a necessary step to gain expertise.</p>
+
+<p>In a nutshell, I spend my days working and learning as much as I
+can to face both my job, entrepreneur project and feed my Linux
+hunger.</p>
+
+<p><strong>How did you get in contact with the Skolelinux / Debian Edu
+project?</strong></p>
+
+<p>I discovered the <a href="http://www.ltsp.org/">LTSP</a> advantages
+with "Ubuntu 12.04 alternate install" and after a year of use I
+started looking for an alternative. Even though I highly value and
+respect the Ubuntu project, I thought it was necessary for me to
+change to a more robust and stable alternative. As far as I was using
+Debian on my personal laptop I thought it would be fine to install
+Debian and configure an LTSP server myself. Surprised, I discovered
+that the Debian project also supported a kind of Edubuntu equivalent,
+and after having some pain I obtained a Debian Edu network up and
+running. I just loved it.</p>
+
+<p><strong>What do you see as the advantages of Skolelinux / Debian
+Edu?</strong></p>
+
+<p>I found a main advantage in that, once you know "the tips and
+tricks", a new installation just works out of the box. It's the most
+complete alternative I've found to create an LTSP network. All the
+other distributions seems to be made of plastic, Debian Edu seems to
+be made of steel.</p>
+
+<p><strong>What do you see as the disadvantages of Skolelinux / Debian
+Edu?</strong></p>
+
+<p>I found two main disadvantages.</p>
+
+<p>I'm not an expert but I've got notions and I had to spent a considerable
+amount of time trying to bring up a standard network topology. I'm quite
+stubborn and I just worked until I did but I'm sure many people with few
+resources (not big schools, but academies for example) would have switched
+or dropped.</p>
+
+<p>It's amazing how such a complex system like Debian Edu has achieved
+this out-of-the-box state. Even though tweaking without breaking gets
+more difficult, as more factors have to be considered. This can
+discourage many people too.</p>
+
+<p><strong>Which free software do you use daily?</strong></p>
+
+<p>I use Debian, Firefox, Okular, Inkscape, LibreOffice and
+Virtualbox.</p>
+
+
+<p><strong>Which strategy do you believe is the right one to use to
+get schools to use free software?</strong></p>
+
+<p>I don't think there is a need for a particular strategy. The free
+attribute in both "freedom" and "no price" meanings is what will
+really bring free software to schools. In my experience I can think of
+the <a href="http://www.r-project.org/">"R" statistical language</a>; a
+few years a ago was an extremely nerd tool for university people.
+Today it's being increasingly used to teach statistics at many
+different level of studies. I believe free and open software will
+increasingly gain popularity, but I'm sure schools will be one of the
+first scenarios where this will happen.</p>
+</description>
+ </item>
+
projects / homepage.git / blobdiff