1 <?xml version=
"1.0" encoding=
"ISO-8859-1"?>
2 <rss version='
2.0' xmlns:lj='http://www.livejournal.org/rss/lj/
1.0/'
>
4 <title>Petter Reinholdtsen - Entries from March
2017</title>
5 <description>Entries from March
2017</description>
6 <link>http://people.skolelinux.org/pere/blog/
</link>
10 <title>Detecting NFS hangs on Linux without hanging yourself...
</title>
11 <link>http://people.skolelinux.org/pere/blog/Detecting_NFS_hangs_on_Linux_without_hanging_yourself___.html
</link>
12 <guid isPermaLink=
"true">http://people.skolelinux.org/pere/blog/Detecting_NFS_hangs_on_Linux_without_hanging_yourself___.html
</guid>
13 <pubDate>Thu,
9 Mar
2017 15:
20:
00 +
0100</pubDate>
14 <description><p
>Over the years, administrating thousand of NFS mounting linux
15 computers at the time, I often needed a way to detect if the machine
16 was experiencing NFS hang. If you try to use
<tt
>df
</tt
> or look at a
17 file or directory affected by the hang, the process (and possibly the
18 shell) will hang too. So you want to be able to detect this without
19 risking the detection process getting stuck too. It has not been
20 obvious how to do this. When the hang has lasted a while, it is
21 possible to find messages like these in dmesg:
</p
>
23 <p
><blockquote
>
24 nfs: server nfsserver not responding, still trying
25 <br
>nfs: server nfsserver OK
26 </blockquote
></p
>
28 <p
>It is hard to know if the hang is still going on, and it is hard to
29 be sure looking in dmesg is going to work. If there are lots of other
30 messages in dmesg the lines might have rotated out of site before they
31 are noticed.
</p
>
33 <p
>While reading through the nfs client implementation in linux kernel
34 code, I came across some statistics that seem to give a way to detect
35 it. The om_timeouts sunrpc value in the kernel will increase every
36 time the above log entry is inserted into dmesg. And after digging a
37 bit further, I discovered that this value show up in
38 /proc/self/mountstats on Linux.
</p
>
40 <p
>The mountstats content seem to be shared between files using the
41 same file system context, so it is enough to check one of the
42 mountstats files to get the state of the mount point for the machine.
43 I assume this will not show lazy umounted NFS points, nor NFS mount
44 points in a different process context (ie with a different filesystem
45 view), but that does not worry me.
</p
>
47 <p
>The content for a NFS mount point look similar to this:
</p
>
49 <p
><blockquote
><pre
>
51 device /dev/mapper/Debian-var mounted on /var with fstype ext3
52 device nfsserver:/mnt/nfsserver/home0 mounted on /mnt/nfsserver/home0 with fstype nfs statvers=
1.1
53 opts: rw,vers=
3,rsize=
65536,wsize=
65536,namlen=
255,acregmin=
3,acregmax=
60,acdirmin=
30,acdirmax=
60,soft,nolock,proto=tcp,timeo=
600,retrans=
2,sec=sys,mountaddr=
129.240.3.145,mountvers=
3,mountport=
4048,mountproto=udp,local_lock=all
55 caps: caps=
0x3fe7,wtmult=
4096,dtsize=
8192,bsize=
0,namlen=
255
56 sec: flavor=
1,pseudoflavor=
1
57 events:
61063112 732346265 1028140 35486205 16220064 8162542 761447191 71714012 37189 3891185 45561809 110486139 4850138 420353 15449177 296502 52736725 13523379 0 52182 9016896 1231 0 0 0 0 0
58 bytes:
166253035039 219519120027 0 0 40783504807 185466229638 11677877 45561809
59 RPC iostats version:
1.0 p/v:
100003/
3 (nfs)
60 xprt: tcp
925 1 6810 0 0 111505412 111480497 109 2672418560317 0 248 53869103 22481820
63 GETATTR:
61063106 61063108 0 9621383060 6839064400 453650 77291321 78926132
64 SETATTR:
463469 463470 0 92005440 66739536 63787 603235 687943
65 LOOKUP:
17021657 17021657 0 3354097764 4013442928 57216 35125459 35566511
66 ACCESS:
14281703 14290009 5 2318400592 1713803640 1709282 4865144 7130140
67 READLINK:
125 125 0 20472 18620 0 1112 1118
68 READ:
4214236 4214237 0 715608524 41328653212 89884 22622768 22806693
69 WRITE:
8479010 8494376 22 187695798568 1356087148 178264904 51506907 231671771
70 CREATE:
171708 171708 0 38084748 46702272 873 1041833 1050398
71 MKDIR:
3680 3680 0 773980 993920 26 23990 24245
72 SYMLINK:
903 903 0 233428 245488 6 5865 5917
73 MKNOD:
80 80 0 20148 21760 0 299 304
74 REMOVE:
429921 429921 0 79796004 61908192 3313 2710416 2741636
75 RMDIR:
3367 3367 0 645112 484848 22 5782 6002
76 RENAME:
466201 466201 0 130026184 121212260 7075 5935207 5961288
77 LINK:
289155 289155 0 72775556 67083960 2199 2565060 2585579
78 READDIR:
2933237 2933237 0 516506204 13973833412 10385 3190199 3297917
79 READDIRPLUS:
1652839 1652839 0 298640972 6895997744 84735 14307895 14448937
80 FSSTAT:
6144 6144 0 1010516 1032192 51 9654 10022
81 FSINFO:
2 2 0 232 328 0 1 1
82 PATHCONF:
1 1 0 116 140 0 0 0
83 COMMIT:
0 0 0 0 0 0 0 0
85 device binfmt_misc mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc
87 </pre
></blockquote
></p
>
89 <p
>The key number to look at is the third number in the per-op list.
90 It is the number of NFS timeouts experiences per file system
91 operation. Here
22 write timeouts and
5 access timeouts. If these
92 numbers are increasing, I believe the machine is experiencing NFS
93 hang. Unfortunately the timeout value do not start to increase right
94 away. The NFS operations need to time out first, and this can take a
95 while. The exact timeout value depend on the setup. For example the
96 defaults for TCP and UDP mount points are quite different, and the
97 timeout value is affected by the soft, hard, timeo and retrans NFS
98 mount options.
</p
>
100 <p
>The only way I have been able to get working on Debian and RedHat
101 Enterprise Linux for getting the timeout count is to peek in /proc/.
103 <ahref=
"http://docs.oracle.com/cd/E19253-
01/
816-
4555/netmonitor-
12/index.html
">Solaris
104 10 System Administration Guide: Network Services
</a
>, the
'nfsstat -c
'
105 command can be used to get these timeout values. But this do not work
106 on Linux, as far as I can tell. I
107 <ahref=
"http://bugs.debian.org/
857043">asked Debian about this
</a
>,
108 but have not seen any replies yet.
</p
>
110 <p
>Is there a better way to figure out if a Linux NFS client is
111 experiencing NFS hangs? Is there a way to detect which processes are
112 affected? Is there a way to get the NFS mount going quickly once the
113 network problem causing the NFS hang has been cleared? I would very
114 much welcome some clues, as we regularly run into NFS hangs.
</p
>
119 <title>How does it feel to be wiretapped, when you should be doing the wiretapping...
</title>
120 <link>http://people.skolelinux.org/pere/blog/How_does_it_feel_to_be_wiretapped__when_you_should_be_doing_the_wiretapping___.html
</link>
121 <guid isPermaLink=
"true">http://people.skolelinux.org/pere/blog/How_does_it_feel_to_be_wiretapped__when_you_should_be_doing_the_wiretapping___.html
</guid>
122 <pubDate>Wed,
8 Mar
2017 11:
50:
00 +
0100</pubDate>
123 <description><p
>So the new president in the United States of America claim to be
124 surprised to discover that he was wiretapped during the election
125 before he was elected president. He even claim this must be illegal.
126 Well, doh, if it is one thing the confirmations from Snowden
127 documented, it is that the entire population in USA is wiretapped, one
128 way or another. Of course the president candidates were wiretapped,
129 alongside the senators, judges and the rest of the people in USA.
</p
>
131 <p
>Next, the Federal Bureau of Investigation ask the Department of
132 Justice to go public rejecting the claims that Donald Trump was
133 wiretapped illegally. I fail to see the relevance, given that I am
134 sure the surveillance industry in USA believe they have all the legal
135 backing they need to conduct mass surveillance on the entire
138 <p
>There is even the director of the FBI stating that he never saw an
139 order requesting wiretapping of Donald Trump. That is not very
140 surprising, given how the FISA court work, with all its activity being
141 secret. Perhaps he only heard about it?
</p
>
143 <p
>What I find most sad in this story is how Norwegian journalists
144 present it. In a news reports the other day in the radio from the
145 Norwegian National broadcasting Company (NRK), I heard the journalist
146 claim that
'the FBI denies any wiretapping
', while the reality is that
147 'the FBI denies any illegal wiretapping
'. There is a fundamental and
148 important difference, and it make me sad that the journalists are
149 unable to grasp it.
</p
>
151 <p
><strong
>Update
2017-
03-
13:
</strong
> Look like
152 <a href=
"https://theintercept.com/
2017/
03/
13/rand-paul-is-right-nsa-routinely-monitors-americans-communications-without-warrants/
">The
153 Intercept report that US Senator Rand Paul confirm what I state above
</a
>.
</p
>
158 <title>Norwegian Bokmål translation of The Debian Administrator
's Handbook complete, proofreading in progress
</title>
159 <link>http://people.skolelinux.org/pere/blog/Norwegian_Bokm_l_translation_of_The_Debian_Administrator_s_Handbook_complete__proofreading_in_progress.html
</link>
160 <guid isPermaLink=
"true">http://people.skolelinux.org/pere/blog/Norwegian_Bokm_l_translation_of_The_Debian_Administrator_s_Handbook_complete__proofreading_in_progress.html
</guid>
161 <pubDate>Fri,
3 Mar
2017 14:
50:
00 +
0100</pubDate>
162 <description><p
>For almost a year now, we have been working on making a Norwegian
163 Bokmål edition of
<a href=
"https://debian-handbook.info/
">The Debian
164 Administrator
's Handbook
</a
>. Now, thanks to the tireless effort of
165 Ole-Erik, Ingrid and Andreas, the initial translation is complete, and
166 we are working on the proof reading to ensure consistent language and
167 use of correct computer science terms. The plan is to make the book
168 available on paper, as well as in electronic form. For that to
169 happen, the proof reading must be completed and all the figures need
170 to be translated. If you want to help out, get in touch.
</p
>
172 <p
><a href=
"http://people.skolelinux.org/pere/debian-handbook/debian-handbook-nb-NO.pdf
">A
174 fresh PDF edition
</a
> in A4 format (the final book will have smaller
175 pages) of the book created every morning is available for
176 proofreading. If you find any errors, please
177 <a href=
"https://hosted.weblate.org/projects/debian-handbook/
">visit
178 Weblate and correct the error
</a
>. The
179 <a href=
"http://l.github.io/debian-handbook/stat/nb-NO/index.html
">state
180 of the translation including figures
</a
> is a useful source for those
181 provide Norwegian bokmål screen shots and figures.
</p
>
186 <title>Unlimited randomness with the ChaosKey?
</title>
187 <link>http://people.skolelinux.org/pere/blog/Unlimited_randomness_with_the_ChaosKey_.html
</link>
188 <guid isPermaLink=
"true">http://people.skolelinux.org/pere/blog/Unlimited_randomness_with_the_ChaosKey_.html
</guid>
189 <pubDate>Wed,
1 Mar
2017 20:
50:
00 +
0100</pubDate>
190 <description><p
>A few days ago I ordered a small batch of
191 <a href=
"http://altusmetrum.org/ChaosKey/
">the ChaosKey
</a
>, a small
192 USB dongle for generating entropy created by Bdale Garbee and Keith
193 Packard. Yesterday it arrived, and I am very happy to report that it
194 work great! According to its designers, to get it to work out of the
195 box, you need the Linux kernel version
4.1 or later. I tested on a
196 Debian Stretch machine (kernel version
4.9), and there it worked just
197 fine, increasing the available entropy very quickly. I wrote a small
198 test oneliner to test. It first print the current entropy level,
199 drain /dev/random, and then print the entropy level for five seconds.
200 Here is the situation without the ChaosKey inserted:
</p
>
202 <blockquote
><pre
>
203 % cat /proc/sys/kernel/random/entropy_avail; \
204 dd bs=
1M if=/dev/random of=/dev/null count=
1; \
205 for n in $(seq
1 5); do \
206 cat /proc/sys/kernel/random/entropy_avail; \
212 28 byte kopiert,
0,
000264565 s,
106 kB/s
219 </pre
></blockquote
>
221 <p
>The entropy level increases by
3-
4 every second. In such case any
222 application requiring random bits (like a HTTPS enabled web server)
223 will halt and wait for more entrpy. And here is the situation with
224 the ChaosKey inserted:
</p
>
226 <blockquote
><pre
>
227 % cat /proc/sys/kernel/random/entropy_avail; \
228 dd bs=
1M if=/dev/random of=/dev/null count=
1; \
229 for n in $(seq
1 5); do \
230 cat /proc/sys/kernel/random/entropy_avail; \
236 104 byte kopiert,
0,
000487647 s,
213 kB/s
243 </pre
></blockquote
>
245 <p
>Quite the difference. :) I bought a few more than I need, in case
246 someone want to buy one here in Norway. :)
</p
>
248 <p
>Update: The dongle was presented at Debconf last year. You might
249 find
<a href=
"https://debconf16.debconf.org/talks/
94/
">the talk
250 recording illuminating
</a
>. It explains exactly what the source of
251 randomness is, if you are unable to spot it from the schema drawing
252 available from the ChaosKey web site linked at the start of this blog
projects / homepage.git / blob