- <title>Over et halvt år før offentlig korrespondanse journalføres - statistikk for 2014</title>
- <link>http://people.skolelinux.org/pere/blog/Over_et_halvt__r_f_r_offentlig_korrespondanse_journalf_res___statistikk_for_2014.html</link>
- <guid isPermaLink="true">http://people.skolelinux.org/pere/blog/Over_et_halvt__r_f_r_offentlig_korrespondanse_journalf_res___statistikk_for_2014.html</guid>
- <pubDate>Fri, 16 Jan 2015 13:10:00 +0100</pubDate>
- <description><p>Den norske tjenesten <a href="https://www.oep.no/">Offentlig
-Elektronisk Postjournal</a> er en samlet webtjeneste som gir borgerne
-innsyn i hva de ulike statlige aktørene har av post inn og ut. Jeg
-har fulgt med på den en stund, og ville dele litt statistikk med mine
-lesere om hvor flinke ulike instanser er til å holde postjournalen
-oppdatert.</p>
-
-<p>Denne oversikt viser hvor stor andel av dokumentene journalført i
-2014 som har mer enn 180 dagers differanse mellom dokumentdato og
-journalføringsdato (liggetid). Dvs. hvor stor andel dokumenter som
-blir journalført veldig sent. Et eksempel på en slik journalføring er
-et <a href="https://oep.no/search/resultSingle.html?journalPostId=10232315">261
-år gammelt hemmeligstemplet dokument</a> ført i postjournalen til
-Forsvarsdepartementet 2014-04-23.</p>
-
-<table border="1">
-<tr> <th>prosent</th> <th># > 180d liggetid</th> <th>Totalt</th> <th>Instans</th> </tr>
-<tr><td>0.0</td> <td>0</td> <td>49478</td> <td>Fylkesmannen i Oslo og Akershus</td></tr>
-<tr><td>0.1</td> <td>4</td> <td>5019</td> <td>Norsk kulturråd</td></tr>
-<tr><td>0.1</td> <td>19</td> <td>20433</td> <td>Fylkesmannen i Østfold</td></tr>
-<tr><td>0.1</td> <td>5</td> <td>3484</td> <td>Medietilsynet</td></tr>
-<tr><td>0.1</td> <td>25</td> <td>17289</td> <td>Statens landbruksforvaltning</td></tr>
-<tr><td>0.2</td> <td>41</td> <td>23638</td> <td>Lotteri- og stiftelsestilsynet</td></tr>
-<tr><td>0.2</td> <td>49</td> <td>26367</td> <td>Direktoratet for byggkvalitet</td></tr>
-<tr><td>0.2</td> <td>39</td> <td>20950</td> <td>Finansdepartementet</td></tr>
-<tr><td>0.2</td> <td>28</td> <td>12848</td> <td>Statens jernbanetilsyn</td></tr>
-<tr><td>0.2</td> <td>25</td> <td>10643</td> <td>Statsministerens kontor</td></tr>
-<tr><td>0.3</td> <td>12</td> <td>4515</td> <td>Nasjonalt organ for kvalitet i utdanningen</td></tr>
-<tr><td>0.3</td> <td>204</td> <td>67600</td> <td>Landbruksdirektoratet Oslo</td></tr>
-<tr><td>0.3</td> <td>16</td> <td>4645</td> <td>Konkurransetilsynet</td></tr>
-<tr><td>0.3</td> <td>21</td> <td>6072</td> <td>Datatilsynet</td></tr>
-<tr><td>0.4</td> <td>64</td> <td>17513</td> <td>Fylkesmannen i Sogn og Fjordane</td></tr>
-<tr><td>0.4</td> <td>40</td> <td>10290</td> <td>Petroleumstilsynet</td></tr>
-<tr><td>0.4</td> <td>77</td> <td>19398</td> <td>Politidirektoratet</td></tr>
-<tr><td>0.4</td> <td>76</td> <td>18714</td> <td>Fylkesmannen i Vest-Agder</td></tr>
-<tr><td>0.4</td> <td>100</td> <td>24068</td> <td>Helse- og omsorgsdepartementet</td></tr>
-<tr><td>0.4</td> <td>758</td> <td>170980</td> <td>Arbeidstilsynet</td></tr>
-<tr><td>0.5</td> <td>198</td> <td>43323</td> <td>Justervesenet</td></tr>
-<tr><td>0.5</td> <td>53</td> <td>10777</td> <td>Forbrukerombudet</td></tr>
-<tr><td>0.5</td> <td>195</td> <td>38701</td> <td>Luftfartstilsynet</td></tr>
-<tr><td>0.5</td> <td>96</td> <td>17972</td> <td>Brønnøysundregistrene</td></tr>
-<tr><td>0.5</td> <td>64</td> <td>11686</td> <td>Landbruks- og matdepartementet</td></tr>
-<tr><td>0.6</td> <td>43</td> <td>7703</td> <td>Direktoratet for mineralforvaltning med Bergmesteren for Svalbard</td></tr>
-<tr><td>0.6</td> <td>158</td> <td>27719</td> <td>Fylkesmannen i Nordland</td></tr>
-<tr><td>0.6</td> <td>80</td> <td>13824</td> <td>Olje- og energidepartementet</td></tr>
-<tr><td>0.6</td> <td>133</td> <td>22730</td> <td>Fylkesmannen i Vestfold</td></tr>
-<tr><td>0.6</td> <td>437</td> <td>74397</td> <td>Arkivverket</td></tr>
-<tr><td>0.6</td> <td>46</td> <td>7809</td> <td>Statens strålevern</td></tr>
-<tr><td>0.6</td> <td>73</td> <td>11812</td> <td>Oljedirektoratet</td></tr>
-<tr><td>0.7</td> <td>142</td> <td>21423</td> <td>Barne-, likestillings- og inkluderingsdepartementet</td></tr>
-<tr><td>0.7</td> <td>142</td> <td>20910</td> <td>Fylkesmannen i Finnmark</td></tr>
-<tr><td>0.7</td> <td>129</td> <td>18413</td> <td>Fylkesmannen i Nord-Trøndelag</td></tr>
-<tr><td>0.7</td> <td>164</td> <td>22554</td> <td>Samferdselsdepartementet</td></tr>
-<tr><td>0.7</td> <td>143</td> <td>19579</td> <td>Fylkesmannen i Møre og Romsdal</td></tr>
-<tr><td>0.8</td> <td>97</td> <td>12800</td> <td>Forsvarsdepartementet</td></tr>
-<tr><td>0.8</td> <td>588</td> <td>75542</td> <td>Fiskeridirektoratet</td></tr>
-<tr><td>0.8</td> <td>23</td> <td>2854</td> <td>Landbruksdirektoratet Alta</td></tr>
-<tr><td>0.8</td> <td>262</td> <td>32425</td> <td>Kommunal- og moderniseringsdepartementet</td></tr>
-<tr><td>0.8</td> <td>66</td> <td>7971</td> <td>Sysselmannen på Svalbard</td></tr>
-<tr><td>0.9</td> <td>1324</td> <td>151481</td> <td>Statens vegvesen Region øst</td></tr>
-<tr><td>0.9</td> <td>251</td> <td>28564</td> <td>Fylkesmannen i Troms</td></tr>
-<tr><td>0.9</td> <td>210</td> <td>23881</td> <td>Fylkesmannen i Buskerud</td></tr>
-<tr><td>0.9</td> <td>932</td> <td>105374</td> <td>Statens vegvesen Region vest</td></tr>
-<tr><td>0.9</td> <td>356</td> <td>39358</td> <td>Fylkesmannen i Hordaland</td></tr>
-<tr><td>0.9</td> <td>401</td> <td>43992</td> <td>Finanstilsynet</td></tr>
-<tr><td>0.9</td> <td>154</td> <td>16735</td> <td>Fylkesmannen i Hedmark</td></tr>
-<tr><td>1.1</td> <td>16</td> <td>1516</td> <td>Statens arbeidsmiljøinstitutt</td></tr>
-<tr><td>1.1</td> <td>227</td> <td>21506</td> <td>Kulturdepartementet</td></tr>
-<tr><td>1.1</td> <td>110</td> <td>10397</td> <td>Toll- og avgiftsdirektoratet</td></tr>
-<tr><td>1.1</td> <td>23</td> <td>2164</td> <td>Kunst i offentlige rom</td></tr>
-<tr><td>1.1</td> <td>100</td> <td>9097</td> <td>Havforskningsinstituttet</td></tr>
-<tr><td>1.3</td> <td>790</td> <td>61404</td> <td>Statens vegvesen Region nord</td></tr>
-<tr><td>1.3</td> <td>323</td> <td>25006</td> <td>Kunnskapsdepartementet</td></tr>
-<tr><td>1.3</td> <td>304</td> <td>23402</td> <td>Arbeids- og sosialdepartementet</td></tr>
-<tr><td>1.3</td> <td>966</td> <td>74177</td> <td>Statens vegvesen Region midt</td></tr>
-<tr><td>1.3</td> <td>194</td> <td>14483</td> <td>Fylkesmannen i Aust-Agder</td></tr>
-<tr><td>1.4</td> <td>144</td> <td>10223</td> <td>Vox, nasjonalt fagorgan for kompetansepolitikk</td></tr>
-<tr><td>1.5</td> <td>557</td> <td>37674</td> <td>Nærings- og fiskeridepartementet</td></tr>
-<tr><td>1.5</td> <td>3</td> <td>200</td> <td>Kompetansesenter for distriktsutvikling</td></tr>
-<tr><td>1.7</td> <td>837</td> <td>50535</td> <td>Norges vassdrags- og energidirektorat</td></tr>
-<tr><td>1.7</td> <td>335</td> <td>20151</td> <td>Norsk Filminstitutt</td></tr>
-<tr><td>1.7</td> <td>123</td> <td>7395</td> <td>Domstoladministrasjonen</td></tr>
-<tr><td>1.7</td> <td>1894</td> <td>110353</td> <td>Statens vegvesen Region sør</td></tr>
-<tr><td>1.8</td> <td>75</td> <td>4283</td> <td>Kriminalomsorgsdirektoratet</td></tr>
-<tr><td>1.8</td> <td>160</td> <td>8695</td> <td>Statens lånekasse for utdanning</td></tr>
-<tr><td>1.9</td> <td>329</td> <td>17370</td> <td>Fylkesmannen i Telemark</td></tr>
-<tr><td>2.1</td> <td>242</td> <td>11367</td> <td>Barne-, ungdoms- og familiedirektoratet</td></tr>
-<tr><td>2.1</td> <td>13</td> <td>610</td> <td>Statens reindriftsforvaltning</td></tr>
-<tr><td>2.5</td> <td>709</td> <td>28625</td> <td>Post- og teletilsynet</td></tr>
-<tr><td>2.5</td> <td>67</td> <td>2653</td> <td>Norges geologiske undersøkelse</td></tr>
-<tr><td>2.7</td> <td>1360</td> <td>50974</td> <td>Miljødirektoratet</td></tr>
-<tr><td>3.0</td> <td>1123</td> <td>37790</td> <td>Statens vegvesen Vegdirektoratet</td></tr>
-<tr><td>3.0</td> <td>628</td> <td>20712</td> <td>Riksantikvaren - Direktoratet for kulturminneforvaltning</td></tr>
-<tr><td>3.0</td> <td>1449</td> <td>47640</td> <td>Justis- og beredskapsdepartementet</td></tr>
-<tr><td>3.1</td> <td>468</td> <td>14949</td> <td>Statistisk sentralbyrå</td></tr>
-<tr><td>3.3</td> <td>182</td> <td>5579</td> <td>Direktoratet for økonomistyring</td></tr>
-<tr><td>3.3</td> <td>1233</td> <td>37188</td> <td>Kystverket</td></tr>
-<tr><td>3.5</td> <td>2115</td> <td>60895</td> <td>Direktoratet for samfunnssikkerhet og beredskap</td></tr>
-<tr><td>3.5</td> <td>22</td> <td>620</td> <td>Statens institutt for rusmiddelforskning</td></tr>
-<tr><td>3.7</td> <td>149</td> <td>4082</td> <td>Språkrådet</td></tr>
-<tr><td>3.6</td> <td>442</td> <td>11783</td> <td>Garantiinstituttet for eksportkreditt</td></tr>
-<tr><td>3.8</td> <td>2872</td> <td>75955</td> <td>Statens legemiddelverk</td></tr>
-<tr><td>3.8</td> <td>675</td> <td>17777</td> <td>Statens helsetilsyn</td></tr>
-<tr><td>3.8</td> <td>151</td> <td>3949</td> <td>Statens pensjonskasse</td></tr>
-<tr><td>3.8</td> <td>513</td> <td>13414</td> <td>Nasjonalt folkehelseinstitutt</td></tr>
-<tr><td>4.0</td> <td>746</td> <td>18754</td> <td>Klima- og miljødepartementet</td></tr>
-<tr><td>4.1</td> <td>158</td> <td>3883</td> <td>Nasjonalbiblioteket</td></tr>
-<tr><td>4.2</td> <td>259</td> <td>6146</td> <td>Utlendingsnemnda</td></tr>
-<tr><td>4.3</td> <td>300</td> <td>7058</td> <td>Norsk Akkreditering</td></tr>
-<tr><td>4.4</td> <td>215</td> <td>4840</td> <td>Statens innkrevingssentral</td></tr>
-<tr><td>4.7</td> <td>1517</td> <td>32464</td> <td>Fylkesmannen i Rogaland</td></tr>
-<tr><td>4.7</td> <td>1725</td> <td>36626</td> <td>Forsvarsbygg</td></tr>
-<tr><td>4.8</td> <td>1739</td> <td>36585</td> <td>Statens kartverk</td></tr>
-<tr><td>4.8</td> <td>2685</td> <td>55388</td> <td>Utdanningsdirektoratet</td></tr>
-<tr><td>5.4</td> <td>1219</td> <td>22606</td> <td>Fylkesmannen i Sør-Trøndelag</td></tr>
-<tr><td>5.5</td> <td>90</td> <td>1648</td> <td>Fredskorpset</td></tr>
-<tr><td>5.9</td> <td>1572</td> <td>26854</td> <td>Direktoratet for utviklingssamarbeid</td></tr>
-<tr><td>5.9</td> <td>3827</td> <td>65102</td> <td>Utenriksdepartementet</td></tr>
-<tr><td>5.9</td> <td>5616</td> <td>94471</td> <td>Norges forskningsråd</td></tr>
-<tr><td>6.3</td> <td>475</td> <td>7553</td> <td>Departementenes sikkerhets- og serviceorganisasjon</td></tr>
-<tr><td>6.3</td> <td>114</td> <td>1798</td> <td>Departementenes servicesenter</td></tr>
-<tr><td>6.4</td> <td>769</td> <td>11993</td> <td>Senter for internasjonalisering av utdanning</td></tr>
-<tr><td>6.5</td> <td>8844</td> <td>137014</td> <td>Husbanken</td></tr>
-<tr><td>6.8</td> <td>1802</td> <td>26457</td> <td>Skattedirektoratet</td></tr>
-<tr><td>9.0</td> <td>765</td> <td>8520</td> <td>Direktoratet for nødkommunikasjon</td></tr>
-<tr><td>9.5</td> <td>1643</td> <td>17343</td> <td>Integrerings- og mangfoldsdirektoratet</td></tr>
-<tr><td>11.1</td> <td>1686</td> <td>15122</td> <td>Utlendingsdirektoratet</td></tr>
-<tr><td>11.8</td> <td>2824</td> <td>23941</td> <td>Fylkesmannen i Oppland</td></tr>
-<tr><td>12.1</td> <td>1486</td> <td>12300</td> <td>Direktoratet for forvaltning og IKT</td></tr>
-<tr><td>12.5</td> <td>7998</td> <td>64163</td> <td>Sjøfartsdirektoratet</td></tr>
-<tr><td>14.4</td> <td>12640</td> <td>87574</td> <td>Helsedirektoratet</td></tr>
-<tr><td>15.2</td> <td>7002</td> <td>45956</td> <td>Statsbygg</td></tr>
-<tr><td>18.8</td> <td>61</td> <td>324</td> <td>Reindriftsforvaltningen</td></tr>
-<tr><td>20.7</td> <td>236</td> <td>1140</td> <td>Landinfo
-</table>
-
-<p>Som en ser fra tabellen, så ser de fleste ut til å journalføre det
-meste ganske raskt, men noen har et betydelig antall dokumenter som
-journalføres lenge etter at de er laget/sendt/mottatt. Jeg vet ikke
-hva det kommer av, men antar at det finnes gode rasjonelle grunner for
-å journalføre gamle dokumenter. Jeg er dog er overrasket over at
-andelen er så stor i enkelte virksomheter. Kanskje noen burde se
-nærmere på om det kan bli bedre?</p>
+ <title>Detecting NFS hangs on Linux without hanging yourself...</title>
+ <link>http://people.skolelinux.org/pere/blog/Detecting_NFS_hangs_on_Linux_without_hanging_yourself___.html</link>
+ <guid isPermaLink="true">http://people.skolelinux.org/pere/blog/Detecting_NFS_hangs_on_Linux_without_hanging_yourself___.html</guid>
+ <pubDate>Thu, 9 Mar 2017 15:20:00 +0100</pubDate>
+ <description><p>Over the years, administrating thousand of NFS mounting linux
+computers at the time, I often needed a way to detect if the machine
+was experiencing NFS hang. If you try to use <tt>df</tt> or look at a
+file or directory affected by the hang, the process (and possibly the
+shell) will hang too. So you want to be able to detect this without
+risking the detection process getting stuck too. It has not been
+obvious how to do this. When the hang has lasted a while, it is
+possible to find messages like these in dmesg:</p>
+
+<p><blockquote>
+nfs: server nfsserver not responding, still trying
+<br>nfs: server nfsserver OK
+</blockquote></p>
+
+<p>It is hard to know if the hang is still going on, and it is hard to
+be sure looking in dmesg is going to work. If there are lots of other
+messages in dmesg the lines might have rotated out of site before they
+are noticed.</p>
+
+<p>While reading through the nfs client implementation in linux kernel
+code, I came across some statistics that seem to give a way to detect
+it. The om_timeouts sunrpc value in the kernel will increase every
+time the above log entry is inserted into dmesg. And after digging a
+bit further, I discovered that this value show up in
+/proc/self/mountstats on Linux.</p>
+
+<p>The mountstats content seem to be shared between files using the
+same file system context, so it is enough to check one of the
+mountstats files to get the state of the mount point for the machine.
+I assume this will not show lazy umounted NFS points, nor NFS mount
+points in a different process context (ie with a different filesystem
+view), but that does not worry me.</p>
+
+<p>The content for a NFS mount point look similar to this:</p>
+
+<p><blockquote><pre>
+[...]
+device /dev/mapper/Debian-var mounted on /var with fstype ext3
+device nfsserver:/mnt/nfsserver/home0 mounted on /mnt/nfsserver/home0 with fstype nfs statvers=1.1
+ opts: rw,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,soft,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=129.240.3.145,mountvers=3,mountport=4048,mountproto=udp,local_lock=all
+ age: 7863311
+ caps: caps=0x3fe7,wtmult=4096,dtsize=8192,bsize=0,namlen=255
+ sec: flavor=1,pseudoflavor=1
+ events: 61063112 732346265 1028140 35486205 16220064 8162542 761447191 71714012 37189 3891185 45561809 110486139 4850138 420353 15449177 296502 52736725 13523379 0 52182 9016896 1231 0 0 0 0 0
+ bytes: 166253035039 219519120027 0 0 40783504807 185466229638 11677877 45561809
+ RPC iostats version: 1.0 p/v: 100003/3 (nfs)
+ xprt: tcp 925 1 6810 0 0 111505412 111480497 109 2672418560317 0 248 53869103 22481820
+ per-op statistics
+ NULL: 0 0 0 0 0 0 0 0
+ GETATTR: 61063106 61063108 0 9621383060 6839064400 453650 77291321 78926132
+ SETATTR: 463469 463470 0 92005440 66739536 63787 603235 687943
+ LOOKUP: 17021657 17021657 0 3354097764 4013442928 57216 35125459 35566511
+ ACCESS: 14281703 14290009 5 2318400592 1713803640 1709282 4865144 7130140
+ READLINK: 125 125 0 20472 18620 0 1112 1118
+ READ: 4214236 4214237 0 715608524 41328653212 89884 22622768 22806693
+ WRITE: 8479010 8494376 22 187695798568 1356087148 178264904 51506907 231671771
+ CREATE: 171708 171708 0 38084748 46702272 873 1041833 1050398
+ MKDIR: 3680 3680 0 773980 993920 26 23990 24245
+ SYMLINK: 903 903 0 233428 245488 6 5865 5917
+ MKNOD: 80 80 0 20148 21760 0 299 304
+ REMOVE: 429921 429921 0 79796004 61908192 3313 2710416 2741636
+ RMDIR: 3367 3367 0 645112 484848 22 5782 6002
+ RENAME: 466201 466201 0 130026184 121212260 7075 5935207 5961288
+ LINK: 289155 289155 0 72775556 67083960 2199 2565060 2585579
+ READDIR: 2933237 2933237 0 516506204 13973833412 10385 3190199 3297917
+ READDIRPLUS: 1652839 1652839 0 298640972 6895997744 84735 14307895 14448937
+ FSSTAT: 6144 6144 0 1010516 1032192 51 9654 10022
+ FSINFO: 2 2 0 232 328 0 1 1
+ PATHCONF: 1 1 0 116 140 0 0 0
+ COMMIT: 0 0 0 0 0 0 0 0
+
+device binfmt_misc mounted on /proc/sys/fs/binfmt_misc with fstype binfmt_misc
+[...]
+</pre></blockquote></p>
+
+<p>The key number to look at is the third number in the per-op list.
+It is the number of NFS timeouts experiences per file system
+operation. Here 22 write timeouts and 5 access timeouts. If these
+numbers are increasing, I believe the machine is experiencing NFS
+hang. Unfortunately the timeout value do not start to increase right
+away. The NFS operations need to time out first, and this can take a
+while. The exact timeout value depend on the setup. For example the
+defaults for TCP and UDP mount points are quite different, and the
+timeout value is affected by the soft, hard, timeo and retrans NFS
+mount options.</p>
+
+<p>The only way I have been able to get working on Debian and RedHat
+Enterprise Linux for getting the timeout count is to peek in /proc/.
+But according to
+<ahref="http://docs.oracle.com/cd/E19253-01/816-4555/netmonitor-12/index.html">Solaris
+10 System Administration Guide: Network Services</a>, the 'nfsstat -c'
+command can be used to get these timeout values. But this do not work
+on Linux, as far as I can tell. I
+<ahref="http://bugs.debian.org/857043">asked Debian about this</a>,
+but have not seen any replies yet.</p>
+
+<p>Is there a better way to figure out if a Linux NFS client is
+experiencing NFS hangs? Is there a way to detect which processes are
+affected? Is there a way to get the NFS mount going quickly once the
+network problem causing the NFS hang has been cleared? I would very
+much welcome some clues, as we regularly run into NFS hangs.</p>