blog/archive/2024/03/03.rss

   1 <?xml version="1.0" encoding="ISO-8859-1"?>
   2 <rss version='2.0' xmlns:lj='http://www.livejournal.org/rss/lj/1.0/'>
   3         <channel>
   4                 <title>Petter Reinholdtsen - Entries from March 2024</title>
   5                 <description>Entries from March 2024</description>
   6                 <link>http://www.hungry.com/~pere/blog/</link>
   7
   8
   9         <item>
  10                 <title>Plain text accounting file from your bitcoin transactions</title>
  11                 <link>http://www.hungry.com/~pere/blog/Plain_text_accounting_file_from_your_bitcoin_transactions.html</link>
  12                 <guid isPermaLink="true">http://www.hungry.com/~pere/blog/Plain_text_accounting_file_from_your_bitcoin_transactions.html</guid>
  13                 <pubDate>Thu, 7 Mar 2024 18:00:00 +0100</pubDate>
  14                 <description>&lt;p&gt;A while back I wrote a small script to extract the Bitcoin
  15 transactions in a wallet in the
  16 &lt;ahref=&quot;https://plaintextaccounting.org/&quot;&gt;ledger plain text accounting
  17 format&lt;/a&gt;.  The last few days I spent some time to get it working
  18 better with more special cases.  In case it can be useful for others,
  19 here is a copy:&lt;/p&gt;
  20
  21 &lt;p&gt;&lt;blockquote&gt;&lt;pre&gt;
  22 #!/usr/bin/python3
  23 #  -*- coding: utf-8 -*-
  24 #  Copyright (c) 2023-2024 Petter Reinholdtsen
  25
  26 from decimal import Decimal
  27 import json
  28 import subprocess
  29 import time
  30
  31 import numpy
  32
  33 def format_float(num):
  34     return numpy.format_float_positional(num, trim=&#39;-&#39;)
  35
  36 accounts = {
  37     u&#39;amount&#39; : &#39;Assets:BTC:main&#39;,
  38 }
  39
  40 addresses = {
  41     &#39;&lt;some address&gt;&#39; : &#39;Assets:bankkonto&#39;,
  42     &#39;&lt;some address&gt;&#39; : &#39;Assets:bankkonto&#39;,
  43 }
  44
  45 def exec_json(cmd):
  46     proc = subprocess.Popen(cmd,stdout=subprocess.PIPE)
  47     j = json.loads(proc.communicate()[0], parse_float=Decimal)
  48     return j
  49
  50 def list_txs():
  51     # get all transactions for all accounts / addresses
  52     c = 0
  53     txs = []
  54     txidfee = {}
  55     limit=100000
  56     cmd = [&#39;bitcoin-cli&#39;, &#39;listtransactions&#39;, &#39;*&#39;, str(limit)]
  57     if True:
  58         txs.extend(exec_json(cmd))
  59     else:
  60         # Useful for debugging
  61         with open(&#39;transactions.json&#39;) as f:
  62             txs.extend(json.load(f, parse_float=Decimal))
  63     #print txs
  64     for tx in sorted(txs, key=lambda a: a[&#39;time&#39;]):
  65 #        print tx[&#39;category&#39;]
  66         if &#39;abandoned&#39; in tx and tx[&#39;abandoned&#39;]:
  67             continue
  68         if &#39;confirmations&#39; in tx and 0 &gt;= tx[&#39;confirmations&#39;]:
  69             continue
  70         when = time.strftime(&#39;%Y-%m-%d %H:%M&#39;, time.localtime(tx[&#39;time&#39;]))
  71         if &#39;message&#39; in tx:
  72             desc = tx[&#39;message&#39;]
  73         elif &#39;comment&#39; in tx:
  74             desc = tx[&#39;comment&#39;]
  75         elif &#39;label&#39; in tx:
  76             desc = tx[&#39;label&#39;]
  77         else:
  78             desc = &#39;n/a&#39;
  79         print(&quot;%s %s&quot; % (when, desc))
  80         if &#39;address&#39; in tx:
  81             print(&quot;  ; to bitcoin address %s&quot; % tx[&#39;address&#39;])
  82         else:
  83             print(&quot;  ; missing address in transaction, txid=%s&quot; % tx[&#39;txid&#39;])
  84         print(f&quot;  ; amount={tx[&#39;amount&#39;]}&quot;)
  85         if &#39;fee&#39;in tx:
  86             print(f&quot;  ; fee={tx[&#39;fee&#39;]}&quot;)
  87         for f in accounts.keys():
  88             if f in tx and Decimal(0) != tx[f]:
  89                 amount = tx[f]
  90                 print(&quot;  %-20s   %s BTC&quot; % (accounts[f], format_float(amount)))
  91         if &#39;fee&#39; in tx and Decimal(0) != tx[&#39;fee&#39;]:
  92             # Make sure to list fee used in several transactions only once.
  93             if &#39;fee&#39; in tx and tx[&#39;txid&#39;] in txidfee \
  94                and tx[&#39;fee&#39;] == txidfee[tx[&#39;txid&#39;]]:
  95                 True
  96             else:
  97                 fee = tx[&#39;fee&#39;]
  98                 print(&quot;  %-20s   %s BTC&quot; % (accounts[&#39;amount&#39;], format_float(fee)))
  99                 print(&quot;  %-20s   %s BTC&quot; % (&#39;Expences:BTC-fee&#39;, format_float(-fee)))
 100                 txidfee[tx[&#39;txid&#39;]] = tx[&#39;fee&#39;]
 101
 102         if &#39;address&#39; in tx and tx[&#39;address&#39;] in addresses:
 103             print(&quot;  %s&quot; % addresses[tx[&#39;address&#39;]])
 104         else:
 105             if &#39;generate&#39; == tx[&#39;category&#39;]:
 106                 print(&quot;  Income:BTC-mining&quot;)
 107             else:
 108                 if amount &lt; Decimal(0):
 109                     print(f&quot;  Assets:unknown:sent:update-script-addr-{tx[&#39;address&#39;]}&quot;)
 110                 else:
 111                     print(f&quot;  Assets:unknown:received:update-script-addr-{tx[&#39;address&#39;]}&quot;)
 112
 113         print()
 114         c = c + 1
 115     print(&quot;# Found %d transactions&quot; % c)
 116     if limit == c:
 117         print(f&quot;# Warning: Limit {limit} reached, consider increasing limit.&quot;)
 118
 119 def main():
 120     list_txs()
 121
 122 main()
 123 &lt;/pre&gt;&lt;/blockquote&gt;&lt;/p&gt;
 124
 125 &lt;p&gt;It is more of a proof of concept, and I do not expect it to handle
 126 all edge cases, but it worked for me, and perhaps you can find it
 127 useful too.&lt;/p&gt;
 128
 129 &lt;p&gt;To get a more interesting result, it is useful to map accounts sent
 130 to or received from to accounting accounts, using the
 131 &lt;tt&gt;addresses&lt;/tt&gt; hash.  As these will be very context dependent, I
 132 leave out my list to allow each user to fill out their own list of
 133 accounts.  Out of the box, &#39;ledger reg BTC:main&#39; should be able to
 134 show the amount of BTCs present in the wallet at any given time in the
 135 past.  For other and more valuable analysis, a account plan need to be
 136 set up in the &lt;tt&gt;addresses&lt;/tt&gt; hash.  Here is an example
 137 transaction:&lt;/p&gt;
 138
 139 &lt;p&gt;&lt;blockquote&gt;&lt;pre&gt;
 140 2024-03-07 17:00 Donated to good cause
 141     Assets:BTC:main                           -0.1 BTC
 142     Assets:BTC:main                       -0.00001 BTC
 143     Expences:BTC-fee                       0.00001 BTC
 144     Expences:donations                         0.1 BTC
 145 &lt;/pre&gt;&lt;/blockquote&gt;&lt;/p&gt;
 146
 147 &lt;p&gt;It need a running Bitcoin Core daemon running, as it connect to it
 148 using &lt;tt&gt;bitcoin-cli listtransactions * 100000&lt;/tt&gt; to extract the
 149 transactions listed in the Wallet.&lt;/p&gt;
 150
 151 &lt;p&gt;As usual, if you use Bitcoin and want to show your support of my
 152 activities, please send Bitcoin donations to my address
 153 &lt;b&gt;&lt;a href=&quot;bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&quot;&gt;15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&lt;/a&gt;&lt;/b&gt;.&lt;/p&gt;
 154 </description>
 155         </item>
 156
 157         <item>
 158                 <title>RAID status from LSI Megaraid controllers using free software</title>
 159                 <link>http://www.hungry.com/~pere/blog/RAID_status_from_LSI_Megaraid_controllers_using_free_software.html</link>
 160                 <guid isPermaLink="true">http://www.hungry.com/~pere/blog/RAID_status_from_LSI_Megaraid_controllers_using_free_software.html</guid>
 161                 <pubDate>Sun, 3 Mar 2024 22:40:00 +0100</pubDate>
 162                 <description>&lt;p&gt;The last few days I have revisited RAID setup using the LSI
 163 Megaraid controller.  These are a family of controllers called PERC by
 164 Dell, and is present in several old PowerEdge servers, and I recently
 165 got my hands on one of these.  I had forgotten how to handle this RAID
 166 controller in Debian, so I had to take a peek in the
 167 &lt;a href=&quot;https://wiki.debian.org/LinuxRaidForAdmins&quot;&gt;Debian wiki page
 168 &quot;Linux and Hardware RAID: an administrator&#39;s summary&quot;&lt;/a&gt; to remember
 169 what kind of software is available to configure and monitor the disks
 170 and controller.  I prefer Free Software alternatives to proprietary
 171 tools, as the later tend to fall into disarray once the manufacturer
 172 loose interest, and often do not work with newer Linux Distributions.
 173 Sadly there is no free software tool to configure the RAID setup, only
 174 to monitor it. RAID can provide improved reliability and resilience in
 175 a storage solution, but only if it is being regularly checked and any
 176 broken disks are being replaced in time.  I thus want to ensure some
 177 automatic monitoring is available.&lt;/p&gt;
 178
 179 &lt;p&gt;In the discovery process, I came across a old free software tool to
 180 monitor PERC2, PERC3, PERC4 and PERC5 controllers, which to my
 181 surprise is not present in debian.  To help change that I created a
 182 &lt;a href=&quot;https://bugs.debian.org/1065322&quot;&gt;request for packaging of the
 183 megactl package&lt;/a&gt;, and tried to track down a usable version.
 184 &lt;a href=&quot;https://sourceforge.net/p/megactl/&quot;&gt;The original project
 185 site&lt;/a&gt; is on Sourceforge, but as far as I can tell that project has
 186 been dead for more than 15 years.  I managed to find a
 187 &lt;a href=&quot;https://github.com/hmage/megactl&quot;&gt;more recent fork on
 188 github&lt;/a&gt; from user hmage, but it is unclear to me if this is still
 189 being maintained.  It has not seen much improvements since 2016.  A
 190 &lt;a href=&quot;https://github.com/namiltd/megactl&quot;&gt;more up to date
 191 edition&lt;/a&gt; is a git fork from the original github fork by user
 192 namiltd, and this newer fork seem a lot more promising.  The owner of
 193 this github repository has replied to change proposals within hours,
 194 and had already added some improvements and support for more hardware.
 195 Sadly he is reluctant to commit to maintaining the tool and stated in
 196 &lt;a href=&quot;https://github.com/namiltd/megactl/pull/1&quot;&gt;my first pull
 197 request&lt;/A&gt; that he think a new release should be made based on the
 198 git repository owned by hmage.  I perfectly understand this
 199 reluctance, as I feel the same about maintaining yet another package
 200 in Debian when I barely have time to take care of the ones I already
 201 maintain, but do not really have high hopes that hmage will have time
 202 to spend on it and hope namiltd will change his mind.&lt;/p&gt;
 203
 204 &lt;p&gt;In any case, I created
 205 &lt;a href=&quot;https://salsa.debian.org/debian/megactl&quot;&gt;a draft package&lt;/a&gt;
 206 based on the namiltd edition and put it under the debian group on
 207 salsa.debian.org.  If you own a Dell PowerEdge server with one of the
 208 PERC controllers, or any other RAID controller using the megaraid or
 209 megaraid_sas Linux kernel modules, you might want to check it out.  If
 210 enough people are interested, perhaps the package will make it into
 211 the Debian archive.&lt;/p&gt;
 212
 213 &lt;p&gt;There are two tools provided, megactl for the megaraid Linux kernel
 214 module, and megasasctl for the megaraid_sas Linux kernel module. The
 215 simple output from the command on one of my machines look like this
 216 (yes, I know some of the disks have problems. :).&lt;/p&gt;
 217
 218 &lt;pre&gt;
 219 # megasasctl
 220 a0       PERC H730 Mini           encl:1 ldrv:2  batt:good
 221 a0d0       558GiB RAID 1   1x2  optimal
 222 a0d1      3067GiB RAID 0   1x11 optimal
 223 a0e32s0     558GiB  a0d0  online   errs: media:0  other:19
 224 a0e32s1     279GiB  a0d1  online
 225 a0e32s2     279GiB  a0d1  online
 226 a0e32s3     279GiB  a0d1  online
 227 a0e32s4     279GiB  a0d1  online
 228 a0e32s5     279GiB  a0d1  online
 229 a0e32s6     279GiB  a0d1  online
 230 a0e32s8     558GiB  a0d0  online   errs: media:0  other:17
 231 a0e32s9     279GiB  a0d1  online
 232 a0e32s10    279GiB  a0d1  online
 233 a0e32s11    279GiB  a0d1  online
 234 a0e32s12    279GiB  a0d1  online
 235 a0e32s13    279GiB  a0d1  online
 236
 237 #
 238 &lt;/pre&gt;
 239
 240 &lt;p&gt;In addition to displaying a simple status report, it can also test
 241 individual drives and print the various event logs.  Perhaps you too
 242 find it useful?&lt;/p&gt;
 243
 244 &lt;p&gt;In the packaging process I provided some patches upstream to
 245 improve installation and ensure
 246 &lt;ahref=&quot;https://github.com/namiltd/megactl/pull/2&quot;&gt;a Appstream
 247 metainfo file is provided&lt;/a&gt; to list all supported HW, to allow
 248 &lt;a href=&quot;https://tracker.debian.org/isenkram&quot;&gt;isenkram&lt;/a&gt; to propose
 249 the package on all servers with a relevant PCI card.&lt;/p&gt;
 250
 251 &lt;p&gt;As usual, if you use Bitcoin and want to show your support of my
 252 activities, please send Bitcoin donations to my address
 253 &lt;b&gt;&lt;a href=&quot;bitcoin:15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&quot;&gt;15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b&lt;/a&gt;&lt;/b&gt;.&lt;/p&gt;
 254
 255 </description>
 256         </item>
 257
 258         </channel>
 259 </rss>