- <div class="title"><a href="http://people.skolelinux.org/pere/blog/Where_did_that_package_go___mdash__geolocated_IP_traceroute.html">Where did that package go? — geolocated IP traceroute</a></div>
- <div class="date"> 9th January 2017</div>
- <div class="body"><p>Did you ever wonder where the web trafic really flow to reach the
-web servers, and who own the network equipment it is flowing through?
-It is possible to get a glimpse of this from using traceroute, but it
-is hard to find all the details. Many years ago, I wrote a system to
-map the Norwegian Internet (trying to figure out if our plans for a
-network game service would get low enough latency, and who we needed
-to talk to about setting up game servers close to the users. Back
-then I used traceroute output from many locations (I asked my friends
-to run a script and send me their traceroute output) to create the
-graph and the map. The output from traceroute typically look like
-this:
-
-<p><pre>
-traceroute to www.stortinget.no (85.88.67.10), 30 hops max, 60 byte packets
- 1 uio-gw10.uio.no (129.240.202.1) 0.447 ms 0.486 ms 0.621 ms
- 2 uio-gw8.uio.no (129.240.24.229) 0.467 ms 0.578 ms 0.675 ms
- 3 oslo-gw1.uninett.no (128.39.65.17) 0.385 ms 0.373 ms 0.358 ms
- 4 te3-1-2.br1.fn3.as2116.net (193.156.90.3) 1.174 ms 1.172 ms 1.153 ms
- 5 he16-1-1.cr1.san110.as2116.net (195.0.244.234) 2.627 ms he16-1-1.cr2.oslosda310.as2116.net (195.0.244.48) 3.172 ms he16-1-1.cr1.san110.as2116.net (195.0.244.234) 2.857 ms
- 6 ae1.ar8.oslosda310.as2116.net (195.0.242.39) 0.662 ms 0.637 ms ae0.ar8.oslosda310.as2116.net (195.0.242.23) 0.622 ms
- 7 89.191.10.146 (89.191.10.146) 0.931 ms 0.917 ms 0.955 ms
- 8 * * *
- 9 * * *
-[...]
-</pre></p>
-
-<p>This show the DNS names and IP addresses of (at least some of the)
-network equipment involved in getting the data traffic from me to the
-www.stortinget.no server, and how long it took in milliseconds for a
-package to reach the equipment and return to me. Three packages are
-sent, and some times the packages do not follow the same path. This
-is shown for hop 5, where three different IP addresses replied to the
-traceroute request.</p>
-
-<p>There are many ways to measure trace routes. Other good traceroute
-implementations I use are traceroute (using ICMP packages) mtr (can do
-both ICMP, UDP and TCP) and scapy (python library with ICMP, UDP, TCP
-traceroute and a lot of other capabilities). All of them are easily
-available in <a href="https://www.debian.org/">Debian</a>.</p>
-
-<p>This time around, I wanted to know the geographic location of
-different route points, to visualize how visiting a web page spread
-information about the visit to a lot of servers around the globe. The
-background is that a web site today often will ask the browser to get
-from many servers the parts (for example HTML, JSON, fonts,
-JavaScript, CSS, video) required to display the content. This will
-leak information about the visit to those controlling these servers
-and anyone able to peek at the data traffic passing by (like your ISP,
-the ISPs backbone provider, FRA, GCHQ, NSA and others).</p>
-
-<p>Lets pick an example, the Norwegian parliament web site
-www.stortinget.no. It is read daily by all members of parliament and
-their staff, as well as political journalists, activits and many other
-citizens of Norway. A visit to the www.stortinget.no web site will
-ask your browser to contact 8 other servers: ajax.googleapis.com,
-insights.hotjar.com, script.hotjar.com, static.hotjar.com,
-stats.g.doubleclick.net, www.google-analytics.com,
-www.googletagmanager.com and www.netigate.se. I extracted this by
-asking <a href="http://phantomjs.org/">PhantomJS</a> to visit the
-Stortinget web page and tell me all the URLs PhantomJS downloaded to
-render the page (in HAR format using
-<a href="https://github.com/ariya/phantomjs/blob/master/examples/netsniff.js">their
-netsniff example</a>. I am very grateful to Gorm for showing me how
-to do this). My goal is to visualize network traces to all IP
-addresses behind these DNS names, do show where visitors personal
-information is spread when visiting the page.</p>
-
-<p align="center"><a href="www.stortinget.no-geoip.kml"><img
-src="http://people.skolelinux.org/pere/blog/images/2017-01-09-www.stortinget.no-geoip-small.png" alt="map of combined traces for URLs used by www.stortinget.no using GeoIP"/></a></p>
-
-<p>When I had a look around for options, I could not find any good
-free software tools to do this, and decided I needed my own traceroute
-wrapper outputting KML based on locations looked up using GeoIP. KML
-is easy to work with and easy to generate, and understood by several
-of the GIS tools I have available. I got good help from by NUUG
-colleague Anders Einar with this, and the result can be seen in
-<a href="https://github.com/petterreinholdtsen/kmltraceroute">my
-kmltraceroute git repository</a>. Unfortunately, the quality of the
-free GeoIP databases I could find (and the for-pay databases my
-friends had access to) is not up to the task. The IP addresses of
-central Internet infrastructure would typically be placed near the
-controlling companies main office, and not where the router is really
-located, as you can see from <a href="www.stortinget.no-geoip.kml">the
-KML file I created</a> using the GeoLite City dataset from MaxMind.
-
-<p align="center"><a href="http://people.skolelinux.org/pere/blog/images/2017-01-09-www.stortinget.no-scapy.svg"><img
-src="http://people.skolelinux.org/pere/blog/images/2017-01-09-www.stortinget.no-scapy-small.png" alt="scapy traceroute graph for URLs used by www.stortinget.no"/></a></p>
-
-<p>I also had a look at the visual traceroute graph created by
-<a href="http://www.secdev.org/projects/scapy/">the scrapy project</a>,
-showing IP network ownership (aka AS owner) for the IP address in
-question.
-<a href="http://people.skolelinux.org/pere/blog/images/2017-01-09-www.stortinget.no-scapy.svg">The
-graph display a lot of useful information about the traceroute in SVG
-format</a>, and give a good indication on who control the network
-equipment involved, but it do not include geolocation. This graph
-make it possible to see the information is made available at least for
-UNINETT, Catchcom, Stortinget, Nordunet, Google, Amazon, Telia, Level
-3 Communications and NetDNA.</p>
-
-<p align="center"><a href="https://geotraceroute.com/index.php?node=4&host=www.stortinget.no"><img
-src="http://people.skolelinux.org/pere/blog/images/2017-01-09-www.stortinget.no-geotraceroute-small.png" alt="example geotraceroute view for www.stortinget.no"/></a></p>
-
-<p>In the process, I came across the
-<a href="https://geotraceroute.com/">web service GeoTraceroute</a> by
-Salim Gasmi. Its methology of combining guesses based on DNS names,
-various location databases and finally use latecy times to rule out
-candidate locations seemed to do a very good job of guessing correct
-geolocation. But it could only do one trace at the time, did not have
-a sensor in Norway and did not make the geolocations easily available
-for postprocessing. So I contacted the developer and asked if he
-would be willing to share the code (he refused until he had time to
-clean it up), but he was interested in providing the geolocations in a
-machine readable format, and willing to set up a sensor in Norway. So
-since yesterday, it is possible to run traces from Norway in this
-service thanks to a sensor node set up by
-<a href="https://www.nuug.no/">the NUUG assosiation</a>, and get the
-trace in KML format for further processing.</p>
-
-<p align="center"><a href="http://people.skolelinux.org/pere/blog/images/2017-01-09-www.stortinget.no-geotraceroute-kml-join.kml"><img
-src="http://people.skolelinux.org/pere/blog/images/2017-01-09-www.stortinget.no-geotraceroute-kml-join.png" alt="map of combined traces for URLs used by www.stortinget.no using geotraceroute"/></a></p>
-
-<p>Here we can see a lot of trafic passes Sweden on its way to
-Denmark, Germany, Holland and Ireland. Plenty of places where the
-Snowden confirmations verified the traffic is read by various actors
-without your best interest as their top priority.</p>
-
-<p>Combining KML files is trivial using a text editor, so I could loop
-over all the hosts behind the urls imported by www.stortinget.no and
-ask for the KML file from GeoTraceroute, and create a combined KML
-file with all the traces (unfortunately only one of the IP addresses
-behind the DNS name is traced this time. To get them all, one would
-have to request traces using IP number instead of DNS names from
-GeoTraceroute). That might be the next step in this project.</p>
-
-<p>Armed with these tools, I find it a lot easier to figure out where
-the IP traffic moves and who control the boxes involved in moving it.
-And every time the link crosses for example the Swedish border, we can
-be sure Swedish Signal Intelligence (FRA) is listening, as GCHQ do in
-Britain and NSA in USA and cables around the globe. (Hm, what should
-we tell them? :) Keep that in mind if you ever send anything
-unencrypted over the Internet.</p>
-
-<p>PS: KML files are drawn using
-<a href="http://ivanrublev.me/kml/">the KML viewer from Ivan
-Rublev<a/>, as it was less cluttered than the local Linux application
-Marble. There are heaps of other options too.</p>
+ <div class="title"><a href="http://people.skolelinux.org/pere/blog/Valutakrambod___A_python_and_bitcoin_love_story.html">Valutakrambod - A python and bitcoin love story</a></div>
+ <div class="date">29th September 2018</div>
+ <div class="body"><p>It would come as no surprise to anyone that I am interested in
+bitcoins and virtual currencies. I've been keeping an eye on virtual
+currencies for many years, and it is part of the reason a few months
+ago, I started writing a python library for collecting currency
+exchange rates and trade on virtual currency exchanges. I decided to
+name the end result valutakrambod, which perhaps can be translated to
+small currency shop.</p>
+
+<p>The library uses the tornado python library to handle HTTP and
+websocket connections, and provide a asynchronous system for
+connecting to and tracking several services. The code is available
+from
+<a href="http://github.com/petterreinholdtsen/valutakrambod">github</a>.</p>
+
+</p>There are two example clients of the library. One is very simple and
+list every updated buy/sell price received from the various services.
+This code is started by running bin/btc-rates and call the client code
+in valutakrambod/client.py. The simple client look like this:</p>
+
+<p><blockquote><pre>
+import functools
+import tornado.ioloop
+import valutakrambod
+class SimpleClient(object):
+ def __init__(self):
+ self.services = []
+ self.streams = []
+ pass
+ def newdata(self, service, pair, changed):
+ print("%-15s %s-%s: %8.3f %8.3f" % (
+ service.servicename(),
+ pair[0],
+ pair[1],
+ service.rates[pair]['ask'],
+ service.rates[pair]['bid'])
+ )
+ async def refresh(self, service):
+ await service.fetchRates(service.wantedpairs)
+ def run(self):
+ self.ioloop = tornado.ioloop.IOLoop.current()
+ self.services = valutakrambod.service.knownServices()
+ for e in self.services:
+ service = e()
+ service.subscribe(self.newdata)
+ stream = service.websocket()
+ if stream:
+ self.streams.append(stream)
+ else:
+ # Fetch information from non-streaming services immediately
+ self.ioloop.call_later(len(self.services),
+ functools.partial(self.refresh, service))
+ # as well as regularly
+ service.periodicUpdate(60)
+ for stream in self.streams:
+ stream.connect()
+ try:
+ self.ioloop.start()
+ except KeyboardInterrupt:
+ print("Interrupted by keyboard, closing all connections.")
+ pass
+ for stream in self.streams:
+ stream.close()
+</pre></blockquote></p>
+
+<p>The library client loops over all known "public" services,
+initialises it, subscribes to any updates from the service, checks and
+activates websocket streaming if the service provide it, and if no
+streaming is supported, fetches information from the service and sets
+up a periodic update every 60 seconds. The output from this client
+can look like this:</p>
+
+<p><blockquote><pre>
+Bl3p BTC-EUR: 5687.110 5653.690
+Bl3p BTC-EUR: 5687.110 5653.690
+Bl3p BTC-EUR: 5687.110 5653.690
+Hitbtc BTC-USD: 6594.560 6593.690
+Hitbtc BTC-USD: 6594.560 6593.690
+Bl3p BTC-EUR: 5687.110 5653.690
+Hitbtc BTC-USD: 6594.570 6593.690
+Bitstamp EUR-USD: 1.159 1.154
+Hitbtc BTC-USD: 6594.570 6593.690
+Hitbtc BTC-USD: 6594.580 6593.690
+Hitbtc BTC-USD: 6594.580 6593.690
+Hitbtc BTC-USD: 6594.580 6593.690
+Bl3p BTC-EUR: 5687.110 5653.690
+Paymium BTC-EUR: 5680.000 5620.240
+</pre></blockquote></p>
+
+<p>The exchange order book is tracked in addition to the best buy/sell
+price, for those that need to know the details.</p>
+
+<p>The other example client is focusing on providing a curses view
+with updated buy/sell prices as soon as they are received from the
+services. This code is located in bin/btc-rates-curses and activated
+by using the '-c' argument. Without the argument the "curses" output
+is printed without using curses, which is useful for debugging. The
+curses view look like this:</p>
+
+<p><blockquote><pre>
+ Name Pair Bid Ask Spr Ftcd Age
+ BitcoinsNorway BTCEUR 5591.8400 5711.0800 2.1% 16 nan 60
+ Bitfinex BTCEUR 5671.0000 5671.2000 0.0% 16 22 59
+ Bitmynt BTCEUR 5580.8000 5807.5200 3.9% 16 41 60
+ Bitpay BTCEUR 5663.2700 nan nan% 15 nan 60
+ Bitstamp BTCEUR 5664.8400 5676.5300 0.2% 0 1 1
+ Bl3p BTCEUR 5653.6900 5684.9400 0.5% 0 nan 19
+ Coinbase BTCEUR 5600.8200 5714.9000 2.0% 15 nan nan
+ Kraken BTCEUR 5670.1000 5670.2000 0.0% 14 17 60
+ Paymium BTCEUR 5620.0600 5680.0000 1.1% 1 7515 nan
+ BitcoinsNorway BTCNOK 52898.9700 54034.6100 2.1% 16 nan 60
+ Bitmynt BTCNOK 52960.3200 54031.1900 2.0% 16 41 60
+ Bitpay BTCNOK 53477.7833 nan nan% 16 nan 60
+ Coinbase BTCNOK 52990.3500 54063.0600 2.0% 15 nan nan
+ MiraiEx BTCNOK 52856.5300 54100.6000 2.3% 16 nan nan
+ BitcoinsNorway BTCUSD 6495.5300 6631.5400 2.1% 16 nan 60
+ Bitfinex BTCUSD 6590.6000 6590.7000 0.0% 16 23 57
+ Bitpay BTCUSD 6564.1300 nan nan% 15 nan 60
+ Bitstamp BTCUSD 6561.1400 6565.6200 0.1% 0 2 1
+ Coinbase BTCUSD 6504.0600 6635.9700 2.0% 14 nan 117
+ Gemini BTCUSD 6567.1300 6573.0700 0.1% 16 89 nan
+ Hitbtc+BTCUSD 6592.6200 6594.2100 0.0% 0 0 0
+ Kraken BTCUSD 6565.2000 6570.9000 0.1% 15 17 58
+ Exchangerates EURNOK 9.4665 9.4665 0.0% 16 107789 nan
+ Norgesbank EURNOK 9.4665 9.4665 0.0% 16 107789 nan
+ Bitstamp EURUSD 1.1537 1.1593 0.5% 4 5 1
+ Exchangerates EURUSD 1.1576 1.1576 0.0% 16 107789 nan
+ BitcoinsNorway LTCEUR 1.0000 49.0000 98.0% 16 nan nan
+ BitcoinsNorway LTCNOK 492.4800 503.7500 2.2% 16 nan 60
+ BitcoinsNorway LTCUSD 1.0221 49.0000 97.9% 15 nan nan
+ Norgesbank USDNOK 8.1777 8.1777 0.0% 16 107789 nan
+</pre></blockquote></p>
+
+<p>The code for this client is too complex for a simple blog post, so
+you will have to check out the git repository to figure out how it
+work. What I can tell is how the three last numbers on each line
+should be interpreted. The first is how many seconds ago information
+was received from the service. The second is how long ago, according
+to the service, the provided information was updated. The last is an
+estimate on how often the buy/sell values change.</p>
+
+<p>If you find this library useful, or would like to improve it, I
+would love to hear from you. Note that for some of the services I've
+implemented a trading API. It might be the topic of a future blog
+post.</p>
projects / homepage.git / blobdiff