- <title>Where did that package go? &mdash; geolocated IP traceroute</title>
- <link>http://people.skolelinux.org/pere/blog/Where_did_that_package_go___mdash__geolocated_IP_traceroute.html</link>
- <guid isPermaLink="true">http://people.skolelinux.org/pere/blog/Where_did_that_package_go___mdash__geolocated_IP_traceroute.html</guid>
- <pubDate>Mon, 9 Jan 2017 12:20:00 +0100</pubDate>
- <description><p>Did you ever wonder where the web trafic really flow to reach the
-web servers, and who own the network equipment it is flowing through?
-It is possible to get a glimpse of this from using traceroute, but it
-is hard to find all the details. Many years ago, I wrote a system to
-map the Norwegian Internet (trying to figure out if our plans for a
-network game service would get low enough latency, and who we needed
-to talk to about setting up game servers close to the users. Back
-then I used traceroute output from many locations (I asked my friends
-to run a script and send me their traceroute output) to create the
-graph and the map. The output from traceroute typically look like
-this:
-
-<p><pre>
-traceroute to www.stortinget.no (85.88.67.10), 30 hops max, 60 byte packets
- 1 uio-gw10.uio.no (129.240.202.1) 0.447 ms 0.486 ms 0.621 ms
- 2 uio-gw8.uio.no (129.240.24.229) 0.467 ms 0.578 ms 0.675 ms
- 3 oslo-gw1.uninett.no (128.39.65.17) 0.385 ms 0.373 ms 0.358 ms
- 4 te3-1-2.br1.fn3.as2116.net (193.156.90.3) 1.174 ms 1.172 ms 1.153 ms
- 5 he16-1-1.cr1.san110.as2116.net (195.0.244.234) 2.627 ms he16-1-1.cr2.oslosda310.as2116.net (195.0.244.48) 3.172 ms he16-1-1.cr1.san110.as2116.net (195.0.244.234) 2.857 ms
- 6 ae1.ar8.oslosda310.as2116.net (195.0.242.39) 0.662 ms 0.637 ms ae0.ar8.oslosda310.as2116.net (195.0.242.23) 0.622 ms
- 7 89.191.10.146 (89.191.10.146) 0.931 ms 0.917 ms 0.955 ms
- 8 * * *
- 9 * * *
-[...]
-</pre></p>
-
-<p>This show the DNS names and IP addresses of (at least some of the)
-network equipment involved in getting the data traffic from me to the
-www.stortinget.no server, and how long it took in milliseconds for a
-package to reach the equipment and return to me. Three packages are
-sent, and some times the packages do not follow the same path. This
-is shown for hop 5, where three different IP addresses replied to the
-traceroute request.</p>
-
-<p>There are many ways to measure trace routes. Other good traceroute
-implementations I use are traceroute (using ICMP packages) mtr (can do
-both ICMP, UDP and TCP) and scapy (python library with ICMP, UDP, TCP
-traceroute and a lot of other capabilities). All of them are easily
-available in <a href="https://www.debian.org/">Debian</a>.</p>
-
-<p>This time around, I wanted to know the geographic location of
-different route points, to visualize how visiting a web page spread
-information about the visit to a lot of servers around the globe. The
-background is that a web site today often will ask the browser to get
-from many servers the parts (for example HTML, JSON, fonts,
-JavaScript, CSS, video) required to display the content. This will
-leak information about the visit to those controlling these servers
-and anyone able to peek at the data traffic passing by (like your ISP,
-the ISPs backbone provider, FRA, GCHQ, NSA and others).</p>
-
-<p>Lets pick an example, the Norwegian parliament web site
-www.stortinget.no. It is read daily by all members of parliament and
-their staff, as well as political journalists, activits and many other
-citizens of Norway. A visit to the www.stortinget.no web site will
-ask your browser to contact 8 other servers: ajax.googleapis.com,
-insights.hotjar.com, script.hotjar.com, static.hotjar.com,
-stats.g.doubleclick.net, www.google-analytics.com,
-www.googletagmanager.com and www.netigate.se. I extracted this by
-asking <a href="http://phantomjs.org/">PhantomJS</a> to visit the
-Stortinget web page and tell me all the URLs PhantomJS downloaded to
-render the page (in HAR format using
-<a href="https://github.com/ariya/phantomjs/blob/master/examples/netsniff.js">their
-netsniff example</a>. I am very grateful to Gorm for showing me how
-to do this). My goal is to visualize network traces to all IP
-addresses behind these DNS names, do show where visitors personal
-information is spread when visiting the page.</p>
-
-<p align="center"><a href="www.stortinget.no-geoip.kml"><img
-src="http://people.skolelinux.org/pere/blog/images/2017-01-09-www.stortinget.no-geoip-small.png" alt="map of combined traces for URLs used by www.stortinget.no using GeoIP"/></a></p>
-
-<p>When I had a look around for options, I could not find any good
-free software tools to do this, and decided I needed my own traceroute
-wrapper outputting KML based on locations looked up using GeoIP. KML
-is easy to work with and easy to generate, and understood by several
-of the GIS tools I have available. I got good help from by NUUG
-colleague Anders Einar with this, and the result can be seen in
-<a href="https://github.com/petterreinholdtsen/kmltraceroute">my
-kmltraceroute git repository</a>. Unfortunately, the quality of the
-free GeoIP databases I could find (and the for-pay databases my
-friends had access to) is not up to the task. The IP addresses of
-central Internet infrastructure would typically be placed near the
-controlling companies main office, and not where the router is really
-located, as you can see from <a href="www.stortinget.no-geoip.kml">the
-KML file I created</a> using the GeoLite City dataset from MaxMind.
-
-<p align="center"><a href="http://people.skolelinux.org/pere/blog/images/2017-01-09-www.stortinget.no-scapy.svg"><img
-src="http://people.skolelinux.org/pere/blog/images/2017-01-09-www.stortinget.no-scapy-small.png" alt="scapy traceroute graph for URLs used by www.stortinget.no"/></a></p>
-
-<p>I also had a look at the visual traceroute graph created by
-<a href="http://www.secdev.org/projects/scapy/">the scrapy project</a>,
-showing IP network ownership (aka AS owner) for the IP address in
-question.
-<a href="http://people.skolelinux.org/pere/blog/images/2017-01-09-www.stortinget.no-scapy.svg">The
-graph display a lot of useful information about the traceroute in SVG
-format</a>, and give a good indication on who control the network
-equipment involved, but it do not include geolocation. This graph
-make it possible to see the information is made available at least for
-UNINETT, Catchcom, Stortinget, Nordunet, Google, Amazon, Telia, Level
-3 Communications and NetDNA.</p>
-
-<p align="center"><a href="https://geotraceroute.com/index.php?node=4&host=www.stortinget.no"><img
-src="http://people.skolelinux.org/pere/blog/images/2017-01-09-www.stortinget.no-geotraceroute-small.png" alt="example geotraceroute view for www.stortinget.no"/></a></p>
-
-<p>In the process, I came across the
-<a href="https://geotraceroute.com/">web service GeoTraceroute</a> by
-Salim Gasmi. Its methology of combining guesses based on DNS names,
-various location databases and finally use latecy times to rule out
-candidate locations seemed to do a very good job of guessing correct
-geolocation. But it could only do one trace at the time, did not have
-a sensor in Norway and did not make the geolocations easily available
-for postprocessing. So I contacted the developer and asked if he
-would be willing to share the code (he refused until he had time to
-clean it up), but he was interested in providing the geolocations in a
-machine readable format, and willing to set up a sensor in Norway. So
-since yesterday, it is possible to run traces from Norway in this
-service thanks to a sensor node set up by
-<a href="https://www.nuug.no/">the NUUG assosiation</a>, and get the
-trace in KML format for further processing.</p>
-
-<p align="center"><a href="http://people.skolelinux.org/pere/blog/images/2017-01-09-www.stortinget.no-geotraceroute-kml-join.kml"><img
-src="http://people.skolelinux.org/pere/blog/images/2017-01-09-www.stortinget.no-geotraceroute-kml-join.png" alt="map of combined traces for URLs used by www.stortinget.no using geotraceroute"/></a></p>
-
-<p>Here we can see a lot of trafic passes Sweden on its way to
-Denmark, Germany, Holland and Ireland. Plenty of places where the
-Snowden confirmations verified the traffic is read by various actors
-without your best interest as their top priority.</p>
-
-<p>Combining KML files is trivial using a text editor, so I could loop
-over all the hosts behind the urls imported by www.stortinget.no and
-ask for the KML file from GeoTraceroute, and create a combined KML
-file with all the traces (unfortunately only one of the IP addresses
-behind the DNS name is traced this time. To get them all, one would
-have to request traces using IP number instead of DNS names from
-GeoTraceroute). That might be the next step in this project.</p>
-
-<p>Armed with these tools, I find it a lot easier to figure out where
-the IP traffic moves and who control the boxes involved in moving it.
-And every time the link crosses for example the Swedish border, we can
-be sure Swedish Signal Intelligence (FRA) is listening, as GCHQ do in
-Britain and NSA in USA and cables around the globe. (Hm, what should
-we tell them? :) Keep that in mind if you ever send anything
-unencrypted over the Internet.</p>
-
-<p>PS: KML files are drawn using
-<a href="http://ivanrublev.me/kml/">the KML viewer from Ivan
-Rublev<a/>, as it was less cluttered than the local Linux application
-Marble. There are heaps of other options too.</p>
+ <title>Streaming the Linux desktop to Kodi using VLC and RTSP</title>
+ <link>http://people.skolelinux.org/pere/blog/Streaming_the_Linux_desktop_to_Kodi_using_VLC_and_RTSP.html</link>
+ <guid isPermaLink="true">http://people.skolelinux.org/pere/blog/Streaming_the_Linux_desktop_to_Kodi_using_VLC_and_RTSP.html</guid>
+ <pubDate>Thu, 12 Jul 2018 02:00:00 +0200</pubDate>
+ <description><p>PS: See
+<ahref="http://people.skolelinux.org/pere/blog/Simple_streaming_the_Linux_desktop_to_Kodi_using_GStreamer_and_RTP.html">the
+followup post</a> for a even better approach.</p>
+
+<p>A while back, I was asked by a friend how to stream the desktop to
+my projector connected to Kodi. I sadly had to admit that I had no
+idea, as it was a task I never had tried. Since then, I have been
+looking for a way to do so, preferable without much extra software to
+install on either side. Today I found a way that seem to kind of
+work. Not great, but it is a start.</p>
+
+<p>I had a look at several approaches, for example
+<a href="https://github.com/mfoetsch/dlna_live_streaming">using uPnP
+DLNA as described in 2011</a>, but it required a uPnP server, fuse and
+local storage enough to store the stream locally. This is not going
+to work well for me, lacking enough free space, and it would
+impossible for my friend to get working.</p>
+
+<p>Next, it occurred to me that perhaps I could use VLC to create a
+video stream that Kodi could play. Preferably using
+broadcast/multicast, to avoid having to change any setup on the Kodi
+side when starting such stream. Unfortunately, the only recipe I
+could find using multicast used the rtp protocol, and this protocol
+seem to not be supported by Kodi.</p>
+
+<p>On the other hand, the rtsp protocol is working! Unfortunately I
+have to specify the IP address of the streaming machine in both the
+sending command and the file on the Kodi server. But it is showing my
+desktop, and thus allow us to have a shared look on the big screen at
+the programs I work on.</p>
+
+<p>I did not spend much time investigating codeces. I combined the
+rtp and rtsp recipes from
+<a href="https://wiki.videolan.org/Documentation:Streaming_HowTo/Command_Line_Examples/">the
+VLC Streaming HowTo/Command Line Examples</a>, and was able to get
+this working on the desktop/streaming end.</p>
+
+<blockquote><pre>
+vlc screen:// --sout \
+ '#transcode{vcodec=mp4v,acodec=mpga,vb=800,ab=128}:rtp{dst=projector.local,port=1234,sdp=rtsp://192.168.11.4:8080/test.sdp}'
+</pre></blockquote>
+
+<p>I ssh-ed into my Kodi box and created a file like this with the
+same IP address:</p>
+
+<blockquote><pre>
+echo rtsp://192.168.11.4:8080/test.sdp \
+ > /storage/videos/screenstream.m3u
+</pre></blockquote>
+
+<p>Note the 192.168.11.4 IP address is my desktops IP address. As far
+as I can tell the IP must be hardcoded for this to work. In other
+words, if someone elses machine is going to do the steaming, you have
+to update screenstream.m3u on the Kodi machine and adjust the vlc
+recipe. To get started, locate the file in Kodi and select the m3u
+file while the VLC stream is running. The desktop then show up in my
+big screen. :)</p>
+
+<p>When using the same technique to stream a video file with audio,
+the audio quality is really bad. No idea if the problem is package
+loss or bad parameters for the transcode. I do not know VLC nor Kodi
+enough to tell.</p>
+
+<p><strong>Update 2018-07-12</strong>: Johannes Schauer send me a few
+succestions and reminded me about an important step. The "screen:"
+input source is only available once the vlc-plugin-access-extra
+package is installed on Debian. Without it, you will see this error
+message: "VLC is unable to open the MRL 'screen://'. Check the log
+for details." He further found that it is possible to drop some parts
+of the VLC command line to reduce the amount of hardcoded information.
+It is also useful to consider using cvlc to avoid having the VLC
+window in the desktop view. In sum, this give us this command line on
+the source end
+
+<blockquote><pre>
+cvlc screen:// --sout \
+ '#transcode{vcodec=mp4v,acodec=mpga,vb=800,ab=128}:rtp{sdp=rtsp://:8080/}'
+</pre></blockquote>
+
+<p>and this on the Kodi end<p>
+
+<blockquote><pre>
+echo rtsp://192.168.11.4:8080/ \
+ > /storage/videos/screenstream.m3u
+</pre></blockquote>
+
+<p>Still bad image quality, though. But I did discover that streaming
+a DVD using dvdsimple:///dev/dvd as the source had excellent video and
+audio quality, so I guess the issue is in the input or transcoding
+parts, not the rtsp part. I've tried to change the vb and ab
+parameters to use more bandwidth, but it did not make a
+difference.</p>
+
+<p>I further received a suggestion from Einar Haraldseid to try using
+gstreamer instead of VLC, and this proved to work great! He also
+provided me with the trick to get Kodi to use a multicast stream as
+its source. By using this monstrous oneliner, I can stream my desktop
+with good video quality in reasonable framerate to the 239.255.0.1
+multicast address on port 1234:
+
+<blockquote><pre>
+gst-launch-1.0 ximagesrc use-damage=0 ! video/x-raw,framerate=30/1 ! \
+ videoconvert ! queue2 ! \
+ x264enc bitrate=8000 speed-preset=superfast tune=zerolatency qp-min=30 \
+ key-int-max=15 bframes=2 ! video/x-h264,profile=high ! queue2 ! \
+ mpegtsmux alignment=7 name=mux ! rndbuffersize max=1316 min=1316 ! \
+ udpsink host=239.255.0.1 port=1234 ttl-mc=1 auto-multicast=1 sync=0 \
+ pulsesrc device=$(pactl list | grep -A2 'Source #' | \
+ grep 'Name: .*\.monitor$' | cut -d" " -f2|head -1) ! \
+ audioconvert ! queue2 ! avenc_aac ! queue2 ! mux.
+</pre></blockquote>
+
+<p>and this on the Kodi end<p>
+
+<blockquote><pre>
+echo udp://@239.255.0.1:1234 \
+ > /storage/videos/screenstream.m3u
+</pre></blockquote>
+
+<p>Note the trick to pick a valid pulseaudio source. It might not
+pick the one you need. This approach will of course lead to trouble
+if more than one source uses the same multicast port and address.
+Note the ttl-mc=1 setting, which limit the multicast packages to the
+local network. If the value is increased, your screen will be
+broadcasted further, one network "hop" for each increase (read up on
+multicast to learn more. :)!</p>
+
+<p>Having cracked how to get Kodi to receive multicast streams, I
+could use this VLC command to stream to the same multicast address.
+The image quality is way better than the rtsp approach, but gstreamer
+seem to be doing a better job.</p>
+
+<blockquote><pre>
+cvlc screen:// --sout '#transcode{vcodec=mp4v,acodec=mpga,vb=800,ab=128}:rtp{mux=ts,dst=239.255.0.1,port=1234,sdp=sap}'
+</pre></blockquote>
projects / homepage.git / blobdiff