Difference between revisions of "TTNHABBridge"

From RevSpace
Jump to navigation Jump to search
(next steps)
(Huge cleanup)
Line 46: Line 46:
 
https://github.com/bertrik/ttnhabbridge
 
https://github.com/bertrik/ttnhabbridge
  
It's nowhere near ready yet.
 
 
The README explains the tool chain setup.
 
The README explains the tool chain setup.
 +
 +
Using this source code, the basic functionality works.
 +
 +
There is no user guide yet, but the .properties settings file has a one-line comment explaining what each setting does.
  
 
=== Tasks ===
 
=== Tasks ===
Line 65: Line 68:
 
** decode the payload into latitude/longitude/altitude, and encode it into a habhub ASCII sentence with correct CRC, see https://ukhas.org.uk/communication:protocol
 
** decode the payload into latitude/longitude/altitude, and encode it into a habhub ASCII sentence with correct CRC, see https://ukhas.org.uk/communication:protocol
 
** for each gateway that received the data:
 
** for each gateway that received the data:
*** look up the gateway name/lat/long/alt, through some TTN API (and cache it?)
+
*** send a listener info and a listener telemetry document to the habhub server
*** fake a HAB receiver and make it send the ASCII sentence to the habhub server
+
*** send the payload telemetry (ASCII sentence) to the habhub server
  
 
==== MQTT listener ====
 
==== MQTT listener ====
Probably implemented using fusesource MQTT client or Eclipse paho MQTT client.
+
Implemented using the Eclipse paho MQTT client.
 
We can listen on a certain application and catch *all* traffic for all nodes registered to that application.
 
We can listen on a certain application and catch *all* traffic for all nodes registered to that application.
  
Line 104: Line 107:
  
 
The "[receiver_id]" part can be repeated as many times as there are gateways that received the data.
 
The "[receiver_id]" part can be repeated as many times as there are gateways that received the data.
This is probably also the trickiest part to implement in Jackson, because it uses variable field names.
 
  
 
===== listener upload =====
 
===== listener upload =====
Line 115: Line 117:
  
 
==== Payload encoder/decoder ====
 
==== Payload encoder/decoder ====
This part decodes the binary payload received from the TTN into a standard habitat sentence, like
+
This part decodes the binary payload received from the TTN into a standard habitat sentence.
 +
 
 +
The bridge application currently assumes that the binary payload is encoded according to
 +
[https://github.com/SodaqMoja/SodaqOne-UniversalTracker SodaqOne universal tracker] conventions.
 +
 
 +
The habitat ASCII sentence typically looks like:
 
   $$hadie,181,10:42:10,54.422829,-6.741293,27799.3,1:10*002A
 
   $$hadie,181,10:42:10,54.422829,-6.741293,27799.3,1:10*002A
 +
 +
Mapping:
 +
* payload callsign = name of TTN device
 +
* payload counter = LoRaWAN frame count
 +
* payload time = current date/time of the application (TODO?)
 +
* payload lat/lon/alt = as decoded from the TTN binary payload
 +
* payload checksum = CCITT-CRC16
  
 
==== Gateway cache ====
 
==== Gateway cache ====
 
We don't actually need this, it seems the required gateway data (id/lat/lon/alt) is already provided by the MQTT stream.
 
We don't actually need this, it seems the required gateway data (id/lat/lon/alt) is already provided by the MQTT stream.
 
=== Implementation ===
 
The bridge will be written in Java.
 
 
Libraries to use:
 
* all application settings are kept in a .properties file
 
* slf4j as the logging interface, log4j as the logging implementation
 
* junit/mockito for unit testing
 
* jetty/jersey/jackson for the REST interface towards habitat
 
* zip/tar-file for easy installation using the gradle application plugin
 
* for the MQTT interface: not sure, either eclipse paho or fusesource
 
  
 
== Helpful links ==
 
== Helpful links ==
Line 148: Line 151:
 
[http://habitat.habhub.org/jse/ list of habitat JSON schemas]
 
[http://habitat.habhub.org/jse/ list of habitat JSON schemas]
  
I think/hope the interface above can be approached as a REST service, for which is very easy to create a client (and test server) in Java, using a combination of Jetty/Jersey/Jackson libraries.
+
The habitat interface is accessed as a REST service, using a combination of Jetty/Jersey/Jackson Java libraries.
 
 
== Settings ==
 
Settings likely needed by the bridge application:
 
* TTN gateway API settings
 
** URL of the TTN gateway API
 
* TTN MQTT API settings
 
** URL of the TTN MQTT API
 
** application id of the TTN 'hab' application
 
** application username
 
** application password
 
* habitat database settings
 
** URL of the habitat server
 
** ...
 
  
 
== JSON examples ==
 
== JSON examples ==
Line 193: Line 183:
  
 
=== MQTT data ===
 
=== MQTT data ===
Over MQTT, it looks like this
+
Over MQTT, it looks like this:
(this is a pretty old sample!):
 
<pre>
 
{
 
  "payload": "eyJIZWxsbyI6ImJlcnRyaWtAc2lra2VuLm5sIn0=",
 
  "port": 1,
 
  "counter": 3,
 
  "dev_eui": "0000000019800501",
 
  "metadata": [
 
    {
 
      "frequency": 867.3,
 
      "datarate": "SF12BW125",
 
      "codingrate": "4/5",
 
      "gateway_timestamp": 1954388012,
 
      "gateway_time": "2016-08-06T19:06:05.891844Z",
 
      "channel": 4,
 
      "server_time": "2016-08-06T19:06:05.926706718Z",
 
      "rssi": -118,
 
      "lsnr": -10.5,
 
      "rfchain": 0,
 
      "crc": 1,
 
      "modulation": "LORA",
 
      "gateway_eui": "AA555A00080605B7",
 
      "altitude": 40,
 
      "longitude": 5.16738,
 
      "latitude": 52.08393
 
    }
 
  ]
 
}
 
</pre>
 
  
Simulated uplink (using the console):
 
 
<pre>
 
<pre>
 
{
 
{
Line 231: Line 191:
 
   "hardware_serial": "0004A30B001ADBC5",
 
   "hardware_serial": "0004A30B001ADBC5",
 
   "port": 1,
 
   "port": 1,
   "counter": 0,
+
   "counter": 587,
   "payload_raw": "AA==",
+
   "payload_raw": "g3ybWXkTef8BH1EOzAK1\/wEA5wQT",
  "metadata": {
 
    "time": "2017-08-19T15:23:39.288816687Z"
 
  }
 
}
 
</pre>
 
 
 
From TTN MQTT:
 
<pre>
 
{
 
  "app_id": "ttn-rfm95-ardumin1",
 
  "dev_id": "ttn-rfm95-2",
 
  "hardware_serial": "0034DA568C2A7566",
 
  "port": 1,
 
  "counter": 11641,
 
  "payload_raw": "EHs=",
 
  "payload_fields": {
 
    "value": "2500"
 
  },
 
 
   "metadata": {
 
   "metadata": {
     "time": "2017-08-20T15:28:19.454240399Z",
+
     "time": "2017-08-22T00:36:18.674804288Z",
     "frequency": 868.3,
+
     "frequency": 868.1,
 
     "modulation": "LORA",
 
     "modulation": "LORA",
     "data_rate": "SF9BW125",
+
     "data_rate": "SF7BW125",
 
     "coding_rate": "4\/5",
 
     "coding_rate": "4\/5",
 
     "gateways": [
 
     "gateways": [
 
       {
 
       {
         "gtw_id": "ttn-cgn_philippshof_1",
+
         "gtw_id": "eui-008000000000b8b6",
        "gtw_trusted": true,
+
         "timestamp": 3979529499,
         "timestamp": 3953316716,
+
         "time": "2017-08-22T00:36:18.345616Z",
         "time": "2017-08-20T15:28:18Z",
+
         "channel": 0,
         "channel": 1,
+
         "rssi": -120,
         "rssi": -63,
+
         "snr": -7,
         "snr": 11.5,
 
 
         "rf_chain": 1,
 
         "rf_chain": 1,
         "latitude": 50.950233,
+
         "latitude": 52.0182,
         "longitude": 6.9196253,
+
         "longitude": 4.70844,
         "altitude": 18
+
         "altitude": 27
 
       }
 
       }
 
     ]
 
     ]

Revision as of 08:48, 22 August 2017

Project TTNHABBridge
Ttnhabbridge.png
A software bridge between TTN and the HAB network
Status In progress
Contact bertrik
Last Update 2017-08-22

Status

Wrote some software in Java that talks to the Habitat interface. Payload telemetry upload seems to work, listener telemetry/information upload is not implemented yet (this part of the Habitat interface is very complex and complicated). Haven't started yet on receiving data from TTN over MQTT, but I expect few problems there.

Next steps:

  • testing if it works
    • Register an app, program a Sodaq One with the proper keys
    • Pick a binary telemetry format (e.g. reverse engineer the format from the default sodaq software) and implement a decoder in Java.
    • Drive around with a tracker in range of a TTN gateway and see if it works (balloon visible on the map)!
    • Fill in a flight document and see if the entire chain actually works
  • (lowprio) play with travis for automatic online building (and packaging?)
  • discuss/review with the people at habhub/ukhas/TTN

Introduction

This idea is about using the-things-network as a receiver for amateur balloon telemetry.

Receiving telemetry from amateur balloons is currently typically done on the 434 MHz band using RTTY modulation, with dedicated receivers listening for RTTY-modulated ASCII strings. The operator of each receiver has to prepare his radio setup for receiving the telemetry, by tuning to the correct frequency at the correct time, setting up a dedicated software client that decodes the RTTY modulation and forwards the data to a central system over the internet. The central system accepts data from many such receivers, performs de-duplication, keeps track of who received what and updates a nice graphical map of where each balloon is and where the receivers are. It can do a burst prediction, landing prediction based on a weather model

A network like the-things-network can help a lot, it has a lot of gateways already (in the Netherlands at least..), already performs deduplication. It uses LoRa as a modulation scheme which is much more sensitive and much less susceptible to slight tuning errors than RTTY.

In short, the idea is:

  • you attach a LoRaWAN transmitter to the balloon
  • the LoRaWAN transmitter is pre-configured with a set of keys generated by the TTN
  • the balloon broadcasts its telemetry every once in a while (say a few times per minute) and this is picked up by one or more TTN gateways
  • the bridge software listens for packets received by the TTN and decodes the payload data into an id, latitude, longitude, altitude of the balloon
  • for each packet, we know which gateways received it and where they are. So we can "fake" a client for each gateway and construct an ASCII sentence according to the HAB server conventions
  • the HAB server still sees the same messages like it would if there were many traditional receivers, so doesn't need any modification!

This way, the entire things network can be used to receive balloon telemetry! There is no longer a need for radio operators to be present at their receiver at the exact time the balloon is launched, making manual adjustments, etc. The Netherlands is already covered by many TTN gateways, greatly increasing the chance the balloon telemetry will be picked up.

Software

Source code

Source code is available at: https://github.com/bertrik/ttnhabbridge

The README explains the tool chain setup.

Using this source code, the basic functionality works.

There is no user guide yet, but the .properties settings file has a one-line comment explaining what each setting does.

Tasks

Stuff to do:

  • come up with a simple but flexible way to encode telemetry in a binary packet, to be transmitted over TTN. Perhaps borrow some ideas from how it's done with [1]
  • figure out how to receive data from TTN, this is an MQTT stream -> pick an easy-to-use Java MQTT client library. Perhaps this one: https://github.com/fusesource/mqtt-client ?
  • figure out the protocol between dl-fldigi and the HAB server -> look into https://github.com/ukhas/habitat-cpp-connector , reverse engineer it and create a Java version. It looks like a couchdb-specific database connection, sending JSON messages! -> it seems we can access this interface as if it were a REST service!
  • implement this (in Java for example) and publish it on github!

Modules

The software consists of the following modules.

Main process

The main process of the bridge is something like this:

  • listen to the MQTT stream of the HAB application
  • once we get data:
    • decode the payload into latitude/longitude/altitude, and encode it into a habhub ASCII sentence with correct CRC, see https://ukhas.org.uk/communication:protocol
    • for each gateway that received the data:
      • send a listener info and a listener telemetry document to the habhub server
      • send the payload telemetry (ASCII sentence) to the habhub server

MQTT listener

Implemented using the Eclipse paho MQTT client. We can listen on a certain application and catch *all* traffic for all nodes registered to that application.

Example invocation, using mosquitto_sub:

 bertrik@zenbook:~$ mosquitto_sub -h eu.thethings.network -t '+/devices/+/up' -u 'ttnmapper' -P 'ttn-account-v2.Xc8BFRKeBK5nUhc9ikDcR-sbelgSMdHKnOQKMAiwpgI' -v

Actual result:

 ttnmapper/devices/mapper2/up {"app_id":"ttnmapper","dev_id":"mapper2","hardware_serial":"0004A30B001ADBC5","port":1,"counter":1,"payload_raw":"d4WaWXATAAAAAAAAAAAAAAAAAAD/","metadata":{"time":"2017-08-21T07:02:14.842855925Z","frequency":868.1,"modulation":"LORA","data_rate":"SF7BW125","coding_rate":"4/5","gateways":[{"gtw_id":"eui-008000000000b8b6","timestamp":865620531,"time":"2017-08-21T07:02:14.846095Z","channel":0,"rssi":-119,"snr":-3.8,"rf_chain":1,"latitude":52.0182,"longitude":4.70844,"altitude":27}]}}
 ttnmapper/devices/mapper2/up {"app_id":"ttnmapper","dev_id":"mapper2","hardware_serial":"0004A30B001ADBC5","port":1,"counter":4,"payload_raw":"loeaWW4T2+8BHzYZzAIeAA8A/QUS","metadata":{"time":"2017-08-21T07:11:18.313946438Z","frequency":868.3,"modulation":"LORA","data_rate":"SF7BW125","coding_rate":"4/5","gateways":[{"gtw_id":"eui-008000000000b8b6","timestamp":1409115451,"time":"2017-08-21T07:11:18.338662Z","channel":1,"rssi":-114,"snr":-0.2,"rf_chain":1,"latitude":52.0182,"longitude":4.70844,"altitude":27}]}}

Habitat uploader

payload upload

This uses a HTTP interface, performing a PUT to a certain URL with a certain content:

  • the URL is http://habitat.habhub.org/habitat/_design/payload_telemetry/_update/add_listener/<doc_id>
  • the <doc_id> is created from the telemetry sentence by doing ASCII to BASE64 conversion, then hashing using SHA-256 and encoding as ASCII-hex (as LOWER case!).
  • the HTTP method is a PUT with the following headers:
    • "Accept: application/json"
    • "Content-Type: application/json"
    • "charsets: utf-8"
  • The contents of the PUT is the following JSON structure
{
  "data": {
    "_raw": "[base64 of telemetry sentence]"
  },
  "receivers": {
    "[receiver_id]": {
      "time_created": "[timestamp]",
      "time_uploaded": "[timestamp]"
    }
  }
}

The "[receiver_id]" part can be repeated as many times as there are gateways that received the data.

listener upload

To figure out how to upload information about the receiver station, I captured some traffic between dl-fldigi and habitat, it seems the following happens:

  • a GET is done on http://habitat.habhub.org/_uuids?count=100, this returns a list of 100 UUIDs, where a UUID is the lower-case ascii hex representation of 16 bytes.
  • a PUT is done with a "listener_information" doc to /habitat/<the first UUID>
  • a PUT is done with a "listener_telemetry" doc to /habitat/<the second UUID>

Perhaps implement this with an expiring cache, so listener information/telemetry is uploaded regularly (say every 20 minutes) but not with *every* payload telemetry.

Payload encoder/decoder

This part decodes the binary payload received from the TTN into a standard habitat sentence.

The bridge application currently assumes that the binary payload is encoded according to SodaqOne universal tracker conventions.

The habitat ASCII sentence typically looks like:

 $$hadie,181,10:42:10,54.422829,-6.741293,27799.3,1:10*002A

Mapping:

  • payload callsign = name of TTN device
  • payload counter = LoRaWAN frame count
  • payload time = current date/time of the application (TODO?)
  • payload lat/lon/alt = as decoded from the TTN binary payload
  • payload checksum = CCITT-CRC16

Gateway cache

We don't actually need this, it seems the required gateway data (id/lat/lon/alt) is already provided by the MQTT stream.

Helpful links

From a conversation on #highaltitude:

20:24 < adamgreig> there's a) a python library that's a lot easier to read
20:24 < adamgreig> but b) basically the gist is you just PUT to http://habitat.habhub.org/habitat/_design/payload_telemetry/_update/add_listener/<id> with some stuff
20:24 < adamgreig> http://habitat.readthedocs.io/en/latest/habitat/habitat/habitat/habitat.views.payload_telemetry.html#module-habitat.views.payload_telemetry
20:24 < adamgreig> so you have a new string, you take the sha256 hex digest of the base64 encoded raw data
20:25 < adamgreig> you PUT to that URL with that ID
20:25 < adamgreig> and you include that JSON struture with your callsign/details

C implementation of the interface between the client and the habitat server

list of habitat JSON schemas

The habitat interface is accessed as a REST service, using a combination of Jetty/Jersey/Jackson Java libraries.

JSON examples

TTN console gateway traffic

In the TTN console, under gateway traffic, the 'Event data' looks like this:

{
  "gw_id": "eui-008000000000b8b6",
  "payload": "QONzlgaABAAB8XtDve5FPT4jnzvJ",
  "f_cnt": 4,
  "lora": {
    "spreading_factor": 7,
    "bandwidth": 125,
    "air_time": 56576000
  },
  "coding_rate": "4/5",
  "timestamp": "2017-08-14T12:12:52.614Z",
  "rssi": -103,
  "snr": 3.8,
  "dev_addr": "069673E3",
  "frequency": 868100000
}

TTN console application data

In the TTN console, under Applications, the 'Application data' looks like this:

TODO

I *think* this data contains also the gateways with their id, latitude, longitude, altitude.

MQTT data

Over MQTT, it looks like this:

{
  "app_id": "ttnmapper",
  "dev_id": "mapper2",
  "hardware_serial": "0004A30B001ADBC5",
  "port": 1,
  "counter": 587,
  "payload_raw": "g3ybWXkTef8BH1EOzAK1\/wEA5wQT",
  "metadata": {
    "time": "2017-08-22T00:36:18.674804288Z",
    "frequency": 868.1,
    "modulation": "LORA",
    "data_rate": "SF7BW125",
    "coding_rate": "4\/5",
    "gateways": [
      {
        "gtw_id": "eui-008000000000b8b6",
        "timestamp": 3979529499,
        "time": "2017-08-22T00:36:18.345616Z",
        "channel": 0,
        "rssi": -120,
        "snr": -7,
        "rf_chain": 1,
        "latitude": 52.0182,
        "longitude": 4.70844,
        "altitude": 27
      }
    ]
  }
}