TTNHABBridge: Difference between revisions

From RevSpace
Jump to navigation Jump to search
No edit summary
 
(92 intermediate revisions by the same user not shown)
Line 2: Line 2:
   |Name=TTNHABBridge
   |Name=TTNHABBridge
   |Picture=ttnhabbridge.png
   |Picture=ttnhabbridge.png
   |Omschrijving=A software bridge between TTN and the HAB network
   |Omschrijving=A software bridge between TheThingsNetwork and the UKHAS high-balloon network
   |Status=In progress
   |Status=Completed
   |Contact=bertrik
   |Contact=bertrik
}}
}}


== Status ==
== Status ==
Wrote some software in Java that talks to the Habitat interface. Payload telemetry upload seems to work, listener telemetry/information upload is not implemented yet (this part of the Habitat interface is very complex and complicated). Haven't started yet on receiving data from TTN over MQTT, but I expect few problems there.
This software is a bridge application that accepts high altitude balloon telemetry data from TheThingsNetwork and forwards it to the habhub tracker website.


Next steps:
It has been tested in practice with the 'koppelting1' payload on 2017-8-26.
* get some example JSON from the TTN application MQTT stream
If you look at the [https://tracker.habhub.org/ habhub map] and see a receiver called EUI-xxxxxxxx then that one is probably generated by this software.
* implement the MQTT client and see if we get data
 
* implement the listener info and listener telemetry upload + cache
=== Next steps ===
* testing if it works
Future development:
** Register an app, program a Sodaq One with the proper keys
* add support for running this in a Docker container: has been realized, but not documented yet
** Pick a binary telemetry format (e.g. reverse engineer the format from the default sodaq software) and implement a decoder in Java.
* add support for sondehub, see also https://github.com/bertrik/ttnhabbridge/issues/6
** Drive around with a tracker in range of a TTN gateway and see if it works (balloon visible on the map)!
** need to implement data structures
** need to restructure internals: sondehub uses a JSON structure, not a 'sentence' like UKHAS does


== Introduction ==
== Introduction ==
This idea is about using the-things-network as a receiver for amateur balloon telemetry.
This application uses TheThingsNetwork as a receiver for high-altitude balloon telemetry and forwards it to habhub.org, so it can be shown on their overview map.
The habhub map shows the current position of the balloon, but also shows a prediction (based on weather model) about where the balloon will burst, where it will land, which receivers are currently in range, etc.


Receiving telemetry from amateur balloons is currently typically done on the 434 MHz band using RTTY modulation, with dedicated receivers listening for RTTY-modulated ASCII strings.
In the current situation, high altitude balloons send telemetry on the 434 MHz band using RTTY modulation. This is received by a set of dedicated receivers.
The operator of each receiver has to prepare his radio setup for receiving the telemetry, by tuning to the correct frequency at the correct time, setting up a dedicated software client that decodes the RTTY modulation and forwards the data to a central system over the internet.
The operator of each receiver (typically radio amateurs) has to prepare his radio setup for receiving the telemetry, by tuning to the correct frequency at the correct time, setting up a dedicated software client that decodes the RTTY modulation and forwards the data to a central system over the internet.
The central system accepts data from many such receivers, performs de-duplication, keeps track of who received what and updates a nice graphical map of where each balloon is and where the receivers are. It can do a burst prediction, landing prediction based on a weather model


A network like the-things-network can help a lot, it has a lot of gateways already (in the Netherlands at least..), already performs deduplication.
In the new situation, the existing infrastructure of TheThingsNetwork is used to receive low-bitrate telemetry packets and forward them on the internet.
It uses LoRa as a modulation scheme which is much more sensitive and much less susceptible to slight tuning errors than RTTY.
It uses LoRa as a modulation scheme which allows for very small transmission power and is much less susceptible to slight tuning errors than the currently used RTTY modulation.
Packets are encoded using the LoRaWAN protocol.
Telemetry packets can be received completely autonomously by TTN gateways, there is no need for a radio operator to make adjustments etc.


In short, the idea is:
In short, the idea is:
* you attach a LoRaWAN transmitter to the balloon
* you attach a LoRaWAN transmitter to the balloon, which been pre-configured with a set of keys generated by the TTN
* the LoRaWAN transmitter is pre-configured with a set of keys generated by the TTN
* the balloon broadcasts its telemetry every once in a while (say a few times per minute) and this is picked up by one or more TTN gateways which forward into the TTN infrastructure
* the balloon broadcasts its telemetry every once in a while (say a few times per minute) and this is picked up by one or more TTN gateways
* the bridge software listens for packets received by the TTN and decodes the payload data into an id, latitude, longitude, altitude of the balloon
* the bridge software listens for packets received by the TTN and decodes the payload data into an id, latitude, longitude, altitude of the balloon
* for each packet, we know which gateways received it and where they are. So we can "fake" a client for each gateway and construct an ASCII sentence according to the HAB server conventions
* the bridge software converts each telemetry packet according to 'UKHAS' conventions and forwards it to the habitat.habhub.org server
* the HAB server still sees the same messages like it would if there were many traditional receivers, so doesn't need any modification!
* the bridge software also forwards the TTN gateway locations, so they also appear on the map, along with their name / EUI and antenna altitude
* the habitat.habhub.org server still sees the same messages like it would if there were many traditional receivers, so doesn't need any modification


This way, the entire things network can be used to receive balloon telemetry!
This way, the entire things network can be used to receive balloon telemetry.
There is no longer a need for radio operators to be present at their receiver at the exact time the balloon is launched, making manual adjustments, etc.
There is no longer a need for radio operators to be present at their receiver at the exact time the balloon is launched, making manual adjustments, etc.
The Netherlands is already covered by many TTN gateways, greatly increasing the chance the balloon telemetry will be picked up.
The Netherlands is already covered by many TTN gateways, greatly increasing the chance the balloon telemetry will be picked up.
Line 46: Line 49:
https://github.com/bertrik/ttnhabbridge
https://github.com/bertrik/ttnhabbridge


It's nowhere near ready yet.
The README explains the tool chain setup.
The README explains the tool chain setup.


=== Tasks ===
=== Building the software ===
Stuff to do:
 
* come up with a simple but flexible way to encode telemetry in a binary packet, to be transmitted over TTN. Perhaps borrow some ideas from how it's done with [http://ttnmapper.org/faq.php]
Steps (on Debian Linux):
* figure out how to receive data from TTN, this is an MQTT stream -> pick an easy-to-use Java MQTT client library. Perhaps this one: https://github.com/fusesource/mqtt-client ?
* install a java-11 JDK
* figure out the protocol between dl-fldigi and the HAB server -> look into https://github.com/ukhas/habitat-cpp-connector , reverse engineer it and create a Java version. It looks like a couchdb-specific database connection, sending JSON messages! -> it seems we can access this interface as if it were a REST service!
  sudo apt install openjdk-11-jdk
* implement this (in Java for example) and publish it on github!
* clone the software from github
  git clone https://github.com/bertrik/ttnhabbridge
* enter directory ttnhabbridge
  cd ttnhabbridge
* run the gradle script to build the executable
  ./gradlew assemble (Linux)
  gradlew assemble (Windows)
* the executable is now available at ttnhabbridge/build/distributions/ttnhabbridge.tar and ttnhabbridge.zip
 
=== Running the software using Docker ===
To be documented ...
 
* install docker and docker-compose
  sudo apt install docker-ce ?
* get the source code
  git clone https://github.com/bertrik/ttnhabbridge
* enter the docker directory
  cd ttnhabbridge
  cd docker
* edit the config file
* get the docker image
  docker-compose pull
* run the docker image
  docker-compose up
 
=== Running the software from the zip-file / tar-ball ===
* extract either the tar (Linux) or zip (Windows) that was generated in the build step
* run it once to generate the initial configuration file, this file is called ttnhabbridge.yaml
  cd ttnhabbridge
  bin/ttnhabbridge
* stop it, using CTRL-C
* edit the ttnhabbridge.yaml configuration file with a text editor, enter your application name, keys, payload encoding, etc.
* run it again and verify that the program starts correctly with the new configuration file


=== Modules ===
=== Design ===
The software consists of the following modules.
The software consists of the following modules.


Line 64: Line 98:
* once we get data:
* once we get data:
** decode the payload into latitude/longitude/altitude, and encode it into a habhub ASCII sentence with correct CRC, see https://ukhas.org.uk/communication:protocol
** decode the payload into latitude/longitude/altitude, and encode it into a habhub ASCII sentence with correct CRC, see https://ukhas.org.uk/communication:protocol
** for each gateway that received the data:
** for each TTN gateway that received the data:
*** look up the gateway name/lat/long/alt, through some TTN API (and cache it?)
*** send a listener info document and a listener telemetry document to the habhub server
*** fake a HAB receiver and make it send the ASCII sentence to the habhub server
*** send the payload telemetry (ASCII sentence) to the habhub server


==== MQTT listener ====
==== Habitat uploader ====
Probably implemented using fusesource MQTT client or Eclipse paho MQTT client.
The habitat interface is accessed as a REST service.
We can listen on a certain application and catch *all* traffic for all nodes registered to that application.


Example invocation, using mosquitto_sub:
  mosquitto_sub -h eu.thethings.network -t '+/devices/+/up' -u 'ttnmapper' -P 'ttn-account-v2.Xc8BFRKeBK5nUhc9ikDcR-sbelgSMdHKnOQKMAXXXXX' -v
Example result (data: a 00 byte) by simulating data in the console:
  ttnmapper/devices/mapper2/up {"app_id":"ttnmapper","dev_id":"mapper2","hardware_serial":"0004A30B001ADBC5","port":1,"counter":0,"payload_raw":"AA==","metadata":{"time":"2017-08-19T15:23:39.288816687Z"}}
==== Habitat uploader ====
===== payload upload =====
===== payload upload =====
This uses a HTTP interface, performing a PUT to a certain URL with a certain content:
The application communicates with the habhub server using a HTTP interface, performing a PUT to a certain URL with a certain content:
* the URL is http://habitat.habhub.org/habitat/_design/payload_telemetry/_update/add_listener/<doc_id>
* the URL is http://habitat.habhub.org/habitat/_design/payload_telemetry/_update/add_listener/<doc_id>
* the <doc_id> is created from the telemetry sentence by doing ASCII to BASE64 conversion, then hashing using SHA-256 and encoding as ASCII-hex (as LOWER case!).
* the <doc_id> is created from the telemetry sentence by doing ASCII to BASE64 conversion, then hashing using SHA-256 and encoding as ASCII-hex (as LOWER case!).
Line 102: Line 128:
</pre>
</pre>


The "[receiver_id]" part can be repeated as many times as there are gateways that received the data.
<del>The "[receiver_id]" part can be repeated as many times as there are gateways that received the data.</del> Eh, no, this results in a 500 internal error from habitat!
This is probably also the trickiest part to implement in Jackson, because it uses variable field names.


===== listener upload =====
===== listener upload =====
To figure out how to upload information about the receiver station, I captured some traffic between dl-fldigi and habitat, it seems the following happens:
To figure out how to upload information about the receiver station, I captured some traffic between dl-fldigi and habitat, the following happens:
* a GET is done on http://habitat.habhub.org/_uuids?count=100, this returns a list of 100 UUIDs, where a UUID is the lower-case ascii hex representation of 16 bytes.
* a GET is done on http://habitat.habhub.org/_uuids?count=100, this returns a list of 100 UUIDs, where a UUID is the lower-case ascii hex representation of 16 bytes.
* a PUT is done with a "listener_information" doc to /habitat/<the first UUID>
* a PUT is done with a "listener_information" doc to /habitat/<the first UUID>
* a PUT is done with a "listener_telemetry" doc to /habitat/<the second UUID>
* a PUT is done with a "listener_telemetry" doc to /habitat/<the second UUID>


Perhaps implement this with an expiring cache, so listener information/telemetry is uploaded regularly (say every 20 minutes) but not with *every* payload telemetry.
This is implemented with an expiring cache, so listener information/telemetry is uploaded regularly (say every 20 minutes) but not with *every* payload telemetry.


==== Payload encoder/decoder ====
==== Payload encoder/decoder ====
This part decodes the binary payload received from the TTN into a standard habitat sentence, like
This part decodes the binary payload received from the TTN into a standard habitat sentence.
  $$hadie,181,10:42:10,54.422829,-6.741293,27799.3,1:10*002A


==== Gateway cache ====
The bridge application currently allows the following encodings for the telemetry data, configurable in the application configuration:
We don't actually need this, it seems the required gateway data (id/lat/lon/alt) is already provided by the MQTT stream.
* 'cayenne' (recommended): [https://mydevices.com/cayenne/docs/lora/#lora-cayenne-low-power-payload "cayenne" encoding]
* 'sodaqone': [https://github.com/SodaqMoja/SodaqOne-UniversalTracker SodaqOne universal tracker] conventions.
* 'json': a JSON encoded format


=== Implementation ===
See the table and example below to know which fields are required to construct the habitat ASCII sentence from the LoRa data payload sent by your tracker.
The bridge will be written in Java.


Libraries to use:
The habitat ASCII sentence typically looks like:
* all application settings are kept in a .properties file
  $$ttntest1,64,20:41:10,52.022100,4.693100,20.0,14.0,3.88*1C8A
* slf4j as the logging interface, log4j as the logging implementation
 
* junit/mockito for unit testing
{|class="wikitable"
* jetty/jersey/jackson for the REST interface towards habitat
|+Property mapping
* zip/tar-file for easy installation using the gradle application plugin
|-
* for the MQTT interface: not sure, either eclipse paho or fusesource
!Field value!!SodaqOne raw!!JSON!!Cayenne!!Remark
|-
| ttntest1 || TTN node name || TTN node name || TTN node name || TTN node name
|-
| 64 || LoRaWAN FCNT || LoRaWAN FCNT || LoRaWAN FCNT || Frame count
|-
| 20:41:10 || field 0 || TTN metadata time || TTN metadata time || TTN metadata time
|-
| 52.022100 || field 3 || "lat" || see below || latitude (degrees)
|-
| 4.693100 || field 4 || "lon" || see below || longitude (degrees)
|-
| 20.0 || field 5 || "gpsalt" || see below || altitude (meters)
|-
| 14.0 || field 2 || "temp" || see below || temperature (degrees Celcius)
|-
| 3.88 || field 1 || "vcc" || see below || battery voltage (volts)
|}
 
For Cayenne encoding, the UKHAS sentence starts with the node name, sequence number and time.
The fields following after that are simply all fields as they appear in the Cayenne message.
I recommend to put the GPS position *first* in the cayenne message.
Typically you would put in the Cayenne message: GPS position (latitude/longitude/altitude), temperature, battery voltage.
 
The field 1C8A is the CCITT-CRC16 over the characters between $$ and *, interpreted as bytes using US-ASCII encoding
 
==== Listener cache ====
The cache keeps track of when listener information / telemetry was sent last.
This makes it possible to avoid sending the listener information / telemetry with each payload telemetry document, reducing the load on the server.
 
Listener information / telemetry is sent only:
* when this is the first time we are sending something for this listener
* when it has been more than X minutes ago that we sent listener information / telemetry for this listener, where X is typically 10 minutes or so.


== Helpful links ==
== Helpful links ==
From a conversation on #highaltitude:
<pre>
20:24 < adamgreig> there's a) a python library that's a lot easier to read
20:24 < adamgreig> but b) basically the gist is you just PUT to http://habitat.habhub.org/habitat/_design/payload_telemetry/_update/add_listener/<id> with some stuff
20:24 < adamgreig> http://habitat.readthedocs.io/en/latest/habitat/habitat/habitat/habitat.views.payload_telemetry.html#module-habitat.views.payload_telemetry
20:24 < adamgreig> so you have a new string, you take the sha256 hex digest of the base64 encoded raw data
20:25 < adamgreig> you PUT to that URL with that ID
20:25 < adamgreig> and you include that JSON struture with your callsign/details
</pre>


[http://stuff.dbrooke.me.uk/sp-VpK2Cw.c C implementation of the interface between the client and the habitat server]
[http://stuff.dbrooke.me.uk/sp-VpK2Cw.c C implementation of the interface between the client and the habitat server]
Line 147: Line 194:
[http://habitat.habhub.org/jse/ list of habitat JSON schemas]
[http://habitat.habhub.org/jse/ list of habitat JSON schemas]


I think/hope the interface above can be approached as a REST service, for which is very easy to create a client (and test server) in Java, using a combination of Jetty/Jersey/Jackson libraries.
== Tracker configuration for the TTN-HAB bridge software ==
 
Roughly the following configuration is needed for your tracker to use the TTN-HAB bridge:
* setup your LoRaWAN device for use with TheThingsNetwork
* create a payload configuration document on the habhub.org webpage
* download, configure and run the ttnhabbridge software
 
=== Setting up the tracker for TTN ===
 
Find out the LoRaWAN EUI of your tracker:
* specifically for the SodaqOne, the EUI (LoRaWAN hardware address) is shown at startup of the tracker
* perhaps just make up your own, if you don't use an RN2483?


== Settings ==
Make up your mind about which binary payload format you are going to use:
Settings likely needed by the bridge application:
I recommend the 'cayenne' payload format. It's relatively flexible and is an actual standard. It is supported by ttnhabbridge without needing customisation.
* TTN gateway API settings
You need to put in at least the GPS coordinate in your cayenne message. If you have it available, I would also put in the battery voltage and temperature in it.
** URL of the TTN gateway API
* TTN MQTT API settings
** URL of the TTN MQTT API
** application id of the TTN 'hab' application
** application username
** application password
* habitat database settings
** URL of the habitat server
** ...


== JSON examples ==
Create the TTN application and register your tracker with TTN (<b>Note: this was written for the previous TTNv2 console, to be updated</b>)
=== TTN console gateway traffic ===
* Create an account on TTN, https://account.thethingsnetwork.org/
In the TTN console, under gateway traffic, the 'Event data' looks like this:
* Go to the [https://console.thethingsnetwork.org/ TTN console] and create a new application, or select an existing application you want to add the device to.
<pre>
* If you use Cayenne as the payload format, you can configure this under 'Payload Formats', this allows you to view the raw payload as a kind of JSON format in the TTN console
{
* create a new node/device, the name needs to be equal to the habhub payload name (but cannot use uppercase characters)
  "gw_id": "eui-008000000000b8b6",
* In the device settings screen:
  "payload": "QONzlgaABAAB8XtDve5FPT4jnzvJ",
** under 'Device EUI', fill in the EUI of your LoraWAN tracker
  "f_cnt": 4,
** under 'Activation method' choose <b>ABP</b>
  "lora": {
** <b>disable</b> option 'Frame Counter Checks'
    "spreading_factor": 7,
** click Save
    "bandwidth": 125,
 
    "air_time": 56576000
Copy the network credentials back to your LoRaWAN tracker:
  },
* this means the "device address", "network key" and "application key"
  "coding_rate": "4/5",
* in case you use a SodaqOne serial debug console, type the following:
  "timestamp": "2017-08-14T12:12:52.614Z",
** copy the Network Session Key from the TTN console and type in the SodaqOne serial debug console 'key=<key>'
  "rssi": -103,
** copy the App Session Key from the TTN console and type in the SodaqOne serial debug console 'app=<key>'
  "snr": 3.8,
** copy the device id from the TTN console and type in the SodaqOne serial debug console 'dev=<deviceid>' (is this right?)
  "dev_addr": "069673E3",
* configure the following additional settings in the SodaqOne:
  "frequency": 868100000
** enable GPS: 'gps=1'
}
** set GPS fix interval: 'fi=1'
</pre>
** set GPS fix timeout: 'gft=30'
** set num coords to upload: 'num=1'
** set min GPS satellite count: 'sat=1'
** enable ABP mode (disable OTAA): 'otaa=0'
** disable ADR: 'adr=0'
** set spreading factor 7: 'sf=7'
 
=== Creating a payload configuration document ===
The payload configuration document allows habhub to decode the UKHAS sentence back into meaningful data items,
like which fields contain latitude, longitude, altitude, etc.
 
Steps:
* Go to the habhub habitat website and [http://habitat.habhub.org/genpayload/ <b>create new</b> payload document]
* under 'payload name' fill in the callsign (e.g. 'ttntest1'), <b>this has to match exactly with the TTN node name</b>
** add a parser configuration, using the <b>new format wizard</b>
** enter the following example string: $$ttntest1,64,20:41:10,52.022100,4.693100,20.0,14.0,3.88*1C8A
* the fields are the following:
** sentence id
** time
** <b>all of the Cayenne fields</b> in the order they appear in the Cayenne message.
* click save


=== TTN console application data ===
So, this contains the basic position information, plus two fields for temperature and battery level.
In the TTN console, under Applications, the 'Application data' looks like this:
<pre>
TODO
</pre>
<i>I *think* this data contains also the gateways with their id, latitude, longitude, altitude.</i>


=== MQTT data ===
=== Configuring the TTN-HAB bridge software ===
Over MQTT, it looks like this
* get the release zip/tar file and unzip it in some place
(this is a pretty old sample!):
* make sure you have Java 8 installed
<pre>
* Edit the configuration file (ttnhabbridge.yaml)
{
** thethingsnetwork.url is the URL of the TheThingsNetwork MQTT server, use 'eu1' for Europe
  "payload": "eyJIZWxsbyI6ImJlcnRyaWtAc2lra2VuLm5sIn0=",
** thethingsnetwork.user is the name of your TTN application
  "port": 1,
** thethingsnetwork.pass is the API key (with access right for reading application uplinks) of your TTN application
  "counter": 3,
** thethingsnetwork.topic is the MQTT topic that messages appear on
  "dev_eui": "0000000019800501",
** helium.url is the URL of the Helium MQTT server, leave this empty if you don't use Helium
  "metadata": [
* start the bridge software from the command line, either using the .bat file (for windows) or the .sh file (for Linux)
    {
* if you use Linux with systemd, you can use the ttnhabbridge.service file to run it as a service, see instructions inside this file
      "frequency": 867.3,
      "datarate": "SF12BW125",
      "codingrate": "4/5",
      "gateway_timestamp": 1954388012,
      "gateway_time": "2016-08-06T19:06:05.891844Z",
      "channel": 4,
      "server_time": "2016-08-06T19:06:05.926706718Z",
      "rssi": -118,
      "lsnr": -10.5,
      "rfchain": 0,
      "crc": 1,
      "modulation": "LORA",
      "gateway_eui": "AA555A00080605B7",
      "altitude": 40,
      "longitude": 5.16738,
      "latitude": 52.08393
    }
  ]
}
</pre>


Simulated uplink (using the console):
Example with TheThingsNetwork and Helium:
<pre>
<pre>
{
---
   "app_id": "ttnmapper",
thethingsnetwork:
   "dev_id": "mapper2",
   url: "tcp://eu1.cloud.thethings.network"
   "hardware_serial": "0004A30B001ADBC5",
   user: "icss-lora-tracker@ttn"
   "port": 1,
   pass: "NNSXS.etcetera"
   "counter": 0,
   topic: "v3/+/devices/+/up"
   "payload_raw": "AA==",
helium:
   "metadata": {
   url: "tcp://beta.medadnewman.co.uk:1887"
    "time": "2017-08-19T15:23:39.288816687Z"
   user: "medad"
   }
  pass: "thesecretpassword"
}
   topic: "helium/+/rx"
habitat:
  url: "http://habitat.habhub.org"
   timeout: 60
gwCacheExpirationTime: 600
payloadEncoding: "cayenne"
</pre>
</pre>

Latest revision as of 15:25, 21 April 2022

Project TTNHABBridge
Ttnhabbridge.png
A software bridge between TheThingsNetwork and the UKHAS high-balloon network
Status Completed
Contact bertrik
Last Update 2022-04-21

Status

This software is a bridge application that accepts high altitude balloon telemetry data from TheThingsNetwork and forwards it to the habhub tracker website.

It has been tested in practice with the 'koppelting1' payload on 2017-8-26. If you look at the habhub map and see a receiver called EUI-xxxxxxxx then that one is probably generated by this software.

Next steps

Future development:

  • add support for running this in a Docker container: has been realized, but not documented yet
  • add support for sondehub, see also https://github.com/bertrik/ttnhabbridge/issues/6
    • need to implement data structures
    • need to restructure internals: sondehub uses a JSON structure, not a 'sentence' like UKHAS does

Introduction

This application uses TheThingsNetwork as a receiver for high-altitude balloon telemetry and forwards it to habhub.org, so it can be shown on their overview map. The habhub map shows the current position of the balloon, but also shows a prediction (based on weather model) about where the balloon will burst, where it will land, which receivers are currently in range, etc.

In the current situation, high altitude balloons send telemetry on the 434 MHz band using RTTY modulation. This is received by a set of dedicated receivers. The operator of each receiver (typically radio amateurs) has to prepare his radio setup for receiving the telemetry, by tuning to the correct frequency at the correct time, setting up a dedicated software client that decodes the RTTY modulation and forwards the data to a central system over the internet.

In the new situation, the existing infrastructure of TheThingsNetwork is used to receive low-bitrate telemetry packets and forward them on the internet. It uses LoRa as a modulation scheme which allows for very small transmission power and is much less susceptible to slight tuning errors than the currently used RTTY modulation. Packets are encoded using the LoRaWAN protocol. Telemetry packets can be received completely autonomously by TTN gateways, there is no need for a radio operator to make adjustments etc.

In short, the idea is:

  • you attach a LoRaWAN transmitter to the balloon, which been pre-configured with a set of keys generated by the TTN
  • the balloon broadcasts its telemetry every once in a while (say a few times per minute) and this is picked up by one or more TTN gateways which forward into the TTN infrastructure
  • the bridge software listens for packets received by the TTN and decodes the payload data into an id, latitude, longitude, altitude of the balloon
  • the bridge software converts each telemetry packet according to 'UKHAS' conventions and forwards it to the habitat.habhub.org server
  • the bridge software also forwards the TTN gateway locations, so they also appear on the map, along with their name / EUI and antenna altitude
  • the habitat.habhub.org server still sees the same messages like it would if there were many traditional receivers, so doesn't need any modification

This way, the entire things network can be used to receive balloon telemetry. There is no longer a need for radio operators to be present at their receiver at the exact time the balloon is launched, making manual adjustments, etc. The Netherlands is already covered by many TTN gateways, greatly increasing the chance the balloon telemetry will be picked up.

Software

Source code

Source code is available at: https://github.com/bertrik/ttnhabbridge

The README explains the tool chain setup.

Building the software

Steps (on Debian Linux):

  • install a java-11 JDK
 sudo apt install openjdk-11-jdk
  • clone the software from github
 git clone https://github.com/bertrik/ttnhabbridge
  • enter directory ttnhabbridge
 cd ttnhabbridge
  • run the gradle script to build the executable
 ./gradlew assemble (Linux)
 gradlew assemble (Windows)
  • the executable is now available at ttnhabbridge/build/distributions/ttnhabbridge.tar and ttnhabbridge.zip

Running the software using Docker

To be documented ...

  • install docker and docker-compose
 sudo apt install docker-ce ?
  • get the source code
 git clone https://github.com/bertrik/ttnhabbridge
  • enter the docker directory
 cd ttnhabbridge
 cd docker
  • edit the config file
  • get the docker image
 docker-compose pull
  • run the docker image
 docker-compose up

Running the software from the zip-file / tar-ball

  • extract either the tar (Linux) or zip (Windows) that was generated in the build step
  • run it once to generate the initial configuration file, this file is called ttnhabbridge.yaml
 cd ttnhabbridge
 bin/ttnhabbridge
  • stop it, using CTRL-C
  • edit the ttnhabbridge.yaml configuration file with a text editor, enter your application name, keys, payload encoding, etc.
  • run it again and verify that the program starts correctly with the new configuration file

Design

The software consists of the following modules.

Main process

The main process of the bridge is something like this:

  • listen to the MQTT stream of the HAB application
  • once we get data:
    • decode the payload into latitude/longitude/altitude, and encode it into a habhub ASCII sentence with correct CRC, see https://ukhas.org.uk/communication:protocol
    • for each TTN gateway that received the data:
      • send a listener info document and a listener telemetry document to the habhub server
      • send the payload telemetry (ASCII sentence) to the habhub server

Habitat uploader

The habitat interface is accessed as a REST service.

payload upload

The application communicates with the habhub server using a HTTP interface, performing a PUT to a certain URL with a certain content:

  • the URL is http://habitat.habhub.org/habitat/_design/payload_telemetry/_update/add_listener/<doc_id>
  • the <doc_id> is created from the telemetry sentence by doing ASCII to BASE64 conversion, then hashing using SHA-256 and encoding as ASCII-hex (as LOWER case!).
  • the HTTP method is a PUT with the following headers:
    • "Accept: application/json"
    • "Content-Type: application/json"
    • "charsets: utf-8"
  • The contents of the PUT is the following JSON structure
{
  "data": {
    "_raw": "[base64 of telemetry sentence]"
  },
  "receivers": {
    "[receiver_id]": {
      "time_created": "[timestamp]",
      "time_uploaded": "[timestamp]"
    }
  }
}

The "[receiver_id]" part can be repeated as many times as there are gateways that received the data. Eh, no, this results in a 500 internal error from habitat!

listener upload

To figure out how to upload information about the receiver station, I captured some traffic between dl-fldigi and habitat, the following happens:

  • a GET is done on http://habitat.habhub.org/_uuids?count=100, this returns a list of 100 UUIDs, where a UUID is the lower-case ascii hex representation of 16 bytes.
  • a PUT is done with a "listener_information" doc to /habitat/<the first UUID>
  • a PUT is done with a "listener_telemetry" doc to /habitat/<the second UUID>

This is implemented with an expiring cache, so listener information/telemetry is uploaded regularly (say every 20 minutes) but not with *every* payload telemetry.

Payload encoder/decoder

This part decodes the binary payload received from the TTN into a standard habitat sentence.

The bridge application currently allows the following encodings for the telemetry data, configurable in the application configuration:

See the table and example below to know which fields are required to construct the habitat ASCII sentence from the LoRa data payload sent by your tracker.

The habitat ASCII sentence typically looks like: 

 $$ttntest1,64,20:41:10,52.022100,4.693100,20.0,14.0,3.88*1C8A
Property mapping
Field value SodaqOne raw JSON Cayenne Remark
ttntest1 TTN node name TTN node name TTN node name TTN node name
64 LoRaWAN FCNT LoRaWAN FCNT LoRaWAN FCNT Frame count
20:41:10 field 0 TTN metadata time TTN metadata time TTN metadata time
52.022100 field 3 "lat" see below latitude (degrees)
4.693100 field 4 "lon" see below longitude (degrees)
20.0 field 5 "gpsalt" see below altitude (meters)
14.0 field 2 "temp" see below temperature (degrees Celcius)
3.88 field 1 "vcc" see below battery voltage (volts)

For Cayenne encoding, the UKHAS sentence starts with the node name, sequence number and time. The fields following after that are simply all fields as they appear in the Cayenne message. I recommend to put the GPS position *first* in the cayenne message. Typically you would put in the Cayenne message: GPS position (latitude/longitude/altitude), temperature, battery voltage.

The field 1C8A is the CCITT-CRC16 over the characters between $$ and *, interpreted as bytes using US-ASCII encoding

Listener cache

The cache keeps track of when listener information / telemetry was sent last. This makes it possible to avoid sending the listener information / telemetry with each payload telemetry document, reducing the load on the server.

Listener information / telemetry is sent only:

  • when this is the first time we are sending something for this listener
  • when it has been more than X minutes ago that we sent listener information / telemetry for this listener, where X is typically 10 minutes or so.

Helpful links

C implementation of the interface between the client and the habitat server

list of habitat JSON schemas

Tracker configuration for the TTN-HAB bridge software

Roughly the following configuration is needed for your tracker to use the TTN-HAB bridge:

  • setup your LoRaWAN device for use with TheThingsNetwork
  • create a payload configuration document on the habhub.org webpage
  • download, configure and run the ttnhabbridge software

Setting up the tracker for TTN

Find out the LoRaWAN EUI of your tracker:

  • specifically for the SodaqOne, the EUI (LoRaWAN hardware address) is shown at startup of the tracker
  • perhaps just make up your own, if you don't use an RN2483?

Make up your mind about which binary payload format you are going to use: I recommend the 'cayenne' payload format. It's relatively flexible and is an actual standard. It is supported by ttnhabbridge without needing customisation. You need to put in at least the GPS coordinate in your cayenne message. If you have it available, I would also put in the battery voltage and temperature in it.

Create the TTN application and register your tracker with TTN (Note: this was written for the previous TTNv2 console, to be updated)

  • Create an account on TTN, https://account.thethingsnetwork.org/
  • Go to the TTN console and create a new application, or select an existing application you want to add the device to.
  • If you use Cayenne as the payload format, you can configure this under 'Payload Formats', this allows you to view the raw payload as a kind of JSON format in the TTN console
  • create a new node/device, the name needs to be equal to the habhub payload name (but cannot use uppercase characters)
  • In the device settings screen:
    • under 'Device EUI', fill in the EUI of your LoraWAN tracker
    • under 'Activation method' choose ABP
    • disable option 'Frame Counter Checks'
    • click Save

Copy the network credentials back to your LoRaWAN tracker:

  • this means the "device address", "network key" and "application key"
  • in case you use a SodaqOne serial debug console, type the following:
    • copy the Network Session Key from the TTN console and type in the SodaqOne serial debug console 'key=<key>'
    • copy the App Session Key from the TTN console and type in the SodaqOne serial debug console 'app=<key>'
    • copy the device id from the TTN console and type in the SodaqOne serial debug console 'dev=<deviceid>' (is this right?)
  • configure the following additional settings in the SodaqOne:
    • enable GPS: 'gps=1'
    • set GPS fix interval: 'fi=1'
    • set GPS fix timeout: 'gft=30'
    • set num coords to upload: 'num=1'
    • set min GPS satellite count: 'sat=1'
    • enable ABP mode (disable OTAA): 'otaa=0'
    • disable ADR: 'adr=0'
    • set spreading factor 7: 'sf=7'

Creating a payload configuration document

The payload configuration document allows habhub to decode the UKHAS sentence back into meaningful data items, like which fields contain latitude, longitude, altitude, etc.

Steps:

  • Go to the habhub habitat website and create new payload document
  • under 'payload name' fill in the callsign (e.g. 'ttntest1'), this has to match exactly with the TTN node name
    • add a parser configuration, using the new format wizard
    • enter the following example string: $$ttntest1,64,20:41:10,52.022100,4.693100,20.0,14.0,3.88*1C8A
  • the fields are the following:
    • sentence id
    • time
    • all of the Cayenne fields in the order they appear in the Cayenne message.
  • click save

So, this contains the basic position information, plus two fields for temperature and battery level.

Configuring the TTN-HAB bridge software

  • get the release zip/tar file and unzip it in some place
  • make sure you have Java 8 installed
  • Edit the configuration file (ttnhabbridge.yaml)
    • thethingsnetwork.url is the URL of the TheThingsNetwork MQTT server, use 'eu1' for Europe
    • thethingsnetwork.user is the name of your TTN application
    • thethingsnetwork.pass is the API key (with access right for reading application uplinks) of your TTN application
    • thethingsnetwork.topic is the MQTT topic that messages appear on
    • helium.url is the URL of the Helium MQTT server, leave this empty if you don't use Helium
  • start the bridge software from the command line, either using the .bat file (for windows) or the .sh file (for Linux)
  • if you use Linux with systemd, you can use the ttnhabbridge.service file to run it as a service, see instructions inside this file

Example with TheThingsNetwork and Helium: 

---
thethingsnetwork:
  url: "tcp://eu1.cloud.thethings.network"
  user: "icss-lora-tracker@ttn"
  pass: "NNSXS.etcetera"
  topic: "v3/+/devices/+/up"
helium:
  url: "tcp://beta.medadnewman.co.uk:1887"
  user: "medad"
  pass: "thesecretpassword"
  topic: "helium/+/rx"
habitat:
  url: "http://habitat.habhub.org"
  timeout: 60
gwCacheExpirationTime: 600
payloadEncoding: "cayenne"
Retrieved from ‘https://revspace.nl/wiki/index.php?title=TTNHABBridge&oldid=29577’