TTNHABBridge: Difference between revisions
(→Status) |
|||
Line 12: | Line 12: | ||
== Introduction == | == Introduction == | ||
This | This application uses TheThingsNetwork as a receiver for high-altitude balloon telemetry and forwards it to habhub.org, so it can be shown on their overview map. | ||
The habhub map shows the current position of the balloon, but also shows a prediction (based on weather model) about where the balloon will burst, where it will land, which receivers are currently in range, etc. | |||
In the current situation, high altitude balloons send telemetry on the 434 MHz band using RTTY modulation. Dedicated receivers listening for RTTY-modulated ASCII strings. | |||
The operator of each receiver has to prepare his radio setup for receiving the telemetry, by tuning to the correct frequency at the correct time, setting up a dedicated software client that decodes the RTTY modulation and forwards the data to a central system over the internet. | The operator of each receiver (typically radio amateurs) has to prepare his radio setup for receiving the telemetry, by tuning to the correct frequency at the correct time, setting up a dedicated software client that decodes the RTTY modulation and forwards the data to a central system over the internet. | ||
TheThingsNetwork | In the new situation, the existing infrastructure of TheThingsNetwork is used to receive low-bitrate telemetry packets and forward them on the internet. | ||
It uses LoRa as a modulation scheme which allows for very small transmission power and is much less susceptible to slight tuning errors than the currently used RTTY modulation. | It uses LoRa as a modulation scheme which allows for very small transmission power and is much less susceptible to slight tuning errors than the currently used RTTY modulation. | ||
This means that telemetry packets can be received completely autonomously by TTN gateways, no need for a radio operator to make adjustments etc. | This means that telemetry packets can be received completely autonomously by TTN gateways, no need for a radio operator to make adjustments etc. | ||
Line 26: | Line 26: | ||
* the balloon broadcasts its telemetry every once in a while (say a few times per minute) and this is picked up by one or more TTN gateways which forward into the TTN infrastructure | * the balloon broadcasts its telemetry every once in a while (say a few times per minute) and this is picked up by one or more TTN gateways which forward into the TTN infrastructure | ||
* the bridge software listens for packets received by the TTN and decodes the payload data into an id, latitude, longitude, altitude of the balloon | * the bridge software listens for packets received by the TTN and decodes the payload data into an id, latitude, longitude, altitude of the balloon | ||
* the bridge software converts each telemetry packet according to 'UKHAS' conventions and | * the bridge software converts each telemetry packet according to 'UKHAS' conventions and forwards it to the habitat.habhub.org server | ||
* the bridge software also forwards the TTN gateway locations, so they also appear on the map, along with their name / EUI and antenna altitude | * the bridge software also forwards the TTN gateway locations, so they also appear on the map, along with their name / EUI and antenna altitude | ||
* the habitat.habhub.org server still sees the same messages like it would if there were many traditional receivers, so doesn't need any modification! | * the habitat.habhub.org server still sees the same messages like it would if there were many traditional receivers, so doesn't need any modification! |
Revision as of 06:33, 4 May 2019
Project TTNHABBridge | |
---|---|
![]() | |
A software bridge between TTN and the HAB network | |
Status | Completed |
Contact | bertrik |
Last Update | 2019-05-04 |
Status
A bridge application that accepts balloon telemetry data from TTN and forwards it to habhub has been implemented in Java and works. It has been tested in practice with the 'koppelting1' payload on 2017-8-26.
Introduction
This application uses TheThingsNetwork as a receiver for high-altitude balloon telemetry and forwards it to habhub.org, so it can be shown on their overview map. The habhub map shows the current position of the balloon, but also shows a prediction (based on weather model) about where the balloon will burst, where it will land, which receivers are currently in range, etc.
In the current situation, high altitude balloons send telemetry on the 434 MHz band using RTTY modulation. Dedicated receivers listening for RTTY-modulated ASCII strings. The operator of each receiver (typically radio amateurs) has to prepare his radio setup for receiving the telemetry, by tuning to the correct frequency at the correct time, setting up a dedicated software client that decodes the RTTY modulation and forwards the data to a central system over the internet.
In the new situation, the existing infrastructure of TheThingsNetwork is used to receive low-bitrate telemetry packets and forward them on the internet. It uses LoRa as a modulation scheme which allows for very small transmission power and is much less susceptible to slight tuning errors than the currently used RTTY modulation. This means that telemetry packets can be received completely autonomously by TTN gateways, no need for a radio operator to make adjustments etc.
In short, the idea is:
- you attach a LoRaWAN transmitter to the balloon, which been pre-configured with a set of keys generated by the TTN
- the balloon broadcasts its telemetry every once in a while (say a few times per minute) and this is picked up by one or more TTN gateways which forward into the TTN infrastructure
- the bridge software listens for packets received by the TTN and decodes the payload data into an id, latitude, longitude, altitude of the balloon
- the bridge software converts each telemetry packet according to 'UKHAS' conventions and forwards it to the habitat.habhub.org server
- the bridge software also forwards the TTN gateway locations, so they also appear on the map, along with their name / EUI and antenna altitude
- the habitat.habhub.org server still sees the same messages like it would if there were many traditional receivers, so doesn't need any modification!
This way, the entire things network can be used to receive balloon telemetry! There is no longer a need for radio operators to be present at their receiver at the exact time the balloon is launched, making manual adjustments, etc. The Netherlands is already covered by many TTN gateways, greatly increasing the chance the balloon telemetry will be picked up.
Software
Source code
Source code is available at: https://github.com/bertrik/ttnhabbridge
The README explains the tool chain setup.
Using this source code, the basic functionality works.
There is no user guide yet, but the .properties settings file has a one-line comment explaining what each setting does.
Modules
The software consists of the following modules.
Main process
The main process of the bridge is something like this:
- listen to the MQTT stream of the HAB application
- once we get data:
- decode the payload into latitude/longitude/altitude, and encode it into a habhub ASCII sentence with correct CRC, see https://ukhas.org.uk/communication:protocol
- for each gateway that received the data:
- send a listener info and a listener telemetry document to the habhub server
- send the payload telemetry (ASCII sentence) to the habhub server
MQTT listener
Implemented using the Eclipse paho MQTT client. We can listen on a certain application and catch *all* traffic for all nodes registered to that application.
Example invocation, using mosquitto_sub:
bertrik@zenbook:~$ mosquitto_sub -h eu.thethings.network -t '+/devices/+/up' -u 'ttnmapper' -P 'ttn-account-v2.Xc8BFRKeBK5nUhc9ikDcR-sbelgSMdHKnOQKMAiwpgI' -v
Actual result:
ttnmapper/devices/mapper2/up {"app_id":"ttnmapper","dev_id":"mapper2","hardware_serial":"0004A30B001ADBC5","port":1,"counter":1,"payload_raw":"d4WaWXATAAAAAAAAAAAAAAAAAAD/","metadata":{"time":"2017-08-21T07:02:14.842855925Z","frequency":868.1,"modulation":"LORA","data_rate":"SF7BW125","coding_rate":"4/5","gateways":[{"gtw_id":"eui-008000000000b8b6","timestamp":865620531,"time":"2017-08-21T07:02:14.846095Z","channel":0,"rssi":-119,"snr":-3.8,"rf_chain":1,"latitude":52.0182,"longitude":4.70844,"altitude":27}]}} ttnmapper/devices/mapper2/up {"app_id":"ttnmapper","dev_id":"mapper2","hardware_serial":"0004A30B001ADBC5","port":1,"counter":4,"payload_raw":"loeaWW4T2+8BHzYZzAIeAA8A/QUS","metadata":{"time":"2017-08-21T07:11:18.313946438Z","frequency":868.3,"modulation":"LORA","data_rate":"SF7BW125","coding_rate":"4/5","gateways":[{"gtw_id":"eui-008000000000b8b6","timestamp":1409115451,"time":"2017-08-21T07:11:18.338662Z","channel":1,"rssi":-114,"snr":-0.2,"rf_chain":1,"latitude":52.0182,"longitude":4.70844,"altitude":27}]}}
Habitat uploader
The habitat interface is accessed as a REST service, using a combination of Jetty/Jersey/Jackson Java libraries.
payload upload
This uses a HTTP interface, performing a PUT to a certain URL with a certain content:
- the URL is http://habitat.habhub.org/habitat/_design/payload_telemetry/_update/add_listener/<doc_id>
- the <doc_id> is created from the telemetry sentence by doing ASCII to BASE64 conversion, then hashing using SHA-256 and encoding as ASCII-hex (as LOWER case!).
- the HTTP method is a PUT with the following headers:
- "Accept: application/json"
- "Content-Type: application/json"
- "charsets: utf-8"
- The contents of the PUT is the following JSON structure
{ "data": { "_raw": "[base64 of telemetry sentence]" }, "receivers": { "[receiver_id]": { "time_created": "[timestamp]", "time_uploaded": "[timestamp]" } } }
The "[receiver_id]" part can be repeated as many times as there are gateways that received the data. Eh, no, this results in a 500 internal error from habitat!
listener upload
To figure out how to upload information about the receiver station, I captured some traffic between dl-fldigi and habitat, the following happens:
- a GET is done on http://habitat.habhub.org/_uuids?count=100, this returns a list of 100 UUIDs, where a UUID is the lower-case ascii hex representation of 16 bytes.
- a PUT is done with a "listener_information" doc to /habitat/<the first UUID>
- a PUT is done with a "listener_telemetry" doc to /habitat/<the second UUID>
This is implemented with an expiring cache, so listener information/telemetry is uploaded regularly (say every 20 minutes) but not with *every* payload telemetry.
Payload encoder/decoder
This part decodes the binary payload received from the TTN into a standard habitat sentence.
The bridge application currently allows the following encodings for the telemetry data, configurable in the application properties:
- 'sodaqone': SodaqOne universal tracker conventions.
- 'json': a JSON encoded format
- 'cayenne': "cayenne" encoding
See the table and example below to know which fields are required to construct the habitat ASCII sentence from the LoRa data payload sent by your tracker.
The habitat ASCII sentence typically looks like:
$$ttntest1,64,20:41:10,52.022100,4.693100,20.0,14.0,3.88*1C8A
Field value | SodaqOne raw | JSON | Cayenne | Remark |
---|---|---|---|---|
ttntest1 | TTN node name | TTN node name | TTN node name | TTN node name |
64 | LoRaWAN FCNT | LoRaWAN FCNT | LoRaWAN FCNT | Frame count |
20:41:10 | field 0 | TTN metadata time | TTN metadata time | TTN metadata time |
52.022100 | field 3 | "lat" | latitude from GPS sensor | latitude (degrees) |
4.693100 | field 4 | "lon" | longitude from GPS sensor | longitude (degrees) |
20.0 | field 5 | "gpsalt" | altitude from GPS sensor | altitude (meters) |
14.0 | field 2 | "temp" | value from TEMPERATURE sensor | temperature (degrees Celcius) |
3.88 | field 1 | "vcc" | value from ANALOG_IN sensor | battery voltage (volts) |
The field 1C8A is the CCITT-CRC16 over the characters between $$ and *, interpreted as bytes using US-ASCII encoding
Listener cache
The cache keeps track of when listener information / telemetry was sent last. This makes it possible to avoid sending the listener information / telemetry with each payload telemetry document, reducing the load on the server.
Listener information / telemetry is sent only:
- when this is the first time we are sending something for this listener
- when it has been more than X minutes ago that we sent listener information / telemetry for this listener, where X is typically 10 minutes or so.
Helpful links
From a conversation on #highaltitude:
20:24 < adamgreig> there's a) a python library that's a lot easier to read 20:24 < adamgreig> but b) basically the gist is you just PUT to http://habitat.habhub.org/habitat/_design/payload_telemetry/_update/add_listener/<id> with some stuff 20:24 < adamgreig> http://habitat.readthedocs.io/en/latest/habitat/habitat/habitat/habitat.views.payload_telemetry.html#module-habitat.views.payload_telemetry 20:24 < adamgreig> so you have a new string, you take the sha256 hex digest of the base64 encoded raw data 20:25 < adamgreig> you PUT to that URL with that ID 20:25 < adamgreig> and you include that JSON struture with your callsign/details
C implementation of the interface between the client and the habitat server
JSON examples
MQTT data
Over MQTT, it looks like this:
{ "app_id": "ttnmapper", "dev_id": "mapper2", "hardware_serial": "0004A30B001ADBC5", "port": 1, "counter": 587, "payload_raw": "g3ybWXkTef8BH1EOzAK1\/wEA5wQT", "metadata": { "time": "2017-08-22T00:36:18.674804288Z", "frequency": 868.1, "modulation": "LORA", "data_rate": "SF7BW125", "coding_rate": "4\/5", "gateways": [ { "gtw_id": "eui-008000000000b8b6", "timestamp": 3979529499, "time": "2017-08-22T00:36:18.345616Z", "channel": 0, "rssi": -120, "snr": -7, "rf_chain": 1, "latitude": 52.0182, "longitude": 4.70844, "altitude": 27 } ] } }
Tracker configuration for the TTN-HAB bridge software
Roughly the following configuration is needed for your tracker to use the TTN-HAB bridge:
- setup your LoRaWAN device for use with TheThingsNetwork
- create a payload configuration document on the habhub.org webpage
- download, configure and run the ttnhabbridge software
Setting up the tracker for TTN
Find out the LoRaWAN EUI of your tracker:
- specifically for the SodaqOne, the EUI (LoRaWAN hardware address) is shown at startup of the tracker
- perhaps just make up your own, if you don't use an RN2483?
Make up your mind about which binary payload format you are going to use:
- I personally think the 'cayenne' payload format is very nice, because it's relatively flexible and is an actual standard. You need to put in at least the GPS coordinate in it. If you have it available, I would also put in the battery voltage and temperature in it.
Create the TTN application and register your tracker with TTN:
- Create an account on TTN
- Go to the TTN console and create a new application, or select an existing application you want to add the device to.
- create a new node/device, this needs to be equal to the habhub payload name (but cannot use uppercase characters)
- In the device settings screen:
- under 'Device EUI', fill in the EUI of your LoraWAN tracker
- under 'Activation method' choose ABP
- disable option 'Frame Counter Checks'
- click Save
Copy the network credentials back to your LoRaWAN tracker:
- this means the "device address", "network key" and "application key"
- in case you use a SodaqOne serial debug console, type the following:
- copy the Network Session Key from the TTN console and type in the SodaqOne serial debug console 'key=<key>'
- copy the App Session Key from the TTN console and type in the SodaqOne serial debug console 'app=<key>'
- copy the device id from the TTN console and type in the SodaqOne serial debug console 'dev=<deviceid>' (is this right?)
- configure the following additional settings in the SodaqOne:
- enable ABP mode (disable OTAA), by typing 'otaa=0'
- set spreading factor 7: 'sf=7'
- disable ADR: 'adr=0'
- enable GPS: 'gps=1'
- set GPS fix interval: 'fi=1'
- set GPS fix timeout: 'gft=30'
- set GPS satellite count: 'num=1'
Creating a payload configuration document
Steps:
- Go to the habhub habitat website and create a new payload document
- under 'payload name' fill in the callsign (e.g. 'ttntest1'), this has to match exactly with the TTN device/node name
- add a parser configuration, using the 'new format wizard'
- enter the following example string: $$ttntest1,64,20:41:10,52.022100,4.693100,20.0,14.0,3.88*1C8A
- click save
Configuring the TTN-HAB bridge software
- get the release zip file and unzip it in some place
- make sure you have Java 8 installed
- In the bridge configuration properties file (ttnhabbridge.properties), fill in the following
- ttn.app.id with the name of your TTN application
- ttn.app.key with the key of your TTN application (you can find this in the TTN console)
- ttn.payload.encoding with the type of payload encoding you use ('sodaqone' for their tracker software)
- you can probably leave the other settings at their default value
- start the bridge software from the command line, either using the .bat file (for windows) or the .sh file (for Linux)