Difference between revisions of "LoraWanDustSensor"

From RevSpace
Jump to navigation Jump to search
(The plan)
(Platformio)
 
(71 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
{{Project
 
{{Project
 
   |Name=LoRaWAN dust Sensor
 
   |Name=LoRaWAN dust Sensor
   |Picture=no
+
   |Picture=LoraWanDustSensor.jpg
 
   |Omschrijving=LoRaWAN airborne particulate matter sensor
 
   |Omschrijving=LoRaWAN airborne particulate matter sensor
   |Status=Initializing
+
   |Status=In progress
 
   |Contact=bertrik
 
   |Contact=bertrik
 
   }}
 
   }}
  
 +
== The idea ==
 +
The idea is to create a system consisting of:
 +
* a sensor that measures airborne particulate matter and sends the measurement data using LoRa to TheThingsNetwork.
 +
* a forwarder application that collects the data from TTN and forwards it to luftdaten.info
  
== The plan ==
+
This has been done before by other people, but it appears there is no universal solution.
The plan is to create a system consisting of:
+
I am publishing all source code on github and will put up documentation on this wiki.
* a sensor that measures airborne particulate matter and sends the measurement data using LoRa/TheThingsNetwork to a central location.
+
Everyone invents their own payload format, something more universal like Cayenne LPP would be nice.
* a backend that collects the data from TTN and forwards it to a project like luftdaten.info
+
However I could not find a way to encode particulate matter data using Cayenne, so I'll just invent my own payload format too.
  
This has been done before by other people, but can't find any really good examples:
+
A similar thing has been done by:
* source code location is obscure, I will publish source code on github and put up documentation on this wiki
+
* https://github.com/VekotinVerstas/AQLoRaBurk
* payload format is non-standard, I'd like to use something relatively universal and standard, so I think I will use the Cayenne LPP format.
+
* https://github.com/alexcorvis84/LoRa_MakersAsturias
 +
* TTN Ulm, see https://github.com/verschwoerhaus/ttn-ulm-feinstaub (the sensor code) and https://github.com/verschwoerhaus/ttn-ulm-muecke (the forwarder, in python)
 +
* https://alexander-schnapper.de/2019/04/02/mobile-feinstaubmessung/
 +
* [https://github.com/nijmeijer/TTN_Apeldoorn_in_Data_2018 Apeldoorn-in-data]
 +
* (others...)
  
So I'd like to just re-invent the wheel properly this time.
+
One thing in particular I'd like to do better than existing solutions is to use proper OTAA for the LoRa connection to TTN.
 +
OTAA means over-the-air-activation and is a mechanism to dynamically negotiate communication/encryption keys instead of programmed specifically in each sensor node.
 +
Once the OTAA is done successfully, the node remembers the network id, device address, session keys, etc for future communication.
  
=== Sensor ===
+
This makes it possible to have a single firmware image for all sensor nodes and it simplifies the setup:
The sensor will be an SDS-011, just like in the luftdaten project.
+
* you flash the node with a unified firmware
 +
* the node shows its unique id on the OLED
 +
* at the TTN console, you register a new device with the unique id
 +
* the sensor node receives encryption keys over the air automatically
 +
* done!
  
I will take a glance at how this sensor is being "driven" in this project, e.g. typical on-off times, averaging, etc.
+
(idea: an ESP32 has a wifi connection too, perhaps registering the node can be done fully automatically, over wifi/internet)
  
The node is based on Arduino.
+
=== Next steps ===
To compile the code, platformio is used, probably with the following libraries:
+
* <s>Investigate failing re-joins that seem to happen sometimes after the node has been online for a while</s> the arduino-lmic library has a fix for this now
 +
* Implement a kind of schedule: turn the sds011 on, wait some time, take a measurement, turn it off, wait for some time
 +
** make this match the TTN send schedule? it's useless to do a measurement if we don't have TTN airtime to transmit it
 +
* display SDS011 serial number and date code on the screen
 +
 
 +
=== Links ===
 +
 
 +
Useful links for the TTGO LoRa board:
 +
* https://primalcortex.wordpress.com/2017/11/24/the-esp32-oled-lora-ttgo-lora32-board-and-connecting-it-to-ttn
 
* https://github.com/fcgdam/TTGO_LoRa32
 
* https://github.com/fcgdam/TTGO_LoRa32
 +
* https://ictoblog.nl/2018/01/10/mijn-eerste-chinese-esp32-verbonden-met-the-things-network
 +
* [https://github.com/akrasnoshchok/LoRa/blob/master/esp32_heltec_v2/weather_station/weather_station.ino Example code that joins TTN by OTAA and saves the OTAA parameters]
 +
* https://github.com/CivicLabsBelgium/lora_particle_sensor
 +
* https://github.com/Cinezaster/ttn2luftdaten_forwarder
 +
 +
== Hardware ==
 +
The node is based on Arduino, in particular a TTGO ESP32 board with onboard SX1276 LoRa chip.
 +
The sensor is an SDS-011, just like in the luftdaten project.
 +
For humidity/temperature, I'd like to use a BME280.
 +
 +
NOTE: https://www.thethingsnetwork.org/community/berlin/post/warning-attention-users-of-ttgo21-v16-boards-labeled-t3_v16-on-pcb-battery-exploded-and-got-on-fire
 +
 +
[https://primalcortex.wordpress.com/tag/lora32/ Page with correct pinout of the ESP32 LoRa board].
 +
 +
Luftdaten uses a cycle time of 145 seconds for the SDS011.
 +
 +
Proposed hardware connections:
 +
* SDS011 5V to ESP32 5V
 +
* SDS011 GND to ESP32 GND
 +
* SDS011 TXD to ESP32 GPIO35 (maybe I can find two suitable pins close together)
 +
* SDS011 RXD to ESP32 GPIO25 (maybe I can find two suitable pins close together)
 +
* BME280 3V todo
 +
* BME280 SDA todo
 +
* BME280 SCL todo
 +
* BME280 GND todo
 +
 +
== Software ==
 +
=== Source code ===
 +
Source code is hosted on github:
 +
* [https://github.com/bertrik/LoraWanPmSensor Arduino node], written in C/Arduino, built using platformio
 +
* [https://github.com/bertrik/LoraLuftdatenForwarder TTN-to-luftdaten forwarder], written in Java, built using gradle
 +
 +
=== Common ===
 +
==== Packet format ====
 +
Proposed structure of packets transferred over LoRa:
 +
* PM10 value, encoded in units of 0.1 ug/m3: 2 bytes, big endian
 +
* PM2.5 value, encoded in units of 0.1 ug/m3: 2 bytes, big endian
 +
* temperature, encoded in units of 0.1 deg C: 2 bytes, signed big endian
 +
* relative humidity, encoded in units of 0.1%, 2 bytes, big endian
 +
Total: 8 bytes
 +
 +
Not present value is 0xFFFF. Encoding is big endian.
 +
 +
Would be nice to use Cayenne for this, but I don't know if Cayenne has an id for particulate matter.
 +
 +
How other projects encode the data:
 +
* TTN Apeldoorn (?): https://github.com/tijnonlijn/RFM-node/blob/master/dustduino_PPD42NS_example.ino#L327 sends 5 bytes
 +
** 1 byte : 0x04
 +
** 2 bytes: PM25(?) big endian
 +
** 2 bytes: PM10(?) big endian
 +
* TTN Ulm: https://github.com/verschwoerhaus/ttn-ulm-feinstaub/blob/master/ttnulmdust/ttnulmdust.ino#L225 sends 8 bytes:
 +
** 2 bytes: P10 (?) big endian (unit 0.01 ug/m3)
 +
** 2 bytes: P25 (?) big endian (unit 0.01 ug/m3)
 +
** 2 bytes: humidity (unit of 0.01% ?)
 +
** 2 bytes: temperature (unit of 0.01 degree Celcius)
 +
* RIVM node, sends 20 bytes
 +
** 1 byte temperature (unit deg Celcius ?)
 +
** 1 byte relative humidity (unit % ?)
 +
** 2 bytes pressure (unit?)
 +
** 2 bytes pm10 (unit?)
 +
** 2 bytes pm25 (unit?)
 +
** 2 bytes op1 (unit?)
 +
** 2 bytes op2 (unit?)
 +
** 4 bytes latitude (unit?)
 +
** 4 bytes longitude (unit?)
 +
* Apeldoorn in data: https://github.com/nijmeijer/TTN_Apeldoorn_in_Data_2018/blob/master/AiD_Dust_2018/AiD_Dust_2018.ino#L184
 +
** 4 bytes: pm2_5 float big endian (unit?)
 +
** 4 bytes: pm10 float big endian (unit?)
 +
** 4 bytes: humidity float big endian (unit?)
 +
** 4 bytes: temperature float big endian (unit?)
 +
 +
A smaller payload means less time in the air, smaller chance of collision with other LoRa packets and more packets per hour.
 +
 +
=== Node ===
 +
Source code for the particulate matter measurement node can be found [https://github.com/bertrik/LoraWanPmSensor on the github page].
 +
 +
==== Platformio ====
 +
To compile and upload the code to the node, platformio is used.
 +
 +
To install platformio (example for Debian):
 +
  sudo apt install python3-pip
 +
  sudo pip3 install platformio
 +
  pio update
 +
 +
To compile and upload:
 +
  pio run -t upload
 +
 +
==== Design ====
 +
The function of the node software is to collect data from the SDS011 (particulate matter) and BME280 (temperature/humidity) at regular intervals,
 +
encode this as a data packet and send it over LoRaWAN towards TheThingsNetwork.
 +
 +
For the LoRaWAN data connection, over-the-air activation (OTAA) is used. I use the following scheme, to keep administration to a minimum:
 +
* The Device EUI is derived from the ESP32 MAC address, the node shows this on its OLED
 +
* The App EUI is generated in the TTN console, it is the same for all nodes
 +
* The App Key is generated in the TTN console, it is the same for all nodes
 +
* The device is registered in the TTN console by the Device EUI (this doesn't happen automatically). Frame counter checks are disabled.
 +
* OTAA is done only once for each node. After that, the OTAA parameters are stored in (simulated) EEPROM.
 +
** A long press on the PRG button restarts the OTAA procedure
 +
* OTAA progress is shown on the OLED
 +
* If OTAA has been done successfully, the node restores the session parameters negotiated during OTAA on next bootup, so it can quickly resume sending data.
 +
* I'm NOT saving the upload frame counter (this would be preferable), just disable the feature in the TTN console.
 +
 +
TODO to figure out:
 +
* What about the channel setup? The node connects using 3 frequencies, but receives a bigger list of frequencies during OTAA JOIN.
 +
 +
I've seen the following from the node, receiving an ADR:
 +
  40829907: engineUpdate, opmode=0x8
 +
  40829935: EV_TXSTART
 +
  40829939: engineUpdate, opmode=0x888
 +
  40830013: TXMODE, freq=868300000, len=25, SF=11, BW=125, CR=4/5, IH=0
 +
  40944876: setupRx1 txrxFlags 0x22 --> 01
 +
  start single rx: now-rxtime: 5
 +
  40945013: RXMODE_SINGLE, freq=868300000, SF=11, BW=125, CR=4/5, IH=0
 +
  rxtimeout: entry: 40951170 rxtime: 40945001 entry-rxtime: 6169 now-entry: 5 rxtime-txend: 63524
 +
  41005584: setupRx2 txrxFlags 0x1 --> 02
 +
  start single rx: now-rxtime: 4
 +
  41005720: RXMODE_SINGLE, freq=869525000, SF=9, BW=125, CR=4/5, IH=0
 +
  41017003: process options (olen=0x5)
 +
  41017012: LinkAdrReq: p1:11 chmap:00ff chpage:00 uprt:01 ans:86
 +
  41017019: ??ack error ack=1 txCnt=0
 +
  41017073: decodeFrame txrxFlags 0x2 --> 22
 +
  41017312: Received downlink, window=RX2, port=-1, ack=1, txrxFlags=0x22
 +
  41017708: EV_TXCOMPLETE (includes waiting for RX windows)
 +
  41018027: engineUpdate, opmode=0x800
  
To do the LoRa stuff, I will probably use an ESP32 with an RFM95 built in.
+
=== Backend ===
The ESP32 is already very similar to the ESP8266 currently used in the luftdaten project.
+
A Java program subscribes to the MQTT stream, decodes the telemetry packets and forwards them to the luftdaten API.
 +
There is no storage of measurement data in the Java application.
  
=== Data collection ===
+
To receive data using mosquitto:
The plan is to use TheThingsNetwork, it publishes the received measurement data back onto an MQTT stream.
+
  mosquitto_sub -h eu.thethings.network -p 1883 -t +/devices/+/up -u particulatematter -P ttn-account-v2.cNaB2zO-nRiXaCUYmSAugzm-BaG_ZSHbEc5KgHNQFsk
  
Sensors join the network using OTAA (instead of ABP), so there is no need for setting up each node individually.  
+
Example upstream data:
 +
  particulatematter/devices/ttgo_mac/up {"app_id":"particulatematter","dev_id":"ttgo_mac","hardware_serial":"000084B14CA4AE30","port":1,"counter":16,"payload_raw":"AAEALAAd/////w==","metadata":{"time":"2019-04-13T08:37:45.338427686Z","frequency":868.3,"modulation":"LORA","data_rate":"SF11BW125","airtime":823296000,"coding_rate":"4/5","gateways":[{"gtw_id":"eui-008000000000b8b6","timestamp":2000599916,"time":"2019-04-13T08:37:45.320735Z","channel":1,"rssi":-115,"snr":-3,"rf_chain":1,"latitude":52.0182,"longitude":4.70844,"altitude":27}]}}
  
I will use a Java program to subscribe to this MQTT stream.
+
Example downstream data:
 +
  particulatematter/devices/ttgo_mac/events/down/sent {"payload":"YPUvASalGgEDEf8AAcqtmOw=","message":{"app_id":"particulatematter","dev_id":"ttgo_mac","port":0},"gateway_id":"eui-008000000000b8b6","config":{"modulation":"LORA","data_rate":"SF9BW125","airtime":164864000,"counter":282,"frequency":869525000,"power":27}}
  
=== Data forwarding ===
+
Gateway API:
I've already developed some Java code that publishes the measurement values towards luftdaten.info.
+
  https://account.thethingsnetwork.org/api/v2/gateways/eui-xxxxxxxxxxx
RIVM has already built their own gateway from luftdaten towards samenmeten, so I won't have to code anything for that.
 

Latest revision as of 15:04, 6 February 2020

Project LoRaWAN dust Sensor
LoraWanDustSensor.jpg
LoRaWAN airborne particulate matter sensor
Status In progress
Contact bertrik
Last Update 2020-02-06

The idea

The idea is to create a system consisting of:

  • a sensor that measures airborne particulate matter and sends the measurement data using LoRa to TheThingsNetwork.
  • a forwarder application that collects the data from TTN and forwards it to luftdaten.info

This has been done before by other people, but it appears there is no universal solution. I am publishing all source code on github and will put up documentation on this wiki. Everyone invents their own payload format, something more universal like Cayenne LPP would be nice. However I could not find a way to encode particulate matter data using Cayenne, so I'll just invent my own payload format too.

A similar thing has been done by:

One thing in particular I'd like to do better than existing solutions is to use proper OTAA for the LoRa connection to TTN. OTAA means over-the-air-activation and is a mechanism to dynamically negotiate communication/encryption keys instead of programmed specifically in each sensor node. Once the OTAA is done successfully, the node remembers the network id, device address, session keys, etc for future communication.

This makes it possible to have a single firmware image for all sensor nodes and it simplifies the setup:

  • you flash the node with a unified firmware
  • the node shows its unique id on the OLED
  • at the TTN console, you register a new device with the unique id
  • the sensor node receives encryption keys over the air automatically
  • done!

(idea: an ESP32 has a wifi connection too, perhaps registering the node can be done fully automatically, over wifi/internet)

Next steps

  • Investigate failing re-joins that seem to happen sometimes after the node has been online for a while the arduino-lmic library has a fix for this now
  • Implement a kind of schedule: turn the sds011 on, wait some time, take a measurement, turn it off, wait for some time
    • make this match the TTN send schedule? it's useless to do a measurement if we don't have TTN airtime to transmit it
  • display SDS011 serial number and date code on the screen

Links

Useful links for the TTGO LoRa board:

Hardware

The node is based on Arduino, in particular a TTGO ESP32 board with onboard SX1276 LoRa chip. The sensor is an SDS-011, just like in the luftdaten project. For humidity/temperature, I'd like to use a BME280.

NOTE: https://www.thethingsnetwork.org/community/berlin/post/warning-attention-users-of-ttgo21-v16-boards-labeled-t3_v16-on-pcb-battery-exploded-and-got-on-fire

Page with correct pinout of the ESP32 LoRa board.

Luftdaten uses a cycle time of 145 seconds for the SDS011.

Proposed hardware connections:

  • SDS011 5V to ESP32 5V
  • SDS011 GND to ESP32 GND
  • SDS011 TXD to ESP32 GPIO35 (maybe I can find two suitable pins close together)
  • SDS011 RXD to ESP32 GPIO25 (maybe I can find two suitable pins close together)
  • BME280 3V todo
  • BME280 SDA todo
  • BME280 SCL todo
  • BME280 GND todo

Software

Source code

Source code is hosted on github:

Common

Packet format

Proposed structure of packets transferred over LoRa:

  • PM10 value, encoded in units of 0.1 ug/m3: 2 bytes, big endian
  • PM2.5 value, encoded in units of 0.1 ug/m3: 2 bytes, big endian
  • temperature, encoded in units of 0.1 deg C: 2 bytes, signed big endian
  • relative humidity, encoded in units of 0.1%, 2 bytes, big endian

Total: 8 bytes

Not present value is 0xFFFF. Encoding is big endian.

Would be nice to use Cayenne for this, but I don't know if Cayenne has an id for particulate matter.

How other projects encode the data:

A smaller payload means less time in the air, smaller chance of collision with other LoRa packets and more packets per hour.

Node

Source code for the particulate matter measurement node can be found on the github page.

Platformio

To compile and upload the code to the node, platformio is used.

To install platformio (example for Debian):

 sudo apt install python3-pip
 sudo pip3 install platformio
 pio update

To compile and upload:

 pio run -t upload

Design

The function of the node software is to collect data from the SDS011 (particulate matter) and BME280 (temperature/humidity) at regular intervals, encode this as a data packet and send it over LoRaWAN towards TheThingsNetwork.

For the LoRaWAN data connection, over-the-air activation (OTAA) is used. I use the following scheme, to keep administration to a minimum:

  • The Device EUI is derived from the ESP32 MAC address, the node shows this on its OLED
  • The App EUI is generated in the TTN console, it is the same for all nodes
  • The App Key is generated in the TTN console, it is the same for all nodes
  • The device is registered in the TTN console by the Device EUI (this doesn't happen automatically). Frame counter checks are disabled.
  • OTAA is done only once for each node. After that, the OTAA parameters are stored in (simulated) EEPROM.
    • A long press on the PRG button restarts the OTAA procedure
  • OTAA progress is shown on the OLED
  • If OTAA has been done successfully, the node restores the session parameters negotiated during OTAA on next bootup, so it can quickly resume sending data.
  • I'm NOT saving the upload frame counter (this would be preferable), just disable the feature in the TTN console.

TODO to figure out:

  • What about the channel setup? The node connects using 3 frequencies, but receives a bigger list of frequencies during OTAA JOIN.

I've seen the following from the node, receiving an ADR:

 40829907: engineUpdate, opmode=0x8
 40829935: EV_TXSTART
 40829939: engineUpdate, opmode=0x888
 40830013: TXMODE, freq=868300000, len=25, SF=11, BW=125, CR=4/5, IH=0
 40944876: setupRx1 txrxFlags 0x22 --> 01
 start single rx: now-rxtime: 5
 40945013: RXMODE_SINGLE, freq=868300000, SF=11, BW=125, CR=4/5, IH=0
 rxtimeout: entry: 40951170 rxtime: 40945001 entry-rxtime: 6169 now-entry: 5 rxtime-txend: 63524
 41005584: setupRx2 txrxFlags 0x1 --> 02
 start single rx: now-rxtime: 4
 41005720: RXMODE_SINGLE, freq=869525000, SF=9, BW=125, CR=4/5, IH=0
 41017003: process options (olen=0x5)
 41017012: LinkAdrReq: p1:11 chmap:00ff chpage:00 uprt:01 ans:86
 41017019: ??ack error ack=1 txCnt=0
 41017073: decodeFrame txrxFlags 0x2 --> 22
 41017312: Received downlink, window=RX2, port=-1, ack=1, txrxFlags=0x22
 41017708: EV_TXCOMPLETE (includes waiting for RX windows)
 41018027: engineUpdate, opmode=0x800

Backend

A Java program subscribes to the MQTT stream, decodes the telemetry packets and forwards them to the luftdaten API. There is no storage of measurement data in the Java application.

To receive data using mosquitto:

 mosquitto_sub -h eu.thethings.network -p 1883 -t +/devices/+/up -u particulatematter -P ttn-account-v2.cNaB2zO-nRiXaCUYmSAugzm-BaG_ZSHbEc5KgHNQFsk

Example upstream data:

 particulatematter/devices/ttgo_mac/up {"app_id":"particulatematter","dev_id":"ttgo_mac","hardware_serial":"000084B14CA4AE30","port":1,"counter":16,"payload_raw":"AAEALAAd/////w==","metadata":{"time":"2019-04-13T08:37:45.338427686Z","frequency":868.3,"modulation":"LORA","data_rate":"SF11BW125","airtime":823296000,"coding_rate":"4/5","gateways":[{"gtw_id":"eui-008000000000b8b6","timestamp":2000599916,"time":"2019-04-13T08:37:45.320735Z","channel":1,"rssi":-115,"snr":-3,"rf_chain":1,"latitude":52.0182,"longitude":4.70844,"altitude":27}]}}

Example downstream data:

 particulatematter/devices/ttgo_mac/events/down/sent {"payload":"YPUvASalGgEDEf8AAcqtmOw=","message":{"app_id":"particulatematter","dev_id":"ttgo_mac","port":0},"gateway_id":"eui-008000000000b8b6","config":{"modulation":"LORA","data_rate":"SF9BW125","airtime":164864000,"counter":282,"frequency":869525000,"power":27}}

Gateway API:

 https://account.thethingsnetwork.org/api/v2/gateways/eui-xxxxxxxxxxx