LoraWanDustSensor: Difference between revisions

From RevSpace
Jump to navigation Jump to search
 
(185 intermediate revisions by the same user not shown)
Line 7: Line 7:
   }}
   }}


== The idea ==
== What is it? ==
The idea is to create a system consisting of:
This is a companion project of [[Sensor-data-bridge]].
* a sensor that measures airborne particulate matter and sends the measurement data using LoRa to TheThingsNetwork.
* a forwarder application that collects the data from TTN and forwards it to luftdaten.info


This has been done before by other people, but it appears there is no universal solution.
The concept consists of:
I am publishing all source code on github and will put up documentation on this wiki.
* a sensor that measures airborne particulate matter and sends the measurement data using LoRaWAN to TheThingsNetwork (TTN)
Everyone invents their own payload format, something more universal like Cayenne LPP would be nice.
* a forwarder application that collects the data from TTN and forwards it to sensor.community (formerly luftdaten), opensensemap, mycayenne dashboard, etc.
However I could not find a way to encode particulate matter data using Cayenne, so I'll just invent my own payload format too.


=== Features ===
* Supported particulate matter sensors (auto-detected):
** <b>Nova Fitness SDS011</b> particulate matter sensor, provides PM10, PM2.5
** <b>Sensirion SPS30</b> particulate matter sensor, provides PM10, PM4.0, PM2.5, PM1.0
** <b>BME280</b> humidity/temperature/pressure sensor, provides humidity, temperature, barometric pressure
* Uses <b>LoRaWAN/TheThingsNetwork</b> for communicating the measurement values:
* Hardware is based on off-the-shelf ESP32+LoRa hardware, specifically
** <b>TTGO LoRa32 v1</b> and
** <b>Heltec LoRa32 v2</b>
* Works with <b>TTNv3</b>, the most recent version of TheThingsNetwork infrastructure
* Uses <b>over-the-air-activation</b> (OTAA), parameters have a sensible default and are stored in EEPROM:
** Device EUI defaults to the ESP32 built-in unique id
** APP EUI and APP key can be configured using the serial port
* LoRaWAN OTAA credentials are shown as a <b>QR-code</b> so they are machine-readable
* Uses LoRaWAN <b>automatic-data-rate</b> (ADR) to optimise data rate and transmit power
* Data upload interval defaults to <b>5 minutes</b> (SF7), scales with spreading-factor to satisfy the fair-access-policy of TheThingsNetwork
* Data is encoded over LoRaWAN as <b>Cayenne LPP</b>, no custom payload decoder needed
* Supports WiFi over-the-air <b>firmware upgrade</b>
I am publishing all source code on github and documentation on this wiki.
=== Similar nodes ===
A similar thing has been done by:
A similar thing has been done by:
* https://github.com/VekotinVerstas/AQLoRaBurk
* https://github.com/VekotinVerstas/AQLoRaBurk
Line 25: Line 44:
* (others...)
* (others...)


One thing in particular I'd like to do better than existing solutions is to use proper OTAA for the LoRa connection to TTN.
One thing in particular that my concept does better than existing solutions is to use proper OTAA for the LoRa connection to TTN.
OTAA means over-the-air-activation and is a mechanism to dynamically negotiate communication/encryption keys instead of programmed specifically in each sensor node.
OTAA means over-the-air-activation and is a mechanism to dynamically negotiate encryption keys and communication settings.
Once the OTAA is done successfully, the node remembers the network id, device address, session keys, etc for future communication.


This makes it possible to have a single firmware image for all sensor nodes and it simplifies the setup:
=== Next steps ===
* you flash the node with a unified firmware
* Automate building and uploading of BIN files on github with github actions/workflows
* the node shows its unique id on the OLED
* Restructure firmware a bit to make the main .ino file smaller
* at the TTN console, you register a new device with the unique id
* Implement a simple command set through LoRaWAN downlink commands, for example:
* the sensor node receives encryption keys over the air automatically
** create a basic command structure: command port, payload encoding, e.g. 1st byte is command id
* done!
** a command to trigger an immediate reboot of the node
** a command to assist in geolocation, performing a WiFi AP scan:
*** Per AP we send 8 bytes: 1) MAC 2) RSSI 3) channel, so a total of 6 APs can be send in 48 bytes


(idea: an ESP32 has a wifi connection too, perhaps registering the node can be done fully automatically, over wifi/internet)
Ideas:
* Base send interval on the exact time spent sending data, investigate if the LoRaWAN device stack supports this -> not currently explicitly supported
* Add support for Davis Airlink local API, more information at https://weatherlink.github.io/airlink-local-api/
* Make measurement values available through bluetooth
* Implement the ESP-Now lamp/display protocol, for easy visualisation, see https://revspace.nl/StofAnanas#Next_generation.
** Start with sending the particulate matter measurements
** Add a suggested colour indication later


=== Next steps ===
==== QR code for TheThingsNetwork credentials ====
* Are we bound by a 5 minute interval towards Luftdaten (dropping off the maps if sending less frequently)?
[[File:Qrcode_ttn.png|thumb|right|LoRaWAN credentials]]
* Implement a kind of schedule: turn the sds011 on, wait some time, take a measurement, turn it off, wait for some time
This idea is to make use of the display to show a QR code with all details needed for configuring the device on TheThingsNetwork (DevEUI, AppEUI, AppKey).
** Luftdaten uses default send schedule of 145 seconds
The QR is shown after startup of the sensor, you can take a picture and send it to person entering the device in the TheThingsNetwork, no risk of typing errors.
** make this match the TTN send schedule? it's useless to do a measurement if we don't have TTN airtime to transmit it
* QR contents will be:
** use the built-in on-off schedule of the sds011
** DevEUI based on ESP32 unique id
** implement calculating the median of accumulated measurements (half of the measurements is higher, other half is lower)
** AppEUI (fixed in the firmware)
* encode using Cayenne encoding, advantage is that it's more or less standard, and you can more easily interpret the data in the TTN console
** AppKey to be randomly generated (and stored in EEPROM), much more secure than a fixed key
* display SDS011 serial number and date code on the screen
* QR code library: https://github.com/ricmoo/QRCode
* Example content should fit in 29x29 pixel version 3 QR code (format "OTAA:JoinEUI:DevEUI:AppKey" as hexadecimal)
<pre>
OTAA:1122334455667788:1122334455667788:112233445566778899AABBCCDDEEFF00
</pre>
* Cool interactive QR code generator: https://www.nayuki.io/page/qr-code-generator-library
* Collection of encodings for various data in QR codes: https://github.com/zxing/zxing/wiki/Barcode-Contents
* https://lora-alliance.org/wp-content/uploads/2020/11/TR005_LoRaWAN_Device_Identification_QR_Codes.pdf
 
==== Bluetooth interface ====
There are now official Bluetooth UUIDs for GATT characteristic and object types for PM10, PM2.5 and PM1.0, see
https://btprodspecificationrefs.blob.core.windows.net/assigned-numbers/Assigned%20Number%20Types/Assigned_Numbers.pdf
 
Assigned numbers:
* 0x2BD5: PM1.0
* 0x2BD6: PM2.5
* 0x2BD7: PM10
* ...


=== Links ===
=== Links ===
Line 57: Line 100:
* https://github.com/CivicLabsBelgium/lora_particle_sensor
* https://github.com/CivicLabsBelgium/lora_particle_sensor
* https://github.com/Cinezaster/ttn2luftdaten_forwarder
* https://github.com/Cinezaster/ttn2luftdaten_forwarder
* https://jackgruber.github.io/2020-04-13-ESP32-DeepSleep-and-LoraWAN-OTAA-join/


== Hardware ==
== Hardware ==
The node is based on Arduino, in particular a TTGO ESP32 board with onboard SX1276 LoRa chip.
The node is based on Arduino-compatible ESP32 boards with LoRaWAN:
The sensor is an SDS-011, just like in the luftdaten project.
* For the processor board, either the
For humidity/temperature, I'd like to use a BME280.
** TTGO LoRa32 V1 ("ttgo-lora32-v1") or the
** Heltec LoRa32 V2 can be used.
* The particulate matter sensor is the SDS011 or the SPS30, the sensor type is auto-detected.
* The humidity/temperature sensor is the BME280 (superior to the DHT11/22), auto-detected.
* An external SSD1306 display (I2C) can be connected, for easier mounting into the casing and avoiding burn-in the built-in OLED, auto-detected.
 
=== Pinout ===
{| class="wikitable"
! [https://primalcortex.files.wordpress.com/2017/11/ttgolorapinout_v2.jpg TTGO LoRa v1] !! [https://resource.heltec.cn/download/WiFi_LoRa_32/WIFI_LoRa_32_V2.pdf Heltec LoRa v2] !! Sensor !! Remark
|-
| 5V || 5V || 5V (SDS pin 3) || SPS30 5v (red), triple-check this, swapping 5V/GND destroys the SDS011
|-
| GND || GND || GND (SDS pin 5) || SPS30 gnd (black), triple-check this, swapping 5V/GND destroys the SDS011
|-
| GPIO23 || GPIO23 || RXD (SDS pin 6) || SPS30 RX (white?)
|-
| GPIO22 || GPIO22 || TXD (SDS pin 7) || SPS30 TX (purple?)
|-
| 3.3V || 3.3V/Vext || BME280 3V || Both Vext and 3.3V can be used
|-
| GND || GND || BME280 GND || ground
|-
| GPIO15 || GPIO15 || BME280 SCL || Piggybacking on the I2C bus to the display
|-
| GPIO4 || GPIO4 || BME280 SDA || Piggybacking on the I2C bus to the display
|}
 
For reference:
* https://randomnerdtutorials.com/esp32-pinout-reference-gpios/
* NOTE: https://www.thethingsnetwork.org/community/berlin/post/warning-attention-users-of-ttgo21-v16-boards-labeled-t3_v16-on-pcb-battery-exploded-and-got-on-fire


NOTE: https://www.thethingsnetwork.org/community/berlin/post/warning-attention-users-of-ttgo21-v16-boards-labeled-t3_v16-on-pcb-battery-exploded-and-got-on-fire
Note: pins 34, 35, 36, 39 are INPUT ONLY!


[https://primalcortex.wordpress.com/tag/lora32/ Page with correct pinout of the ESP32 LoRa board].
=== Display ===
The display on the ESP32 LoRa boards is an 128x64 pixel OLED display.
It wears out after some months, pixels that have been on more than other pixels will become dimmer.


Luftdaten uses a cycle time of 145 seconds for the SDS011.
[https://hackaday.com/2019/04/23/a-year-long-experiment-in-oled-burn-in/ Hackaday article about this effect].
The burn-in/dimming effect is very obvious [https://youtu.be/GWOFF5tMv_A?t=498 here].


Proposed hardware connections:
The firmware tries to auto-detect an external display, and if found, disable the internal display to avoid burn-in.
* SDS011 5V to ESP32 5V
* SDS011 GND to ESP32 GND
* SDS011 TXD to ESP32 GPIO35 (maybe I can find two suitable pins close together)
* SDS011 RXD to ESP32 GPIO25 (maybe I can find two suitable pins close together)
* BME280 3V todo
* BME280 SDA todo
* BME280 SCL todo
* BME280 GND todo


== Software ==
Things to keep in mind when connecting an external display:
=== Source code ===
* You need a 128x64 display with I2C interface (not SPI), four pins VDD/GND/SCL/SDA
* Choose either a full yellow or full cyan OLED. OLEDs with *white* pixels seem to burn in quite quickly. Avoid picking a partial yellow/partial cyan OLED, the QR code with LoRaWAN OTAA credentials cannot be displayed correctly on these.
* The pin order appears to vary, in particular take note of VDD and GND, they can be exactly the other way around between models
 
== Firmware ==
=== Compile and upload the firmware ===
Source code is hosted on github:
Source code is hosted on github:
* [https://github.com/bertrik/LoraWanPmSensor Arduino node], written in C/Arduino, built using platformio
* [https://github.com/bertrik/LoraWanPmSensor Arduino node], written in C/Arduino, built using platformio. This firmware joins TTN by OTAA and sends the measurement data using Cayenne.
* [https://github.com/bertrik/LoraLuftdatenForwarder TTN-to-luftdaten forwarder], written in Java, built using gradle
* [https://github.com/bertrik/LoraLuftdatenForwarder TTN-to-luftdaten forwarder], written in Java, built using gradle. This picks up the Cayenne encoded data and forwards it to the Luftdaten API.


=== Common ===
On Linux, with platformio (the command line tool), instructions for Debian Linux are:
==== Packet format ====
* install platformio:
  sudo apt install python3-pip
  sudo pip3 install platformio
  pio update
* get the source from github:
  git clone https://github.com/bertrik/LoraWanPmSensor
* enter the correct directory:
  cd LoraWanPmSensor
* compile and upload (for TTGO LoRa32 v1 board):
  pio run -e ttgov1 -t upload
or (for Heltec LoRa32 v2 board):
  pio run -e heltecv2 -t upload
* when source code has updated, run the following and perform the previous step again after that
  git pull
 
=== Firmware update ===
[[File:Elegantota.png|thumb|right|Firmware update]]
Once the initial firmware has been loaded into the device, further firmware updates can be done over WiFi:
* Depending on the board you have, check that you have either a ttgov1.bin or heltecv2.bin firmware file
* Connect to the access point set up by the node, it's called 'ESP-XXXXXXXXXXXX', where XX is the same as the LoRaWAN device EUI
* Using a browser, go to http://192.168.4.1 , a firmware selection screen appears as shown on the right
* Select the appropriate BIN file, the firmware will be updated and the board reboots


New idea: encode everything in Cayenne.
=== Payload encoding ===
My firmware uses the Cayenne LPP (low power payload) encoding.
For the SPS30, a custom encoding is used, that consists just of a block of 16-bit values, sent on LoRaWAN port 30.


===== Cayenne =====
===== Cayenne =====
It's reasonably compact, it's a standard format, you can get a preview of the data in the TTN console. Interacts nicely with other platforms.
It's reasonably compact, it's a standard format, you can get a preview of the data in the TTN console. Interacts nicely with other platforms.


HOWEVER:
[https://community.mydevices.com/t/cayenne-lpp-2-0/7510 Specification for Cayenne LPP 2.0]
* A standard analog value has a resolution of 0.01 units, leaving a maximum of 655.35 ug/m3. This value is possibly exceeded in extreme cases (night of new year's day). Alternative: just encode it as a 'digital input' with units of 0.1 ug/m3.
* Still not as compact as direct binary encoding


So proposed format:
Over-the-air payload encoding:
* PM10: digital input (type 1), channel id 100, with value in units of 0.01 ug/m3, saturated to 655.35 ug/m3  
* 4 bytes: PM10: analog input (type 2), channel id 1, with value in units of 0.01 ug/m3, saturated to 327.67 ug/m3  
* PM2.5: digital input (type 1), channel id 25, with value in units of 0.01 ug/m3, saturated to 655.35 ug/m3
* 4 bytes: PM2.5: analog input (type 2), channel id 2, with value in units of 0.01 ug/m3, saturated to 327.67 ug/m3
* Temperature: temperature (type 103), channel 0, with value in units of 0.1 degrees celcius
* 4 bytes: Temperature: temperature (type 103), with value in units of 0.1 degrees celcius
* Humidity: humidity (type 104), channel 1, with value in units of 0.5 %
* 3 bytes: Humidity: humidity (type 104), with value in units of 0.5 %
Particulate matter concentrations higher than 327.67 ug/m3 are saturated to 327.67, this is the maximum that can be represented as analog value in Cayenne.


Example:
Optionally:
  0x64 0x01 0xXX 0xXX  0x19 0x01 0xYY 0xYY  0x00 0x67 0xTT 0xTT  0x01 0x68 0xHH
* 4 bytes: PM1.0: analog input (type 2), channel id 0, with value in units of 0.01 ug/m3, saturated to 327.67 ug/m3
  <== PM10 value ===>  <= PM2.5 value ==>  <== temperature ==>  <= humidity =>
* 4 bytes: Pressure: barometer (type 115), with value in units of 0.1 mbar, or 10 Pa (optional)


Total payload size is 15 bytes. Over the air, LoRaWAN header adds some more data (13 bytes at least).  
A nice thing about Cayenne is that you can simply leave items out if you don't support them, which results in a shorter yet still valid message.


Values encoded are averaged using the median over the accumulated measurements over the measurement interval.
Example:
  0x01 0x01 0xXX 0xXX  0x02 0x01 0xYY 0xYY  0x03 0x67 0xTT 0xTT  0x04 0x68 0xHH
  <== PM10 value ===>  <== PM2.5 value ==>  <== temperature ==>  <= humidity =>


===== Binary =====
<b>Total payload size is 15 bytes.</b> The <b>LoRaWAN header adds 13 bytes</b> (at least).


Proposed structure of packets transferred over LoRa:
Particulate matter values encoded are averaged over the measurement interval.
* PM10 value, encoded in units of 0.1 ug/m3: 2 bytes, big endian
* PM2.5 value, encoded in units of 0.1 ug/m3: 2 bytes, big endian
* temperature, encoded in units of 0.1 deg C: 2 bytes, signed big endian
* relative humidity, encoded in units of 0.1%, 2 bytes, big endian
Total: 8 bytes


Not present value is 0xFFFF. Encoding is big endian.
Update: discovered that the Cayenne standard *does* actually support a "particle concentration" id, this is 125,
 
derived from IPSO id 3325 http://www.openmobilealliance.org/tech/profiles/lwm2m/3325.xml .
Would be nice to use Cayenne for this, but I don't know if Cayenne has an id for particulate matter.
No idea yet though how to specify the type of particle measurement, if there's any convention for PM10, PM2.5, etc.
The resolution is only 1 ppm, with typical sensors delivering 0.1 ppm resolution.


===== Binary =====
How other projects encode the data:
How other projects encode the data:
* TTN Apeldoorn (?): https://github.com/tijnonlijn/RFM-node/blob/master/dustduino_PPD42NS_example.ino#L327 sends 5 bytes
* TTN Apeldoorn (?): https://github.com/tijnonlijn/RFM-node/blob/master/dustduino_PPD42NS_example.ino#L327 sends 5 bytes
Line 144: Line 238:
** 4 bytes latitude (unit?)
** 4 bytes latitude (unit?)
** 4 bytes longitude (unit?)
** 4 bytes longitude (unit?)
* Apeldoorn in data: https://github.com/nijmeijer/TTN_Apeldoorn_in_Data_2018/blob/master/AiD_Dust_2018/AiD_Dust_2018.ino#L184
* Apeldoorn in data: https://github.com/nijmeijer/TTN_Apeldoorn_in_Data_2018/blob/master/AiD_Dust_2018/AiD_Dust_2018.ino#L184 sends 16 bytes:
** 4 bytes: pm2_5 float big endian (unit?)
** 4 bytes: pm2_5 float big endian (unit?)
** 4 bytes: pm10 float big endian (unit?)
** 4 bytes: pm10 float big endian (unit?)
Line 150: Line 244:
** 4 bytes: temperature float big endian (unit?)
** 4 bytes: temperature float big endian (unit?)


A smaller payload means less time in the air, smaller chance of collision with other LoRa packets and more packets per hour.
A smaller payload means less time in the air, smaller chance of collision with other LoRaWAN packets and more packets per hour.
However, there is always an overhead from the LoRaWAN package (minimum 13 bytes), so using the smallest encoding (5 bytes) compared to the largest (16 bytes), reduces the on-air-time by only 23%.


=== LoraWan time budget ===
=== LoraWan time budget ===
The number of bytes per telemetry packet and the number of packets sent per day determine how much of the available "airtime" we use.
TheThingsNetwork states a fair-use-policy of 30 seconds per day total uplink time.
Consider the following:
Consider the following:
* radio regulations generally have a 0.1% duty cycle requirement for each of the 8 frequencies, so according to the legal limits, there is about 86400x0.001x8 = 691s send time at the best case.
* radio regulations generally have a 1% duty cycle requirement for the two bands used by the 8 LoRaWAN (EU) frequencies, so according to the legal limits, there is about (86400s x 1% x 2bands) = 1728 seconds per day send time at the best case.
* TheThingsNetwork has a [https://www.thethingsnetwork.org/forum/t/limitations-data-rate-packet-size-30-seconds-uplink-and-10-messages-downlink-per-day-fair-access-policy-guidelines/1300 FUP of 30s of data upload per day]
* TheThingsNetwork has a [https://www.thethingsnetwork.org/forum/t/limitations-data-rate-packet-size-30-seconds-uplink-and-10-messages-downlink-per-day-fair-access-policy-guidelines/1300 FUP of 30s of data upload per day]. This is actually a *lot* more strict than allowed purely by radio regulations.
* The Luftdaten backend appears to run on a 5 minute interval, or 288 measurements per day.
* The sensor.community backend runs on a 5 minute interval, or 288 measurements per day.
* The default Luftdaten firmware sends new data every 145s by default
* The default sensor.community WiFi firmware sends new data every 145s by default


With the TTN FUP of 30s upload per day, we can spend 30s / 288 = 104 ms on each transmission.
<b>With the TTN FUP of 30s upload per day and 5 minute upload interval, we can spend 30s / 288 = 104 ms on each transmission.</b>


Using the [https://www.thethingsnetwork.org/airtime-calculator LoRaWAN airtime calculator] we can determine which modes can be used.
Using the [https://www.thethingsnetwork.org/airtime-calculator LoRaWAN airtime calculator] we can determine which modes can be used.
So with 288 measurements, we can spend 30s / 288 = 104 ms per packet.
At 15 bytes payload, this is only possible at SF7, the highest LoRa speed.
At 19 bytes payload, this is only possible at SF7, the highest LoRa speed.
Stretching the TTN guideline a bit, say by a factor 2, we can achieve those transfers still only at SF7 and SF8. So you need to be relatively close to a gateway.
Stretching the TTN guideline a bit, say by a factor 2, we can achieve 288 transfers of 19 bytes each at SF7 and SF8. So you need to be relatively close to a gateway.
 
In terms of interval:
* SF7: payload 17 bytes -> 71.9 ms/transmission -> every 3m27s
* SF8: payload 17 bytes -> 123,4 ms/transmission -> every 5m55s
* SF9: payload 17 bytes -> 226,3 ms/transmission -> every 10m32s
* SF10: payload 17 bytes -> 452.6 ms/transmission -> every 21m43s


With a smaller payload, can we use higher spreading factors? :
With a smaller payload, can we use higher spreading factors? :
* skip T/RH/P completely (8 bytes left): SF7 is possible, at SF8 we spend 102 ms per transmission
* smallest possible binary payload is 4 bytes (PM10 and PM2.5, no temperature, no humidity): SF7 takes 51.5 ms, SF8 takes 92.7 ms, SF9 takes 164.9 ms
* skip P only (15 bytes left): SF7 is possible, at SF8 we spend 123 ms per transmission
* smallest Cayenne payload is 8 bytes (PM10 and PM2.5, no temperature, no humidity): SF7 takes 56.6 ms, SF8 takes 102.9 ms, SF9 takes 185.3 ms
 
So, in conclusion: <s>With the FUP of TTN and use of Cayenne encoding, you can just barely send enough data to transport Luftdaten PM data!</s>
 
=== Node ===
Source code for the particulate matter measurement node can be found [https://github.com/bertrik/LoraWanPmSensor on the github page].
 
==== Platformio ====
To compile and upload the code to the node, platformio is used.
 
To install platformio (example for Debian):
  sudo apt install python3-pip
  sudo pip3 install platformio
  pio update
 
To compile and upload:
  pio run -t upload
 
==== Design ====
The function of the node software is to collect data from the SDS011 (particulate matter) and BME280 (temperature/humidity) at regular intervals,
encode this as a data packet and send it over LoRaWAN towards TheThingsNetwork.
 
For the LoRaWAN data connection, over-the-air activation (OTAA) is used. I use the following scheme, to keep administration to a minimum:
* The Device EUI is derived from the ESP32 MAC address, the node shows this on its OLED
* The App EUI is generated in the TTN console, it is the same for all nodes
* The App Key is generated in the TTN console, it is the same for all nodes
* The device is registered in the TTN console by the Device EUI (this doesn't happen automatically). Frame counter checks are disabled.
* OTAA is done only once for each node. After that, the OTAA parameters are stored in (simulated) EEPROM.
** A long press on the PRG button restarts the OTAA procedure
* OTAA progress is shown on the OLED
* If OTAA has been done successfully, the node restores the session parameters negotiated during OTAA on next bootup, so it can quickly resume sending data.
* I'm NOT saving the upload frame counter (this would be preferable), just disable the feature in the TTN console.
 
TODO to figure out:
* What about the channel setup? The node connects using 3 frequencies, but receives a bigger list of frequencies during OTAA JOIN.
 
I've seen the following from the node, receiving an ADR:
  40829907: engineUpdate, opmode=0x8
  40829935: EV_TXSTART
  40829939: engineUpdate, opmode=0x888
  40830013: TXMODE, freq=868300000, len=25, SF=11, BW=125, CR=4/5, IH=0
  40944876: setupRx1 txrxFlags 0x22 --> 01
  start single rx: now-rxtime: 5
  40945013: RXMODE_SINGLE, freq=868300000, SF=11, BW=125, CR=4/5, IH=0
  rxtimeout: entry: 40951170 rxtime: 40945001 entry-rxtime: 6169 now-entry: 5 rxtime-txend: 63524
  41005584: setupRx2 txrxFlags 0x1 --> 02
  start single rx: now-rxtime: 4
  41005720: RXMODE_SINGLE, freq=869525000, SF=9, BW=125, CR=4/5, IH=0
  41017003: process options (olen=0x5)
  41017012: LinkAdrReq: p1:11 chmap:00ff chpage:00 uprt:01 ans:86
  41017019: ??ack error ack=1 txCnt=0
  41017073: decodeFrame txrxFlags 0x2 --> 22
  41017312: Received downlink, window=RX2, port=-1, ack=1, txrxFlags=0x22
  41017708: EV_TXCOMPLETE (includes waiting for RX windows)
  41018027: engineUpdate, opmode=0x800
 
=== Backend ===
A Java program subscribes to the MQTT stream, decodes the telemetry packets and forwards them to the luftdaten API.
There is no storage of measurement data in the Java application.
 
To receive data using mosquitto:
  mosquitto_sub -h eu.thethings.network -p 1883 -t +/devices/+/up -u particulatematter -P ttn-account-v2.cNaB2zO-nRiXaCUYmSAugzm-BaG_ZSHbEc5KgHNQFsk


Example upstream data:
So, in conclusion: <b>With the FUP of TTN and use of Cayenne encoding, you can just barely send enough data to transport PM data!</b>
  particulatematter/devices/ttgo_mac/up {"app_id":"particulatematter","dev_id":"ttgo_mac","hardware_serial":"000084B14CA4AE30","port":1,"counter":16,"payload_raw":"AAEALAAd/////w==","metadata":{"time":"2019-04-13T08:37:45.338427686Z","frequency":868.3,"modulation":"LORA","data_rate":"SF11BW125","airtime":823296000,"coding_rate":"4/5","gateways":[{"gtw_id":"eui-008000000000b8b6","timestamp":2000599916,"time":"2019-04-13T08:37:45.320735Z","channel":1,"rssi":-115,"snr":-3,"rf_chain":1,"latitude":52.0182,"longitude":4.70844,"altitude":27}]}}
Also note that the payload size does not actually differ *that* much, this is because of the LoRaWAN overhead of 13 bytes minimum and pre-amble symbols which are sent anyway.


Example downstream data:
=== Downlink commands ===
  particulatematter/devices/ttgo_mac/events/down/sent {"payload":"YPUvASalGgEDEf8AAcqtmOw=","message":{"app_id":"particulatematter","dev_id":"ttgo_mac","port":0},"gateway_id":"eui-008000000000b8b6","config":{"modulation":"LORA","data_rate":"SF9BW125","airtime":164864000,"counter":282,"frequency":869525000,"power":27}}
I plan the following downlink commands:
* "info", retrieves information on the platform, hardware and firmware version
* "reboot", performs a remote reboot, OTAA will be performed. This command should only be sent UNconfirmed, as the reboot is performed *before* sending the reply back. This could result in a reboot-loop. A reboot request with confirmed bit is therefore ignored.
* "locate", performs a wifi scan so the node can be located, response data contains an 8-byte structure per AP: 6-byte MAC address, 1-byte RSSI, 1-byte channel. A max of 6 APs is reported.


Gateway API:
Principles:
  https://account.thethingsnetwork.org/api/v2/gateways/eui-xxxxxxxxxxx
* command requests are sent over a specific port, say port 100
* responses are sent back on the same port
* commands can be confirmed or unconfirmed, the response will mimic the command in this respect
* commands and responses have a simple structure: first byte is command id, rest is command/response-specific

Latest revision as of 19:25, 8 April 2023

Project LoRaWAN dust Sensor
LoraWanDustSensor.jpg
LoRaWAN airborne particulate matter sensor
Status In progress
Contact bertrik
Last Update 2023-04-08

What is it?

This is a companion project of Sensor-data-bridge.

The concept consists of:

  • a sensor that measures airborne particulate matter and sends the measurement data using LoRaWAN to TheThingsNetwork (TTN)
  • a forwarder application that collects the data from TTN and forwards it to sensor.community (formerly luftdaten), opensensemap, mycayenne dashboard, etc.

Features

  • Supported particulate matter sensors (auto-detected):
    • Nova Fitness SDS011 particulate matter sensor, provides PM10, PM2.5
    • Sensirion SPS30 particulate matter sensor, provides PM10, PM4.0, PM2.5, PM1.0
    • BME280 humidity/temperature/pressure sensor, provides humidity, temperature, barometric pressure
  • Uses LoRaWAN/TheThingsNetwork for communicating the measurement values:
  • Hardware is based on off-the-shelf ESP32+LoRa hardware, specifically
    • TTGO LoRa32 v1 and
    • Heltec LoRa32 v2
  • Works with TTNv3, the most recent version of TheThingsNetwork infrastructure
  • Uses over-the-air-activation (OTAA), parameters have a sensible default and are stored in EEPROM:
    • Device EUI defaults to the ESP32 built-in unique id
    • APP EUI and APP key can be configured using the serial port
  • LoRaWAN OTAA credentials are shown as a QR-code so they are machine-readable
  • Uses LoRaWAN automatic-data-rate (ADR) to optimise data rate and transmit power
  • Data upload interval defaults to 5 minutes (SF7), scales with spreading-factor to satisfy the fair-access-policy of TheThingsNetwork
  • Data is encoded over LoRaWAN as Cayenne LPP, no custom payload decoder needed
  • Supports WiFi over-the-air firmware upgrade

I am publishing all source code on github and documentation on this wiki.

Similar nodes

A similar thing has been done by:

One thing in particular that my concept does better than existing solutions is to use proper OTAA for the LoRa connection to TTN. OTAA means over-the-air-activation and is a mechanism to dynamically negotiate encryption keys and communication settings.

Next steps

  • Automate building and uploading of BIN files on github with github actions/workflows
  • Restructure firmware a bit to make the main .ino file smaller
  • Implement a simple command set through LoRaWAN downlink commands, for example:
    • create a basic command structure: command port, payload encoding, e.g. 1st byte is command id
    • a command to trigger an immediate reboot of the node
    • a command to assist in geolocation, performing a WiFi AP scan:
      • Per AP we send 8 bytes: 1) MAC 2) RSSI 3) channel, so a total of 6 APs can be send in 48 bytes

Ideas:

  • Base send interval on the exact time spent sending data, investigate if the LoRaWAN device stack supports this -> not currently explicitly supported
  • Add support for Davis Airlink local API, more information at https://weatherlink.github.io/airlink-local-api/
  • Make measurement values available through bluetooth
  • Implement the ESP-Now lamp/display protocol, for easy visualisation, see https://revspace.nl/StofAnanas#Next_generation.
    • Start with sending the particulate matter measurements
    • Add a suggested colour indication later

QR code for TheThingsNetwork credentials

LoRaWAN credentials

This idea is to make use of the display to show a QR code with all details needed for configuring the device on TheThingsNetwork (DevEUI, AppEUI, AppKey). The QR is shown after startup of the sensor, you can take a picture and send it to person entering the device in the TheThingsNetwork, no risk of typing errors.

  • QR contents will be:
    • DevEUI based on ESP32 unique id
    • AppEUI (fixed in the firmware)
    • AppKey to be randomly generated (and stored in EEPROM), much more secure than a fixed key
  • QR code library: https://github.com/ricmoo/QRCode
  • Example content should fit in 29x29 pixel version 3 QR code (format "OTAA:JoinEUI:DevEUI:AppKey" as hexadecimal)
OTAA:1122334455667788:1122334455667788:112233445566778899AABBCCDDEEFF00

Bluetooth interface

There are now official Bluetooth UUIDs for GATT characteristic and object types for PM10, PM2.5 and PM1.0, see https://btprodspecificationrefs.blob.core.windows.net/assigned-numbers/Assigned%20Number%20Types/Assigned_Numbers.pdf

Assigned numbers:

  • 0x2BD5: PM1.0
  • 0x2BD6: PM2.5
  • 0x2BD7: PM10
  • ...

Links

Useful links for the TTGO LoRa board:

Hardware

The node is based on Arduino-compatible ESP32 boards with LoRaWAN:

  • For the processor board, either the
    • TTGO LoRa32 V1 ("ttgo-lora32-v1") or the
    • Heltec LoRa32 V2 can be used.
  • The particulate matter sensor is the SDS011 or the SPS30, the sensor type is auto-detected.
  • The humidity/temperature sensor is the BME280 (superior to the DHT11/22), auto-detected.
  • An external SSD1306 display (I2C) can be connected, for easier mounting into the casing and avoiding burn-in the built-in OLED, auto-detected.

Pinout

TTGO LoRa v1 Heltec LoRa v2 Sensor Remark
5V 5V 5V (SDS pin 3) SPS30 5v (red), triple-check this, swapping 5V/GND destroys the SDS011
GND GND GND (SDS pin 5) SPS30 gnd (black), triple-check this, swapping 5V/GND destroys the SDS011
GPIO23 GPIO23 RXD (SDS pin 6) SPS30 RX (white?)
GPIO22 GPIO22 TXD (SDS pin 7) SPS30 TX (purple?)
3.3V 3.3V/Vext BME280 3V Both Vext and 3.3V can be used
GND GND BME280 GND ground
GPIO15 GPIO15 BME280 SCL Piggybacking on the I2C bus to the display
GPIO4 GPIO4 BME280 SDA Piggybacking on the I2C bus to the display

For reference:

Note: pins 34, 35, 36, 39 are INPUT ONLY!

Display

The display on the ESP32 LoRa boards is an 128x64 pixel OLED display. It wears out after some months, pixels that have been on more than other pixels will become dimmer.

Hackaday article about this effect. The burn-in/dimming effect is very obvious here.

The firmware tries to auto-detect an external display, and if found, disable the internal display to avoid burn-in.

Things to keep in mind when connecting an external display:

  • You need a 128x64 display with I2C interface (not SPI), four pins VDD/GND/SCL/SDA
  • Choose either a full yellow or full cyan OLED. OLEDs with *white* pixels seem to burn in quite quickly. Avoid picking a partial yellow/partial cyan OLED, the QR code with LoRaWAN OTAA credentials cannot be displayed correctly on these.
  • The pin order appears to vary, in particular take note of VDD and GND, they can be exactly the other way around between models

Firmware

Compile and upload the firmware

Source code is hosted on github:

  • Arduino node, written in C/Arduino, built using platformio. This firmware joins TTN by OTAA and sends the measurement data using Cayenne.
  • TTN-to-luftdaten forwarder, written in Java, built using gradle. This picks up the Cayenne encoded data and forwards it to the Luftdaten API.

On Linux, with platformio (the command line tool), instructions for Debian Linux are:

  • install platformio:
 sudo apt install python3-pip
 sudo pip3 install platformio
 pio update
  • get the source from github:
 git clone https://github.com/bertrik/LoraWanPmSensor
  • enter the correct directory:
 cd LoraWanPmSensor
  • compile and upload (for TTGO LoRa32 v1 board):
 pio run -e ttgov1 -t upload

or (for Heltec LoRa32 v2 board):

 pio run -e heltecv2 -t upload
  • when source code has updated, run the following and perform the previous step again after that
 git pull

Firmware update

Firmware update

Once the initial firmware has been loaded into the device, further firmware updates can be done over WiFi:

  • Depending on the board you have, check that you have either a ttgov1.bin or heltecv2.bin firmware file
  • Connect to the access point set up by the node, it's called 'ESP-XXXXXXXXXXXX', where XX is the same as the LoRaWAN device EUI
  • Using a browser, go to http://192.168.4.1 , a firmware selection screen appears as shown on the right
  • Select the appropriate BIN file, the firmware will be updated and the board reboots

Payload encoding

My firmware uses the Cayenne LPP (low power payload) encoding. For the SPS30, a custom encoding is used, that consists just of a block of 16-bit values, sent on LoRaWAN port 30.

Cayenne

It's reasonably compact, it's a standard format, you can get a preview of the data in the TTN console. Interacts nicely with other platforms.

Specification for Cayenne LPP 2.0

Over-the-air payload encoding:

  • 4 bytes: PM10: analog input (type 2), channel id 1, with value in units of 0.01 ug/m3, saturated to 327.67 ug/m3
  • 4 bytes: PM2.5: analog input (type 2), channel id 2, with value in units of 0.01 ug/m3, saturated to 327.67 ug/m3
  • 4 bytes: Temperature: temperature (type 103), with value in units of 0.1 degrees celcius
  • 3 bytes: Humidity: humidity (type 104), with value in units of 0.5 %

Particulate matter concentrations higher than 327.67 ug/m3 are saturated to 327.67, this is the maximum that can be represented as analog value in Cayenne.

Optionally:

  • 4 bytes: PM1.0: analog input (type 2), channel id 0, with value in units of 0.01 ug/m3, saturated to 327.67 ug/m3
  • 4 bytes: Pressure: barometer (type 115), with value in units of 0.1 mbar, or 10 Pa (optional)

A nice thing about Cayenne is that you can simply leave items out if you don't support them, which results in a shorter yet still valid message.

Example:

 0x01 0x01 0xXX 0xXX  0x02 0x01 0xYY 0xYY  0x03 0x67 0xTT 0xTT  0x04 0x68 0xHH
 <== PM10 value ===>  <== PM2.5 value ==>  <== temperature ==>  <= humidity =>

Total payload size is 15 bytes. The LoRaWAN header adds 13 bytes (at least).

Particulate matter values encoded are averaged over the measurement interval.

Update: discovered that the Cayenne standard *does* actually support a "particle concentration" id, this is 125, derived from IPSO id 3325 http://www.openmobilealliance.org/tech/profiles/lwm2m/3325.xml . No idea yet though how to specify the type of particle measurement, if there's any convention for PM10, PM2.5, etc. The resolution is only 1 ppm, with typical sensors delivering 0.1 ppm resolution.

Binary

How other projects encode the data:

A smaller payload means less time in the air, smaller chance of collision with other LoRaWAN packets and more packets per hour. However, there is always an overhead from the LoRaWAN package (minimum 13 bytes), so using the smallest encoding (5 bytes) compared to the largest (16 bytes), reduces the on-air-time by only 23%.

LoraWan time budget

The number of bytes per telemetry packet and the number of packets sent per day determine how much of the available "airtime" we use. TheThingsNetwork states a fair-use-policy of 30 seconds per day total uplink time.

Consider the following:

  • radio regulations generally have a 1% duty cycle requirement for the two bands used by the 8 LoRaWAN (EU) frequencies, so according to the legal limits, there is about (86400s x 1% x 2bands) = 1728 seconds per day send time at the best case.
  • TheThingsNetwork has a FUP of 30s of data upload per day. This is actually a *lot* more strict than allowed purely by radio regulations.
  • The sensor.community backend runs on a 5 minute interval, or 288 measurements per day.
  • The default sensor.community WiFi firmware sends new data every 145s by default

With the TTN FUP of 30s upload per day and 5 minute upload interval, we can spend 30s / 288 = 104 ms on each transmission.

Using the LoRaWAN airtime calculator we can determine which modes can be used. At 15 bytes payload, this is only possible at SF7, the highest LoRa speed. Stretching the TTN guideline a bit, say by a factor 2, we can achieve those transfers still only at SF7 and SF8. So you need to be relatively close to a gateway.

In terms of interval:

  • SF7: payload 17 bytes -> 71.9 ms/transmission -> every 3m27s
  • SF8: payload 17 bytes -> 123,4 ms/transmission -> every 5m55s
  • SF9: payload 17 bytes -> 226,3 ms/transmission -> every 10m32s
  • SF10: payload 17 bytes -> 452.6 ms/transmission -> every 21m43s

With a smaller payload, can we use higher spreading factors? :

  • smallest possible binary payload is 4 bytes (PM10 and PM2.5, no temperature, no humidity): SF7 takes 51.5 ms, SF8 takes 92.7 ms, SF9 takes 164.9 ms
  • smallest Cayenne payload is 8 bytes (PM10 and PM2.5, no temperature, no humidity): SF7 takes 56.6 ms, SF8 takes 102.9 ms, SF9 takes 185.3 ms

So, in conclusion: With the FUP of TTN and use of Cayenne encoding, you can just barely send enough data to transport PM data! Also note that the payload size does not actually differ *that* much, this is because of the LoRaWAN overhead of 13 bytes minimum and pre-amble symbols which are sent anyway.

Downlink commands

I plan the following downlink commands:

  • "info", retrieves information on the platform, hardware and firmware version
  • "reboot", performs a remote reboot, OTAA will be performed. This command should only be sent UNconfirmed, as the reboot is performed *before* sending the reply back. This could result in a reboot-loop. A reboot request with confirmed bit is therefore ignored.
  • "locate", performs a wifi scan so the node can be located, response data contains an 8-byte structure per AP: 6-byte MAC address, 1-byte RSSI, 1-byte channel. A max of 6 APs is reported.

Principles:

  • command requests are sent over a specific port, say port 100
  • responses are sent back on the same port
  • commands can be confirmed or unconfirmed, the response will mimic the command in this respect
  • commands and responses have a simple structure: first byte is command id, rest is command/response-specific