Difference between revisions of "Sensor-data-bridge"
(→What is this) |
(→Work in progress) |
||
(39 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
{{Project | {{Project | ||
|Name=LoraLuftdatenForwarder | |Name=LoraLuftdatenForwarder | ||
− | |Picture= | + | |Picture=loraluftdatenforwarder.png |
|Omschrijving=LoRaWAN forwarder for particulate matter data | |Omschrijving=LoRaWAN forwarder for particulate matter data | ||
|Status=In progress | |Status=In progress | ||
Line 8: | Line 8: | ||
== What is this == | == What is this == | ||
− | This is a | + | This is a companion project of [[LoraWanDustSensor]]. |
− | It | + | It takes airborne particulate matter measurement data transferred through TheThingsNetwork and forwards it to online databases: |
+ | * http://sensor.community (formerly Luftdaten), | ||
+ | * http://opensensemap.org and | ||
+ | * https://cayenne.mydevices.com | ||
− | Next steps: | + | Project on Github: https://github.com/bertrik/LoraLuftdatenForwarder |
+ | |||
+ | === Features === | ||
+ | * Picks up particulate matter measurement data received through TheThingsNetwork, using their "v3" infrastructure | ||
+ | * Forwards measurement data to https://sensor.community | ||
+ | * Forwards measurement data to https://opensensemap.org, you can configure the opensense-id by adding a device attribute in TheThingsNetwork console | ||
+ | * Forwards measurement data to https://cayenne.mydevices.com, you configure the username/password/clientid by adding a device attribute in TheThingsNetwork console | ||
+ | * Supports Cayenne payload format for the data encoding, a custom payload format for SPS30 data (includes particle counts) | ||
+ | * Handles particulate matter data (PM10, PM4.0, PM2.5, PM1.0), temperature, humidity, barometric pressure | ||
+ | * Can be run as a systemd service, so it automatically restarts in case the software would crash | ||
+ | |||
+ | === Next steps === | ||
+ | * test support for forwarding of SPS30 data, including particle counts and particle size | ||
+ | * add support for auto-localization through a scan of surrounding WiFi access-points and a geolocation service (google, mozilla), see https://github.com/bertrik/LoraLuftdatenForwarder/tree/master/LoraLuftdatenForwarder/src/main/java/nl/bertriksikken/gls | ||
* add support for NB-IOT modem with t-mobile backend, see my [[Sim7020]] project | * add support for NB-IOT modem with t-mobile backend, see my [[Sim7020]] project | ||
+ | * add support for other backends, e.g. feinstaub-app? | ||
== Requirements == | == Requirements == | ||
Line 28: | Line 45: | ||
* enter the LoraLuftdatenForwarder/gradle directory | * enter the LoraLuftdatenForwarder/gradle directory | ||
cd LoraLuftdatenForwarder/gradle | cd LoraLuftdatenForwarder/gradle | ||
− | * run the gradle script to build the software: | + | * run the gradle script to build the software (Linux): |
./gradlew assemble | ./gradlew assemble | ||
+ | or (Windows): | ||
+ | gradlew assemble | ||
* the application zip & tar is now available in LoraLuftdatenForwarder/LoraLuftdatenForwarder/build/distributions | * the application zip & tar is now available in LoraLuftdatenForwarder/LoraLuftdatenForwarder/build/distributions | ||
Line 45: | Line 64: | ||
== Configuration == | == Configuration == | ||
− | === | + | === Node configuration === |
The particulate matter measurement device needs to send data in the Cayenne format. | The particulate matter measurement device needs to send data in the Cayenne format. | ||
I used the following conventions: | I used the following conventions: | ||
Line 51: | Line 70: | ||
* PM2.5 is encoded as analog value on channel 2 | * PM2.5 is encoded as analog value on channel 2 | ||
* PM1.0 is encoded as analog value on channel 0 (optional) | * PM1.0 is encoded as analog value on channel 0 (optional) | ||
+ | * PM4.0 is encoded as analog value on channel 4 (optional) | ||
* Temperature is encoded using standard Cayenne encoding (optional) | * Temperature is encoded using standard Cayenne encoding (optional) | ||
* Humidity is encoded using standard Cayenne encoding (optional) | * Humidity is encoded using standard Cayenne encoding (optional) | ||
* Barometric pressure is encoded using standard Cayenne encoding (optional) | * Barometric pressure is encoded using standard Cayenne encoding (optional) | ||
− | === TheThingsNetwork === | + | The SPS30 produces mass concentration data in 4 categories, particle count in 5 categories, plus particle size. |
+ | Format: | ||
+ | * 16-bit (big endian) PM1.0 mass concentration (unit 0.1) | ||
+ | * 16-bit (big endian) PM2.5 mass concentration (unit 0.1) | ||
+ | * 16-bit (big endian) PM4.0 mass concentration (unit 0.1) | ||
+ | * 16-bit (big endian) PM10 mass concentration (unit 0.1) | ||
+ | * 16-bit (big endian) PM0.5 number concentration (unit 1) | ||
+ | * 16-bit (big endian) PM1.0 number concentration (unit 1) | ||
+ | * 16-bit (big endian) PM2.5 number concentration (unit 1) | ||
+ | * 16-bit (big endian) PM4.0 number concentration (unit 1) | ||
+ | * 16-bit (big endian) PM10 number concentration (unit 1) | ||
+ | * 16-bit (big endian) typical particle size (unit nm) | ||
+ | |||
+ | A total of 20 bytes. This is sent on LoRaWAN port 30. | ||
+ | Temperature and humidity is not encoded. | ||
+ | |||
+ | === TheThingsNetwork application/device configuration === | ||
[[File:TtnEditApiKey.png|thumb|right|TTN API key rights]] | [[File:TtnEditApiKey.png|thumb|right|TTN API key rights]] | ||
Line 63: | Line 99: | ||
* You need an 'application', create a new one, or use an existing one | * You need an 'application', create a new one, or use an existing one | ||
* Within the application you need a 'device', so create a new one, or use an existing one: | * Within the application you need a 'device', so create a new one, or use an existing one: | ||
− | ** Use OTAA | + | ** Use OTAA, LoRaMac version 1.0.3 |
** Enter the device EUI as displayed on the display | ** Enter the device EUI as displayed on the display | ||
** Use the application keys as specified in my [[LoraWanDustSensor]] page | ** Use the application keys as specified in my [[LoraWanDustSensor]] page | ||
Line 70: | Line 106: | ||
** NOTE: you have only one chance to copy this key somewhere, so copy/paste it locally to a text file or something | ** NOTE: you have only one chance to copy this key somewhere, so copy/paste it locally to a text file or something | ||
− | === | + | === LoraLuftdatenForwarder configuration === |
To configure the application: | To configure the application: | ||
* Start the application without a configuration file, this will create a default template, stop the application again | * Start the application without a configuration file, this will create a default template, stop the application again | ||
Line 81: | Line 117: | ||
--- | --- | ||
ttn: | ttn: | ||
− | + | mqtt_url: "tcp://eu1.cloud.thethings.network" | |
− | |||
identity_server_url: "https://eu1.cloud.thethings.network" | identity_server_url: "https://eu1.cloud.thethings.network" | ||
identity_server_timeout: 20 | identity_server_timeout: 20 | ||
apps: | apps: | ||
- name: "particulatematter" | - name: "particulatematter" | ||
− | |||
key: "NNSXS......." | key: "NNSXS......." | ||
encoding: "CAYENNE" | encoding: "CAYENNE" | ||
Line 96: | Line 130: | ||
url: "https://api.opensensemap.org" | url: "https://api.opensensemap.org" | ||
timeout: 20 | timeout: 20 | ||
− | |||
− | |||
− | |||
</pre> | </pre> | ||
Line 109: | Line 140: | ||
TODO | TODO | ||
* Go to https://devices.sensor.community/ and log in | * Go to https://devices.sensor.community/ and log in | ||
− | * Register a node with id 'TTN-<device-EUI-as-shown-on-display>' | + | * Register a node with id 'TTN-<device-EUI-as-shown-on-display>' (without the spaces or hyphens, e.g. 'TTN-0000547AF1BF713C') |
* Register it with the proper configuration, e.g. SDS011 with BME280 | * Register it with the proper configuration, e.g. SDS011 with BME280 | ||
=== Opensensemap === | === Opensensemap === | ||
− | + | * Go to opensensemap.org and log in | |
− | * Create | + | ** Create an opensense node with the proper configuration |
− | * Copy the opensensenmap 'box id' | + | ** Copy the opensensenmap 'box id', a long hexadecimal string |
− | * | + | * Go the TTN console: https://console.cloud.thethings.network/ and log in |
− | ** | + | ** Add an attribute for the device, under 'General settings', name = 'opensense-id', value = boxid that you copied from opensensemap.org |
− | + | * The mapping from TTN-id to boxid is refreshed by the forwarder once an hour, so within an hour the forwarding to opensensemap.org starts | |
− | * The boxid | + | |
+ | === Cayenne myDevices === | ||
+ | * Go to https://cayenne.mydevices.com/ and log in | ||
+ | ** Add a new node, TODO | ||
+ | * Go to TTN console and log in | ||
+ | ** Add attributes for the device, under 'General settings' | ||
+ | *** attribute name = 'mydevices-username', value = MQTT username from cayenne mydevices dashboard | ||
+ | *** attribute name = 'mydevices-password', value = MQTT password from cayenne mydevices dashboard | ||
+ | *** attribute name = 'mydevices-clientid', value = Client ID from cayenne mydevices dashboard | ||
+ | |||
+ | == Work in progress == | ||
+ | |||
+ | === Forwarding noise data === | ||
+ | |||
+ | How it is encoded in the sensor.community firmware: | ||
+ | * Three values are sent in the JSON to sensor.community: | ||
+ | ** "noise_LAeq", value in dB(A), meaning? | ||
+ | ** "noise_LA_min", value in dB(A), some kind of minimum | ||
+ | ** "noise_LA_max", value in dB(A), some kind of maximum | ||
+ | |||
+ | I think these can be encoded in Cayenne as a simple analog value (which has a range of approximately -327..327 with a resolution of 0.01. | ||
+ | Just need to assign a channel number to it. | ||
+ | |||
+ | Values are read from the noise sensor as follows: | ||
+ | * call to dnms_calculate_leq() | ||
+ | * call to dnms_read_data_ready(&data_ready) returns 0 if OK and (data_ready != 0) | ||
+ | * call to dnms_read_leq(&dnms_values) returns 0 if OK | ||
+ | * firmware applies a "correction" by adding a fixed offset | ||
+ | |||
+ | Measurement values are encoded as 32-bit units, interpreted as 32-bit floats. | ||
+ | |||
+ | References: | ||
+ | * DNMS is read at https://github.com/opendata-stuttgart/sensors-software/blob/beta/airrohr-firmware/airrohr-firmware.ino#L3392 | ||
+ | * DNMS is formatted in the JSON at https://github.com/opendata-stuttgart/sensors-software/blob/beta/airrohr-firmware/airrohr-firmware.ino#L3405 | ||
+ | * DNMS I2C code is at https://github.com/opendata-stuttgart/sensors-software/blob/beta/airrohr-firmware/dnms_i2c.cpp |
Revision as of 17:27, 16 June 2021
Project LoraLuftdatenForwarder | |
---|---|
LoRaWAN forwarder for particulate matter data | |
Status | In progress |
Contact | bertrik |
Last Update | 2021-06-16 |
Contents
What is this
This is a companion project of LoraWanDustSensor.
It takes airborne particulate matter measurement data transferred through TheThingsNetwork and forwards it to online databases:
- http://sensor.community (formerly Luftdaten),
- http://opensensemap.org and
- https://cayenne.mydevices.com
Project on Github: https://github.com/bertrik/LoraLuftdatenForwarder
Features
- Picks up particulate matter measurement data received through TheThingsNetwork, using their "v3" infrastructure
- Forwards measurement data to https://sensor.community
- Forwards measurement data to https://opensensemap.org, you can configure the opensense-id by adding a device attribute in TheThingsNetwork console
- Forwards measurement data to https://cayenne.mydevices.com, you configure the username/password/clientid by adding a device attribute in TheThingsNetwork console
- Supports Cayenne payload format for the data encoding, a custom payload format for SPS30 data (includes particle counts)
- Handles particulate matter data (PM10, PM4.0, PM2.5, PM1.0), temperature, humidity, barometric pressure
- Can be run as a systemd service, so it automatically restarts in case the software would crash
Next steps
- test support for forwarding of SPS30 data, including particle counts and particle size
- add support for auto-localization through a scan of surrounding WiFi access-points and a geolocation service (google, mozilla), see https://github.com/bertrik/LoraLuftdatenForwarder/tree/master/LoraLuftdatenForwarder/src/main/java/nl/bertriksikken/gls
- add support for NB-IOT modem with t-mobile backend, see my Sim7020 project
- add support for other backends, e.g. feinstaub-app?
Requirements
You need the following:
- a server that is always on and connected to the internet, can be Linux or Windows
- a Java installation (JDK to compile), at least version 8
- some configuration on TheThingsNetwork side
- some configuration of my application (YAML file)
Compilation
To compile the software:
- clone the software from my github archive
git clone https://github.com/bertrik/LoraLuftdatenForwarder.git
- enter the LoraLuftdatenForwarder/gradle directory
cd LoraLuftdatenForwarder/gradle
- run the gradle script to build the software (Linux):
./gradlew assemble
or (Windows):
gradlew assemble
- the application zip & tar is now available in LoraLuftdatenForwarder/LoraLuftdatenForwarder/build/distributions
To update to the latest version:
- Update software from github archive:
git pull
- perform the last two steps above again
Installation
Unzip the distribution file somewhere on your system. I put it in my home directory, for example
cd tar xvf code/LoraLuftdatenForwarder/LoraLuftdatenForwarder/build/distributions/LoraLuftdatenForwarder.tar
Configuration
Node configuration
The particulate matter measurement device needs to send data in the Cayenne format. I used the following conventions:
- PM10 is encoded as analog value on channel 1
- PM2.5 is encoded as analog value on channel 2
- PM1.0 is encoded as analog value on channel 0 (optional)
- PM4.0 is encoded as analog value on channel 4 (optional)
- Temperature is encoded using standard Cayenne encoding (optional)
- Humidity is encoded using standard Cayenne encoding (optional)
- Barometric pressure is encoded using standard Cayenne encoding (optional)
The SPS30 produces mass concentration data in 4 categories, particle count in 5 categories, plus particle size. Format:
- 16-bit (big endian) PM1.0 mass concentration (unit 0.1)
- 16-bit (big endian) PM2.5 mass concentration (unit 0.1)
- 16-bit (big endian) PM4.0 mass concentration (unit 0.1)
- 16-bit (big endian) PM10 mass concentration (unit 0.1)
- 16-bit (big endian) PM0.5 number concentration (unit 1)
- 16-bit (big endian) PM1.0 number concentration (unit 1)
- 16-bit (big endian) PM2.5 number concentration (unit 1)
- 16-bit (big endian) PM4.0 number concentration (unit 1)
- 16-bit (big endian) PM10 number concentration (unit 1)
- 16-bit (big endian) typical particle size (unit nm)
A total of 20 bytes. This is sent on LoRaWAN port 30. Temperature and humidity is not encoded.
TheThingsNetwork application/device configuration
You need to define an 'application' on TheTheThingsNetwork.
- Go the TTN console: https://console.cloud.thethings.network/ and log in
- You need an 'application', create a new one, or use an existing one
- Within the application you need a 'device', so create a new one, or use an existing one:
- Use OTAA, LoRaMac version 1.0.3
- Enter the device EUI as displayed on the display
- Use the application keys as specified in my LoraWanDustSensor page
- You need an API key
- Create this on the TTN console, grant individual rights as shown in the screenshot
- NOTE: you have only one chance to copy this key somewhere, so copy/paste it locally to a text file or something
LoraLuftdatenForwarder configuration
To configure the application:
- Start the application without a configuration file, this will create a default template, stop the application again
cd LoraLuftdatenForwarder bin/LoraLuftdatenForwarder (ctrl-C)
- Edit the loraluftdatenforwarder.yaml file, example:
--- ttn: mqtt_url: "tcp://eu1.cloud.thethings.network" identity_server_url: "https://eu1.cloud.thethings.network" identity_server_timeout: 20 apps: - name: "particulatematter" key: "NNSXS......." encoding: "CAYENNE" luftdaten: url: "https://api.sensor.community" timeout: 20 opensense: url: "https://api.opensensemap.org" timeout: 20
So:
- enter the name of your application
- enter the TTN API key you saved earlier
- other defaults are probably OK
Sensor.community
TODO
- Go to https://devices.sensor.community/ and log in
- Register a node with id 'TTN-<device-EUI-as-shown-on-display>' (without the spaces or hyphens, e.g. 'TTN-0000547AF1BF713C')
- Register it with the proper configuration, e.g. SDS011 with BME280
Opensensemap
- Go to opensensemap.org and log in
- Create an opensense node with the proper configuration
- Copy the opensensenmap 'box id', a long hexadecimal string
- Go the TTN console: https://console.cloud.thethings.network/ and log in
- Add an attribute for the device, under 'General settings', name = 'opensense-id', value = boxid that you copied from opensensemap.org
- The mapping from TTN-id to boxid is refreshed by the forwarder once an hour, so within an hour the forwarding to opensensemap.org starts
Cayenne myDevices
- Go to https://cayenne.mydevices.com/ and log in
- Add a new node, TODO
- Go to TTN console and log in
- Add attributes for the device, under 'General settings'
- attribute name = 'mydevices-username', value = MQTT username from cayenne mydevices dashboard
- attribute name = 'mydevices-password', value = MQTT password from cayenne mydevices dashboard
- attribute name = 'mydevices-clientid', value = Client ID from cayenne mydevices dashboard
- Add attributes for the device, under 'General settings'
Work in progress
Forwarding noise data
How it is encoded in the sensor.community firmware:
- Three values are sent in the JSON to sensor.community:
- "noise_LAeq", value in dB(A), meaning?
- "noise_LA_min", value in dB(A), some kind of minimum
- "noise_LA_max", value in dB(A), some kind of maximum
I think these can be encoded in Cayenne as a simple analog value (which has a range of approximately -327..327 with a resolution of 0.01. Just need to assign a channel number to it.
Values are read from the noise sensor as follows:
- call to dnms_calculate_leq()
- call to dnms_read_data_ready(&data_ready) returns 0 if OK and (data_ready != 0)
- call to dnms_read_leq(&dnms_values) returns 0 if OK
- firmware applies a "correction" by adding a fixed offset
Measurement values are encoded as 32-bit units, interpreted as 32-bit floats.
References:
- DNMS is read at https://github.com/opendata-stuttgart/sensors-software/blob/beta/airrohr-firmware/airrohr-firmware.ino#L3392
- DNMS is formatted in the JSON at https://github.com/opendata-stuttgart/sensors-software/blob/beta/airrohr-firmware/airrohr-firmware.ino#L3405
- DNMS I2C code is at https://github.com/opendata-stuttgart/sensors-software/blob/beta/airrohr-firmware/dnms_i2c.cpp