Difference between revisions of "Sensor-data-bridge"

From RevSpace
Jump to navigation Jump to search
(Opensensemap)
(Work in progress)
(34 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
{{Project
 
{{Project
 
   |Name=LoraLuftdatenForwarder
 
   |Name=LoraLuftdatenForwarder
   |Picture=nopicture.jpg
+
   |Picture=loraluftdatenforwarder.png
 
   |Omschrijving=LoRaWAN forwarder for particulate matter data
 
   |Omschrijving=LoRaWAN forwarder for particulate matter data
 
   |Status=In progress
 
   |Status=In progress
Line 8: Line 8:
  
 
== What is this ==
 
== What is this ==
This is a Java application that sends airborne particulate matter measurement data from TheThingsNetwork and forwards it to sensor.community (formerly Luftdaten).
+
This is a companion project of [[LoraWanDustSensor]].
  
It is used in combination with my [[LoraWanDustSensor]]: the sensor sends measurement data to TheThingsNetwork, this application picks up the data from TheThingsNetwork and forwards it to sensor.community and to opensensemap.org.
+
It takes airborne particulate matter measurement data transferred through TheThingsNetwork and forwards it to online databases:
 +
* http://sensor.community (formerly Luftdaten),
 +
* http://opensensemap.org and
 +
* https://cayenne.mydevices.com
  
Next steps:
+
Project on Github: https://github.com/bertrik/LoraLuftdatenForwarder
* clean out TTNv2 support, it is obsoleted by TTNv3
+
 
 +
=== Features ===
 +
* Picks up particulate matter measurement data received through TheThingsNetwork, using their "v3" infrastructure
 +
* Forwards measurement data to https://sensor.community
 +
* Forwards measurement data to https://opensensemap.org, you can configure the opensense-id by adding a device attribute in TheThingsNetwork console
 +
* Forwards measurement data to https://cayenne.mydevices.com, you configure the username/password/clientid by adding a device attribute in TheThingsNetwork console
 +
* Supports Cayenne payload format for the data encoding, a custom payload format for SPS30 data (includes particle counts)
 +
* Handles particulate matter data (PM10, PM4.0, PM2.5, PM1.0), temperature, humidity, barometric pressure
 +
* Can be run as a systemd service, so it automatically restarts in case the software would crash
 +
 
 +
=== Next steps ===
 +
* test support for forwarding of SPS30 data, including particle counts and particle size
 +
* add support for auto-localization through a scan of surrounding WiFi access-points and a geolocation service (google, mozilla), see https://github.com/bertrik/LoraLuftdatenForwarder/tree/master/LoraLuftdatenForwarder/src/main/java/nl/bertriksikken/gls
 
* add support for NB-IOT modem with t-mobile backend, see my [[Sim7020]] project
 
* add support for NB-IOT modem with t-mobile backend, see my [[Sim7020]] project
 +
* add support for other backends, e.g. feinstaub-app?
  
 
== Requirements ==
 
== Requirements ==
Line 29: Line 45:
 
* enter the LoraLuftdatenForwarder/gradle directory
 
* enter the LoraLuftdatenForwarder/gradle directory
 
   cd LoraLuftdatenForwarder/gradle
 
   cd LoraLuftdatenForwarder/gradle
* run the gradle script to build the software:
+
* run the gradle script to build the software (Linux):
 
   ./gradlew assemble
 
   ./gradlew assemble
 +
or (Windows):
 +
  gradlew assemble
 
* the application zip & tar is now available in LoraLuftdatenForwarder/LoraLuftdatenForwarder/build/distributions
 
* the application zip & tar is now available in LoraLuftdatenForwarder/LoraLuftdatenForwarder/build/distributions
  
Line 52: Line 70:
 
* PM2.5 is encoded as analog value on channel 2
 
* PM2.5 is encoded as analog value on channel 2
 
* PM1.0 is encoded as analog value on channel 0 (optional)
 
* PM1.0 is encoded as analog value on channel 0 (optional)
 +
* PM4.0 is encoded as analog value on channel 4 (optional)
 
* Temperature is encoded using standard Cayenne encoding (optional)
 
* Temperature is encoded using standard Cayenne encoding (optional)
 
* Humidity is encoded using standard Cayenne encoding (optional)
 
* Humidity is encoded using standard Cayenne encoding (optional)
 
* Barometric pressure is encoded using standard Cayenne encoding (optional)
 
* Barometric pressure is encoded using standard Cayenne encoding (optional)
 +
 +
The SPS30 produces mass concentration data in 4 categories, particle count in 5 categories, plus particle size.
 +
Format:
 +
* 16-bit (big endian) PM1.0 mass concentration (unit 0.1)
 +
* 16-bit (big endian) PM2.5 mass concentration (unit 0.1)
 +
* 16-bit (big endian) PM4.0 mass concentration (unit 0.1)
 +
* 16-bit (big endian) PM10 mass concentration (unit 0.1)
 +
* 16-bit (big endian) PM0.5 number concentration (unit 1)
 +
* 16-bit (big endian) PM1.0 number concentration (unit 1)
 +
* 16-bit (big endian) PM2.5 number concentration (unit 1)
 +
* 16-bit (big endian) PM4.0 number concentration (unit 1)
 +
* 16-bit (big endian) PM10 number concentration (unit 1)
 +
* 16-bit (big endian) typical particle size (unit nm)
 +
 +
A total of 20 bytes. This is sent on LoRaWAN port 30.
 +
Temperature and humidity is not encoded.
  
 
=== TheThingsNetwork application/device configuration ===
 
=== TheThingsNetwork application/device configuration ===
Line 64: Line 99:
 
* You need an 'application', create a new one, or use an existing one
 
* You need an 'application', create a new one, or use an existing one
 
* Within the application you need a 'device', so create a new one, or use an existing one:
 
* Within the application you need a 'device', so create a new one, or use an existing one:
** Use OTAA
+
** Use OTAA, LoRaMac version 1.0.3
 
** Enter the device EUI as displayed on the display
 
** Enter the device EUI as displayed on the display
 
** Use the application keys as specified in my [[LoraWanDustSensor]] page
 
** Use the application keys as specified in my [[LoraWanDustSensor]] page
Line 82: Line 117:
 
---
 
---
 
ttn:
 
ttn:
   mqtt_url_v2: "tcp://eu.thethings.network"
+
   mqtt_url: "tcp://eu1.cloud.thethings.network"
  mqtt_url_v3: "tcp://eu1.cloud.thethings.network"
 
 
   identity_server_url: "https://eu1.cloud.thethings.network"
 
   identity_server_url: "https://eu1.cloud.thethings.network"
 
   identity_server_timeout: 20
 
   identity_server_timeout: 20
 
   apps:
 
   apps:
 
   - name: "particulatematter"
 
   - name: "particulatematter"
    version: "V3"
 
 
     key: "NNSXS......."
 
     key: "NNSXS......."
 
     encoding: "CAYENNE"
 
     encoding: "CAYENNE"
Line 97: Line 130:
 
   url: "https://api.opensensemap.org"
 
   url: "https://api.opensensemap.org"
 
   timeout: 20
 
   timeout: 20
  ids:
 
    "0ttn-deveui-here": "put-opensense-boxid-here"
 
    "0004A30B001E00BE": "5eca4a0fd0545b001c0e5d46"
 
 
</pre>
 
</pre>
  
Line 110: Line 140:
 
TODO
 
TODO
 
* Go to https://devices.sensor.community/ and log in
 
* Go to https://devices.sensor.community/ and log in
* Register a node with id 'TTN-<device-EUI-as-shown-on-display>'
+
* Register a node with id 'TTN-<device-EUI-as-shown-on-display>' (without the spaces or hyphens, e.g. 'TTN-0000547AF1BF713C')
 
* Register it with the proper configuration, e.g. SDS011 with BME280
 
* Register it with the proper configuration, e.g. SDS011 with BME280
  
Line 118: Line 148:
 
** Copy the opensensenmap 'box id', a long hexadecimal string
 
** Copy the opensensenmap 'box id', a long hexadecimal string
 
* Go the TTN console: https://console.cloud.thethings.network/ and log in
 
* Go the TTN console: https://console.cloud.thethings.network/ and log in
** Add an attribute for the device, under 'General settings', name = 'opensense-id', value = 'box id' (without the quotes) that you copied from opensensemap.org
+
** Add an attribute for the device, under 'General settings', name = 'opensense-id', value = boxid that you copied from opensensemap.org
* The boxid should be picked up automatically by the forwarder, within an hour
+
* The mapping from TTN-id to boxid is refreshed by the forwarder once an hour, so within an hour the forwarding to opensensemap.org starts
 +
 
 +
=== Cayenne myDevices ===
 +
* Go to https://cayenne.mydevices.com/ and log in
 +
** Add a new node, TODO
 +
* Go to TTN console and log in
 +
** Add attributes for the device, under 'General settings'
 +
*** attribute name = 'mydevices-username', value = MQTT username from cayenne mydevices dashboard
 +
*** attribute name = 'mydevices-password', value = MQTT password from cayenne mydevices dashboard
 +
*** attribute name = 'mydevices-clientid', value = Client ID from cayenne mydevices dashboard
 +
 
 +
== Work in progress ==
 +
 
 +
=== Forwarding noise data ===
 +
 
 +
How it is encoded in the sensor.community firmware:
 +
* Three values are sent in the JSON to sensor.community:
 +
** "noise_LAeq", value in dB(A), meaning?
 +
** "noise_LA_min", value in dB(A), some kind of minimum
 +
** "noise_LA_max", value in dB(A), some kind of maximum
 +
 
 +
I think these can be encoded in Cayenne as a simple analog value (which has a range of approximately -327..327 with a resolution of 0.01.
 +
Just need to assign a channel number to it.
 +
 
 +
Values are read from the noise sensor as follows:
 +
* call to dnms_calculate_leq()
 +
* call to dnms_read_data_ready(&data_ready) returns 0 if OK and (data_ready != 0)
 +
* call to dnms_read_leq(&dnms_values) returns 0 if OK
 +
* firmware applies a "correction" by adding a fixed offset
 +
 
 +
Measurement values are encoded as 32-bit units, interpreted as 32-bit floats.
 +
 
 +
References:
 +
* DNMS is read at https://github.com/opendata-stuttgart/sensors-software/blob/beta/airrohr-firmware/airrohr-firmware.ino#L3392
 +
* DNMS is formatted in the JSON at https://github.com/opendata-stuttgart/sensors-software/blob/beta/airrohr-firmware/airrohr-firmware.ino#L3405
 +
* DNMS I2C code is at https://github.com/opendata-stuttgart/sensors-software/blob/beta/airrohr-firmware/dnms_i2c.cpp

Revision as of 17:27, 16 June 2021

Project LoraLuftdatenForwarder
Loraluftdatenforwarder.png
LoRaWAN forwarder for particulate matter data
Status In progress
Contact bertrik
Last Update 2021-06-16

What is this

This is a companion project of LoraWanDustSensor.

It takes airborne particulate matter measurement data transferred through TheThingsNetwork and forwards it to online databases:

Project on Github: https://github.com/bertrik/LoraLuftdatenForwarder

Features

  • Picks up particulate matter measurement data received through TheThingsNetwork, using their "v3" infrastructure
  • Forwards measurement data to https://sensor.community
  • Forwards measurement data to https://opensensemap.org, you can configure the opensense-id by adding a device attribute in TheThingsNetwork console
  • Forwards measurement data to https://cayenne.mydevices.com, you configure the username/password/clientid by adding a device attribute in TheThingsNetwork console
  • Supports Cayenne payload format for the data encoding, a custom payload format for SPS30 data (includes particle counts)
  • Handles particulate matter data (PM10, PM4.0, PM2.5, PM1.0), temperature, humidity, barometric pressure
  • Can be run as a systemd service, so it automatically restarts in case the software would crash

Next steps

Requirements

You need the following:

  • a server that is always on and connected to the internet, can be Linux or Windows
  • a Java installation (JDK to compile), at least version 8
  • some configuration on TheThingsNetwork side
  • some configuration of my application (YAML file)

Compilation

To compile the software:

  • clone the software from my github archive
 git clone https://github.com/bertrik/LoraLuftdatenForwarder.git
  • enter the LoraLuftdatenForwarder/gradle directory
 cd LoraLuftdatenForwarder/gradle
  • run the gradle script to build the software (Linux):
 ./gradlew assemble

or (Windows):

 gradlew assemble
  • the application zip & tar is now available in LoraLuftdatenForwarder/LoraLuftdatenForwarder/build/distributions

To update to the latest version:

  • Update software from github archive:
 git pull
  • perform the last two steps above again

Installation

Unzip the distribution file somewhere on your system. I put it in my home directory, for example

 cd
 tar xvf code/LoraLuftdatenForwarder/LoraLuftdatenForwarder/build/distributions/LoraLuftdatenForwarder.tar

Configuration

Node configuration

The particulate matter measurement device needs to send data in the Cayenne format. I used the following conventions:

  • PM10 is encoded as analog value on channel 1
  • PM2.5 is encoded as analog value on channel 2
  • PM1.0 is encoded as analog value on channel 0 (optional)
  • PM4.0 is encoded as analog value on channel 4 (optional)
  • Temperature is encoded using standard Cayenne encoding (optional)
  • Humidity is encoded using standard Cayenne encoding (optional)
  • Barometric pressure is encoded using standard Cayenne encoding (optional)

The SPS30 produces mass concentration data in 4 categories, particle count in 5 categories, plus particle size. Format:

  • 16-bit (big endian) PM1.0 mass concentration (unit 0.1)
  • 16-bit (big endian) PM2.5 mass concentration (unit 0.1)
  • 16-bit (big endian) PM4.0 mass concentration (unit 0.1)
  • 16-bit (big endian) PM10 mass concentration (unit 0.1)
  • 16-bit (big endian) PM0.5 number concentration (unit 1)
  • 16-bit (big endian) PM1.0 number concentration (unit 1)
  • 16-bit (big endian) PM2.5 number concentration (unit 1)
  • 16-bit (big endian) PM4.0 number concentration (unit 1)
  • 16-bit (big endian) PM10 number concentration (unit 1)
  • 16-bit (big endian) typical particle size (unit nm)

A total of 20 bytes. This is sent on LoRaWAN port 30. Temperature and humidity is not encoded.

TheThingsNetwork application/device configuration

TTN API key rights

You need to define an 'application' on TheTheThingsNetwork.

  • Go the TTN console: https://console.cloud.thethings.network/ and log in
  • You need an 'application', create a new one, or use an existing one
  • Within the application you need a 'device', so create a new one, or use an existing one:
    • Use OTAA, LoRaMac version 1.0.3
    • Enter the device EUI as displayed on the display
    • Use the application keys as specified in my LoraWanDustSensor page
  • You need an API key
    • Create this on the TTN console, grant individual rights as shown in the screenshot
    • NOTE: you have only one chance to copy this key somewhere, so copy/paste it locally to a text file or something

LoraLuftdatenForwarder configuration

To configure the application:

  • Start the application without a configuration file, this will create a default template, stop the application again
 cd LoraLuftdatenForwarder
 bin/LoraLuftdatenForwarder
 (ctrl-C)
  • Edit the loraluftdatenforwarder.yaml file, example:
---
ttn:
  mqtt_url: "tcp://eu1.cloud.thethings.network"
  identity_server_url: "https://eu1.cloud.thethings.network"
  identity_server_timeout: 20
  apps:
  - name: "particulatematter"
    key: "NNSXS......."
    encoding: "CAYENNE"
luftdaten:
  url: "https://api.sensor.community"
  timeout: 20
opensense:
  url: "https://api.opensensemap.org"
  timeout: 20

So:

  • enter the name of your application
  • enter the TTN API key you saved earlier
  • other defaults are probably OK

Sensor.community

TODO

  • Go to https://devices.sensor.community/ and log in
  • Register a node with id 'TTN-<device-EUI-as-shown-on-display>' (without the spaces or hyphens, e.g. 'TTN-0000547AF1BF713C')
  • Register it with the proper configuration, e.g. SDS011 with BME280

Opensensemap

  • Go to opensensemap.org and log in
    • Create an opensense node with the proper configuration
    • Copy the opensensenmap 'box id', a long hexadecimal string
  • Go the TTN console: https://console.cloud.thethings.network/ and log in
    • Add an attribute for the device, under 'General settings', name = 'opensense-id', value = boxid that you copied from opensensemap.org
  • The mapping from TTN-id to boxid is refreshed by the forwarder once an hour, so within an hour the forwarding to opensensemap.org starts

Cayenne myDevices

  • Go to https://cayenne.mydevices.com/ and log in
    • Add a new node, TODO
  • Go to TTN console and log in
    • Add attributes for the device, under 'General settings'
      • attribute name = 'mydevices-username', value = MQTT username from cayenne mydevices dashboard
      • attribute name = 'mydevices-password', value = MQTT password from cayenne mydevices dashboard
      • attribute name = 'mydevices-clientid', value = Client ID from cayenne mydevices dashboard

Work in progress

Forwarding noise data

How it is encoded in the sensor.community firmware:

  • Three values are sent in the JSON to sensor.community:
    • "noise_LAeq", value in dB(A), meaning?
    • "noise_LA_min", value in dB(A), some kind of minimum
    • "noise_LA_max", value in dB(A), some kind of maximum

I think these can be encoded in Cayenne as a simple analog value (which has a range of approximately -327..327 with a resolution of 0.01. Just need to assign a channel number to it.

Values are read from the noise sensor as follows:

  • call to dnms_calculate_leq()
  • call to dnms_read_data_ready(&data_ready) returns 0 if OK and (data_ready != 0)
  • call to dnms_read_leq(&dnms_values) returns 0 if OK
  • firmware applies a "correction" by adding a fixed offset

Measurement values are encoded as 32-bit units, interpreted as 32-bit floats.

References: