Sensor-data-bridge: Difference between revisions

From RevSpace
Jump to navigation Jump to search
No edit summary
 
(85 intermediate revisions by the same user not shown)
Line 1: Line 1:
{{Project
  |Name=sensor-data-bridge
  |Picture=loraluftdatenforwarder.png
  |Omschrijving=Collects sensor data, forwards it to sensor.community/opensense/etc
  |Status=Completed
  |Contact=bertrik
  }}
== What is this ==
== What is this ==
This is a Java application that sends airborne particulate matter measurement data from TheThingsNetwork and forwards it to sensor.community (formerly Luftdaten).
This is a companion project of [[LoraWanDustSensor]].
 
It  takes airborne particulate matter measurement data transferred through TheThingsNetwork and forwards it to online databases:
* http://sensor.community (formerly Luftdaten),
* http://opensensemap.org and
* <s>https://cayenne.mydevices.com</s>
 
Project on Github: https://github.com/bertrik/sensor-data-bridge
 
Unofficial sensor.community API documentation: https://api-sensor-community.bessarabov.com/
 
=== Features ===
* Picks up particulate matter measurement data received through TheThingsNetwork, using their "v3" infrastructure
* Forwards measurement data to https://sensor.community
* Forwards measurement data to https://opensensemap.org, you can configure the opensense-id by adding a device attribute in TheThingsNetwork console
* -Forwards measurement data to https://cayenne.mydevices.com, you configure the username/password/clientid by adding a device attribute in TheThingsNetwork console-
* Supports Cayenne payload format for the data encoding, a custom payload format for SPS30 data (includes particle counts)
* Supports JSON payload format for the data encoding, you can specify in the config file which JSON fields are used
* Handles particulate matter data (PM10, PM4.0, PM2.5, PM1.0), temperature, humidity, barometric pressure
* Handles sound/noise data (LA EQ, and min/max value)
* Can be run as a systemd service, so it automatically restarts in case the software would crash, server is restarted, etc


It is used in combination with my [[LoraWanDustSensor]]: the sensor sends measurement data to TheThingsNetwork, this application picks up the data from TheThingsNetwork and forwards it to sensor.community and to opensensemap.org.
=== Next steps ===
* add support for geolocation through a scan of surrounding WiFi access-points and a geolocation service (google, mozilla)
* add support for NB-IOT modem with t-mobile backend, see my [[Sim7020]] project
* add support for other backends, e.g. feinstaub-app?


== Requirements ==
== Requirements ==
You need the following:
You need the following:
* a server that is always on and connected to the internet, can be Linux or Windows
* a server that is always on and connected to the internet, can be Linux or Windows
* a Java installation (JDK to compile), at least version 8
* a Java installation (<b>JDK</b> to compile), at least version 17
* some configuration on TheThingsNetwork side
* some configuration on TheThingsNetwork side
* some configuration of my application (YAML file)
* some configuration of my application (YAML file)


== Compilation ==
== Compilation ==
(not needed if you use the docker image)
To compile the software:
To compile the software:
* clone the software from my github archive
* clone the software from my github archive
   git clone https://github.com/bertrik/LoraLuftdatenForwarder.git
   git clone https://github.com/bertrik/sensor-data-bridge.git
* enter the LoraLuftdatenForwarder/gradle directory
* enter the application directory
cd LoraLuftdatenForwarder/gradle
  cd sensor-data-bridge
* run the gradle script to build the software
* run the gradle script to build the software (Linux):
./gradlew assemble
  ./gradlew assemble
* the application zip is now available in LoraLuftdatenForwarder/LoraLuftdatenForwarder/build/distributions
or (Windows):
  gradlew assemble
* the application zip & tar is now available in sensor-data-bridge/sensor-data-bridge/build/distributions


To update to the latest version:
To update to the latest version:
* Update software from github archive:
* Update software from github archive:
   git pull
   git pull
* perform the steps above again
* perform the last two steps above again


== Installation ==
== Installation ==
=== run as docker/podman container ===
How to run in docker/podman:
* Install docker and docker-compose and add your user account to the 'docker' group (Linux):
  sudo apt install docker-ce docker-compose
  sudo usermod -aG docker <username>
* Get the code from github:
  git clone https://github.com/bertrik/sensor-data-bridge
* Enter the 'docker' directory and pull the docker image from github:
  cd sensor-data-bridge
  cd docker
  docker-compose pull
* Edit the config files with your own settings:
  vi sensor-data-bridge.yaml
  (or use nano, gedit, whatever)
* Start the container:
  docker-compose up
=== installation from raw tar ===
Unzip the distribution file somewhere on your system.
Unzip the distribution file somewhere on your system.
I put it in my home directory, for example
I put it in my home directory, for example
   cd
   cd
   tar xvf code/LoraLuftdatenForwarder/LoraLuftdatenForwarder/build/distributions/LoraLuftdatenForwarder.tar
   tar xvf code/sensor-data-bridge/sensor-data-bridge/build/distributions/sensor-data-bridge.tar


== Configuration ==
== Configuration ==


=== Device side ===
=== Node configuration ===
The particulate matter measurement device needs to send data in the Cayenne format.
The particulate matter measurement device needs to send data in the Cayenne format.
I used the following conventions:
I used the following conventions:
Line 40: Line 92:
* PM2.5 is encoded as analog value on channel 2
* PM2.5 is encoded as analog value on channel 2
* PM1.0 is encoded as analog value on channel 0 (optional)
* PM1.0 is encoded as analog value on channel 0 (optional)
* PM4.0 is encoded as analog value on channel 4 (optional)
* Temperature is encoded using standard Cayenne encoding (optional)
* Temperature is encoded using standard Cayenne encoding (optional)
* Humidity is encoded using standard Cayenne encoding (optional)
* Humidity is encoded using standard Cayenne encoding (optional)
* Barometric pressure is encoded using standard Cayenne encoding (optional)
* Barometric pressure is encoded using standard Cayenne encoding (optional)


=== TheThingsNetwork ===
The SPS30 produces mass concentration data in 4 categories, particle count in 5 categories, plus particle size.
Format:
* 16-bit (big endian) PM1.0 mass concentration (unit 0.1)
* 16-bit (big endian) PM2.5 mass concentration (unit 0.1)
* 16-bit (big endian) PM4.0 mass concentration (unit 0.1)
* 16-bit (big endian) PM10 mass concentration (unit 0.1)
* 16-bit (big endian) PM0.5 number concentration (unit 1)
* 16-bit (big endian) PM1.0 number concentration (unit 1)
* 16-bit (big endian) PM2.5 number concentration (unit 1)
* 16-bit (big endian) PM4.0 number concentration (unit 1)
* 16-bit (big endian) PM10 number concentration (unit 1)
* 16-bit (big endian) typical particle size (unit nm)
 
A total of 20 bytes. This is sent on LoRaWAN port 30.
Temperature and humidity is not encoded.
 
In case your node uses a payload decoder in the TTN console, you can configure the sensor-data-bridge with the name of the JSON property:
* 'path' is a pointer inside the decoded JSON payload that contains the value, e.g. "lc/avg"
* 'item' is an internal id in the sensor-data-bridge, e.g. "PM10", see https://github.com/bertrik/sensor-data-bridge/blob/master/sensor-data-bridge/src/main/java/nl/bertriksikken/pm/ESensorItem.java
 
=== TheThingsNetwork application/device configuration ===
[[File:TtnEditApiKey.png|thumb|right|TTN API key rights]]
 
You need to define an 'application' on TheTheThingsNetwork.
You need to define an 'application' on TheTheThingsNetwork.


* Go the TTN console: https://console.cloud.thethings.network/ and log in
* Go the TTN console: https://console.cloud.thethings.network/ and log in
* You need an 'application', create a new one, or use an existing one
* You need an 'application', create a new one, or use an existing one
* Within the application, ou need a 'device', create a new one, or use an existing one:
* Within the application you need a 'device', so create a new one, or use an existing one:
** OTAA
** Use OTAA, LoRaMac version 1.0.3
** Use the application keys as specified in my [[https://revspace.nl/LoraWanDustSensor]] page
** Enter the device EUI as displayed on the display
** Enter the device EUI as displayed on the display
** Use the application keys as specified in my [[LoraWanDustSensor]] page
* You need an API key
* You need an API key
** Create this on the TTN console, for now just create a key that grants all rights
** Create this on the TTN console, grant individual rights as shown in the screenshot
** NOTE: you have only one chance to copy this key somewhere, so copy/paste it locally to a text file or something
** NOTE: you have only one chance to copy this key somewhere, so copy/paste it locally to a text file or something


=== This application ===
=== Configuration ===
To configure the application:
To configure the application:
* Start the application without a configuration file, this will create a default template, stop the application again
* Start the application without a configuration file, this will create a default template, stop the application again
   cd LoraLuftdatenForwarder
   cd sensor-data-bridge
   bin/LoraLuftdatenForwarder
   bin/sensor-data-bridge
   (ctrl-C)
   (ctrl-C)
* Edit the loraluftdatenforwarder.yaml file
* Edit the configuration YAML file, example:
 
<pre>
---
ttn:
  mqtt_url: "tcp://eu1.cloud.thethings.network"
  identity_server_url: "https://eu1.cloud.thethings.network"
  identity_server_timeout: 20
  apps:
  - name: "particulatematter"
    key: "NNSXS......."
    encoding: "CAYENNE"
senscom:
  url: "https://api.sensor.community"
  timeout: 20
opensense:
  url: "https://api.opensensemap.org"
  timeout: 20
</pre>
 
So:
* enter the name of your application
* enter the TTN API key you saved earlier
* other defaults are probably OK


=== Sensor.community ===
=== Sensor.community ===
TODO
TODO
* Go to https://devices.sensor.community/ and log in
* Go to https://devices.sensor.community/ and log in
* Register a node with id 'TTN-<device-EUI-as-shown-on-display>'
* Register a node with id 'TTN-<device-EUI-as-shown-on-display>' (without the spaces or hyphens, e.g. 'TTN-0000547AF1BF713C')
* Register it with the proper configuration, e.g. SDS011 with BME280
* Register it with the proper configuration, e.g. SDS011 with BME280
* In the TTN console, add a device attribute with name 'senscom-id' and value 'TTN-<device-EUI-as-shown-on-display>'


=== Opensensemap ===
=== Opensensemap ===
TODO
* Go to opensensemap.org and log in
* Create a node with the proper configuration
** Create an opensense node with the proper configuration
* Copy the opensensenmap 'box id'
** Copy the opensensenmap 'box id', a long hexadecimal string
* Add an attribute in the TTN console:
* Go the TTN console: https://console.cloud.thethings.network/ and log in
** Go to the TTN console, General settings of the device
** Add an attribute for the device, under 'General settings', name = 'opensense-id', value = boxid that you copied from opensensemap.org
** Add attribute, name = 'opensense-id', value = box id that you just copied
* The mapping from TTN-id to boxid is refreshed by the forwarder once an hour, so within an hour the forwarding to opensensemap.org starts
* The boxid should be picked up automatically by the forwarder, within an hour
 
== Development ==
 
== Forwarding noise data ==
How it is encoded in the sensor.community firmware:
* Three values are sent in the JSON to sensor.community:
** "noise_LAeq", value in dB(A)
** "noise_LA_min", value in dB(A)
** "noise_LA_max", value in dB(A)
 
Values are read from the noise sensor as follows:
* call to dnms_calculate_leq()
* call to dnms_read_data_ready(&data_ready) returns 0 if OK and (data_ready != 0)
* call to dnms_read_leq(&dnms_values) returns 0 if OK
* firmware applies a "correction" by adding a fixed offset
 
Measurement values are encoded as 32-bit units, interpreted as 32-bit floats.
 
References:
* DNMS is read at https://github.com/opendata-stuttgart/sensors-software/blob/beta/airrohr-firmware/airrohr-firmware.ino#L3392
* DNMS is formatted in the JSON at https://github.com/opendata-stuttgart/sensors-software/blob/beta/airrohr-firmware/airrohr-firmware.ino#L3405
* DNMS I2C code is at https://github.com/opendata-stuttgart/sensors-software/blob/beta/airrohr-firmware/dnms_i2c.cpp

Latest revision as of 22:17, 10 March 2024

Project sensor-data-bridge
Loraluftdatenforwarder.png
Collects sensor data, forwards it to sensor.community/opensense/etc
Status Completed
Contact bertrik
Last Update 2024-03-10

What is this

This is a companion project of LoraWanDustSensor.

It takes airborne particulate matter measurement data transferred through TheThingsNetwork and forwards it to online databases:

Project on Github: https://github.com/bertrik/sensor-data-bridge

Unofficial sensor.community API documentation: https://api-sensor-community.bessarabov.com/

Features

  • Picks up particulate matter measurement data received through TheThingsNetwork, using their "v3" infrastructure
  • Forwards measurement data to https://sensor.community
  • Forwards measurement data to https://opensensemap.org, you can configure the opensense-id by adding a device attribute in TheThingsNetwork console
  • -Forwards measurement data to https://cayenne.mydevices.com, you configure the username/password/clientid by adding a device attribute in TheThingsNetwork console-
  • Supports Cayenne payload format for the data encoding, a custom payload format for SPS30 data (includes particle counts)
  • Supports JSON payload format for the data encoding, you can specify in the config file which JSON fields are used
  • Handles particulate matter data (PM10, PM4.0, PM2.5, PM1.0), temperature, humidity, barometric pressure
  • Handles sound/noise data (LA EQ, and min/max value)
  • Can be run as a systemd service, so it automatically restarts in case the software would crash, server is restarted, etc

Next steps

  • add support for geolocation through a scan of surrounding WiFi access-points and a geolocation service (google, mozilla)
  • add support for NB-IOT modem with t-mobile backend, see my Sim7020 project
  • add support for other backends, e.g. feinstaub-app?

Requirements

You need the following:

  • a server that is always on and connected to the internet, can be Linux or Windows
  • a Java installation (JDK to compile), at least version 17
  • some configuration on TheThingsNetwork side
  • some configuration of my application (YAML file)

Compilation

(not needed if you use the docker image) To compile the software:

  • clone the software from my github archive
 git clone https://github.com/bertrik/sensor-data-bridge.git
  • enter the application directory
 cd sensor-data-bridge
  • run the gradle script to build the software (Linux):
 ./gradlew assemble

or (Windows):

 gradlew assemble
  • the application zip & tar is now available in sensor-data-bridge/sensor-data-bridge/build/distributions

To update to the latest version:

  • Update software from github archive:
 git pull
  • perform the last two steps above again

Installation

run as docker/podman container

How to run in docker/podman:

  • Install docker and docker-compose and add your user account to the 'docker' group (Linux):
 sudo apt install docker-ce docker-compose
 sudo usermod -aG docker <username>
  • Get the code from github:
 git clone https://github.com/bertrik/sensor-data-bridge
  • Enter the 'docker' directory and pull the docker image from github:
 cd sensor-data-bridge
 cd docker
 docker-compose pull
  • Edit the config files with your own settings:
 vi sensor-data-bridge.yaml
 (or use nano, gedit, whatever)
  • Start the container:
 docker-compose up

installation from raw tar

Unzip the distribution file somewhere on your system. I put it in my home directory, for example

 cd
 tar xvf code/sensor-data-bridge/sensor-data-bridge/build/distributions/sensor-data-bridge.tar

Configuration

Node configuration

The particulate matter measurement device needs to send data in the Cayenne format. I used the following conventions:

  • PM10 is encoded as analog value on channel 1
  • PM2.5 is encoded as analog value on channel 2
  • PM1.0 is encoded as analog value on channel 0 (optional)
  • PM4.0 is encoded as analog value on channel 4 (optional)
  • Temperature is encoded using standard Cayenne encoding (optional)
  • Humidity is encoded using standard Cayenne encoding (optional)
  • Barometric pressure is encoded using standard Cayenne encoding (optional)

The SPS30 produces mass concentration data in 4 categories, particle count in 5 categories, plus particle size. Format:

  • 16-bit (big endian) PM1.0 mass concentration (unit 0.1)
  • 16-bit (big endian) PM2.5 mass concentration (unit 0.1)
  • 16-bit (big endian) PM4.0 mass concentration (unit 0.1)
  • 16-bit (big endian) PM10 mass concentration (unit 0.1)
  • 16-bit (big endian) PM0.5 number concentration (unit 1)
  • 16-bit (big endian) PM1.0 number concentration (unit 1)
  • 16-bit (big endian) PM2.5 number concentration (unit 1)
  • 16-bit (big endian) PM4.0 number concentration (unit 1)
  • 16-bit (big endian) PM10 number concentration (unit 1)
  • 16-bit (big endian) typical particle size (unit nm)

A total of 20 bytes. This is sent on LoRaWAN port 30. Temperature and humidity is not encoded.

In case your node uses a payload decoder in the TTN console, you can configure the sensor-data-bridge with the name of the JSON property:

TheThingsNetwork application/device configuration

TTN API key rights

You need to define an 'application' on TheTheThingsNetwork.

  • Go the TTN console: https://console.cloud.thethings.network/ and log in
  • You need an 'application', create a new one, or use an existing one
  • Within the application you need a 'device', so create a new one, or use an existing one:
    • Use OTAA, LoRaMac version 1.0.3
    • Enter the device EUI as displayed on the display
    • Use the application keys as specified in my LoraWanDustSensor page
  • You need an API key
    • Create this on the TTN console, grant individual rights as shown in the screenshot
    • NOTE: you have only one chance to copy this key somewhere, so copy/paste it locally to a text file or something

Configuration

To configure the application:

  • Start the application without a configuration file, this will create a default template, stop the application again
 cd sensor-data-bridge
 bin/sensor-data-bridge
 (ctrl-C)
  • Edit the configuration YAML file, example:
---
ttn:
  mqtt_url: "tcp://eu1.cloud.thethings.network"
  identity_server_url: "https://eu1.cloud.thethings.network"
  identity_server_timeout: 20
  apps:
  - name: "particulatematter"
    key: "NNSXS......."
    encoding: "CAYENNE"
senscom:
  url: "https://api.sensor.community"
  timeout: 20
opensense:
  url: "https://api.opensensemap.org"
  timeout: 20

So:

  • enter the name of your application
  • enter the TTN API key you saved earlier
  • other defaults are probably OK

Sensor.community

TODO

  • Go to https://devices.sensor.community/ and log in
  • Register a node with id 'TTN-<device-EUI-as-shown-on-display>' (without the spaces or hyphens, e.g. 'TTN-0000547AF1BF713C')
  • Register it with the proper configuration, e.g. SDS011 with BME280
  • In the TTN console, add a device attribute with name 'senscom-id' and value 'TTN-<device-EUI-as-shown-on-display>'

Opensensemap

  • Go to opensensemap.org and log in
    • Create an opensense node with the proper configuration
    • Copy the opensensenmap 'box id', a long hexadecimal string
  • Go the TTN console: https://console.cloud.thethings.network/ and log in
    • Add an attribute for the device, under 'General settings', name = 'opensense-id', value = boxid that you copied from opensensemap.org
  • The mapping from TTN-id to boxid is refreshed by the forwarder once an hour, so within an hour the forwarding to opensensemap.org starts

Development

Forwarding noise data

How it is encoded in the sensor.community firmware:

  • Three values are sent in the JSON to sensor.community:
    • "noise_LAeq", value in dB(A)
    • "noise_LA_min", value in dB(A)
    • "noise_LA_max", value in dB(A)

Values are read from the noise sensor as follows:

  • call to dnms_calculate_leq()
  • call to dnms_read_data_ready(&data_ready) returns 0 if OK and (data_ready != 0)
  • call to dnms_read_leq(&dnms_values) returns 0 if OK
  • firmware applies a "correction" by adding a fixed offset

Measurement values are encoded as 32-bit units, interpreted as 32-bit floats.

References: