Stofradar: Difference between revisions
mNo edit summary |
|||
(34 intermediate revisions by the same user not shown) | |||
Line 1: | Line 1: | ||
{{Project | {{Project | ||
|Name=Stofradar | |Name=Stofradar | ||
|Picture= | |Picture=stofradar.png | ||
|Omschrijving=Visualizing | |Omschrijving=Visualizing airborne particulate matter concentrations on a map | ||
|Status= | |Status=Completed | ||
|Contact=bertrik | |Contact=bertrik | ||
}} | }} | ||
== Introduction == | == Introduction == | ||
This page is about creating a 'stofradar' image of atmospheric particulate matter concentrations based on the raw data measured by the | This page is about creating a 'stofradar' image of atmospheric particulate matter concentrations based on the raw data measured by the sensor.community network, | ||
see [http://www.stofradar.nl www.stofradar.nl]. | see [http://www.stofradar.nl www.stofradar.nl]. | ||
Line 20: | Line 16: | ||
See also my [[DustSensor]] page. | See also my [[DustSensor]] page. | ||
The website [https:// | The website [https://sensor.community sensor.community] is an initiative to allow citizens to participate in measuring atmospheric particulate matter concentration using an inexpensive and [https://sensor.community/nl/sensor-bouwen/ easy to build sensor]. | ||
They collect this data, calculate 5 minute and daily averages and publish it again as open data. | They collect this data, calculate 5 minute and daily averages and publish it again as open data. | ||
The total number of sensors is > | The total number of sensors is > 12000 worldwide, most of them in Germany, Bulgaria, Belgium, Austria, Sweden. | ||
The Netherlands has | The Netherlands has > 2000 sensors. See also [https://stats.sensor.community/]. | ||
Future activities: | Future activities: | ||
* | * play with different algorithm: | ||
* plot official [https://www.luchtmeetnet.nl/ luchtmeetnet] measurements on the map too, access to the API has been implemented | ** for each pixel consider only stations with a certain radius (say 10 km), calculate the median, convert median to color | ||
** should more naturally filter out outliers and is actually a bit similar to what sensor.community uses. | |||
* | * plot official [https://www.luchtmeetnet.nl/ luchtmeetnet] measurements on the map too, access to the API has been implemented [https://github.com/bertrik/luftdatenmapper/tree/master/luftdatenmapper/src/main/java/nl/bertriksikken/luchtmeetnet/api work in progress] | ||
* <s>base humidity on BME280 sensors only, ignore DHT11/22 type sensors</s> | |||
* add a water mark | |||
* | |||
== Visualisation == | == Visualisation == | ||
Line 40: | Line 33: | ||
=== Background map === | === Background map === | ||
The map background on stofradar.nl is based on https://mapsvg.com/maps/netherlands | |||
The map projection used is the '''equirectangular projection''' (EPSG-32662), | |||
so I can easily map a pixel back to a latitude/longitude. | so I can easily map a pixel back to a latitude/longitude. | ||
=== Data filtering === | |||
There is only very minimal data filtering. Sensor measurements are taken into account as follows: | |||
* Sensors from an area 2x2 times bigger than the area visualized are considered for visualisation | |||
* Sensors marked as 'indoor' are ignored | |||
* Sensors with a measurement value smaller than 0 are ignored | |||
* The top percent of highest PM2.5 concentrations is discarded, this mostly takes care of outliers caused by defective sensors | |||
* When sensor data is not available in the past 5 minutes, data from a previous measurement interval is used, up to 1 hour old | |||
* A (small) number of sensors that are known to always report a very high value are not considered (blacklisted) | |||
=== Interpolation === | === Interpolation === | ||
Line 55: | Line 54: | ||
To calculate the distance, I use a very simple approximation: | To calculate the distance, I use a very simple approximation: | ||
* calculate the "middle" of the map (average latitude/longitude between top-left and bottom-right); | * calculate the "middle" of the map (average latitude/longitude between top-left and bottom-right); | ||
* calculate the "km-per-degree-latitude" at the middle for latitude as | * calculate the "km-per-degree-latitude" at the middle for latitude as 40075 km / 360 degrees; | ||
* calculate the "km-per-degree-longitude" at the middle for longitude as the number above multiplied with cos(latitude); | * calculate the "km-per-degree-longitude" at the middle for longitude as the number above multiplied with cos(latitude); | ||
* determine the difference in longitude and the difference in latitude; | * determine the difference in longitude and the difference in latitude; | ||
* convert both to km using the factors calculated earlier; | * convert both to km using the factors calculated earlier; | ||
* calculate the [https://en.wikipedia.org/wiki/Euclidean_distance euclidean distance]. | * calculate the [https://en.wikipedia.org/wiki/Euclidean_distance euclidean distance]. | ||
Pixels that are not within a certain distance of any sensor station (e.g. | Pixels that are not within a certain distance of any sensor station (e.g. 10 km) are rendered as grayscale, to indicate a geographic limit of each sensor. | ||
Only sensors within a reasonable range of the map are taken into account, currently this is an area of | Only sensors within a reasonable range of the map are taken into account, currently this is an area of 4 times (2x2) the visible area. | ||
PM10 values | PM10 values < 0 ug/m3 are ignored in the software. | ||
=== Colour range === | === Colour range === | ||
The colours I'm using | [[File:luchtmeetnet_lki.png|right|thumb|Luchtmeetnet ranges]] | ||
The colours I'm using are based on the scale used for air quality index from luchtmeetnet with data from RIVM, | |||
see https://www.luchtmeetnet.nl/informatie/luchtkwaliteit/luchtkwaliteitsindex-(lki) | |||
The input value is the PM2.5 concentration. | |||
Values in between these levels are interpolated linearly with respect to the RGB colour value and alpha channel. | Values in between these levels are interpolated linearly with respect to the RGB colour value and alpha channel. | ||
=== Correction for high humidity === | === Correction for high humidity === | ||
Line 84: | Line 80: | ||
The effect become really significant above approximately 70% humidity. | The effect become really significant above approximately 70% humidity. | ||
An interesting idea is to try to compensate for this effect, since the | An interesting idea is to try to compensate for this effect, since the sensor.community sensor has an onboard humidity-sensor. | ||
Some papers/links about this: | Some papers/links about this: | ||
* https://www.samenmetenaanluchtkwaliteit.nl/sites/default/files/2018-07/Status_SDS011_12juli18.pdf | * https://www.samenmetenaanluchtkwaliteit.nl/sites/default/files/2018-07/Status_SDS011_12juli18.pdf | ||
Line 93: | Line 89: | ||
* it combines formulas and coefficients from different sources where relative humidity has different units. One paper seems to use an RH-value from 0 to 100, while another uses a kind of normalized relative humidity (from 0 to 1). You cannot just use the same coefficients if the unit is different. | * it combines formulas and coefficients from different sources where relative humidity has different units. One paper seems to use an RH-value from 0 to 100, while another uses a kind of normalized relative humidity (from 0 to 1). You cannot just use the same coefficients if the unit is different. | ||
* it claims a humidity correction for PM10 with coefficients that is not found in the source paper. | * it claims a humidity correction for PM10 with coefficients that is not found in the source paper. | ||
=== Animation === | === Animation === | ||
Besides an image with current data from the last 5 minutes, every hour two animations are created: | |||
* GIF animation composed of hourly images over the past 24 hours | |||
* WEBM animation composed of 5-minute images over the past 24 hours | |||
* GIF | |||
You can click on the GIF animation to view the WEBM animation. | |||
== Software == | == Software == | ||
See the [https://github.com/bertrik/luftdatenmapper github page] for the source code. | See the [https://github.com/bertrik/luftdatenmapper github page] for the source code. |
Revision as of 07:08, 15 October 2021
Project Stofradar | |
---|---|
Visualizing airborne particulate matter concentrations on a map | |
Status | Completed |
Contact | bertrik |
Last Update | 2021-10-15 |
Introduction
This page is about creating a 'stofradar' image of atmospheric particulate matter concentrations based on the raw data measured by the sensor.community network, see www.stofradar.nl.
The focus is on raw visualisation of the source data, only the most minimal attempt is made to "validate" the data. Sensor measurements and sensor locations are basically uncontrolled, since we cannot tell if a particular sensor is defective or has an unusual position that affects its measurements.
See also my DustSensor page.
The website sensor.community is an initiative to allow citizens to participate in measuring atmospheric particulate matter concentration using an inexpensive and easy to build sensor. They collect this data, calculate 5 minute and daily averages and publish it again as open data. The total number of sensors is > 12000 worldwide, most of them in Germany, Bulgaria, Belgium, Austria, Sweden. The Netherlands has > 2000 sensors. See also [1].
Future activities:
- play with different algorithm:
- for each pixel consider only stations with a certain radius (say 10 km), calculate the median, convert median to color
- should more naturally filter out outliers and is actually a bit similar to what sensor.community uses.
- plot official luchtmeetnet measurements on the map too, access to the API has been implemented work in progress
base humidity on BME280 sensors only, ignore DHT11/22 type sensors- add a water mark
Visualisation
The general idea is to create an image, with a map at the background and the atmospheric particulate matter concentration overlaid on top.
Background map
The map background on stofradar.nl is based on https://mapsvg.com/maps/netherlands
The map projection used is the equirectangular projection (EPSG-32662), so I can easily map a pixel back to a latitude/longitude.
Data filtering
There is only very minimal data filtering. Sensor measurements are taken into account as follows:
- Sensors from an area 2x2 times bigger than the area visualized are considered for visualisation
- Sensors marked as 'indoor' are ignored
- Sensors with a measurement value smaller than 0 are ignored
- The top percent of highest PM2.5 concentrations is discarded, this mostly takes care of outliers caused by defective sensors
- When sensor data is not available in the past 5 minutes, data from a previous measurement interval is used, up to 1 hour old
- A (small) number of sensors that are known to always report a very high value are not considered (blacklisted)
Interpolation
Since we only have data at a set of discrete points, the concentration at other points is estimated by combining data from all sensors using inverse distance weighting, in particular using the distance *squared* as the weighing factor in a weighted average. So a nearby sensor has a large effect and a far away sensor has very little effect, contributing only a little bit to the global average.
To calculate the distance, I use a very simple approximation:
- calculate the "middle" of the map (average latitude/longitude between top-left and bottom-right);
- calculate the "km-per-degree-latitude" at the middle for latitude as 40075 km / 360 degrees;
- calculate the "km-per-degree-longitude" at the middle for longitude as the number above multiplied with cos(latitude);
- determine the difference in longitude and the difference in latitude;
- convert both to km using the factors calculated earlier;
- calculate the euclidean distance.
Pixels that are not within a certain distance of any sensor station (e.g. 10 km) are rendered as grayscale, to indicate a geographic limit of each sensor.
Only sensors within a reasonable range of the map are taken into account, currently this is an area of 4 times (2x2) the visible area.
PM10 values < 0 ug/m3 are ignored in the software.
Colour range
The colours I'm using are based on the scale used for air quality index from luchtmeetnet with data from RIVM, see https://www.luchtmeetnet.nl/informatie/luchtkwaliteit/luchtkwaliteitsindex-(lki)
The input value is the PM2.5 concentration.
Values in between these levels are interpolated linearly with respect to the RGB colour value and alpha channel.
Correction for high humidity
Humidity generally seems to cause an overestimation of PM measurements for measurements done with a "particle counting" type of PM sensor. The effect become really significant above approximately 70% humidity.
An interesting idea is to try to compensate for this effect, since the sensor.community sensor has an onboard humidity-sensor. Some papers/links about this:
- https://www.samenmetenaanluchtkwaliteit.nl/sites/default/files/2018-07/Status_SDS011_12juli18.pdf
- https://www.researchgate.net/publication/320474792_Influence_of_Humidity_on_the_Accuracy_of_Low-Cost_Particulate_Matter_Sensors
- https://github.com/opendata-stuttgart/meta/wiki/Luftfeuchte-Korrektur
However, I see the following problems with the formulas and coefficient in the opendata-stuttgart link above:
- it combines formulas and coefficients from different sources where relative humidity has different units. One paper seems to use an RH-value from 0 to 100, while another uses a kind of normalized relative humidity (from 0 to 1). You cannot just use the same coefficients if the unit is different.
- it claims a humidity correction for PM10 with coefficients that is not found in the source paper.
Animation
Besides an image with current data from the last 5 minutes, every hour two animations are created:
- GIF animation composed of hourly images over the past 24 hours
- WEBM animation composed of 5-minute images over the past 24 hours
You can click on the GIF animation to view the WEBM animation.
Software
See the github page for the source code.