Difference between revisions of "Stofradar"

From RevSpace
Jump to navigation Jump to search
(Created page with "== Introduction == This is about my plan to create a 'stofradar' image of fine dust concentrations based on the raw data measured by the luftdaten.info network. luftdaten.inf...")
 
(Visualisation)
(29 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
  {{Project
 +
  |Name=Stofradar
 +
  |Picture=Stofradar3.png
 +
  |Omschrijving=Visualizing atmospheric particulate matter concentrations on a map
 +
  |Status=In progress
 +
  |Contact=bertrik
 +
  }}
 +
 
== Introduction ==
 
== Introduction ==
This is about my plan to create a 'stofradar' image of fine dust concentrations based on the raw data measured by the luftdaten.info network.
+
This page is about creating a 'stofradar' image of atmospheric particulate matter concentrations based on the raw data measured by the luftdaten.info network,
 +
see [http://www.stofradar.nl www.stofradar.nl].
  
luftdaten.info is an initiative to allow citizens measure fine dust concentration using an inexpensive and easy to build fine dust sensor.
+
See also my [[DustSensor]] page.
 +
 
 +
luftdaten.info is an initiative to allow citizens to measure atmospheric particulate matter concentration using an inexpensive and [https://luftdaten.info/nl/sensor-bouwen/ easy to build sensor].
 
They collect this data, calculate 5 minute and daily averages and publish it again as open data.
 
They collect this data, calculate 5 minute and daily averages and publish it again as open data.
The total number of sensors is about 5000 worldwide, most of them in germany, bulgaria, belgium, austria, sweden.
+
The total number of sensors is about 5000 worldwide, most of them in Germany, Bulgaria, Belgium, Austria, Sweden.
The netherlands has about 100 sensors.
+
The Netherlands has about 100 sensors.
  
I want to create images / animations based on this data.
+
Future activities:
 +
* get an OSM background map, once their export functionality works again, https://www.openstreetmap.org/export still broken .... :(
 +
* fix fine alignment between map and dust overlay
 +
* put all of the images from one day into an animation
 +
* also create a map of a larger area, including germany (it has a lot of data points)
  
 
== Visualisation ==
 
== Visualisation ==
The general idea is to create an image, with a map at the background and the fine dust concentration overlaid on top.
+
The general idea is to create an image, with a map at the background and the atmospheric particulate matter concentration overlaid on top.
 +
 
 +
=== Background map ===
 +
Pages to investigate:
 +
* https://wiki.openstreetmap.org/wiki/OSM_on_Paper
 +
* http://maps.stamen.com/m2i/#toner-background/600:800/6/52.200/5.300
 +
* https://developer.mapquest.com/documentation/open/static-map-api/v5/examples/basic/map-bounding-box/
  
TODO:
+
What I need/want is to be able to specify the 'bounding box' and use a '''equirectangular projection''' ,
* get a nice background map, I prefer the equirectangular projection since that is easy to calculate. A black-and-white map would be nice so the overlay adds all the colour.
+
so I can easily map a pixel back to a latitude/longitude.
* find a way to combine the background with the dust concentration overlay: imagemagick?
 
* find a way to combine images over time into an animated gif: imagemagick?
 
  
 
=== Interpolation ===
 
=== Interpolation ===
 
Since we only have data at a set of discrete points, the concentration at other points is estimated by combining data from all sensors using
 
Since we only have data at a set of discrete points, the concentration at other points is estimated by combining data from all sensors using
[https://en.wikipedia.org/wiki/Inverse_distance_weighting inverse distance weighting], in particular using the distance squared as a weighing factor in a weighted average.
+
[https://en.wikipedia.org/wiki/Inverse_distance_weighting inverse distance weighting], in particular using the distance *squared* as the weighing factor in a weighted average.
So a nearby sensor has a large effect and a far away sensor has very little effect only contributing a bit to the global average.
+
So a nearby sensor has a large effect and a far away sensor has very little effect, contributing only a little bit to the global average.
  
To calculate the distance, I use a very simple approximation:
+
To calculate the distance<sup>2</sup>, I use a very simple approximation:
 
* determine the difference in longitude and latitude
 
* determine the difference in longitude and latitude
* apply an 'aspect ratio' factor of cos(latitude) to the longitude difference. I use the latitude of the approximate middle of the netherlands to calculate this aspect ratio factor.  
+
* apply an 'aspect ratio' factor of cos(latitude) to the longitude difference. I use the latitude of the approximate middle of the netherlands (52.2 lat, 5.3 lon) to calculate this aspect ratio factor.  
* distance squared = (difference latitude)squared + (difference longitude)squared
+
* distance<sup>2</sup> = (difference latitude)<sup>2</sup> + (difference longitude)<sup>2</sup>
A better way would be to use the 'great-circle-distance' and possibly even account for the fact that the earth is not perfectly spherical, but I think my approximation suffices and makes the calculation faster.
+
A better way would be to use the  
 +
[https://en.wikipedia.org/wiki/Great-circle_distance 'great-circle-distance'] and possibly even account for the fact that the earth is not perfectly spherical, but I like to start simple and this makes the calculation a lot faster.
 +
 
 +
PM10 values higher than 500 ug/m3 are simply ignored in the software.
  
 
=== Colour range ===
 
=== Colour range ===
 +
The colours I'm using (a kind of inverted spectral range from blue to red):
 +
*  0 ug/m3: fully transparent white (#FFFFFF)
 +
*  25 ug/m3: semi-transparent cyan (#00FFFF)
 +
*  50 ug/m3: semi-transparent yellow (#FFFF00). This is the threshold for the maximum daily average in the Netherlands.
 +
* 100 ug/m3: semi-transparent red (#FF0000)
 +
* 200 ug/m3 and higher: semi-transparent purple (#FF00FF)
 +
Values in between these levels are interpolated linearly with respect to the RGB colour value and alpha channel.
  
The colours I'll probably be using (a kind of spectral range):
+
This scale is approximately logarithmic, with each step being twice as big as the previous one.
*  0 ug/m3: fully transparent white (#FFFFFF.00)
 
*  25 ug/m3: quarter transparent cyan (#00FFFF.40)
 
*  50 ug/m3: quarter transparent yellow (#FFFF00.40)
 
* 100 ug/m3: quarter transparent red (#FF0000.40)
 
* 200 ug/m3 and higher: quarter transparent purple (#FF00FF.40)
 
With interpolation for RGB and alpha for values between these levels.
 
  
The level of 50 ug/m3 has been agreed on as a level that should not be exceeded as the yearly average in the Netherlands.
+
=== Compositing ===
 +
I use imagemagick for this, for example:
 +
  composite -compose over -geometry 600x800 20180605_210100.json.png netherlands.png output.png
 +
where netherlands.png is an 600x800 opaque black-and-white image of the map of the netherlands
 +
and 20180605_210100.json.png is an 60x80 image of dust concentrations with an alpha channel
  
This scale is approximately logarithmic, with each step being twice as big as the previous one.
+
== Software ==
 +
See the [https://github.com/bertrik/luftdatenmapper github page] for the source code.

Revision as of 08:20, 7 November 2018

Project Stofradar
Stofradar3.png
Visualizing atmospheric particulate matter concentrations on a map
Status In progress
Contact bertrik
Last Update 2018-11-07

Introduction

This page is about creating a 'stofradar' image of atmospheric particulate matter concentrations based on the raw data measured by the luftdaten.info network, see www.stofradar.nl.

See also my DustSensor page.

luftdaten.info is an initiative to allow citizens to measure atmospheric particulate matter concentration using an inexpensive and easy to build sensor. They collect this data, calculate 5 minute and daily averages and publish it again as open data. The total number of sensors is about 5000 worldwide, most of them in Germany, Bulgaria, Belgium, Austria, Sweden. The Netherlands has about 100 sensors.

Future activities:

  • get an OSM background map, once their export functionality works again, https://www.openstreetmap.org/export still broken .... :(
  • fix fine alignment between map and dust overlay
  • put all of the images from one day into an animation
  • also create a map of a larger area, including germany (it has a lot of data points)

Visualisation

The general idea is to create an image, with a map at the background and the atmospheric particulate matter concentration overlaid on top.

Background map

Pages to investigate:

What I need/want is to be able to specify the 'bounding box' and use a equirectangular projection , so I can easily map a pixel back to a latitude/longitude.

Interpolation

Since we only have data at a set of discrete points, the concentration at other points is estimated by combining data from all sensors using inverse distance weighting, in particular using the distance *squared* as the weighing factor in a weighted average. So a nearby sensor has a large effect and a far away sensor has very little effect, contributing only a little bit to the global average.

To calculate the distance2, I use a very simple approximation:

  • determine the difference in longitude and latitude
  • apply an 'aspect ratio' factor of cos(latitude) to the longitude difference. I use the latitude of the approximate middle of the netherlands (52.2 lat, 5.3 lon) to calculate this aspect ratio factor.
  • distance2 = (difference latitude)2 + (difference longitude)2

A better way would be to use the 'great-circle-distance' and possibly even account for the fact that the earth is not perfectly spherical, but I like to start simple and this makes the calculation a lot faster.

PM10 values higher than 500 ug/m3 are simply ignored in the software.

Colour range

The colours I'm using (a kind of inverted spectral range from blue to red):

  • 0 ug/m3: fully transparent white (#FFFFFF)
  • 25 ug/m3: semi-transparent cyan (#00FFFF)
  • 50 ug/m3: semi-transparent yellow (#FFFF00). This is the threshold for the maximum daily average in the Netherlands.
  • 100 ug/m3: semi-transparent red (#FF0000)
  • 200 ug/m3 and higher: semi-transparent purple (#FF00FF)

Values in between these levels are interpolated linearly with respect to the RGB colour value and alpha channel.

This scale is approximately logarithmic, with each step being twice as big as the previous one.

Compositing

I use imagemagick for this, for example:

 composite -compose over -geometry 600x800 20180605_210100.json.png netherlands.png output.png

where netherlands.png is an 600x800 opaque black-and-white image of the map of the netherlands and 20180605_210100.json.png is an 60x80 image of dust concentrations with an alpha channel

Software

See the github page for the source code.