Stofradar - Revision history

Bertrik Sikken: /* Interpolation */

2022-12-13T13:53:39Z

Interpolation

← Older revision		Revision as of 13:53, 13 December 2022
Line 84:		Line 84:
	** more rules to be determined ...		** more rules to be determined ...
	* for every pixel, consider only the stations within a certain radius, say 10km for the netherlands map		* for every pixel, consider only the stations within a certain radius, say 10km for the netherlands map
	* consider only ~~sensor values from~~ stations meeting some threshold plausibility value (~~to be determined~~)		* for '''interpolated''' pixels, consider only stations meeting some threshold plausibility value (configurable)
	* ~~render~~ pixels directly around a station ~~with the~~ color ~~appropriate for~~ the station sensor value, even if ~~they are~~ not plausible		* for pixels directly around a station, override color corrresponding to the station sensor value, even if it is not plausible

	=== Colour range ===		=== Colour range ===

Bertrik Sikken: /* Interpolation */

2022-12-13T09:13:15Z

Interpolation

← Older revision		Revision as of 09:13, 13 December 2022
Line 63:		Line 63:
	Since we only have data at a set of discrete points, the concentration at other points is estimated by combining data from all sensors using		Since we only have data at a set of discrete points, the concentration at other points is estimated by combining data from all sensors using
	[https://en.wikipedia.org/wiki/Inverse_distance_weighting inverse distance weighting], in particular using the distance squared as the weighing factor in a weighted average.		[https://en.wikipedia.org/wiki/Inverse_distance_weighting inverse distance weighting], in particular using the distance squared as the weighing factor in a weighted average.
	So a nearby sensor has a large effect and a far away sensor has ~~very little~~ effect~~, contributing only a little bit to the global average~~.		So a nearby sensor has a large effect and a far away sensor has a small effect.

	To calculate the distance, I use a very simple approximation:		To calculate the distance, I use a very simple approximation:
Line 78:		Line 78:

	New plan to improve the situation of stations with anomalous data having a large influence:		New plan to improve the situation of stations with anomalous data having a large influence:
			* discard obviously wrong sensor values completely, e.g. check PM2.5 against the following rule: 0 <= PM2.5 <= PM10
			* apply a plausibility score to each sensor, the basis for this is the plausibility score (0-10) already calculated by samenmeten
			* for sensors not considered in the samenmeten plausibility score (e.g. using PMS7003 or SPS30 sensors), calculate our own plausiblity
			** in top 1% percentile: score lower
			** more rules to be determined ...
	* for every pixel, consider only the stations within a certain radius, say 10km for the netherlands map		* for every pixel, consider only the stations within a certain radius, say 10km for the netherlands map
	* ~~if there are 3 or more stations in range:~~		* consider only sensor values from stations meeting some threshold plausibility value (to be determined)
	** interpolate with inverse-distance-weighing using the values ~~in range, but disregard the~~ stations ~~with highest and lowest~~ value		* render pixels directly around a station with the color appropriate for the station sensor value, even if they are not plausible
	** else (~~less than 3 stations~~) ~~don't interpolate, this means transparent color~~
	* ~~if within some minimum distance from~~ a station~~, render pixels~~ with ~~flat~~ color ~~according to its measurement~~ value (even ~~when it is an outlier)~~

	=== Colour range ===		=== Colour range ===

Bertrik Sikken: /* Stale data */

2022-11-29T12:36:07Z

Stale data

← Older revision		Revision as of 12:36, 29 November 2022
Line 29:		Line 29:

	=== Stale data ===		=== Stale data ===
	~~Perhaps~~ the ~~next things to tackle~~ is ~~how to handle stale data~~.		Sensor.community nodes typically provides data at high time resolution, a few minutes, with a 'heartbeat' update of 5 minutes.
			Some data sources provide relatively stale data, data is only updated once per hour and once the application picks it up the data is already over an hour old.
			Data sources with such different latencies cannot be combined properly in a single near-realtime image.

	~~Some~~ data ~~sources are not as frequent~~ (~~5 min) as~~ the sensor~~.community data~~.		This is handled by providing with each image a maximum age of the data.
	For ~~example~~, ~~some data from samenmeten undergoes several stages~~ of ~~aggregation~~, ~~is only updated hourly and is often over three hours old already.~~		For the Rotterdam visualisation this is a couple of hours (most of the sensor are from the luchtclub project, which has high latency).
			For the Netherlands visualisation, this is a maximum of one hour, meaning that data from Rotterdam is not visible on the national map.
	~~Perhaps invent some kind of weighing factor so~~ that ~~stale/old~~ data is ~~still used, but does~~ not ~~deteriorate~~ the ~~overall quality when combined with other data:~~
	* there is already a mechanism in place to weigh local PM concentrations by distance to measurement points
	* perhaps also add a mechanism to account for weighing by age of the measurement?
	* for example simply use age as a weighing factor: a 5-minute old measurement from sensor.~~community is regarded as 24 times more important than a 2-hour old samenmeten measurement~~

	== Visualisation ==		== Visualisation ==

Bertrik Sikken: /* Interpolation */

2022-11-25T23:34:31Z

Interpolation

← Older revision		Revision as of 23:34, 25 November 2022
Line 78:		Line 78:

	Only sensors within a reasonable range of the map are taken into account, currently this is an area of 4 times (2x2) the visible area.		Only sensors within a reasonable range of the map are taken into account, currently this is an area of 4 times (2x2) the visible area.

			New plan to improve the situation of stations with anomalous data having a large influence:
			* for every pixel, consider only the stations within a certain radius, say 10km for the netherlands map
			* if there are 3 or more stations in range:
			** interpolate with inverse-distance-weighing using the values in range, but disregard the stations with highest and lowest value
			** else (less than 3 stations) don't interpolate, this means transparent color
			* if within some minimum distance from a station, render pixels with flat color according to its measurement value (even when it is an outlier)

	=== Colour range ===		=== Colour range ===

Bertrik Sikken: /* Data filtering */

2022-06-10T23:01:35Z

Data filtering

← Older revision		Revision as of 23:01, 10 June 2022
Line 52:		Line 52:
	* sensor.commnunity (5 minute data)		* sensor.commnunity (5 minute data)
	* RIVM samenmeten data portal (60 minute data)		* RIVM samenmeten data portal (60 minute data)
			* meetjestad

	There is only very minimal data filtering. Sensor measurements are taken into account as follows:		There is only very minimal data filtering. Sensor measurements are taken into account as follows:

Bertrik Sikken: /* Introduction */

2022-05-19T10:58:59Z

Introduction

← Older revision		Revision as of 10:58, 19 May 2022
Line 11:		Line 11:
	see [http://www.stofradar.nl www.stofradar.nl].		see [http://www.stofradar.nl www.stofradar.nl].

	Visualisation of citizen-science data by RIVM can be found [https://samenmeten.rivm.nl/animatie/index-both.php here].		Visualisation of citizen-science data by RIVM in the samenmeten project can be found [https://samenmeten.rivm.nl/animatie/index-both.php here].

	The focus is on raw visualisation of the source data, only the most minimal attempt is made to "validate" the data.		The focus is on raw visualisation of the source data, only the most minimal attempt is made to "validate" the data.

Bertrik Sikken at 10:58, 19 May 2022

2022-05-19T10:58:44Z

← Older revision		Revision as of 10:58, 19 May 2022
Line 10:		Line 10:
	This page is about creating a 'stofradar' image of atmospheric particulate matter concentrations based on the raw data measured by the sensor.community network,		This page is about creating a 'stofradar' image of atmospheric particulate matter concentrations based on the raw data measured by the sensor.community network,
	see [http://www.stofradar.nl www.stofradar.nl].		see [http://www.stofradar.nl www.stofradar.nl].

			Visualisation of citizen-science data by RIVM can be found [https://samenmeten.rivm.nl/animatie/index-both.php here].

	The focus is on raw visualisation of the source data, only the most minimal attempt is made to "validate" the data.		The focus is on raw visualisation of the source data, only the most minimal attempt is made to "validate" the data.

Bertrik Sikken: /* Introduction */

2022-05-15T11:04:28Z

Introduction

@@ Line 24: / Line 24: @@
 * add a water mark
 * add some kind of slider to indicate progress on the GIF image, or even allow the user to slide back and forth
 == Visualisation ==

Bertrik Sikken: /* Animation */

2022-05-06T16:10:01Z

Animation

← Older revision		Revision as of 16:10, 6 May 2022
Line 95:		Line 95:
	Besides an image with current data from the last 5 minutes, every hour two animations are created:		Besides an image with current data from the last 5 minutes, every hour two animations are created:
	* GIF animation composed of hourly images over the past 24 hours		* GIF animation composed of hourly images over the past 24 hours
	* ~~WEBM~~ animation composed of 5-minute images over the past 24 hours		* MP4 animation composed of 5-minute images over the past 24 hours

	You can click on the GIF animation to view the ~~WEBM~~ animation.		You can click on the GIF animation to view the MP4 animation.

	== Software ==		== Software ==
	See the [https://github.com/bertrik/stofradar github page] for the source code.		See the [https://github.com/bertrik/stofradar github page] for the source code.

Bertrik Sikken: /* Introduction */

2022-05-06T16:01:27Z

Introduction

← Older revision		Revision as of 16:01, 6 May 2022
Line 22:		Line 22:

	Future activities:		Future activities:
	* plot official [https://www.luchtmeetnet.nl/ luchtmeetnet] measurements on the map too, access to the API has been implemented [https://github.com/bertrik/stofradar/tree/master/luftdatenmapper/src/main/java/nl/bertriksikken/luchtmeetnet/api work in progress]
	* play with different algorithm:
	** for each pixel consider only stations with a certain radius (say 10 km), calculate the median, convert median to color
	** should more naturally filter out outliers and is actually a bit similar to what sensor.community uses.
	* add a water mark		* add a water mark
	* add some kind of slider to indicate progress on the GIF image, or even allow the user to slide back and forth		* add some kind of slider to indicate progress on the GIF image, or even allow the user to slide back and forth