EspWifiTracker

Project EspWifiTracker
Location-tracker based on WiFi station ids
Status	In progress
Contact	bertrik
Last Update	2016-04-11

Status

Status:

• 2016-04-10 Made my first actual scan and processed the data using the Google location service. I'm using pin D5 to decide between "run mode" (scan, log, sleep) and "debug mode" (accept commands from the serial port)
• 2016-03-26 I've been thinking about using the WiFiManager component, this component presents you with a web page to configure stuff, like the name/password of the AP to connect to the internet. It also allows you to configure extra properties. I could put the URL of the upload server in these extra properties (and just HTTP GET the rest of the settings from the server).
• 2016-03-25 I got submission of wifi scan results to google working, Mozilla appears to use a compatible format (i.e. the google JSON format with some extensions).
• 2016-03-17 Can't really get the wake-up using the DS3231 to work, the interrupt pin appears to be active all the time, keeping the ESP in reset. I think I'll just use the ESP built-in wake-up mechanism.
• 2016-03-16 Working on a prototype using a NodeMCU with a DS3231 RTC

Next steps:

• Make scanning WiFi and storing it on the file system just work without worrying about how to upload data yet.
• Make data uploading from the ESP work reliably, right now it can't seem to handle large uploads
• Make some nice Java service to accept the JSON data, store/convert the data to coordinates and host it somewhere
• create a kind of blacklist/whitelist mechanism that scores each scanned AP for suitability to connect to the internet, based on AP name / security type / whether we know the password / etc. A negative score could mean that it should definitely NOT try to connect using a specific AP. The AP with the highest score is tried first

Open challenges:

• first of all make a working prototype, not necessarily optimized yet for size / appearance / battery usage / etc
• when using "WiFi hotspots": figure out how to handle so-called captive portals. Many open access points spoof DNS after connecting to the access point and re-direct you to a webpage where you either need to log in or click some 'I agree' button. It's easy for a human to understand this, possibly not so easy for an embedded device.
• figure out how to run everything from a battery, like can we run it directly off an LiFePO4 battery, or use a lithium-cobalt battery with an LDO, etc.
• make it lower power: so we can use a smaller battery or get longer tracking time
• make it smaller and lighter: so it can be attached to smaller items (e.g. a migrating bat)

Introduction

The idea of this project is to create a simple and low-cost WiFi location tracker.

It tracks location by periodically scanning for WiFi station ids, storing those in internal memory and using one of the various location services (google, mozilla, openwlanmap, etc.) to determine a geographical location from this.

This can be used to track various things, for example:

• track your own bicycle
• know where your cat goes
• monitor movement of wild animals for research purposes, like bats or stone martens.

The basic operating principle is this:

• the tracker is in deep-sleep most of the time, keeping only a low-power real-time-clock running.
• every once in a while (say once a day) it wakes up, takes a reading of all WiFi stations in range and their signal strength. It stores these in internal memory.
• if the tracker wakes up to find an open WiFi station (or one that it knows the WiFi password of), it tries to upload all previous readings to a server.
• the server resolves the uploaded data into geographical locations using a service like Google or the Mozilla Location Service.

Tracker prototype

I'm working on a prototype of this.

Hardware

I'm basing this off an inexpensive NodeMCU clone. The NodeMCU is basically an ESP8266-module on a break-out board with an USB-serial converter and a proper 3.3V power supply. The ESP8266 is a WiFi-capable microcontroller, capable of running at 80 MHz, it has 1 MB (or more) of flash onboard which enough spare flash sectors to hold a file system.

Pin D0 (GPIO16/WAKEn) is connected to RST to support wake-up-by-reset with the ESP internal deep sleep mode.

To keep time, I'm using a ZS-042 clock module, containing a Maxim DS3231 RTC chip (datasheet). This module is attached to the NodeMCU through the I2C-bus on pins D1 (GPIO5/SCL) and D2 (GPIO4/SDA). I removed the power LED on board the ZS-042 module, since one of the reasons to have an RTC is to save current and the power LED wastes current unnecessarily. It would be very nice if the RTC could wake up the ESP8266 using the alarm wake-up feature, but I cannot get this to work: the alarm output always seems to be low, keeping the ESP8266 in reset.

Pin D5 (GPIO14) is used to distinguish between run mode and debug mode during reset/wake-up. When this pin is pulled low (e.g. by a wire to GND), the tracker goes into debug mode. If this pin is left floating, an internal pull-up makes it high and the tracker goes into run mode.

NodeMCU schematic

Software

The NodeMCU can be conveniently programmed from the "Arduino" development environment using the Arduino core for ESP8266 WiFi chip

This already comes with a library for WiFi that allows you to scan for WiFi stations

Other libraries used:

• Makuna RTC
• SPIFFS included into the esp-arduino environment

My prototype software can be found at: https://github.com/bertrik/wifitracker

It runs a simple command interpreter, type 'help' at 115200 bps to get a list of supported commands.

AP selection

I'm thinking of a kind of whitelist/blacklist mechanism to select an AP for uploading data. The plan is to score each scanned AP against this list and then select the one with the highest score, excluding APs which have a negative score. The first match in the list determines the score, so the list is ordered from most specific to least specific. The score is based on network name and encryption type.

This way, we could select and AP on a preference order, for example:

• first, try a secure network that we know the password of;
• next, try an open network that we know is open to the internet (probably very rare...);
• ignore any open network that we know is not really open or uses a captive portal (like "KPN Fon")
• try any open network that is unknown to us.

For example, the entries in the list could look like this:

• {score=100, ssid="my_home_network", pass="my_home_pass"}
• {score=100, ssid="my_work_network", pass="my_work_pass"}
• {score=-100, ssid="KPN Fon", pass=""}
• {score=-100, ssid="WiFi in de trein", pass=""}
• {score=-100, ssid="McDonalds FREE WIFI", pass=""}
• {score=0, ssid="*", pass=""}

where an entry with an empty password represents an open network.

main program flow

The ESP wakes from sleep on reset:

• read the RTC to determine if it's time to do a scan, else go to sleep again
• scan wifi networks
• append the results of the scan to file, the file name is based on the current date.
• score each network for suitability for upload, if none is found, go to sleep again
• connect to the network and probe for actual internet connectivity
• upload the log files one by one, delete the file if successful
• download new settings files
• try to perform an NTP sync and update the RTC if successful
• go to sleep.

data structures

The plan is to use JSON for data to be uploaded. The log file is also formatted as JSON, so during upload we can simply upload the log file without further processing. The advantage is that JSON is readable to humans, disadvantage is that it needs more space than a binary format. The log file contains one line per wifi scan, example of JSON:

{"deviceid":"18FE34XXXXXX","datetime":"2016-03-21 12:34:56","scan":[{"mac":"123456787878","rssi":-49},{"mac":"785634121212","rssi":-42}]}

The device id is the ESP MAC address. Times are in UTC.

The Google Location Service seems to be interested in the channel too (not sure why), we could also log this.

For the settings file, I'm thinking of a simple ini/properties file structure, with KEY=VALUE pairs in it, with empty lines and lines starting with "#" being ignored.

Server backend

The server backend receives uploads from the tracker and resolves the WiFi station ids into locations by using a location service, for example:

• Google maps geolocation API
• Mozilla Location Service
• OpenWLANmap.org / OpenWifi.su
• see also this WiFi positioning page on Wikipedia.

The MAC address of the ESP8266 should be unique and can be used to identify a specific tracker.

Communication between the tracker and the server is always initiated by the tracker. The protocol is REST (HTTP) and the format is JSON. I'm thinking of writing this in Java, with the use of Jackson 2.0 (JSON), Jersey 2.0 (REST) and Jetty libraries.

The REST service could have two endpoints:

• one to upload collected WiFi data from the tracker to the server
• one to download configuration data from the server to the tracker

Configuration data can be made specific for each tracker, and could consist of things like:

• a new periodic wake-up schedule
• list of known WiFi stations + passwords
• WiFi station whitelist, list of stations which are known to be really open to the internet
• WiFi station blacklist, list of stations which are open (unencrypted), but not really open to the internet