Issue 118 (physical-computation#127)

* Added README.md and fixed bug in boot.c

* Made adding number of config errors easier

* Made structure of bitfield easier to parse

* minor style changes

* checkpoint

* part way to unifying flash

* moved to general flash read

* at45db write set to general flash write

* Added device headers to glaux target in Makefile.

* all changes made, without testing. Compiles

* checkpoint

* checkpoint

* checkpoint

* checkpoint

* read and write work

* 10s test works

* Updated instructions in src/boot/ksdk1.1.0/README.md

* works? but the RTC does not seem to work

* Use an enum for bitfield

* Added saving of hour, min and sec from RV8803C7

* reverted setup.conf and jlink command scripts to master

* Added flag to control metadata, and removed from printing

* Changed sleep time to 3600

* Added extra quiet mode and removed gWarpWriteToFlash

* Build works when flash is disabled

* Updated menudelay in writeallsensorstoflash and printallsensors to use kWarpPowerModeVLPS mode

* Use 0 time delay in Warp variant writeAllSensorsToFlash when boot to CSV_STREAM

Makefile

@@ -65,10 +65,17 @@ glaux:
 	cp src/boot/ksdk1.1.0/config.h					build/ksdk1.1/work/boards/Glaux
 	cp src/boot/ksdk1.1.0/glaux.h					build/ksdk1.1/work/boards/Glaux
 	cp src/boot/ksdk1.1.0/CMakeLists-Glaux.txt			build/ksdk1.1/work/demos/Glaux/armgcc/Glaux/CMakeLists.txt
-	cp src/boot/ksdk1.1.0/devBME680.*				build/ksdk1.1/work/demos/Glaux/src/
 	cp src/boot/ksdk1.1.0/devIS25xP.*				build/ksdk1.1/work/demos/Glaux/src/
 	cp src/boot/ksdk1.1.0/devRV8803C7.*				build/ksdk1.1/work/demos/Glaux/src/
 	cp src/boot/ksdk1.1.0/devBME680.*				build/ksdk1.1/work/demos/Glaux/src/
+	cp src/boot/ksdk1.1.0/devADXL362.h				build/ksdk1.1/work/demos/Glaux/src/
+	cp src/boot/ksdk1.1.0/devAMG8834.h				build/ksdk1.1/work/demos/Glaux/src/
+	cp src/boot/ksdk1.1.0/devMMA8451Q.h				build/ksdk1.1/work/demos/Glaux/src/
+	cp src/boot/ksdk1.1.0/devMAG3110.h				build/ksdk1.1/work/demos/Glaux/src/
+	cp src/boot/ksdk1.1.0/devL3GD20H.h				build/ksdk1.1/work/demos/Glaux/src/
+	cp src/boot/ksdk1.1.0/devBMX055.h				build/ksdk1.1/work/demos/Glaux/src/
+	cp src/boot/ksdk1.1.0/devCCS811.h				build/ksdk1.1/work/demos/Glaux/src/
+	cp src/boot/ksdk1.1.0/devHDC1000.h				build/ksdk1.1/work/demos/Glaux/src/
 	cd build/ksdk1.1/work/lib/ksdk_platform_lib/armgcc/KL03Z4 && ./clean.sh; ./build_release.sh
 	cd build/ksdk1.1/work/demos/Glaux/armgcc/Glaux && ./clean.sh; ./build_release.sh
 	@echo "\n\nNow, run\n\n\tmake load-glaux\n\n"
@@ -105,5 +112,8 @@ load-glaux:
 connect-warp:
 	$(JLINKPATH) -device MKL03Z32XXX4 -if SWD -speed 10000 -CommanderScript tools/scripts/connect.jlink.commands
 
+connect-glaux:
+	$(JLINKPATH) -device MKL03Z32XXX4 -if SWD -speed 10000 -CommanderScript tools/scripts/connect.jlink.commands
+
 clean:
 	rm -rf build/ksdk1.1/work

setup.conf

@@ -1,3 +1,2 @@
 JLINKPATH	=	<full path to JLink binary>
 ARMGCC_DIR	= 	/usr/
-

src/boot/ksdk1.1.0/README.md

@@ -1,5 +1,5 @@
 ## Overview
-The Warp firmware is intended to be a demonstration and meassurement environment for testing the Warp hardware. 
+The Warp firmware is intended to be a demonstration and meassurement environment for testing the Warp hardware.
 
 It provides facilities that we use to perform tests on the hardware such as activating the different programmable voltage regulator output voltages (16 different supply voltage levels),
 activating the programmable I2C pull-ups to different values (65536 different settings), changing the I2C and SPI bit rate, changing the Cortex-M0 clock frequency,
@@ -8,7 +8,7 @@ and so on. Having a menu interface allows us to perform various experiments with
 The Warp firmware is a tool for experimentation. You can also use it as a baseline for building real applications by modifying it to remove the menu-driven functionality and linking in only the sensor drivers you need.
 
 The core of the firmware is in `warp-kl03-ksdk1.1-boot.c`. The drivers for the individual sensors are in `devXXX.c` for sensor `XXX`. For example,
-`devADXL362.c` for the ADXL362 3-axis accelerometer. The section below briefly describes all the source files in this directory. 
+`devADXL362.c` for the ADXL362 3-axis accelerometer. The section below briefly describes all the source files in this directory.
 
 
 ## Source File Descriptions
@@ -112,3 +112,110 @@ Implements functionality related to enabling the different low-power modes of th
 
 ##### `warp.h`
 Constant and data structure definitions.
+
+## Flash protocol
+_This is currently valid only for the Warp variant, since this was written only for the AT45DB_
+
+This section describes the protocol that is used to store measurements into flash. Let us first discuss some terminology.
+
+The Warp hardware comes with a selection of _sensors_ which can be activated by setting the appropriate flag in `config.h`. An example of a sensor is the `MAG3110`. Each sensor produces a number of _readings_. Each such reading consists of a signed integer; the maximum size of a reading will depend on the sensor.
+
+A collection of such readings from the activated sensors is a _measurement_. A measurement is an ordered sequence of readings from all the activated sensors. The `writeAllSensorsToFlash` function in `boot.c` generates such a measurement, and writes them to flash.
+
+When printing to `stdout`, the `printSensorData<sensor>` function of the sensor (found in `dev<sensor>.h`) is called. Here `<sensor>` is `MAG3110` for example. This function simply sequentially prints out readings as characters. Since we don't need to do any thing further, this process is fairly straightforward.
+
+In order to write to flash, the `writeAllSensorsToFlash` function creates an array of bytes that corresponds to the measurement. For each activated sensor `<sensor>`, its `appendSensorData<sensor>` function is called to append its readings to a buffer.
+
+The purpose of writing to flash is to eventually read back and print out the measurements in a form that is similar to if we had printed to `stdout` in the first place. If we kept the configuration of active sensors the same, we know exactly how many bytes correspond to a measurement, and the byte configuration of readings from any active sensor, allowing us to sequentially decode measurements easily.
+
+However, for ease of use, we have made it is possible to write to flash with different configurations of the sensors, across multiple calls to `writeAllSensorsToFlash`. This means that, when reading from the flash, we need to know which sensors were active when a particular measurement was written. This is done by first creating a 16b bitfield in front of each measurement which encodes the active sensor configuration.
+
+We can see that this is done in `writeAllSensorsToFlash`, as soon as an active sensor is configured. For example, the following is an excerpt from `boot.c`:
+```C
+#if (WARP_BUILD_ENABLE_DEVMAG3110)
+	numberOfConfigErrors += configureSensorMAG3110(
+		0x00, /*	Payload: DR 000, OS 00, 80Hz, ADC 1280, Full 16bit, standby mode
+							 to set up register*/
+		0xA0, /*	Payload: AUTO_MRST_EN enable, RAW value without offset */
+		0x10);
+
+	sensorBitField = sensorBitField | 0b1000000;
+#endif
+```
+Note `sensorBitField = sensorBitField | 0b100000;`, which activates the bit that corresponds to the `MAG3110` sensor.
+
+When reading, this bitfield is first decoded to know exactly which sensors were active, and therefore how many bytes make up this measurement. Once those bytes have been read and decoded, the next measurement is decoded, read and printed out in the same way.
+
+While this protocol makes writing and reading to flash more useful, it also means that any one adding new sensors need to follow the following guidelines to make sure that the protocol is not violated.
+
+### The structure of the sensor configuration bitfield.
+In the current version,  13 out of the 16 bits are used. They correspond to the following sensors being active.
+
+* `0b0000000000000001`: 	the measurement number,
+* `0b0000000000000010`: 	the `RTC->TSR` time stamp,
+* `0b0000000000000100`: 	the `RTC->TPR` time stamp,
+* `0b0000000000001000`:	the `ADXL362` sensor,
+* `0b0000000000010000`:	the `AMG8834` sensor,
+* `0b0000000000100000`:	the `MMA8451Q` sensor,
+* `0b0000000001000000`:	the `MAG3110` sensor,
+* `0b0000000010000000`:	the `L3GD20H` sensor,
+* `0b0000000100000000`:	the `BME680` sensor,
+* `0b0000001000000000`:	the `BMX055` sensor,
+* `0b0000010000000000`:	the `CCS811` sensor,
+* `0b0000100000000000`:	the `HDC1000` sensor,
+* `0b1000000000000000`:	the number of config errors.
+
+*As an important note, this bitfield represents the order in which the readings from the sensors are used to create a measurement.* The first element represents the least significant bit. For example, if the bitfield is `0b00000000100101111`, then the following readings are expected:
+
+1) 4B of the measurement number,
+2) 4B of the `RTC->TSR` time stamp,
+3) 4B of the `RTC->TPR` time stamp,
+4) 8B (4 readings of 2B) of `ADXL362` sensor readings,
+5) 6B (3 readings of 2B) of `MMA8451Q` sensor readings, and
+6) 12B (3 readings of 4B)of `BME680` sensor readings.
+
+### Writing a new sensor
+When writing a new sensor, the following guidelines should be followed in addition to other common sense steps:
+
+1) Write both a `printSensorData<sensor>` and `appendSensorData<sensor>` function. The former should print out the readings to `stdout`, and the latter should append the readings to a buffer. The function signatures should be as follows; look at the existing sensors for examples of what should be inside them:
+	```C
+	void printSensorData<sensor>(bool hexModeFlag);
+	uint8_t appendSensorData<sensor>(uint8_t* buf);
+	```
+
+	The output of `appendSensorData<sensor>` is the number of bytes that were appended by the function.
+
+2) Inside `dev<sensor>.h`, define the following 3 constant variables, with their appropriate values. As a sanity check, `numberOfReadingsPerMeasurement<sensor> * bytesPerReading<sensor> = bytesPerMeasurement<sensor>`.
+	```C
+	const uint8_t bytesPerMeasurement<sensor> = <appropriate value>;
+	const uint8_t bytesPerReading<sensor> = <appropriate value>;
+	const uint8_t numberOfReadingsPerMeasurement<sensor> = <appropriate value>;
+	```
+
+3) Decide on which free bit of the bitfield can be assigned to this sensor. Then, in `printAllSensors`, add the following line right after its configuration:
+	```C
+	sensorBitField = sensorBitField | 0b<bitfield value>;
+	```
+
+4) Following the pattern of the `writeAllSensorsToFlash` function, use the `appnedSensorData<sensor>` functions to append the readings to the buffer, respectively. *Note that you must do this in the order that is specified by the bitfield.* Otherwise, the data won't be stored in the correct order, and will be impossible to decode when reading from flash.
+
+	When writing the `appendSensorData<sensor>` function, you must use the following template:
+	```C
+	bytesWrittenIndex += appendSensorData<sensor>(flashWriteBuf + bytesWrittenIndex);
+	```
+	This is to ensure that the buffer is appended to correctly.
+
+
+
+5) Add a decoding case to the `writeAllSensorsToFlash` function in `boot.c`, in the correct order. For example, if the new sensor is designated the 13th bit, then the following should be added AFTER the 12th case:
+	```C
+	if (sensorBitField & 0b1000000000000) {
+		numberOfSensorsFound++;
+		if (numberOfSensorsFound - 1 == sensorIndex) {
+		*sizePerReading = bytesPerReading<sensor>;
+		*numberOfReadings = numberOfReadingsPerMeasurement<sensor>;
+		return;
+		}
+	}
+	```
+	Note that `bytesPerReading<sensor>` and `numberOfReadingsPerMeasurement<sensor>` are the constants defined in Step 2.