Brainstorm better data storage/retrieval methods

[yoos]

Currently, Logger serially dumps all generated data to the SD card. One issue is that the system time is not synchronized to the loops, which leads to entries in the same control loop being logged with different timestamps. Parsing is also fairly easy, but it needs to be a unified solution that sanity checks data and outputs the pertinent few minutes of flight to plots and tables.

[kylc]

I think the remote communication and logging utilities need to be rethought. Here's an option I've been thinking about:

Instead of directly publishing to the rate limited message streams in the controller, register all logged variables in a central repository and push them to the logger and remote comms in batches at a fixed rate.

Here's some pseudocode of what I'm thinking:

class MyController {
    LoggedVariable<double> yawEstimate;
    LoggedVariable<double> pitchEstimate;
    LoggedVariable<double> rollEstimate;

    ...

    double someCalculation = yawEstimate.get() + pitchEstimate.get() + rollEstimate.get();

    double someValue = ...;
    yawEstimate.set(someValue);
};

template ...
class LoggedVariable {
    LoggedVariable(string name/path?) {
        LoggerRepository.register(name);
    }

    void set(...) {
        LoggerRepository.update(name, value);
    }
};

class LoggerRepository {
    // Call off main thread at some fixed rate
    void publishLogger() {
        for var in loggedVars {
            log var
        }
    }

    // Call off main thread at some fixed rate, probably much slower than
    // onboard logger
    void publishRemoteComms() {
        for var in loggedVars {
            transmit var
        }
    }
};

[yoos]

If I understand correctly, this pseudocode would push all registered variables to the logger at a fixed rate, and if the logging rate is greater than the rate at which some variables are updated, some variables would be logged twice with the same (stale) value - I'd like to avoid that.

Let's start a list of requirements for the logger/comm system, assuming they will be more or less the same system:

• Filesystem logging and communications should employ the same data protocol
• The logger should be able to log all data generated in firmware without downsampling
• The logger should not add jitter to the 1kHz control loop
• The logger should not require a giant 83kB buffer to get around the filesystem I/O latency
• Logging rate should be independent of data generation rate

My general wish is to be able to write anything (within the protocol) to the logger's buffer and have it magically stored on the SD card without interfering with the core control loop. Given the SD card's unpredictable latency and the amount of free clock cycles we have, it would be nice to use up all idle time writing to file with nonblocking file I/O.

[kylc]

We can avoid sending stale values by adding a stale flag to each logged variable which is set on each call to set(). But let's not worry about this until we realize that we're actually writing too much data, because it's nice to be able to look at any slice in time and see the robot's state without having to backtrack to the most recent update.

Everything else I agree with. For SD card latency, there are three options:

1. Track down and kill the latency. Not possible if this is actual hardware/write latency.
2. Have a big buffer (not ideal).
3. Reduce logging rate. This could be done statically or dynamically depending on buffer load.

Could we also experiment with buffering to flash? That does add an additional layer of complexity, though.

[yoos]

For some reason, the filesystem Logger refactor (cf0648e) and RegisteredVariable together cause the firmware to hang.

Either change by itself does not cause the firmware to hang. The declaration of the test variable loopCounter in the control thread is enough to trigger the hang.