better config handling

[nliaudat]

I mean actual config.ini is not optimal and called by many functions across every parts of the software.
Access to sdcard is very slow and degrades overall performance.

I see several option to do it better :

• Use an ini parser at startup which loads every values to global extern const variable like that (in answer#3)
• Use the NVS partition instead of config.ini

I've seen many good ini parser in github:
https://github.com/pulzed/mINI
https://github.com/benhoyt/inih
https://github.com/brofield/simpleini

But I prefer the NVS way as it already exists an NVS partition with 16k free space

What do you think of that ?

[caco3]

I agree it would be nicer to have a single place where we read the ini file. How ever it is a lot of work to refactor and test this. If you want to do it, feel free.

I don't think we should move from config.ini to NVS. I often swap devices but keep the SD-Card. That then no longer would be possible! Also there are people who edit the config file manually, that also would not be possible anymore! Backup and restore would also be more difficult.

Overall, you would not gain much performance with this! The config.ini file gets only read at initialization!

[nliaudat]

config.ini file gets only read at initialization!

Not really :

bool ClassFlowAlignment::LoadReferenceAlignmentValues(void)
{
    FILE* pFile;
    char zw[1024];
    string zwvalue;
    std::vector<string> splitted;  


    pFile = fopen(FileStoreRefAlignment.c_str(), "r");
...

bool ClassFlowAlignment::ReadParameter(FILE* pfile, string& aktparamgraph)
{

[jomjol]

config.ini file gets only read at initialization!

Not really :

bool ClassFlowAlignment::LoadReferenceAlignmentValues(void)
{
    FILE* pFile;
    char zw[1024];
    string zwvalue;
    std::vector<string> splitted;  


    pFile = fopen(FileStoreRefAlignment.c_str(), "r");
...

bool ClassFlowAlignment::ReadParameter(FILE* pfile, string& aktparamgraph)
{

@nliaudat : this is not true. Please check what is behind FileStoreRefAlignment. This is not config.ini:

    FileStoreRefAlignment = "/sdcard/config/align.txt";

[nliaudat]

@nliaudat : this is not true. Please check what is behind FileStoreRefAlignment. This is not config.ini:

Ok, but it's hard to understand the real flow.... everything function call doflow() in other part.
The main I wanted to suggest is the that every fopen() should be replace by in memory call.

[jomjol]

This write and read is needed, as it implements the fast autoalign mode, which needs to store some values from the last run, also for a reboot.

[jomjol]

I also agree to caco3, that I don't prefer to store informations on the device itself. Just moving the sd-card takes everything you need with it.

[nliaudat]

The ReadParameter(FILE* pfile, string &aktparamgraph) is called 9x in the flow processing.
Each time is time lost to read a file, parse it to retrieve value. A global varial definition is needed !

ClassFlow.cpp  56,17: bool ClassFlow::ReadParameter(FILE* pfile, string &aktparamgraph)
ClassFlowAlignment.cpp  63,26: bool ClassFlowAlignment::ReadParameter(FILE* pfile, string& aktparamgraph)
ClassFlowCNNGeneral.cpp  293,27: bool ClassFlowCNNGeneral::ReadParameter(FILE* pfile, string& aktparamgraph)
ClassFlowControll.cpp  298,18:             cfc->ReadParameter(pFile, line);
ClassFlowInfluxDB.cpp  70,25: bool ClassFlowInfluxDB::ReadParameter(FILE* pfile, string& aktparamgraph)
ClassFlowMakeImage.cpp  67,26: bool ClassFlowMakeImage::ReadParameter(FILE* pfile, string& aktparamgraph)
ClassFlowMQTT.cpp  88,21: bool ClassFlowMQTT::ReadParameter(FILE* pfile, string& aktparamgraph)
ClassFlowPostProcessing.cpp  494,31: bool ClassFlowPostProcessing::ReadParameter(FILE* pfile, string& aktparamgraph)
ClassFlowWriteList.cpp  38,26: bool ClassFlowWriteList::ReadParameter(FILE* pfile, string& aktparamgraph)

The fopen is called 37x in first run :

components\jomjol_configfile\configFile.cpp
  16,13:     pFile = fopen(config.c_str(), "r");

components\jomjol_controlcamera\ClassControllCamera.cpp
  405,17:     FILE * fp = fopen(nm.c_str(), "wb");
  408,26:     fprintf(stderr, "fopen() failed for '%s'\n", nm.c_str());
  671,16:     FILE *fd = fopen("/sdcard/demo/files.txt", "r");
  713,17:     FILE * fp = fopen(filename, "rb");

components\jomjol_fileserver_ota\server_file.cpp
  222,20:     FILE *fd = fopen("/sdcard/html/file_server.html", "r");
  358,10:     fd = fopen(currentfilename.c_str(), "r");
  444,10:     fd = fopen(currentfilename.c_str(), "r");
  572,10:     fd = fopen(filepath, "r");
  662,10:     fd = fopen(filepath, "w");
  1001,38:    FILE* fpTargetFile = fopen(filename_zw.c_str(), "wb");
  1097,34:    FILE* fpTargetFile = fopen(zw.c_str(), "wb");

components\jomjol_fileserver_ota\server_help.cpp
  42,16:     FILE *fd = fopen(filename.c_str(), "r");

components\jomjol_fileserver_ota\server_ota.cpp
  109,18:     if ((pfile = fopen("/sdcard/update.txt", "r")) == NULL)
  172,15:     FILE* f = fopen(fn.c_str(), "rb");     // previously only "r
  449,21:     pfile = fopen("/sdcard/update.txt", "w");
  578,19:     FILE* pfile = fopen("/sdcard/reboot.txt", "w");

components\jomjol_flowcontroll\ClassFlowAlignment.cpp
  295,13:     pFile = fopen(FileStoreRefAlignment.c_str(), "w");
  337,13:     pFile = fopen(FileStoreRefAlignment.c_str(), "r");

components\jomjol_flowcontroll\ClassFlowControll.cpp
  280,13:     pFile = fopen(config.c_str(), "r");

components\jomjol_flowcontroll\ClassFlowPostProcessing.cpp
  134,13:     pFile = fopen(FilePreValue.c_str(), "r");
  257,13:     pFile = fopen(FilePreValue.c_str(), "w");

components\jomjol_helper\Helper.cpp
  307,23: 	FILE* fpSourceFile = fopen(from.c_str(), "rb");
  322,23: 	FILE* fpSourceFile = fopen(filename.c_str(), "rb");
  336,23: 	FILE* fpSourceFile = fopen(fn.c_str(), "rb");
  361,23: 	FILE* fpSourceFile = fopen(input.c_str(), "rb");
  368,23: 	FILE* fpTargetFile = fopen(output.c_str(), "wb");

components\jomjol_logfile\ClassLogFile.cpp
  51,13:     pFile = fopen(logpath.c_str(), "a+");
  212,31:    logFileAppendHandle = fopen(logpath.c_str(), "a+");
  222,27:    logFileAppendHandle = fopen(logpath.c_str(), "a+");

components\jomjol_tfliteclass\CTfLiteClass.cpp
  257,19:    FILE* f = fopen(_fn.c_str(), "rb");     // previously only "r

components\jomjol_tfliteclass\server_tflite.cpp
  47,18:     if ((pfile = fopen("/sdcard/reboot.txt", "r")) == NULL)

code\components\jomjol_wlan\read_wlanini.cpp
  61,13:      pFile = fopen(fn.c_str(), "r");
  219,13:     pFile = fopen(fn.c_str(), "r");
  261,13:     pFile = fopen(fn.c_str(), "w+");
  291,13:     pFile = fopen(fn.c_str(), "r");
  333,13:     pFile = fopen(fn.c_str(), "w+");

[caco3]

A global varial definition is needed

You are welcome to look into it. I already see that there are themes where you have a much better understanding than me (eg. platformio.ini. I more focus on the thinks I understand better and where I can put my skills in.

There are a lot of corners where improvement is possible.
But we also have other work which is more urgent.

[bucklevision]

Am I using this incorrectly? I run once every 15 minutes. I don't care how fast the process is so long as it's faster than my read interval, even at 5 mins interval I wasn't over-running.

[caco3]

please create a separate discussion for your question.

[bucklevision]

It is not a separate question, it is a rhetorical question, it is a comment on this question. Does anyone need to measure more often than every 10 minutes? And if not, why does the card access speed matter?

[nliaudat]

It is not a separate question, it is a rhetorical question, it is a comment on this question. Does anyone need to measure more often than every 10 minutes? And if not, why does the card access speed matter?

You probably have a car and use it only 4 time a day, but you don't want it's engine start with 5 min delay. That's the same rhetorical question :)

[caco3]

@bucklevision If you would have opened a separate discussion witch a subject matching what you would like to speak about, others would find it and could contribute as well.

As for the interval, I have it on 2 minutes. This gives me a much higher accuracy than 15 minutes, eg. it is easier to see when I took a shower and how much water it took. The ESP is running the whole time anyway, so it doesn't really cost more power.

[bucklevision]

It is not a separate question, it is a rhetorical question, it is a comment on this question. Does anyone need to measure more often than every 10 minutes? And if not, why does the card access speed matter?

You probably have a car and use it only 4 time a day, but you don't want it's engine start with 5 min delay. That's the same rhetorical question :)

That's an awful analogy, they're two totally different use cases. You don't fire up the esp32 every time you want a reading, you plug it in and leave itso the config read only occurs once, and you don't have any time dependence on the reading, so long as it reads at the intervals specified, you turn it on and leave it running. The car is totally different, you turn the thing off between uses and you care about driving it straight away. 😂