My new device running V13.0.7 runs terribly slow

[dsteinkopf]

I've got it running and working, but it is very slow and unstable:

• ping times (from extern to esp32) are about 1000 ms averagy.
• web UI takes >20s up to minutes to response, frequent timeouts
• CPU temperature reported as ab 50°C
• about once a day, it totaly seems so hang and a power cylcle helps most of the times.

What could be the reason. What could I do?

BTW. My other device (still 13.0.1) has been fine for some weeks now: ping about 20-100ms, UI much faster, CPU about 20°C

[caco3]

how is the RSSI?
20˚C seems rater low, I always see >50!
Please provide the logfile!
As for the speed, was that device before faster with another version?

[dsteinkopf]

The RSSI of the watermeter ESP32 is -76 dBm which seems much "too low". The AP (Fritzbox) is located less than 3 meters away in the same room. And the gasmeter (= the device which is working fine) ESP32 is 30 cm away and has RSSI -51 dBm.
Temperatur of gasmeter is changing around 22°C, watermeter around 47°C.

Logfile of watermeter (debug level) from System -> Logviewer:
https://gist.github.com/dsteinkopf/bba2897a8d13c63f32e12ec3ae0c6c80

This is still from running 13.0.7 - I'll try to update today.

Thank you very much!

[dsteinkopf]

I've upgraded to 13.0.8 just now and my first impression is much better: UI is faster, ping times avg 190ms. Temp still is near 50°, RSSI also unchanged.

[dsteinkopf]

Maybe we (you) could wait if this version resolved my problem before investing more time in investigation.

[dsteinkopf]

Update: It seemed to work fine for more than 12 hours. Then (no event or reason I can see) it stopped sending data. Several reboot and power cycles later it is working again. But again, slow and seemingly unreliable: Once the Uptime was a very large number (> 1e6). MQTT messages seem to get lost.

E.g ping time (look at sequence numbers and loss rate)

# ping 192.168.40.96
PING 192.168.40.96 (192.168.40.96) 56(84) bytes of data.
64 bytes from 192.168.40.96: icmp_seq=1 ttl=255 time=2898 ms
64 bytes from 192.168.40.96: icmp_seq=2 ttl=255 time=1890 ms
64 bytes from 192.168.40.96: icmp_seq=3 ttl=255 time=866 ms
64 bytes from 192.168.40.96: icmp_seq=12 ttl=255 time=1526 ms
64 bytes from 192.168.40.96: icmp_seq=20 ttl=255 time=1346 ms
64 bytes from 192.168.40.96: icmp_seq=21 ttl=255 time=318 ms
64 bytes from 192.168.40.96: icmp_seq=28 ttl=255 time=1770 ms
64 bytes from 192.168.40.96: icmp_seq=32 ttl=255 time=898 ms
^C
--- 192.168.40.96 ping statistics ---
36 packets transmitted, 8 received, 77.7778% packet loss, time 35665ms
rtt min/avg/max/mdev = 318.069/1438.907/2898.251/736.230 ms, pipe 3

This (?) makes the UI nearly unusable.

Could it be that (for some reason) the Wifi signal is so bad that the ESP spends all CPU on receiving and (re)sending? (But - as I said: The wifi router is 3m away in the same room and another ESP32 has good signal...)

I'll keep watching this...

[caco3]

can you provide a debug log?

Maybe your esp32 is bad or there is an issue with the SD-Card.

[dsteinkopf]

[0d15h37m30s] 2022-12-22T00:41:57	<WRN>	[MQTT IF] Failed to publish topic 'homeassistant/sensor/watermeter/CPUtemp/config', re-trying...
[0d15h37m30s] 2022-12-22T00:41:57	<ERR>	[MQTT IF] Failed to publish topic 'homeassistant/sensor/watermeter/CPUtemp/config', skipping all MQTT publishings in this round!
[1d04h20m36s] 2022-12-22T13:25:03	<ERR>	[C FIND TEMPL] Failed to load /sdcard/config/ref1.jpg! Is it corrupted?
[1d04h35m49s] 2022-12-22T13:40:16	<ERR>	[CNN] Can't read model file /config/dig-cont_0600_s3.tflite`

-> I'll first try a different SD-Card, then let's see... Thank you for the hint.

[caco3]

[1d04h20m36s] 2022-12-22T13:25:03 [C FIND TEMPL] Failed to load /sdcard/config/ref1.jpg! Is it corrupted?
[1d04h35m49s] 2022-12-22T13:40:16 [CNN] Can't read model file /config/dig-cont_0600_s3.tflite`

This indicates that your SD-Card had an issue.

[dsteinkopf]

Now, the system has been running on a new SD for some days: No change that I am able to realize. Now, my "favourite" theory is a broken or problematic WIFI on by ESP32Cam module. Most of the time, it does NOT connect to the access point (FritzBox) that is 2m nearby but to a different in a different root behind thick wall(s). When I block this device on the other access points, then the ESP connects to the FritzBox but the connection seems the be equally bad or even worse:

708 packets transmitted, 524 received, +42 errors, 25.9887% packet loss, time 713330ms
rtt min/avg/max/mdev = 7.442/4243.214/18626.795/5284.607 ms

Any idea or hint?

[mihainn]

just broke that resistance near wifi antenna you will see that interface will run much better, i solder that pins to use external antenna and work flowless

[caco3]

Most of the time, it does NOT connect to the access point (FritzBox) that is 2m nearby but to a different in a different root behind thick wall(s).

Do you use a Wifi Mesh?

Schau dir mal #1671 an

@jomjol hat das vor einigen Wochen ins rolling eingebaut.

[dsteinkopf]

Yes/no. I am using a Fritzbox (located near the ESPs in the cellar) plus a TP-Link Omada system with some EAP routers (in the house, some meters and walls away).

thank you very much, I’ll try that soon. - but I’m still wondering why my other esp32cam reading the gasmeter is working well(30cm away in the same room).

[caco3]

maybe the USB log can provide more information (accesspoint, ...)

[github-actions[bot]]

You can use the latest Automatic Build of the the rolling branch. It might already contain a fix for your issue.
Pick the most top passing entry (it has a green circle with a tick in it), then scoll down to the Artifacts and download the file named update_*.

[github-actions[bot]]

You can use the Web Console to get USB log from the device.
The USB log contains more information about the startup and operation of the device than the normal Web UI log
When you copy the log into herm, please make sure to use Fenced code blocks by wrapping it into separate lines with ```, see https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks#fenced-code-blocks

[dsteinkopf]

I've just succeeded to upload the current rolling version (ec8de62). I do not see any difference :-(
To get a "USB log" I will have to connect the device to my PC using a serial adapter, correct? (I am asking because this will not be easy - at the place the water meter is located...)

[caco3]

To get a "USB log" I will have to connect the device to my PC using a serial adapter, correct?

Yes

[caco3]

If the ping still is as slow, then it most likely is a connection issue.

On the device, ping is handled separately and by the framework itself, the application has no impact on it.

[f-r-e-d-r-i-k]

I have the same issue - only tested 13.0.8 and one sd-card so far. Testing close to router, a UniFi Dream Router. This is my first esp32 device and my first try.
I don't get past the Initial setup since the pages loads very slowly. High ping as well.

Is your device identical to the other one you have that is working?
I'm using https://www.amazon.se/dp/B093GSCBWJ - which is confirmed to be working by another user.
And sd-card: Verbatim Premium 16GB V10 class 10

I might try with an earlier app version.

[f-r-e-d-r-i-k]

The problem for me seems to be my router. I tested to change the esp32s wifi connection to use my mobile device hotspot - it works great.
Will check router config next...

[f-r-e-d-r-i-k]

After trying many settings I found one that solved the problem for me; changing my router from having WiFi channel "Auto" to "1". The Auto selection used channel 11. There are some articles advicing to use lower channels for ESP devices, for example: https://olimex.wordpress.com/2021/12/10/avoid-wifi-channel-12-13-14-when-working-with-esp-devices/

[dsteinkopf]

I am currently doing some experiments, hoping that this will somehow help:

1. I took the device out from the cellar and attached a serial console.
   • Now everything (except the recognition itself because of a missing water meter) seemed to work well:
   • ping times most of the time around 100ms, sometimes higher sometime less than 10ms. (Note that now, the device is located 2 floors away from the FritzBox - I verified that it is really connected to it.)
2. After about 5-10 minutes my bad problem suddenly reappeared. I don't know what caused this.
   • ((pressing reset did not help)). Edit: That's not correct. Pressing reset helps.
3. disconnecting power and everything (+ reconnecting) helped. Now "everying" seems ok again.
   • I am waiting and clicking now to see if the problem reappears.

To summarize the differences: A. Different power supply (but now probably the worse), B. Different location (far away from the AP), C. serial console connected.

BTW. No helpful output on the serial console. Only some (very few) lines about the digitisation/recognotion process.

[caco3]

you could use the demo mode, see https://jomjol.github.io/AI-on-the-edge-device-docs/Demo-Mode/

[dsteinkopf]

After having the described experience that the same device running the same software is able to run smoothly, I moved everything back to the cellar. Then I discovered another difference: The way the device was mounted covered the ESP CPU box nearly completely. Maybe this resulted in some heat, pressure or whatever.

Now, I tried a slightly different way of mounting... And for more than 12 hours, it is running fine, now! I will keep watching, of course.

This is far from being a good explanation. I don't know at all, what could be the real problem, but currently I am just hoping that it will continue to run stable.

[Pewibe]

Are you able to describe your situation today? How is the CPU temp today? Any suggestions for what helped really?

[dsteinkopf]

Unchanged since then. I still think (and doubt at the same time) that the way of mointing helped. Temperature is still incrased, reliability, too.

Are you having the same problem?

[Phoenix17785]

Hi,

I could flash the firmware without any problems.

But when it comes to the WLAN step it doesn't show the WLAN Network name AI-on-the-Edge?