Disk usage issue

[peteruithoven]

We have a disk space available check in the save sketch api function, but it has happened multiple times that we found boxes with 0 kb available. If this happens boxes boxes become unresponsive, sometimes you can't even ssh into them. (You can still go into failsave mode).
Our current check:
https://github.com/Doodle3D/doodle3d-firmware/blob/master/src/rest/api/api_sketch.lua#L121-L127

[peteruithoven]

Not sure what is happening on these boxes. getFreeDiskSpace() should return 0, (since df / -k | awk '$3 ~ /[0-9]+/ { print $4 }' returns 0 on a box with this issue), which should have triggered the check and prevented saving more sketches. It should even keep 81920 bytes free.
There might be something else that filled that free space, but what could that be?

[peteruithoven]

I'm trying to reproduce this on a WiFi-Box. I've saved multiple drawings, and it seems to properly prevent saving skeches that would bring the free space below 80k, even when I make very simple sketches.

[peteruithoven]

After saving so many sketches that only 80k was left it still rebooted properly. I can't figure out what else we're saving on the hard disk that could fill that space. Settings can hardly have an impact.

[peteruithoven]

We've listed the files and their disk usage using du:
WiFi-Box with 0kb remaining:
486B84-diskusage-du.txt
WiFi-Box with 80kb remaining:
4CCE92-diskusage-du.txt

[peteruithoven]

The WiFi-Box with 0kb remaining has 1.5MB of sketches.
WiFi-Box with 80kb remaining has 1.6M of sketches.

[peteruithoven]

I've edited the diskusage-du files, putting the filename first, sorted both alphabetically and created a diff. This should contain something interesting. This is a diff from 4CCE92 to 486B84. I've removed the /proc/ and /sys/files as they are virtual files, not taking up disk space.

diff --git a/4CCE92-diskusage-du-3.txt b/486B84-diskusage-du-3.txt
index 368dd47..2177e08 100644
--- a/4CCE92-diskusage-du-3.txt
+++ b/486B84-diskusage-du-3.txt
@@ -1,5 +1,6 @@
-/  15323
+/  18739
 /bin  517
+/bin/ash  0
 /bin/busybox  282
 /bin/cat  0
 /bin/chgrp  0
@@ -55,15 +56,10 @@
 /bin/vi  0
 /bin/zcat  0
 /dev  0
-/dev/bus  0
-/dev/bus/usb  0
-/dev/bus/usb/001  0
-/dev/bus/usb/001/001  0
 /dev/console  0
 /dev/cpu_dma_latency  0
 /dev/full  0
 /dev/kmsg  0
-/dev/log  0
 /dev/mem  0
 /dev/mtd0  0
 /dev/mtd0ro  0
@@ -98,28 +94,24 @@
 /dev/urandom  0
 /dev/watchdog  0
 /dev/zero  0
-/etc  102
+/etc  132
 /etc/TZ  0
 /etc/banner  1
 /etc/banner.default  1
-/etc/config  13
+/etc/config  10
 /etc/config/dhcp  1
 /etc/config/dropbear  1
 /etc/config/firewall  4
-/etc/config/network  1
 /etc/config/system  1
-/etc/config/ubootenv  0
-/etc/config/uhttpd  1
-/etc/config/wifibox  4
-/etc/config/wireless  1
+/etc/config/uhttpd  4
+/etc/config/wifibox  1
 /etc/crontabs  0
 /etc/device_info  1
 /etc/diag.sh  5
 /etc/dnsmasq.conf  2
-/etc/dropbear  2
-/etc/dropbear/dropbear_dss_host_key  1
-/etc/dropbear/dropbear_rsa_host_key  1
-/etc/ethers  0
+/etc/dropbear  0
+/etc/dropbear/dropbear_dss_host_key  0
+/etc/dropbear/dropbear_rsa_host_key  0
 /etc/firewall.user  1
 /etc/fstab  0
 /etc/group  1
@@ -184,7 +176,6 @@
 /etc/openwrt_version  1
 /etc/opkg.conf  1
 /etc/passwd  1
-/etc/passwd-  1
 /etc/preinit  1
 /etc/profile  1
 /etc/protocols  3
@@ -224,11 +215,23 @@
 /etc/resolv.conf  0
 /etc/services  3
 /etc/shadow  1
-/etc/shadow-  1
 /etc/shells  1
 /etc/sysctl.conf  1
 /etc/sysupgrade.conf  1
-/etc/uci-defaults  0
+/etc/uci-defaults  36
+/etc/uci-defaults/01_leds  17
+/etc/uci-defaults/02_network  11
+/etc/uci-defaults/03_network-switchX-migration  2
+/etc/uci-defaults/03_network-vlan-migration  1
+/etc/uci-defaults/04_led_migration  2
+/etc/uci-defaults/09_fix-seama-header  1
+/etc/uci-defaults/09_fix-trx-header  1
+/etc/uci-defaults/10_migrate-shadow  1
+/etc/uci-defaults/11_migrate-sysctl  1
+/etc/uci-defaults/12_network-generate-ula  1
+/etc/uci-defaults/30_uboot-envtools  1
+/etc/uci-defaults/odhcpd.defaults  1
+/etc/uci-defaults/setup-wifibox.sh  1
 /etc/wifibox-version  1
 /lib  2383
 /lib/ar71xx.sh  12
@@ -362,89 +365,110 @@
 /lib/wifi  3
 /lib/wifi/mac80211.sh  3
 /mnt  0
-/overlay  1693
-/overlay/etc  23
-/overlay/etc/config  13
-/overlay/etc/config/dhcp  1
-/overlay/etc/config/dropbear  1
-/overlay/etc/config/firewall  4
-/overlay/etc/config/network  1
-/overlay/etc/config/system  1
-/overlay/etc/config/ubootenv  0
-/overlay/etc/config/uhttpd  1
-/overlay/etc/config/wifibox  4
-/overlay/etc/config/wireless  1
-/overlay/etc/dropbear  2
-/overlay/etc/dropbear/dropbear_dss_host_key  1
-/overlay/etc/dropbear/dropbear_rsa_host_key  1
-/overlay/etc/ethers  0
-/overlay/etc/passwd  1
-/overlay/etc/passwd-  1
-/overlay/etc/shadow  1
-/overlay/etc/shadow-  1
-/overlay/etc/uci-defaults  7
-/overlay/etc/uci-defaults/01_leds  1
-/overlay/etc/uci-defaults/02_network  1
-/overlay/etc/uci-defaults/03_network-switchX-migration  1
-/overlay/etc/uci-defaults/03_network-vlan-migration  1
-/overlay/etc/uci-defaults/04_led_migration  1
-/overlay/etc/uci-defaults/09_fix-seama-header  1
-/overlay/etc/uci-defaults/09_fix-trx-header  1
-/overlay/etc/uci-defaults/10_migrate-shadow  1
-/overlay/etc/uci-defaults/11_migrate-sysctl  1
-/overlay/etc/uci-defaults/12_network-generate-ula  1
-/overlay/etc/uci-defaults/30_uboot-envtools  1
-/overlay/etc/uci-defaults/odhcpd.defaults  1
-/overlay/etc/uci-defaults/setup-wifibox.sh  1
-/overlay/root  1664
-/overlay/root/sketches  1664
-/overlay/root/sketches/00001.svg  18
-/overlay/root/sketches/00002.svg  13
-/overlay/root/sketches/00003.svg  174
-/overlay/root/sketches/00004.svg  40
-/overlay/root/sketches/00005.svg  160
-/overlay/root/sketches/00006.svg  154
-/overlay/root/sketches/00007.svg  186
-/overlay/root/sketches/00008.svg  221
-/overlay/root/sketches/00009.svg  62
-/overlay/root/sketches/00010.svg  112
-/overlay/root/sketches/00011.svg  120
-/overlay/root/sketches/00012.svg  121
-/overlay/root/sketches/00013.svg  20
-/overlay/root/sketches/00014.svg  39
-/overlay/root/sketches/00015.svg  62
-/overlay/root/sketches/00016.svg  30
-/overlay/root/sketches/00017.svg  25
-/overlay/root/sketches/00018.svg  25
-/overlay/root/sketches/00019.svg  25
-/overlay/root/sketches/00020.svg  25
-/overlay/root/sketches/00021.svg  1
-/overlay/root/sketches/00022.svg  1
-/overlay/root/sketches/00023.svg  1
-/overlay/root/sketches/00024.svg  1
-/overlay/root/sketches/00025.svg  1
-/overlay/root/sketches/00026.svg  1
-/overlay/root/sketches/00027.svg  2
-/overlay/root/sketches/00028.svg  2
-/overlay/root/sketches/00029.svg  2
-/overlay/root/sketches/00030.svg  2
-/overlay/root/sketches/00031.svg  2
-/overlay/root/sketches/00032.svg  2
-/overlay/root/sketches/00033.svg  2
-/overlay/root/sketches/00034.svg  2
-/overlay/root/sketches/00035.svg  3
-/overlay/root/sketches/00036.svg  3
+/overlay  1501
+/overlay/root  1501
+/overlay/root/sketches  1501
+/overlay/root/sketches/00009.svg  20
+/overlay/root/sketches/00010.svg  1
+/overlay/root/sketches/00011.svg  1
+/overlay/root/sketches/00012.svg  1
+/overlay/root/sketches/00013.svg  1
+/overlay/root/sketches/00014.svg  1
+/overlay/root/sketches/00015.svg  1
+/overlay/root/sketches/00016.svg  15
+/overlay/root/sketches/00017.svg  26
+/overlay/root/sketches/00018.svg  54
+/overlay/root/sketches/00019.svg  54
+/overlay/root/sketches/00020.svg  9
+/overlay/root/sketches/00021.svg  7
+/overlay/root/sketches/00022.svg  7
+/overlay/root/sketches/00023.svg  41
+/overlay/root/sketches/00024.svg  56
+/overlay/root/sketches/00025.svg  56
+/overlay/root/sketches/00026.svg  4
+/overlay/root/sketches/00027.svg  7
+/overlay/root/sketches/00028.svg  10
+/overlay/root/sketches/00029.svg  21
+/overlay/root/sketches/00030.svg  23
+/overlay/root/sketches/00031.svg  20
+/overlay/root/sketches/00032.svg  51
+/overlay/root/sketches/00033.svg  81
+/overlay/root/sketches/00034.svg  81
+/overlay/root/sketches/00035.svg  5
+/overlay/root/sketches/00036.svg  6
 /overlay/root/sketches/00037.svg  3
-/overlay/root/sketches/00038.svg  3
-/overlay/root/sketches/00039.svg  3
-/overlay/root/sketches/00040.svg  3
-/overlay/usr  7
-/overlay/usr/share  7
-/overlay/usr/share/lua  7
-/overlay/usr/share/lua/wifibox  7
-/overlay/usr/share/lua/wifibox/rest  7
-/overlay/usr/share/lua/wifibox/rest/api  7
-/overlay/usr/share/lua/wifibox/rest/api/api_sketch.lua  7
+/overlay/root/sketches/00038.svg  14
+/overlay/root/sketches/00039.svg  47
+/overlay/root/sketches/00040.svg  50
+/overlay/root/sketches/00041.svg  50
+/overlay/root/sketches/00042.svg  50
+/overlay/root/sketches/00043.svg  49
+/overlay/root/sketches/00044.svg  72
+/overlay/root/sketches/00045.svg  5
+/overlay/root/sketches/00046.svg  88
+/overlay/root/sketches/00047.svg  88
+/overlay/root/sketches/00048.svg  9
+/overlay/root/sketches/00049.svg  36
+/overlay/root/sketches/00050.svg  3
+/overlay/root/sketches/00051.svg  1
+/overlay/root/sketches/00052.svg  1
+/overlay/root/sketches/00053.svg  1
+/overlay/root/sketches/00054.svg  1
+/overlay/root/sketches/00055.svg  3
+/overlay/root/sketches/00056.svg  2
+/overlay/root/sketches/00057.svg  26
+/overlay/root/sketches/00058.svg  6
+/overlay/root/sketches/00059.svg  6
+/overlay/root/sketches/00060.svg  7
+/overlay/root/sketches/00061.svg  59
+/overlay/root/sketches/00062.svg  38
+/overlay/root/sketches/00063.svg  55
+/overlay/root/sketches/00064.svg  1
+/overlay/root/sketches/00065.svg  2
+/overlay/root/sketches/00066.svg  2
+/overlay/root/sketches/00067.svg  2
+/overlay/root/sketches/00068.svg  2
+/overlay/root/sketches/00069.svg  2
+/overlay/root/sketches/00070.svg  2
+/overlay/root/sketches/00071.svg  2
+/overlay/root/sketches/00072.svg  2
+/overlay/root/sketches/00073.svg  2
+/overlay/root/sketches/00074.svg  1
+/overlay/root/sketches/00075.svg  2
+/overlay/root/sketches/00076.svg  1
+/overlay/root/sketches/00077.svg  2
+/overlay/root/sketches/00078.svg  2
+/overlay/root/sketches/00079.svg  4
+/overlay/root/sketches/00080.svg  4
+/overlay/root/sketches/00081.svg  1
+/overlay/root/sketches/00082.svg  1
+/overlay/root/sketches/00083.svg  1
+/overlay/root/sketches/00084.svg  1
+/overlay/root/sketches/00085.svg  1
+/overlay/root/sketches/00086.svg  2
+/overlay/root/sketches/00087.svg  2
+/overlay/root/sketches/00088.svg  2
+/overlay/root/sketches/00089.svg  2
+/overlay/root/sketches/00090.svg  1
+/overlay/root/sketches/00091.svg  14
+/overlay/root/sketches/00092.svg  1
+/overlay/root/sketches/00093.svg  1
+/overlay/root/sketches/00094.svg  1
+/overlay/root/sketches/00095.svg  2
+/overlay/root/sketches/00096.svg  1
+/overlay/root/sketches/00097.svg  1
+/overlay/root/sketches/00098.svg  2
+/overlay/root/sketches/00099.svg  2
+/overlay/root/sketches/00100.svg  2
+/overlay/root/sketches/00101.svg  2
+/overlay/root/sketches/00102.svg  2
+/overlay/root/sketches/00103.svg  1
+/overlay/root/sketches/00104.svg  2
+/overlay/root/sketches/00105.svg  3
+/overlay/root/sketches/00106.svg  1
+/overlay/root/sketches/00107.svg  5
+/overlay/root/sketches/00108.svg  1
+/overlay/root/sketches/00109.svg  3
 /proc  0
@@ -5127,49 +3722,110 @@
 /rom/www/redirect.html  2
 /rom/www/settings.html  13
 /rom/www/sketches  0
-/root  1665
+/root  1502
 /root/.profile  1
-/root/sketches  1664
-/root/sketches/00001.svg  18
-/root/sketches/00002.svg  13
-/root/sketches/00003.svg  174
-/root/sketches/00004.svg  40
-/root/sketches/00005.svg  160
-/root/sketches/00006.svg  154
-/root/sketches/00007.svg  186
-/root/sketches/00008.svg  221
-/root/sketches/00009.svg  62
-/root/sketches/00010.svg  112
-/root/sketches/00011.svg  120
-/root/sketches/00012.svg  121
-/root/sketches/00013.svg  20
-/root/sketches/00014.svg  39
-/root/sketches/00015.svg  62
-/root/sketches/00016.svg  30
-/root/sketches/00017.svg  25
-/root/sketches/00018.svg  25
-/root/sketches/00019.svg  25
-/root/sketches/00020.svg  25
-/root/sketches/00021.svg  1
-/root/sketches/00022.svg  1
-/root/sketches/00023.svg  1
-/root/sketches/00024.svg  1
-/root/sketches/00025.svg  1
-/root/sketches/00026.svg  1
-/root/sketches/00027.svg  2
-/root/sketches/00028.svg  2
-/root/sketches/00029.svg  2
-/root/sketches/00030.svg  2
-/root/sketches/00031.svg  2
-/root/sketches/00032.svg  2
-/root/sketches/00033.svg  2
-/root/sketches/00034.svg  2
-/root/sketches/00035.svg  3
-/root/sketches/00036.svg  3
+/root/sketches  1501
+/root/sketches/00009.svg  20
+/root/sketches/00010.svg  1
+/root/sketches/00011.svg  1
+/root/sketches/00012.svg  1
+/root/sketches/00013.svg  1
+/root/sketches/00014.svg  1
+/root/sketches/00015.svg  1
+/root/sketches/00016.svg  15
+/root/sketches/00017.svg  26
+/root/sketches/00018.svg  54
+/root/sketches/00019.svg  54
+/root/sketches/00020.svg  9
+/root/sketches/00021.svg  7
+/root/sketches/00022.svg  7
+/root/sketches/00023.svg  41
+/root/sketches/00024.svg  56
+/root/sketches/00025.svg  56
+/root/sketches/00026.svg  4
+/root/sketches/00027.svg  7
+/root/sketches/00028.svg  10
+/root/sketches/00029.svg  21
+/root/sketches/00030.svg  23
+/root/sketches/00031.svg  20
+/root/sketches/00032.svg  51
+/root/sketches/00033.svg  81
+/root/sketches/00034.svg  81
+/root/sketches/00035.svg  5
+/root/sketches/00036.svg  6
 /root/sketches/00037.svg  3
-/root/sketches/00038.svg  3
-/root/sketches/00039.svg  3
-/root/sketches/00040.svg  3
+/root/sketches/00038.svg  14
+/root/sketches/00039.svg  47
+/root/sketches/00040.svg  50
+/root/sketches/00041.svg  50
+/root/sketches/00042.svg  50
+/root/sketches/00043.svg  49
+/root/sketches/00044.svg  72
+/root/sketches/00045.svg  5
+/root/sketches/00046.svg  88
+/root/sketches/00047.svg  88
+/root/sketches/00048.svg  9
+/root/sketches/00049.svg  36
+/root/sketches/00050.svg  3
+/root/sketches/00051.svg  1
+/root/sketches/00052.svg  1
+/root/sketches/00053.svg  1
+/root/sketches/00054.svg  1
+/root/sketches/00055.svg  3
+/root/sketches/00056.svg  2
+/root/sketches/00057.svg  26
+/root/sketches/00058.svg  6
+/root/sketches/00059.svg  6
+/root/sketches/00060.svg  7
+/root/sketches/00061.svg  59
+/root/sketches/00062.svg  38
+/root/sketches/00063.svg  55
+/root/sketches/00064.svg  1
+/root/sketches/00065.svg  2
+/root/sketches/00066.svg  2
+/root/sketches/00067.svg  2
+/root/sketches/00068.svg  2
+/root/sketches/00069.svg  2
+/root/sketches/00070.svg  2
+/root/sketches/00071.svg  2
+/root/sketches/00072.svg  2
+/root/sketches/00073.svg  2
+/root/sketches/00074.svg  1
+/root/sketches/00075.svg  2
+/root/sketches/00076.svg  1
+/root/sketches/00077.svg  2
+/root/sketches/00078.svg  2
+/root/sketches/00079.svg  4
+/root/sketches/00080.svg  4
+/root/sketches/00081.svg  1
+/root/sketches/00082.svg  1
+/root/sketches/00083.svg  1
+/root/sketches/00084.svg  1
+/root/sketches/00085.svg  1
+/root/sketches/00086.svg  2
+/root/sketches/00087.svg  2
+/root/sketches/00088.svg  2
+/root/sketches/00089.svg  2
+/root/sketches/00090.svg  1
+/root/sketches/00091.svg  14
+/root/sketches/00092.svg  1
+/root/sketches/00093.svg  1
+/root/sketches/00094.svg  1
+/root/sketches/00095.svg  2
+/root/sketches/00096.svg  1
+/root/sketches/00097.svg  1
+/root/sketches/00098.svg  2
+/root/sketches/00099.svg  2
+/root/sketches/00100.svg  2
+/root/sketches/00101.svg  2
+/root/sketches/00102.svg  2
+/root/sketches/00103.svg  1
+/root/sketches/00104.svg  2
+/root/sketches/00105.svg  3
+/root/sketches/00106.svg  1
+/root/sketches/00107.svg  5
+/root/sketches/00108.svg  1
+/root/sketches/00109.svg  3
 /sbin  356
 /sbin/askfirst  3
 /sbin/devmem  0
@@ -8473,211 +6393,22 @@
-/tmp  180
-/tmp/.uci  0
-/tmp/.uci/wifibox  0
-/tmp/TZ  4
-/tmp/d3d-signinstatus.txt  4
-/tmp/dhcp.leases  4
-/tmp/dnsmasq.d  0
-/tmp/etc  4
-/tmp/etc/dnsmasq.conf  4
-/tmp/hosts  4
-/tmp/hosts/dhcp  4
+/tmp  3920
+/tmp/.failsafe  4
+/tmp/backup.bin  3904
+/tmp/failsafe  0
+/tmp/failsafe_button  4
 /tmp/lock  0
-/tmp/log  0
-/tmp/log/lastlog  0
-/tmp/log/wtmp  0
-/tmp/networkstatus.txt  4
 /tmp/overlay  0
-/tmp/resolv.conf  4
-/tmp/resolv.conf.auto  0
-/tmp/run  36
-/tmp/run/config.md5  4
-/tmp/run/dnsmasq.br-lan.dhcp  4
-/tmp/run/dnsmasq.pid  4
-/tmp/run/dnsmasq.wlan0.dhcp  4
-/tmp/run/dropbear.1.pid  4
-/tmp/run/fw3.state  4
-/tmp/run/hostapd  0
-/tmp/run/hostapd-phy0.conf  4
-/tmp/run/hostapd/wlan0  0
-/tmp/run/ntpd.pid  4
-/tmp/run/ubus.sock  0
-/tmp/run/wifi-phy0.pid  4
-/tmp/state  4
-/tmp/state/network  4
+/tmp/run  0
+/tmp/state  0
 /tmp/sysinfo  8
 /tmp/sysinfo/board_name  4
 /tmp/sysinfo/model  4
-/tmp/wifibox.log  104
 /usr  1714
 /usr/bin  37
 /usr/bin/[  0
@@ -9084,4 +6815,4 @@
 /www/redirect.html  2
 /www/settings.html  13
 /www/sketches  0
-total  15323
+total  18739

There are mostly files missing, maybe because 486B84 is in failsave mode. There are some extra files in /etc/uci-defaults/ but that's only 17kb.

[peteruithoven]

I thought maybe extra files where added during a upgrade, but after running sysupgrade (keeping configuration) with the latest version, nothing much happened, it might have even created space.

[olijf]

How can the wifibox with 0kb diskspace remaining have 15MB of sketches stored? As the wifibox has only 4MB of flash (and even less available for sketches in the OverlayFS filesystem)
Do things get swapped around to the ram disk maybe?

[peteruithoven]

Looking at the disk usage file (which contains I'm assuming the kilobytes per file) and I count up the size of all the sketches I get 1526, so 1.5MB.

Logging in to the box with this issue, through failsave, and after mounting root I ran disk usage on the sketches folder, this gave 1.5MB:

# du -h -c /root/sketches
1.5M    /root/sketches

So that 15MB was probably a typo, I'll correct that comment.

Yep I still have the box, so anyone working on this should probably check it out.

[olijf]

I would like to take a look at it. Maybe this not entirely related: According to the specs the WiFi-Box has only 851kb of physical storage available (which is aligned to around 832KB). Using some smart compression on the filesystem (LZMA2) its able to squeeze out a little bit more, but only on some filetypes (svg sketches are pretty good though because they contain a lot of redundant code, but this is unpredictable, as binaries or other filetypes yield different results. I was able to compress 7MB of XML random data (simulating svg sketches) to around 500KB. If our WiFi-Box only contained sketches in the rootfs_data partition we would have plenty of space for sketches (if all sketches where comparable). But this real world example shows that somehow the memory got filled up.
The reason why JFFS2 (LZMA filesystem) is bad to use is that we can not predict the real size before it is compressed. Thus I recommend setting a lower value than the actual value.
In lua the getFreeDiskSpace() function can not predict that it has 82KB left. It can only hope it does compress to something usable. I don't have a real solution here, but maybe we can do something with this information...

[peteruithoven]

You're saying that because data like the sketches is compressed (on LZMA2) it's hard to predict (for df) how much free disk space will remain? Sot it might be wrong?
Maybe there is a trick here:
http://unix.stackexchange.com/questions/157097/how-to-know-disk-space-occupied-by-packages-in-openwrt