size_t calculations can overflow leading to memory errors, memleaks

[tigusoft-vm]

If we pass a huge string m, with m.size() almost the maximum size (maximum value of size_t), then calculations will overflow e.g. in

size_t mlen = m.size() + crypto_box_ZEROBYTES;
unsigned char mpad[mlen];

then e.g. mlen=5 so we allocate just 5 bytes but we later iterate more bytes because loops do not take such overflow into account:

for (size_t i = 0;i < crypto_secretbox_ZEROBYTES;++i) mpad[i] = 0;

this will write 0 into indexes e.g. 0 to 15, causing UB, e.g. stack overwrite.

For example, if size_t would be 16 bit, a string of length (2^16)-20 causes this problems.

This error is hopefully impossible to exploit in practice, because you need to actually have a string taking almost all possible memory, program would instead just crash OOM while creating the string in first place.

We propose that program instead should use safe function with assert for all such operations to check against any overflows when calculating sizes.

EDIT: on many implementations max_size that string can have is like /2 or /4 or less of maximum value of size_t, making it not possible to exploit there.