Binning for mutual info calculation

[pmorerio]

Hi, I have been running your code with the option -activation_function=1, i.e. with ReLUs according to the README. However, I noticed that binning for mutual info calculation is performed in between -1 and 1 independent of the activation function.

IDNNs/idnns/information/information_process.py

Line 183 in c4abb1d

bins = np.linspace(-1, 1, num_of_bins)

However, this looks coherent with tanh activations only.

[pmorerio]

Hi again. I changed the line of code above in order to have a bin interval which should better reflect the asymmetric range of values of ReLUs (approximately; I may be missing some bigger values).

bins = np.linspace(0, 10, num_of_bins)

Since this increases the range, I also considered more bins with -num_of_bins=100.

This is what I get.

[choasma]

i think somewhat you have similar figure compared with the reviewing paper
https://openreview.net/forum?id=ry_WPG-A-
see Figure 1. (B)
while there's few comments below and worth to check

[pmorerio]

@gladiator8072 yes indeed. Thanks. I have read the paper and the comments. I am inclined to think that the compression phase is to be ascribed to the saturation of the tanh and to this wrong binning for the ReLU. What do you think?

[choasma]

not_so_sure, except bin range, the number of bins might cause some noise effects and also relu is unbound on positive side, implies that the range of such direction is not fixed.
actually they also defends the relu result in the comments: https://goo.gl/VrRm5W. which shows the relu also has compression phase. i guess the repo just not quite up-to-date
(as the comment is pushed at 04/Jan where the repo is few months before updated)

please let me know you find some secret :D

[pmorerio]

Thanks, I read that document. Naftali Tishby and @ravidziv state that the authors of the ICLR paper

don’t know how to estimate mutual information correctly

However, they do not explain how such information should be estimated in practice, and the code they provide here certainly does a wrong estimate for ReLUs. It would be great to have an update of this repo which replicates the results they show. This would clarify everything and shed some light.

[pmorerio]

@gladiator8072 The code for that ICLR paper is here https://github.com/artemyk/ibsgd

[choasma]

awesome! i'm going to study this!! yeah... they didn't mention the correct estimation very detail

[ravidziv]

Hi,
Sorry for the late response
There are a may papers right now the try to calculate the mutual information with ReLU. It is a very complicated problem. This paper explains why the calculations of Saxe et el is not a good estimation. You can look at this paper for example how to calculate mutual information in networks, they used PixelCNN++ do estimate the information on MNIST and convolutional networks.