Implement TimeDistributed Pruning

[santacml]

Hello,

I have been using this library for pruning nodes in a time-distributed network. It works great, thank you for implementing it!

In order to work with my net, I implemented many, small changes. Some changes are straightforward and some are not very elegant (such as an optional timeDistributedLayer flag kwarg). I would love your opinions on the changes and how we could improve them.

There are some big things this commit is missing, tracing back to actually calculating which nodes to prune. The issues are:

• The output of a TimeDistributed layer must be aggregated along the time axis somehow. My methodology was to average along the time axis and then do APoZ (and some other algorithms) to determine the nodes to prune, but deciding when to perform that average is difficult.
• My dataset has variable-length time-series data that comes in the form of a generator, which causes predict_generator to error. Because each file is a different length, predict_generator will error when it tries to use np.concatenate() at the very end of the function.

I therefore had to make some modifications to predict_generator in the keras source to get it to work with identify.py. To do this, I made a function locally that is the same as predict_generator but takes the average along the time axis before the concatenate. That way, identify doesn't have to worry at all about the time axis.

This solution is gross, but I'm not sure what else to do. The options I see are:

• Implement the above and attempt to put it in the Keras source
• Make a predict function in identify.py that can handle both variable-length data and averaging along the time axis. This isn't hard, but it results in a lot of duplicate code with the Keras source

Let me know what your thoughts are.

Thanks,

Michael Santacroce

[santacml]

Also just a comment, all the test cases passed on my computer that pass in the regular version of kerassurgeon.

[santacml]

Bump - any thoughts on this addition?

I've got more additions ready for some additional pruning methods, but they depend on this one.

[gerwin3]

These changes are great and have helped me a lot. Would be nice to have them in master.

[santacml]

These changes are great and have helped me a lot. Would be nice to have them in master.

Thanks, I'm glad to hear! I had honestly forgotten about this pull request - I was pretty unfamiliar with collaborating on git so the code in the original pull request was pretty gross. I have just cleaned up identify.py a bit.

I have a new pruning algorithm I'd love to put in this repo (once it is approved for public release by my research funder) but I think it may no longer be in active development.

[gerwin3]

Yeah, seems like it's not actively maintained anymore which is a shame because it's currently the best pruning library out there for Keras.

Would be great to see an alternative!