Add ops.random.alpha_dropout and layers.AlphaDropout
#18940
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #18940 +/- ##
==========================================
+ Coverage 79.51% 79.57% +0.05%
==========================================
Files 336 337 +1
Lines 34975 35056 +81
Branches 6865 6872 +7
==========================================
+ Hits 27812 27896 +84
+ Misses 5585 5583 -2
+ Partials 1578 1577 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
keras/backend/jax/random.py
Outdated
| @@ -81,6 +82,24 @@ def dropout(inputs, rate, noise_shape=None, seed=None): | |||
| ) | |||
|
|
|||
|
|
|||
| def alpha_dropout(inputs, rate, noise_shape=None, seed=None): | |||
| noise_shape = _get_concrete_noise_shape(inputs, noise_shape) | |||
| alpha = 1.6732632423543772848170429916717 | |||
There was a problem hiding this comment.
Any specific reason for having more than 6-7 decimal points?
There was a problem hiding this comment.
In practice it will be cast to float32 so some precision will be lost. The number above is just taken from the paper.
|
|
||
| def call(self, inputs, training=False): | ||
| if training and self.rate > 0: | ||
| return backend.random.alpha_dropout( |
There was a problem hiding this comment.
We can implement the layer in terms of backend ops and random.uniform. There is no real need for the alpha_dropout op -- only the layer would get used (even then, it's fairly niche usage).
There was a problem hiding this comment.
Sure, I've made the changes.
google-ml-butler
bot
added
kokoro:force-run
ready to pull
google-ml-butler
bot
removed
awaiting review
ready to pull
Related to #18467
This PR should benefit the users who want to apply the self-normalizing property to their models.
The implementation is taken from:
https://github.com/keras-team/keras/blob/v2.14.0/keras/layers/regularization/alpha_dropout.py#L28-L104