Checkpoint Ablation for Flamby #71
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… heart disease implementations. Work towards ablation study.
emersodb
changed the title
emersodb
commented
Nov 18, 2023
| @@ -50,7 +54,7 @@ def __init__( | |||
| self.alpha_learning_rate = alpha_learning_rate | |||
| self.client_number = client_number | |||
|
|
|||
| def get_dataloader(self, config: Config) -> Tuple[DataLoader, DataLoader]: | |||
There was a problem hiding this comment.
Somehow a typo crept into this APFL client
emersodb
commented
Nov 18, 2023
| @@ -23,9 +23,9 @@ def __init__( | |||
| metric_meter_type: MetricMeterType = MetricMeterType.AVERAGE, | |||
| checkpointer: Optional[TorchCheckpointer] = None, | |||
| temperature: Optional[float] = 0.5, | |||
| perfcl_loss_weights: Optional[Tuple[float, float]] = (0.0, 0.0), | |||
| cos_sim_loss_weight: Optional[float] = 0.0, | |||
| contrastive_loss_weight: Optional[float] = 0.0, | |||
There was a problem hiding this comment.
If these are not None, then some of these losses (at least perfcl) are still computed, but just "zeroed" out. This is problematic if you want to use asymmetric latent spaces in a FENDA architecture.
yc7z
reviewed
Nov 21, 2023
yc7z
reviewed
Nov 21, 2023
yc7z
reviewed
Nov 21, 2023
yc7z
reviewed
Nov 21, 2023
|
Looks good to me. Only a few comments on some small typos |
yc7z
approved these changes
Nov 21, 2023
Thanks for pointing out the helper message. Will fix! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adding in the ability to perform "non-federated checkpointing" for the FLamby experiments. This facilitates the checkpoint ablation study that was added to the ICLR paper.
Fixed two small bugs as well. The first was the dataloader method name for APFL (had a typo) and the second was defaulting the auxiliary FENDA losses to zeros rather than
None. This prevents using non-symmetric latent spaces in FENDA, because you need symmetric latent spaces for some the auxiliary losses and they are computed even if the coefficients are zero.