-
Notifications
You must be signed in to change notification settings - Fork 158
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
6c0badb
commit 2c77f60
Showing
8 changed files
with
25 additions
and
25 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,6 @@ | ||
# Managing Data Through Various TFHE-rs Versions | ||
|
||
In what follows, the process to manage data when upgrading the TFHE-rs version (starting from the 0.4.2 release) is given. This page details the methods to make data, which have initially been generated with an older version of TFHE-rs, usable with a newer version. | ||
In what follows, the process to manage data when upgrading the TFHE-rs version (starting from the 0.4.3 release) is given. This page details the methods to make data, which have initially been generated with an older version of TFHE-rs, usable with a newer version. | ||
|
||
## Forward Compatibility Strategy | ||
|
||
|
@@ -18,12 +18,12 @@ To be able to use older serialized data with newer versions, the following is do | |
In practice, if we take the 0.5 release as a concrete example, here is what will happen: | ||
|
||
- 0.5.0 is released with breaking changes to the serialization; | ||
- 0.4.2 has [email protected] as optional dependency gated by the `forward_compatibility` feature; | ||
- Conversion code is added to 0.4.2, if possible without any user input, but some data migration will likely require some information to be provided by the developer writing the migration code; | ||
- 0.4.2 is released. | ||
- 0.4.3 has [email protected] as optional dependency gated by the `forward_compatibility` feature; | ||
- Conversion code is added to 0.4.3, if possible without any user input, but some data migration will likely require some information to be provided by the developer writing the migration code; | ||
- 0.4.3 is released. | ||
|
||
{% hint style="info" %} | ||
Note that if you do not need forward compatibility 0.4.2 will be equivalent to 0.4.1 from a usability perspective and you can safely update. | ||
Note that if you do not need forward compatibility 0.4.3 will be equivalent to 0.4.1 from a usability perspective and you can safely update. | ||
Note also that the 0.5.0 has no knowledge of previous releases. | ||
{% endhint %} | ||
|
||
|
@@ -36,10 +36,10 @@ Examples to migrate data: | |
An `Application` uses TFHE-rs 0.4.1 and needs/wants to upgrade to 0.5.0 to benefit from various improvements. | ||
|
||
Example timeline of the data migration or `Bulk Data Migration`: | ||
- A new transition version of the `Application` is compiled with the 0.4.2 release of TFHE-rs; | ||
- A new transition version of the `Application` is compiled with the 0.4.3 release of TFHE-rs; | ||
- The transition version of the `Application` adds code to read previously stored data, convert it to the proper format for 0.5.0 and save it back to disk; | ||
- The service enters a maintenance period (if relevant); | ||
- Migration of data from 0.4.2 to 0.5.0 is done with the transition version of the `Application`, note that depending on the volume of data this transition can take a significant amount of time; | ||
- Migration of data from 0.4.3 to 0.5.0 is done with the transition version of the `Application`, note that depending on the volume of data this transition can take a significant amount of time; | ||
- The updated version of the `Application` is compiled with the 0.5.0 release of TFHE-rs and put in production; | ||
- Service is resumed with the updated `Application` (if relevant). | ||
|
||
|
@@ -48,18 +48,18 @@ The above case is describing a simple use case, where only a single version of d | |
In order to manage more complicated cases, another method called `Migrate On Read` can be used. | ||
|
||
Here is an example timeline where data is migrated only as needed with the `Migrate On Read` approach: | ||
- A new version of the `Application` is compiled, it has [email protected].2 as dependency (the dependency will have to be renamed to avoid conflicts, a possible name is to use the major version like `tfhe_0_4`) and [email protected] which will not be renamed and can be accessed as `tfhe` | ||
- A new version of the `Application` is compiled, it has [email protected].3 as dependency (the dependency will have to be renamed to avoid conflicts, a possible name is to use the major version like `tfhe_0_4`) and [email protected] which will not be renamed and can be accessed as `tfhe` | ||
- Code to manage reading the data is added to the `Application`: | ||
- The code determines whether the data was saved with the 0.4 `Application` or the 0.5 `Application`, if the data is already up to date with the 0.5 format it can be loaded right away, if it's in the 0.4 format the `Application` can check if an updated version of the data is already available in the 0.5 format and loads that if it's available, otherwise it converts the data to 0.5, saves the converted data to avoid having to convert it every time it is accessed and continue processing with the 0.5 data | ||
|
||
The above is more complicated to manage as data will be present on disk with several versions, however it allows to run the service continuously or near-continuously once the new `Application` is deployed (it will require careful routing or error handling as nodes with outdated `Application` won't be able to process the 0.5 data). | ||
|
||
Also, if required, several version of TFHE-rs can be "chained" to upgrade very old data to newer formats. | ||
The above pattern can be extended to have `tfhe_0_4` ([email protected].2 renamed), `tfhe_0_5` ([email protected] renamed) and `tfhe` being [email protected], this will require special handling from the developers so that their protocol can handle data from 0.4.2, 0.5.0 and 0.6.0 using all the conversion tooling from the relevant version. | ||
The above pattern can be extended to have `tfhe_0_4` ([email protected].3 renamed), `tfhe_0_5` ([email protected] renamed) and `tfhe` being [email protected], this will require special handling from the developers so that their protocol can handle data from 0.4.3, 0.5.0 and 0.6.0 using all the conversion tooling from the relevant version. | ||
|
||
E.g., if some computation requires data from version 0.4.2 a conversion function could be called `upgrade_data_from_0_4_to_0_6` and do: | ||
E.g., if some computation requires data from version 0.4.3 a conversion function could be called `upgrade_data_from_0_4_to_0_6` and do: | ||
|
||
- read data from 0.4.2 | ||
- read data from 0.4.3 | ||
- convert to 0.5.0 format using `tfhe_0_4` | ||
- convert to 0.6.0 format using `tfhe_0_5` | ||
- save to disk in 0.6.0 format | ||
|
@@ -80,8 +80,8 @@ edition = "2021" | |
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html | ||
|
||
[dependencies] | ||
# The project used tfhe 0.4.1, it now depends on the 0.4.2 with the forward compatibility code | ||
tfhe_0_4 = { package = "tfhe", version = "0.4.2", features = [ | ||
# The project used tfhe 0.4.1, it now depends on the 0.4.3 with the forward compatibility code | ||
tfhe_0_4 = { package = "tfhe", version = "0.4.3", features = [ | ||
"x86_64-unix", | ||
"boolean", | ||
"shortint", | ||
|
@@ -137,6 +137,6 @@ The noise level here is set at `usize::MAX` on a 64 bits system, it corresponds | |
|
||
## Breaking changes and additional migration information | ||
|
||
The main breaking change going from 0.4.2 to 0.5.0 with respect to data migration is that the High Level API dropped support for `shortint`. The `boolean` format has changed to use `integer`'s `BooleanBlock` under the hood. | ||
The main breaking change going from 0.4.3 to 0.5.0 with respect to data migration is that the High Level API dropped support for `shortint`. The `boolean` format has changed to use `integer`'s `BooleanBlock` under the hood. | ||
|
||
This means that any data coming from the High Level API which previously used `boolean` or `shortint` is not supported for the data migration. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters