-
Notifications
You must be signed in to change notification settings - Fork 2
Design documentation
This project was develop by Julian Raufelder and Jonas Reinwald as an assignment in the context of the lecture PiR (Programmieren in Rust - Programming in Rust), held by Professor Mächtel at the University of Applied Science in Constance. The goal was to give students the chance to get a better understanding of a new programming language (Rust) and to help them understand how to collaboratively work on an open source team project.
This repository aims to provide a cross-plattform backup solution for Android phones with a small (and in the future hopefully growing) but robust set of features for the rust community. Meanwhile, the goal of this wiki page is to provide insight into the design thoughts, the development process and the difficulties we were having along the way.
A packaged version of this code can be found on crates.io. More information about the usage can be found in the readme.
The main reason for developing adbackup is the fact that we are using Android phones ourselves and find it to be really important to backup our data. Unfortunately, we didn’t find any existing tool fitting our requirements:
- recoveries (e.g. TWRP): They don't have the ability to backup data storage. Furthermore installing a recovery to make backups isn't that convenient, especially for the community.
- Android apps: There are a few Android apps, for example Titanium Backup, but they are all running in their own sandbox. Such apps can only backup their own content and the shared data like photos etc. (without root access). Rooting the phone to create backups is not a viable option for most people in the community.
- adb: Using
adbdirectly is kind of a hassle. There exist a few scripts or simple tools (Windows only, e.g. Ultimate Backup Tool) which can create backups but the data management behind them, like incremental backups etc., isn't implemented.
Because of this we decided to write our own MIT-licensed incremental backup tool using adb and the simple backup mechanism of Android. Furthermore we are willing to provide a side-channel for pulling or pushing (optional transparently without inserting the data to the backup database) directories directly from or to the Android device.
The crate has several features which can be invoked either programmatically or by specifying the respective keywords when using the cli tool. This also reveals the first notable design decision: We split our work into a module / library part and a cli part. The cli part is a thin wrapper around our module - it basically only handles user interaction on the console (parsing params, printing help, ...) and uses the module part itself to carry out any real work. There are a few reasons for providing not just the module but also the cli wrapper:
- Users can start using the crate quickly and test out features before integrating it into their own work
- People without programming experience (i.e. people wo would not be able to use the module part of the crate) should be able to benefit from our work
- Implementing it as a thin wrapper is a good test for our module API and gives us valuable feedback (shortcomings, awkward usage) about it
The module part is where most of the work went into, it currently (v0.5.x) has the following sub-modules / features:
- adb_command: Building and executing calls to
adb, the tool used to interact with the Android phone(s). - backup: Building and executing backup commands (full backup with apps and optionally their apks, shared storage, and system apps or just individual apps) and getting a list of apps installed on a phone is done here. Uses adb_command.
- database
- management: Handles connections with device specific databases and the insertion / retrieval of data.
- migration: Used to ensure our databases are in a valid state (have the correct version) and to migrate databases from an older version to the current one.
- devices: Generates a list of all attached Android devices and information about them. Uses adb_command.
- file_transfer: Can pull or push single files and directories from / to a device. Uses adb_command.
- lib: Central API interface of the module. Uses all other sub-modules listed here and evaluates their return values / potential errors.
- logging: Configuration for the logging mechanism. Used at the initialization of the module.
- restore: Counterpart to
backup, restores Android backups to the device. Uses adb_command.
- clap: Used for specifying and handling all command line parameters.
- failure: Brings strong error management strategies. Because we are using a lot of different other crates (all with their own specific error-types) and because we also created a few of error-types of our own, an error-management crate is essential.
- fern: A logging framework which we have configured to log to a file when being used as a dependency of another rust project and to the console (stdout) when being invoked from our cli tool.
- rusqlite: Rustqlite offers bindings for SQLite (with an instance of a native SQLite client already bundled within). We use it to store complete backups (and in the future single files) in a SQLite database for easier access and centralized safekeeping of all data.
Especially a backup suite has to be as reliable as possible. A high test coverage is essential to proof the quality of the code. With this mindset we started developing adbackup, put on Travis (Linux & macOS) and Appveyour (Windows) for continuous integration to build the program and to execute our tests on every commit.
The main idea was to split our tests into three different categories:
- unit: Directly in the source file. Used to test the functions unit for unit.
- integration: Tests are placed in a separate file. Test the functionality of
adbackupfrom inside but with a greater view than an unit e.g. call the backup command and check the result. - blackbox: Just as with integration, all tests are placed in a separate file. These tests use rusts
CommandAPI to execute the possible commands ofadbackupand check the results.
Most of the integration- & blackbox- and also a lot of unit-test require the presence of an Android device. In other projects we already used an Android emulator for executing tests, locally and on Travis but only when developing / building an Android application.
While developing adbackup, we came across a problem using Travis and Appveyor in combination with such an emulator:
At the moment it is not possible to start an Android emulator on Appveyor in general and on Travis only if the project target is Android. On the other hand, Travis doesn't support multi-targets right now so if we choose Android as the target, we can not build rust applications. It is really sad that it is not possible to execute all adbackup commands and check the results with this setup without having an Android device on the server (CI) side.
We decided to write unit tests for functions where no emulator is involved or where we can mock the response/data from the device. For all other tests which depend on an Android devices, we created an issue to periodically check if Travis and Appveyor will fix the limitations and as soon as that happens, we will tighten up the missing tests: #20
adbackup is available on crates.io and as soon as we have a first release version 1.0.0, we will host the binaries for each version and operation system in this repository on github: releases
A good backup solution has to provide incremental backups. We wanted to implement this feature from the get-go, but a few rather unfortunate issues encountered during development prevented us from doing so:
- The backup created by Android phones / adb is encrypted with AES/CBC/PKCS5Padding for which we need a respective decryption method. We could not find an actively developed rust crypto crate which offers this functionality (but the ring crate might soon be able to, see: https://github.com/briansmith/ring/issues/573 & https://github.com/briansmith/ring/issues/588). This forced us to rely on a third party java tool, the Android Backup Extractor, to handle the decryption of the backup, which in turn meant potential users would have to download or build an executable of the extractor and also download the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files if they were not yet running Java 9.
- Once decrypted an Android backup is basically a tar-archive. To extract these archives we wanted to use the tar-rs create. Unfortunately, the Android archives could contain files with file names which are not valid for every operating system
adbackupis intended to run on (e.g.:on Windows). This results in a few non-extractable files for every backup and in turn could lead to corrupted data when restoring the (incomplete) backup if these files were essential for certain apps or the entire system to run. To circumvent this in the future we either need to find a crate which can replace / map invalid file name characters when extracting files from archives or request / implement this feature in thetar-rscrate.
On its own, the first issue would be bad enough, but combined with the second, more severe point we could not in good conscience publish the crate with the incremental backup feature enabled. As a compromise and because we didn't just want to delete code which could become useful in the near future we decided to create an experimental-features-branch (https://github.com/DonatJR/adbackup/tree/experimental-features).
We intend to run this branch in parallel with develop and host all as of yet unstable features / code snippets on or under it.
The development process was fun and insightful and provided us with a lot of experience of both rust and the design of a library / crate. The latter is especially true, as we invested a lot of time into thinking about our technical design & its implementation and into the research required for arriving at a (hopefully) sound result.
There were a few struggles while writing code as the rust compiler is uncharacteristically strict, but we still see this as a good thing as it always results in a better product in the end.
The fact there is such a big setup process required to use adbackup is something we don't like at all but which cannot be helped in the current Android ecosystem. We will try to improve this or find workarounds to get users started even faster.
Incremental backups have a high priority for us and we hope to get them working really soon. We intend to actively work with the rust community (mainly the ring and tar-rs crates) to get the respective requirements for this to work and would of course also appreciate any involvment of the community on adbackup (pointing out issues, suggest new features, ...).
Also, we do want to develop a GUI at some point and maybe assess a few different rust GUI crates / libraries while we are at it.