From 6f803155eca14c2a1cd624015eb7facd85a95485 Mon Sep 17 00:00:00 2001 From: Klemens Muthmann Date: Wed, 24 Apr 2024 16:03:08 +0200 Subject: [PATCH] Move to Markdown and add Documentation Moves the adoc readme to the markdown format, as only this will be processable by Kotlin KDoc. Also adds explanations on the current Cyface binary format. --- README.adoc | 97 ----------------------------------- README.md | 143 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 143 insertions(+), 97 deletions(-) delete mode 100644 README.adoc create mode 100644 README.md diff --git a/README.adoc b/README.adoc deleted file mode 100644 index 1dbab17..0000000 --- a/README.adoc +++ /dev/null @@ -1,97 +0,0 @@ -= Serialization - -image:https://img.shields.io/badge/vert.x-4.1.2-purple.svg[link="https://vertx.io"] -image:https://github.com/cyface-de/serialization/actions/workflows/gradle_build.yml/badge.svg[link="https://github.com/cyface-de/serialization/actions/workflows/gradle_build.yml"] -image:https://github.com/cyface-de/serialization/actions/workflows/gradle_publish.yml/badge.svg[link="https://github.com/cyface-de/serialization/actions/workflows/gradle_publish.yml"] - -This application represents the https://cyface.de[Cyface] serialization software. - -It is used to de-/serialize the Cyface Binary Format which contains traffic data from Cyface measurement devices, such as our sensor box or our smartphone application. - -Our smartphone SDK is available as GPL application for https://github.com/cyface-de/android-backend[Android] and https://github.com/cyface-de/ios-backend[iOS] (or as https://github.com/cyface-de/ios-podspecs[Podspec]) as well. - -If you require this software under a closed source license for you own projects, please https://www.cyface.de/#kontakt[contact us]. - -Changes between versions are found in the link:https://github.com/cyface-de/serialization/releases[Release Section]. - -The project uses link:https://gradle.org/[Gradle] as the build system. -It is set up as a Gradle multi project build. -There are two general categories of sub-projects. -One are executable programs fulfilling a certain task of the Cyface backend. -Others are libraries shared by several Cyface executables. -The executables are grouped in the sub-project `executables` while the libraries are grouped in `libs`. -The following sections provide an overview about those projects. - -== Overview - -.link:#_libraries[Libraries] -* link:#_model[Model] -* link:#_serializer[Serializer] - -.General information -* link:#_release_a_new_version[Release a new Version] -* link:#_publishing_artifacts_to_github_packages_manually[Publishing Artifacts to GitHub Packages manually] -* link:#_licensing[Licensing] - - -== Libraries - -[#_model] -=== Model - -Model classes shared between multiple projects. - -[#_serializer] -=== Serializer - -Serialization wrappers for the Cyface Binary Format. - - -[#_release_a_new_version] -== Release a new Version - -See https://github.com/cyface-de/data-collector#release-a-new-version[Cyface Collector Readme] - -* `version` in root `build.gradle` is automatically set by the CI -* Just tag the release and push the tag to Github -* The Github package is automatically published when a new version is tagged and pushed by our -https://github.com/cyface-de/serialization/actions[Github Actions] to -the https://github.com/cyface-de/serialization/packages[Github Registry] -* The tag is automatically marked as a 'new Release' on https://github.com/cyface-de/serialization/releases[Github] - - -[#_publishing_artifacts_to_github_packages_manually] -== Publishing artifacts to GitHub Packages manually - -The artifacts produced by this project are distributed via link:https://github.com/features/packages[GitHubPackages]. -Before you can publish artifacts you need to rename `gradle.properties.template` to `gradle.properties` and enter your GitHub credentials. -How to obtain these credentials is described link:https://help.github.com/en/github/managing-packages-with-github-packages/about-github-packages#about-tokens[here]. - -To publish a new version of an artifact you need to: - -1. Increase the version number of the sub-project within the `build.gradle` file -2. Call `./gradlew publish` - -This will upload a new artifact to GitHub packages with the new version. -GitHub Packages will not accept to overwrite an existing version or to upload a lower version. -This project uses link:https://semver.org/[semantic versioning]. - - -[#_licensing] -== Licensing -Copyright 2018-2023 Cyface GmbH - -This file is part of the Cyface Serialization. - -The Cyface Serialization is free software: you can redistribute it and/or modify -it under the terms of the GNU General Public License as published by -the Free Software Foundation, either version 3 of the License, or -(at your option) any later version. - -The Cyface Serialization is distributed in the hope that it will be useful, -but WITHOUT ANY WARRANTY; without even the implied warranty of -MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -GNU General Public License for more details. - -You should have received a copy of the GNU General Public License -along with the Cyface Serialization. If not, see http://www.gnu.org/licenses/. diff --git a/README.md b/README.md new file mode 100644 index 0000000..bae7a2d --- /dev/null +++ b/README.md @@ -0,0 +1,143 @@ += Serialization + +![https://vertx.io](https://img.shields.io/badge/vert.x-4.1.2-purple.svg) +![https://github.com/cyface-de/serialization/actions/workflows/gradle_build.yml](https://github.com/cyface-de/serialization/actions/workflows/gradle_build.yml/badge.svg) +![https://github.com/cyface-de/serialization/actions/workflows/gradle_publish.yml](https://github.com/cyface-de/serialization/actions/workflows/gradle_publish.yml/badge.svg) + +This application represents the [Cyface](https://cyface.de) serialization software. + +It is used to de-/serialize the Cyface Binary Format which contains traffic data from Cyface measurement devices, such as our sensor box or our smartphone application. + +Our smartphone SDK is available as GPL application for [Android](https://github.com/cyface-de/android-backend) and [iOS](https://github.com/cyface-de/ios-backend) as well. + +If you require this software under a closed source license for you own projects, please [contact us](https://www.cyface.de/#kontakt). + +Changes between versions are found in the [Release Section](https://github.com/cyface-de/serialization/releases). + +The project uses [Gradle](https://gradle.org/) as the build system. +It is set up as a Gradle multi project build. +The following sections provide an overview of the details of each sub-projects. + +Overview +======== + +* [Model](#Model) +* [Serializer](#Serializer) +* [Release a new Version](#Release-A-New-Version) +* [Publishing Artifacts to GitHub Packages manually](#Publishing-Artifacts-To-Github-Packages-Manually) +* [Licensing](#Licensing) + +Model +===== + +Model classes shared between multiple projects. + +Serializer +========== + +This project provides a library to transform raw captured data, usually produced on a smartphone, into the Cyface binary format. +The following explanation is a description of what this library does. +Reading them is only important if you need to understand the binary format produced by this project. +If you have no interest in the details, just use our `deserializer` to deserialize binaries produced by this project. + +## Storage Format + +The data is usually placed into a file using ZLIB compression, with compression level 5 and the nowrap flag set to true. +The reason for using this level and the non standard value for the nowrap flag being that on iOS these are the standard unchangeable values. +Since there is no way to change them on iOS, compression level and nowrap are set for compatibility reasons. + +## The Cyface Binary Format +The structure of a file in the Cyface Binary Format in the most recent version is based on a Protobuf serializer. +Please use the appropriate Protobuf deserializer to decode such a file. +For an example on how to achieve this see the deserializer sub-project of this project or our [Protobuf Schemes](https://github.com/cyface-de/protos). +However values provided to this serializer should be preprocessed, according to the following specification. + +The first two bytes specify the version of the Cyface Binary Format using Big Endian. +The current version is **version 3**. +So the header should read `0000 0000 0000 0011`. + +The following sections contain the three data areas: *Events*, *Locations* and *Sensor Data*. + +Before going into the detailed descriptions it is important to understand the epoch format, fixed point values and run length encoding. + +The epoch format is a timestamp as a 64-Bit integer counting milliseconds since *01/01/1970 00:00:00,000*. + +Fixed point values are integers representing a floating point value with an implicit comma at a certain point. +A fixed point value of `1234` with two places after the comma should be handled like `12.34` after deserialization. + +Finally run length encoding stores values not with their actual value but by difference to the previous value in a timeline. +So `1.83792349,4.378237850,2.75263407` becomes `1838,2540,915`. + +### Events +Events are user interactions, that occurred with the measuring device. +They consist of a tuple of three values. +The first is the type of the event, given by the protobuf scheme. +A type can be for example a user pressing the start button on the measuring device. +The second value is the events timestamp in epoch format. +The third is an optional event value. +The modality type changed event for example requires the value of the new transportation mode, if the user switches from walking to using a bus, for example. + +### Location Records +Location records are stored as tuples of five values. +The first is a timestamp in epoch format using run length encoding. +The second is the latitude value as a fixed point value with 6 places after the comma and run length encoding. +The third is the longitude value, which is encoded exactly as the latitude. +The fourth is the GNSS accuracy value using two places after the command and run length encoding. +The fifth finally is the traveling speed in meters per second using two places after the comma and run length encoding. + +### Sensor Data +Sensor data is stored in batches, as this data is also captured and saved using batches. +There is one section for accelerations, one for rotations and one for directions. +All three are formatted very similar. + +Each batch contains multiple tuples of a timestamp in epoch format with run length encoding and a fixed point value for each of the three space dimensions *x*, *y* and *z*. +The fixed point value is an integer using run length encoding with three places after the comma for accelerations and rotations and two places for directions. + +Release a new Version +===================== + +See [Cyface Collector Readme](https://github.com/cyface-de/data-collector#release-a-new-version) + +* `version` in root `build.gradle` is automatically set by the CI +* Just tag the release and push the tag to Github +* The Github package is automatically published when a new version is tagged and pushed by our +[Github Actions](https://github.com/cyface-de/serialization/actions) to +the [Github Registry](https://github.com/cyface-de/serialization/packages) +* The tag is automatically marked as a 'new Release' on [Github](https://github.com/cyface-de/serialization/releases) + + +Publishing artifacts to GitHub Packages manually +================================================ + +The artifacts produced by this project are distributed via [GitHubPackages](https://github.com/features/packages). +Before you can publish artifacts you need to rename `gradle.properties.template` to `gradle.properties` and enter your GitHub credentials. +How to obtain these credentials is described [here](https://help.github.com/en/github/managing-packages-with-github-packages/about-github-packages#about-tokens). + +To publish a new version of an artifact you need to: + +1. Increase the version number of the sub-project within the `build.gradle` file +2. Call `./gradlew publish` + +This will upload a new artifact to GitHub packages with the new version. +GitHub Packages will not accept to overwrite an existing version or to upload a lower version. +This project uses [semantic versioning](https://semver.org/). + +Licensing +========= + +Copyright 2018-2024 Cyface GmbH + +This file is part of the Cyface Serialization. + +The Cyface Serialization is free software: you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation, either version 3 of the License, or +(at your option) any later version. + +The Cyface Serialization is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with the Cyface Serialization. If not, see http://www.gnu.org/licenses/.