Skip to content

Commit

Permalink
[RFR-471] Add api/v4 with OAuth2 (#158)
Browse files Browse the repository at this point in the history
* Fix docker configuration

* WIP: upload works

* [RFR-520] Get user id for upload

* Remove login endpoint from openapi yml

* [RFR-526] Add API v4 and move deprecated classes to v6 package

* Add openapi v3

* Replace v3 mentions with v4

* [RFR-538] Inject keycloak parameters into config

* [RFR-543] Mock Keycloak server in tests

* Cleanup

* Fix openapi documenation

* Cleanup tests for invalid credentials

* Substitute keycloak for oauth

* Fix build

* [RFR-266] Fix CI
  • Loading branch information
hb0 authored Jun 14, 2023
1 parent 7084f83 commit 48848fa
Show file tree
Hide file tree
Showing 55 changed files with 1,350 additions and 462 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/gradle_build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,8 @@ jobs:
docker-compose up -d
sleep 10
# Grep exits with exit code 0 when the API is started and fails with exit code 1 otherwise
echo "Try to reach the Collector API at http://localhost:8080/api/v3/ ..."
curl -s http://localhost:8080/api/v3/ | grep -q "Cyface Data Collector" #; echo $?
echo "Try to reach the Collector API at http://localhost:8080/api/v4/ ..."
curl -s http://localhost:8080/api/v4/ | grep -q "Cyface Data Collector" #; echo $?
env:
USERNAME: ${{ github.actor }}
PASSWORD: ${{ secrets.GITHUB_TOKEN }}
Expand All @@ -96,4 +96,4 @@ jobs:
docker logs collector_api
cat logs/collector.log
cat logs/collector-out.log
curl http://localhost:8080/api/v3/
curl http://localhost:8080/api/v4/
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ gradle.properties

# Cyface Temporary Files
**/uploadFolder
**/file-uploads
**/.vertx
**/conf.json
**/*.log
Expand Down
21 changes: 13 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ Now build the system as described in the "Building" section above:
Then simply run `docker-compose up` inside `build/docker`:
`cd build/docker/ && docker-compose up -d`

This calls docker to bring up a Mongo-database container and a container running the Cyface data collector API. The Collector API is by default available via port 8080. This means if you boot up everything using the default settings, the Collector API is accessible via `http://localhost:8080/api/v3/`.
This calls docker to bring up a Mongo-database container and a container running the Cyface data collector API. The Collector API is by default available via port 8080. This means if you boot up everything using the default settings, the Collector API is accessible via `http://localhost:8080/api/v4/`.

**ATTENTION: The docker setup should only be used for development purposes.**
It exposes the Cyface data collector as well as the ports of both Mongo database instances freely on the local network.
Expand Down Expand Up @@ -131,29 +131,34 @@ The following parameters are supported:
* **jwt.public:** The path of the file containing the public key used to sign JWT keys.
* **http.port:** The port the API is available at.
* **http.host:** The hostname under which the Cyface Data Collector is running. This can be something like `localhost`.
* **http.endpoint:** The path to the endpoint the Cyface Data Collector is running. This can be something like `/api/v3`.
* **http.port.management:** The port the management API is available at.
* **http.endpoint:** The path to the endpoint the Cyface Data Collector. This defaults to `/api/v4`.
* **mongo.db:** Settings for a Mongo database storing information about all the users capable of logging into the system and all data uploaded via the Cyface data collector. This defaults to a Mongo database available at `mongodb://127.0.0.1:27017`. The value of this should be a JSON object configured as described [here](https://vertx.io/docs/vertx-mongo-client/java/#_configuring_the_client).
* **admin.user:** The username of a default administration account which is created if it does not exist upon start up.
* **admin.password:** The password for the default administration account.
* **salt.path:** The path to a salt file used to encrypt passwords stored in the user database even stronger.
* **salt:** A salt value that may be used instead of the salt from salt.path. You must make sure that either the salt or the salt.path parameter are used. If both are specified the application startup will fail.
* **metrics.enabled:** Set to either `true` or `false`. If `true` the collector API publishes metrics using micrometer. These metrics are accessible by a [Prometheus](https://prometheus.io/) server (Which you need to set up yourself) at port `8081`.
* **http.port.management:** The port running the management API responsible for creating user accounts.
* **jwt.expiration**: The time it takes for a JWT token to expire in seconds. If a JWT token expires, clients need to acquire a new one via username and password authentication. Setting this time too short requires sending the username and password more often. This makes it easier for malicious parties to intercept and brute force usernames and passwords. However long time JWT tokens may be captured as well and used for malicious purposes.
* **upload.expiration:** The time an interrupted upload is stored for continueation in the future in milliseconds. If this time expires, the upload must start from the beginning.
* **jwt.expiration**: The time it takes for a JWT token to expire in seconds. If a JWT token expires, clients need to acquire a new one via username and password authentication. Setting this time too short requires sending the username and password more often. This makes it easier for malicious parties to intercept and brute force usernames and passwords. However, long time JWT tokens may be captured as well and used for malicious purposes.
* **upload.expiration:** The time an interrupted upload is stored for continuation in the future in milliseconds. If this time expires, the upload must start from the beginning.
* **measurement.payload.limit:** The size of a measurement in bytes up to which it is accepted as a single upload. Larger measurements are transmitted in chunks.
* **storage-type:** The type of storage to use for the uploaded data. Currently either `gridfs` or `google` is supported. The following parameter are required:
* **storage-type:** The type of storage to use for the uploaded data. Currently, either `gridfs` or `google` is supported. The following parameter are required:
* **gridfs**
* **type:** Must be `gridfs` in this case.
* **uploads-folder:** The relative or absolute path to a folder, to store temporary not finished uploads on the local harddrive before upload of the complete data blob to GridFS upon completion.
* **uploads-folder:** The relative or absolute path to a folder, to store temporary not finished uploads on the local hard drive before upload of the complete data blob to GridFS upon completion.
* **google**
* **type:** Must be `google` in this case.
* **collection-name:** The name of a Mongo collection to store an uploads metadata.
* **collection-name:** The name of a Mongo collection to store uploads' metadata into.
* **project-identifier:** A Google Cloud Storage project identifier to where the upload bucket is located.
* **bucket-name:** The Google Cloud Storage bucket name to load the data into.
* **credentials-file:** A credentials file used to authenticate with the Google Cloud Storage account used to upload the data to the Cloud.
* **paging-size:** The number of buckets to load per request, when iterating through all the data uploaded. Large numbers require fewer requests but more memory.
* **auth-type:** The type of authentication service to use. Currently, either `mocked` or `oauth` is supported. Defaults to `oauth`. Both require the following parameters:
* **oauth.callback**: The callback URL you entered in your provider admin console. This defaults to `http://localhost:8080/callback`.
* **oauth.client**: The name of the oauth client to contact. This defaults to `collector`.
* **oauth.secret**: The secret of the oauth client to contact.
* **oauth.site**: The Root URL for the provider without trailing slashes. This defaults to `https://auth.cyface.de:8443/realms/{tenant}`.
* **oauth.tenant**: The name of the oauth realm to contact. This defaults to `rfr`.

#### Running from Command Line

Expand Down
5 changes: 3 additions & 2 deletions build.gradle.kts
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ extra["mongoDriverVersion"] = "4.8.0"
extra["micrometerVersion"] = "1.10.6"
extra["commonsLangVersion"] = "3.12.0"
extra["logbackVersion"] = "1.4.6"
extra["cyfaceApiVersion"] = "2.1.2"
extra["cyfaceApiVersion"] = "3.0.0"
extra["cyfaceSerializationVersion"] = "2.3.6"
extra["gradleWrapperVersion"] = "7.6.1"
extra["googleCloudLibrariesVersion"] = "26.12.0"
Expand Down Expand Up @@ -154,7 +154,8 @@ dependencies {
// Authentication
implementation("io.vertx:vertx-auth-common:${project.extra["vertxVersion"]}")
implementation("io.vertx:vertx-auth-mongo:${project.extra["vertxVersion"]}")
implementation("io.vertx:vertx-auth-jwt:${project.extra["vertxVersion"]}")
implementation("io.vertx:vertx-auth-jwt:${project.extra["vertxVersion"]}") // Remove when dropping api/v3
implementation("io.vertx:vertx-auth-oauth2:${project.extra["vertxVersion"]}")

// Monitoring + Metrics
implementation("io.vertx:vertx-micrometer-metrics:${project.extra["vertxVersion"]}")
Expand Down
10 changes: 8 additions & 2 deletions conf.json.template
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,19 @@
"admin.password":"secret",
"http.port":8080,
"http.host":"localhost",
"http.endpoint":"/api/v3",
"http.endpoint":"/api/v4",
"http.port.management":13371,
"salt": "cyface_salt",
"upload.expiration": 60000,
"measurement.payload.limit": 104857600,
"storage-type": {
"type": "gridfs",
"uploads-folder": "file-uploads"
}
},
"auth-type": "oauth",
"oauth.callback":"http://localhost:8080/callback",
"oauth.client":"collector",
"oauth.secret":"SECRET",
"oauth.site":"https://auth.cyface.de:8443/realms/{tenant}",
"oauth.tenant":"rfr"
}
2 changes: 1 addition & 1 deletion doc/index.html

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion src/main/docker/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,6 @@ networks:
# Configure the IP Address range to not conflict with Deutsche Bahn Wi-Fi
- subnet: 172.72.0.1/24
- subnet: 2001:3984:3989::/64
# Shared docker network. Allows "exporter" docker container to access APIs without exposing them to the outside world
# Shared docker network. Allows "provider" docker container to access APIs without exposing them to the outside world
database:
name: cyface-collector_database
45 changes: 41 additions & 4 deletions src/main/docker/entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ SERVICE_NAME="Cyface Collector API"
main() {
loadJwtParameters
loadSaltParameters
loadAuthParameters
loadApiParameters
loadCollectorParameters
loadConfig
Expand Down Expand Up @@ -66,10 +67,41 @@ loadApiParameters() {
fi

if [ -z $CYFACE_API_ENDPOINT ]; then
CYFACE_API_ENDPOINT="/api/v3/"
CYFACE_API_ENDPOINT="/api/v4/"
fi
}

loadAuthParameters() {
if [ -z "$CYFACE_AUTH_TYPE" ]; then
CYFACE_AUTH_TYPE="oauth"
fi
if [ -z "$CYFACE_OAUTH_CALLBACK" ]; then
# FIXME: only use http if this stays internal (localhost)
CYFACE_OAUTH_CALLBACK="http://localhost:8080/callback"
fi
if [ -z "$CYFACE_OAUTH_CLIENT" ]; then
CYFACE_OAUTH_CLIENT="collector"
fi

if [ -z CYFACE_OAUTH_SECRET ]; then
echo "Unable to find OAuth client secret. Please set the environment variable CYFACE_OAUTH_SECRET to an appropriate value! API will not start!"
exit 1
fi

if [ -z "$CYFACE_OAUTH_SITE" ]; then
CYFACE_OAUTH_SITE="https://auth.cyface.de:8443/realms/{tenant}"
fi
if [ -z "$CYFACE_OAUTH_TENANT" ]; then
CYFACE_OAUTH_TENANT="rfr"
fi

echo "Using Auth type: $CYFACE_AUTH_TYPE"
echo "Using OAuth callback $CYFACE_OAUTH_CALLBACK"
echo "Using OAuth client $CYFACE_OAUTH_CLIENT"
echo "Using OAuth site $CYFACE_OAUTH_SITE"
echo "Using OAuth tenant $CYFACE_OAUTH_TENANT"
}

loadCollectorParameters() {
# JWT Expiration time
if [ -z $JWT_EXPIRATION_TIME_SECONDS ]; then
Expand Down Expand Up @@ -126,11 +158,16 @@ loadConfig() {
\"salt\":\"cyface-salt\",\
\"upload.expiration\":60000,\
\"measurement.payload.limit\":104857600,\
\"metrics.enabled\": false,\
\"storage-type\":{\
\"type\":\"gridfs\",\
\"uploads-folder\":\"file-uploads\"\
}
\"uploads-folder\":\"file-uploads\"\
},\
\"auth-type\":\"$CYFACE_AUTH_TYPE\",
\"oauth.callback\":\"$CYFACE_OAUTH_CALLBACK\",\
\"oauth.client\":\"$CYFACE_OAUTH_CLIENT\",\
\"oauth.secret\":\"$CYFACE_OAUTH_SECRET\",\
\"oauth.site\":\"$CYFACE_OAUTH_SITE\",\
\"oauth.tenant\":\"$CYFACE_OAUTH_TENANT\"\
}"
}

Expand Down
38 changes: 38 additions & 0 deletions src/main/kotlin/de/cyface/collector/auth/AuthHandlerBuilder.kt
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
/*
* Copyright 2023 Cyface GmbH
*
* This file is part of the Cyface Data Collector.
*
* The Cyface Data Collector is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* The Cyface Data Collector is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with the Cyface Data Collector. If not, see <http://www.gnu.org/licenses/>.
*/
package de.cyface.collector.auth

import io.vertx.core.Future
import io.vertx.ext.web.handler.OAuth2AuthHandler

/**
* Interface for the builder which creates an [OAuth2AuthHandler] to allow mocking.
*
* @author Armin Schnabel
* @version 1.0.0
* @since 7.0.0
*/
interface AuthHandlerBuilder {

/**
* Start the creation process of a [AuthHandlerBuilder] and provide a [Future], that will be notified about
* successful or failed completion.
*/
fun create(): Future<OAuth2AuthHandler>
}
87 changes: 87 additions & 0 deletions src/main/kotlin/de/cyface/collector/auth/MockedHandlerBuilder.kt
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
/*
* Copyright 2023 Cyface GmbH
*
* This file is part of the Cyface Data Collector.
*
* The Cyface Data Collector is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* The Cyface Data Collector is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with the Cyface Data Collector. If not, see <http://www.gnu.org/licenses/>.
*/
package de.cyface.collector.auth

import io.vertx.core.Future
import io.vertx.core.json.JsonObject
import io.vertx.ext.auth.impl.UserImpl
import io.vertx.ext.web.Route
import io.vertx.ext.web.RoutingContext
import io.vertx.ext.web.handler.OAuth2AuthHandler
import java.util.UUID

/**
* Mocked OAuth2 builder which creates an OAuth2 handler for testing.
*
* @author Armin Schnabel
* @version 1.0.0
* @since 7.0.0
*/
class MockedHandlerBuilder : AuthHandlerBuilder {

override fun create(): Future<OAuth2AuthHandler> {
val handler: OAuth2AuthHandler = object : OAuth2AuthHandler {
override fun handle(event: RoutingContext) {
val principal = JsonObject()
.put("username", "test-user")
.put("sub", UUID.randomUUID()) // user id
val user = UserImpl(principal, JsonObject())
event.setUser(user)

// From AuthenticationHandlerImpl.handle @ `authenticate(ctx, authN -> {..})`
// event.session()?.regenerateId() - this leads to SessionExpired exception, thus, commented out
// proceed with the router
if (!event.request().isEnded) {
event.request().resume()
}
postAuthentication(event)
}

// From AuthenticationHandlerInternal
private fun postAuthentication(event: RoutingContext) {
event.next()
}

override fun extraParams(extraParams: JsonObject?): OAuth2AuthHandler {
return this
}

override fun withScope(scope: String?): OAuth2AuthHandler {
return this
}

override fun withScopes(scopes: MutableList<String>?): OAuth2AuthHandler {
return this
}

override fun prompt(prompt: String?): OAuth2AuthHandler {
return this
}

override fun pkceVerifierLength(length: Int): OAuth2AuthHandler {
return this
}

override fun setupCallback(route: Route?): OAuth2AuthHandler {
return this
}
}
return Future.succeededFuture(handler)
}
}
62 changes: 62 additions & 0 deletions src/main/kotlin/de/cyface/collector/auth/OAuth2HandlerBuilder.kt
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
/*
* Copyright 2023 Cyface GmbH
*
* This file is part of the Cyface Data Collector.
*
* The Cyface Data Collector is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* The Cyface Data Collector is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with the Cyface Data Collector. If not, see <http://www.gnu.org/licenses/>.
*/
package de.cyface.collector.auth

import io.vertx.core.Future
import io.vertx.core.Promise
import io.vertx.core.Vertx
import io.vertx.ext.auth.oauth2.OAuth2Options
import io.vertx.ext.auth.oauth2.providers.KeycloakAuth
import io.vertx.ext.web.Router
import io.vertx.ext.web.handler.OAuth2AuthHandler
import java.net.URL

/**
* Keycloak OAuth2 builder which creates an OAuth2 handler.
*
* @author Armin Schnabel
* @version 1.0.0
* @since 7.0.0
* @property vertx
* @property apiRouter
* @property callbackUrl The callback URL you entered in your provider admin console.
* @property options the oauth configuration.
*/
class OAuth2HandlerBuilder(
private val vertx: Vertx,
private val apiRouter: Router,
private val callbackUrl: URL,
private val options: OAuth2Options,
) : AuthHandlerBuilder {

override fun create(): Future<OAuth2AuthHandler> {
val promise = Promise.promise<OAuth2AuthHandler>()

KeycloakAuth.discover(vertx, options)
.onSuccess {
val callbackAddress = apiRouter.get(callbackUrl.path)
val oauth2Handler = OAuth2AuthHandler.create(vertx, it, callbackUrl.toURI().toString())
.setupCallback(callbackAddress)
promise.complete(oauth2Handler)
}
.onFailure { promise.fail(it) }

return promise.future()
}
}
Loading

0 comments on commit 48848fa

Please sign in to comment.