Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] New storage using Time Series Server #96

Open
wants to merge 62 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
6165f6c
Init timeseries server-based storage
bhorvilleur-artelys Apr 23, 2021
07feb06
Create AFS storage using TimeSeriesServer (WIP)
bhorvilleur-artelys May 6, 2021
8fd48e4
Add time series server project to master pom
armic-artelys May 6, 2021
6afa4f8
Test TimeSeriesServerAppStorage with a MapDB implementation for gener…
bhorvilleur-artelys May 10, 2021
b99902e
Bugfixes on create and search time series methods:
bhorvilleur-artelys May 11, 2021
c355abc
TimeSeriesServerAppStorage : add publish timeseries method
bhorvilleur-artelys May 11, 2021
554d157
Bugfix call publish timeseries method
bhorvilleur-artelys May 11, 2021
b05fe4c
TimeSeriesServerAppStorage : add fetch timeseries method
bhorvilleur-artelys May 11, 2021
79c809c
Adapt AbstractAppStorageTest to storages which do not handle chunk co…
bhorvilleur-artelys May 18, 2021
8028d83
Specialize TimeSeriesServer in Double time series
bhorvilleur-artelys May 18, 2021
b79dbd4
Merge remote-tracking branch 'origin/draft_ts_server' into draft_ts_s…
armic-artelys May 25, 2021
0bde97f
Checkstyle fix
bhorvilleur-artelys May 26, 2021
5f718f1
AFS time series server : customize the "AFS" app
bhorvilleur-artelys May 26, 2021
2ad5175
Merge remote-tracking branch 'origin/draft_ts_server' into draft_ts_s…
armic-artelys May 27, 2021
99d825f
Code style fixes
armic-artelys Jun 1, 2021
59828bb
Create a special module for timeseries config services
armic-artelys Jun 1, 2021
4c08ce7
Initialization for TS server app file system configuration components
armic-artelys Jun 1, 2021
0908732
Add unit tests to TS server config components
armic-artelys Jun 1, 2021
d7bd572
Setup a conf.yaml file for tests
armic-artelys Jun 1, 2021
d0c65b7
Rework ts server storage classes to setup the modules separation (sto…
armic-artelys Jun 1, 2021
b819782
Add new module in parent pom
armic-artelys Jun 1, 2021
e99a63e
Rework mapdb app file system configuration to allow delegate provider…
armic-artelys Jun 1, 2021
c0895f5
Add support for String time series in time series server AppStorage
armic-artelys Jun 17, 2021
4c55a33
Fix code to match powsybl standards
armic-artelys Jun 18, 2021
3c876f0
Fix checkstyle + remove missing dependency
pl-buiquang Jul 7, 2021
3439ca7
Prepare release v3.4.0
Nov 16, 2020
a8a7bbe
Prepare next release v3.5.0
Nov 16, 2020
fe303a1
Remove ifnotexist when inserting node_data_name (#65)
pl-buiquang Jan 15, 2021
150b2df
Use java 11 for sonar analysis (#70)
pl-buiquang Feb 8, 2021
97a8ffd
Springboot based implementation of AFS backend (#63)
sylvlecl Feb 18, 2021
6f73aa1
Back to springfox v2.9.2, v2.10.5 is broken (#76)
sylvlecl Feb 19, 2021
789b69a
Release v3.5.0
sylvlecl Mar 2, 2021
540d534
Prepare next release v3.6.0
sylvlecl Mar 2, 2021
5847cbc
[Iissue-78] Add NetworkListener on virtual case (#79)
marifunf Apr 16, 2021
8a603cc
Init timeseries server-based storage
bhorvilleur-artelys Apr 23, 2021
e3e2453
Create AFS storage using TimeSeriesServer (WIP)
bhorvilleur-artelys May 6, 2021
328c1c4
Add time series server project to master pom
armic-artelys May 6, 2021
3fa9d97
Test TimeSeriesServerAppStorage with a MapDB implementation for gener…
bhorvilleur-artelys May 10, 2021
fd0c955
Bugfixes on create and search time series methods:
bhorvilleur-artelys May 11, 2021
95ce4ba
TimeSeriesServerAppStorage : add publish timeseries method
bhorvilleur-artelys May 11, 2021
e47ffba
Bugfix call publish timeseries method
bhorvilleur-artelys May 11, 2021
715ba5f
TimeSeriesServerAppStorage : add fetch timeseries method
bhorvilleur-artelys May 11, 2021
7ce55d7
Adapt AbstractAppStorageTest to storages which do not handle chunk co…
bhorvilleur-artelys May 18, 2021
30f47ee
Specialize TimeSeriesServer in Double time series
bhorvilleur-artelys May 18, 2021
f09a560
Checkstyle fix
bhorvilleur-artelys May 26, 2021
b738bff
AFS time series server : customize the "AFS" app
bhorvilleur-artelys May 26, 2021
d3cd820
Code style fixes
armic-artelys Jun 1, 2021
955ead9
Create a special module for timeseries config services
armic-artelys Jun 1, 2021
5b63954
Initialization for TS server app file system configuration components
armic-artelys Jun 1, 2021
99c6152
Add unit tests to TS server config components
armic-artelys Jun 1, 2021
ff05947
Setup a conf.yaml file for tests
armic-artelys Jun 1, 2021
8c0ebb3
Rework ts server storage classes to setup the modules separation (sto…
armic-artelys Jun 1, 2021
63aca9a
Add new module in parent pom
armic-artelys Jun 1, 2021
cc36235
Rework mapdb app file system configuration to allow delegate provider…
armic-artelys Jun 1, 2021
8317832
Add support for String time series in time series server AppStorage
armic-artelys Jun 17, 2021
291b4ef
Fix code to match powsybl standards
armic-artelys Jun 18, 2021
9216dad
Fix checkstyle + remove missing dependency
pl-buiquang Jul 7, 2021
74e501b
Add missing jackson provider dependency
armic-artelys Jul 9, 2021
4d29d36
Fix fetch results subtypes handling in TS storage delegate : retrieve…
armic-artelys Jul 9, 2021
2c449b0
Add missing test dependency : assertj-core
armic-artelys Jul 9, 2021
f4aa998
Merge remote-tracking branch 'origin/draft_ts_server' into draft_ts_s…
bhorvilleur-artelys Jul 21, 2021
91d6495
TimeSeriesDAO now trigger IOException when failing to publish data
armic-artelys Jul 29, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Initialization for TS server app file system configuration components
Signed-off-by: Arthur Michaut <[email protected]>
armic-artelys committed Jul 9, 2021
commit 5b63954952570515f8cdef84136fd4a0ab7e9ea9
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
package com.powsybl.afs.timeseriesserver;

import com.powsybl.afs.AppFileSystem;
import com.powsybl.afs.storage.AppStorage;

/**
* @author [email protected]
*/
public class TSServerAppFileSystem extends AppFileSystem {

public TSServerAppFileSystem(final String name, final boolean remotelyAccessible, final AppStorage storage) {
super(name, remotelyAccessible, storage);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
package com.powsybl.afs.timeseriesserver;

import com.powsybl.afs.mapdb.MapDbAppFileSystemConfig;
import com.powsybl.afs.storage.AbstractAppFileSystemConfig;
import com.powsybl.commons.config.PlatformConfig;
import lombok.Getter;
import lombok.Setter;

import java.net.URI;
import java.net.URISyntaxException;
import java.util.List;

/**
* App file system configuration for time series server app storage
*
* @author [email protected]
*/
public class TSServerAppFileSystemConfig extends AbstractAppFileSystemConfig<TSServerAppFileSystemConfig> {

private static final String TSSERVER_APP_FILE_SYSTEM = "tsserver-app-file-system";
private static final String DRIVE_NAME = "drive-name";
private static final String REMOTELY_ACCESSIBLE = "remotely-accessible";
private static final boolean DEFAULT_REMOTELY_ACCESSIBLE = false;
private static final String HOST = "host";
private static final String PORT = "port";
private static final String SCHEME = "scheme";
private static final String APP = "app";
private static final String DEFAULT_APP = "AFS";
public static final String DELEGATE_MAPDB_APP_FILE_SYSTEM = "delegate-mapdb-app-file-system";

@Setter
private String host;

@Setter
private int port;

@Setter
private String scheme;

@Getter
@Setter
private String app = DEFAULT_APP;

@Getter
@Setter
private MapDbAppFileSystemConfig delegateConfig;

public TSServerAppFileSystemConfig(final String driveName, final boolean remotelyAccessible) {
super(driveName, remotelyAccessible);
}

public static TSServerAppFileSystemConfig load() {
return load(PlatformConfig.defaultConfig());
}

public static TSServerAppFileSystemConfig load(final PlatformConfig platformConfig) {
return platformConfig.getOptionalModuleConfig(TSSERVER_APP_FILE_SYSTEM)
.map(moduleConfig -> {
final String driveName;
if (moduleConfig.hasProperty(DRIVE_NAME)) {
driveName = moduleConfig.getStringProperty(DRIVE_NAME);
} else {
throw new IllegalArgumentException("Please provide a drive name for timeseries server app file system configuration");
}
final boolean remotelyAccessible = moduleConfig.getBooleanProperty(REMOTELY_ACCESSIBLE, DEFAULT_REMOTELY_ACCESSIBLE);
final TSServerAppFileSystemConfig config = new TSServerAppFileSystemConfig(driveName, remotelyAccessible);
if (moduleConfig.hasProperty(HOST)) {
config.setHost(moduleConfig.getStringProperty(HOST));
}
if (moduleConfig.hasProperty(PORT)) {
config.setPort(moduleConfig.getIntProperty(PORT));
}
if (moduleConfig.hasProperty(SCHEME)) {
config.setScheme(moduleConfig.getStringProperty(SCHEME));
}
config.setApp(moduleConfig.getStringProperty(APP, DEFAULT_APP));
// Load delegate config
final List<MapDbAppFileSystemConfig> delegateConfig = MapDbAppFileSystemConfig.load(platformConfig.getModuleConfig(DELEGATE_MAPDB_APP_FILE_SYSTEM));
if (delegateConfig.size() != 1) {
throw new IllegalArgumentException("A single mapdb delegate configuration is necessary");
}
config.setDelegateConfig(delegateConfig.get(0));
return config;
})
.orElse(null);
}

/**
* @return target URI for timeseries service
* @throws URISyntaxException if information does not allow to build a proper URI
*/
public URI getURI() throws URISyntaxException {
return new URI(scheme, null, host, port, null, null, null);
}

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
package com.powsybl.afs.timeseriesserver;

import com.google.auto.service.AutoService;
import com.powsybl.afs.AppFileSystem;
import com.powsybl.afs.AppFileSystemProvider;
import com.powsybl.afs.AppFileSystemProviderContext;
import com.powsybl.afs.mapdb.storage.MapDbAppStorage;
import com.powsybl.afs.storage.AbstractAppStorage;
import com.powsybl.afs.storage.EventsBus;
import com.powsybl.afs.timeseriesserver.storage.TimeSeriesServerAppStorage;

import java.net.URI;
import java.net.URISyntaxException;
import java.nio.file.Path;
import java.util.Collections;
import java.util.List;

/**
* @author [email protected]
*/
@AutoService(AppFileSystemProvider.class)
public class TSServerAppFileSystemProvider implements AppFileSystemProvider {

private final TSServerAppFileSystemConfig configuration;

private final TimeSeriesServerAppStorage.TimeSeriesServerAppStorageProvider<URI, String, AbstractAppStorage, TimeSeriesServerAppStorage> storageProvider;

private final MapDbAppStorage.MapDbAppStorageProvider<String, Path, EventsBus, MapDbAppStorage> delegateStorageProvider;

public TSServerAppFileSystemProvider() {
this(
TSServerAppFileSystemConfig.load(),
TimeSeriesServerAppStorage::new,
(name, path, eventsStore) -> MapDbAppStorage.createMmapFile(name, path.toFile(), eventsStore)
);
}

public TSServerAppFileSystemProvider(final TSServerAppFileSystemConfig configuration,
final TimeSeriesServerAppStorage.TimeSeriesServerAppStorageProvider<URI, String, AbstractAppStorage, TimeSeriesServerAppStorage> provider,
final MapDbAppStorage.MapDbAppStorageProvider<String, Path, EventsBus, MapDbAppStorage> delegateStorageProvider) {
this.configuration = configuration;
this.storageProvider = provider;
this.delegateStorageProvider = delegateStorageProvider;
}

@Override
public List<AppFileSystem> getFileSystems(final AppFileSystemProviderContext context) {
return Collections.singletonList(getFileSystem(context));
}

/**
* @param context a context holding necessary event bus
* @return a single time series server app file system, built using internal configuration and provided context
*/
private TSServerAppFileSystem getFileSystem(final AppFileSystemProviderContext context) {
// Make an URI form configuration
URI uri;
try {
uri = configuration.getURI();
} catch (URISyntaxException e) {
throw new IllegalStateException("Could not build a proper target URI for time series server");
}
MapDbAppStorage delegateAppStorage = delegateStorageProvider.apply(configuration.getDriveName(), configuration.getDelegateConfig().getDbFile(), context.getEventsBus());
return new TSServerAppFileSystem(
configuration.getDriveName(),
configuration.isRemotelyAccessible(),
storageProvider.apply(uri, configuration.getApp(), delegateAppStorage)
);
}
}