Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Springboot 3.3.3 migration #41

Closed
wants to merge 52 commits into from
Closed
Show file tree
Hide file tree
Changes from 50 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
2f5e41d
Refactor duplication endpoint
anistouri Apr 17, 2024
4e225e5
Fix TU
anistouri Apr 18, 2024
5a13fc9
Code refactoring
anistouri Apr 24, 2024
3075cda
remove whitespace
anistouri Apr 24, 2024
7f458a3
Merge pull request #28 from powsybl/refactor-duplication-elements
anistouri May 2, 2024
35106ac
Parent 19: update liquibase maven plugin
anistouri May 3, 2024
7ba466c
Merge pull request #32 from powsybl/upgrade-to-parent-19
anistouri May 3, 2024
c200713
Update to powsybl dependencies 2024.1.0 (#31)
etiennehomer May 17, 2024
933e75d
Enable Prometheus in every microservice (#33)
antoinebhs May 29, 2024
93b531c
Fix case names when exported
achour94 Jun 10, 2024
700852a
Fix case names when exported
achour94 Jun 10, 2024
92d5796
Fix case names when exported
achour94 Jun 14, 2024
40e669e
Fix case names when exported #34
achour94 Jun 14, 2024
ca90c16
update to powsybl-ws-dependencies v2.11.0 (#35)
AbdelHedhili Jun 14, 2024
d3b815d
replace OffsetDateTime with Instant
anistouri Jun 20, 2024
17b5b1f
add comment
anistouri Jun 20, 2024
7438f1d
Merge pull request #36 from powsybl/replace-LocalDateTime-by-Instant
anistouri Jun 20, 2024
6b5761c
refactor
souissimai Jul 17, 2024
abd04b8
upgrade to powsybl-dependencies 2024.2.0
FranckLecuyer Jul 22, 2024
3b1f5b2
Merge pull request #37 from powsybl/rename_file_when_converting_case
souissimai Jul 22, 2024
bbe78ca
Merge pull request #38 from powsybl/upgrade_to_powsybl_dependencies_2…
FranckLecuyer Jul 29, 2024
49f5a76
add indexed attribute
jamal-khey Jul 29, 2024
eb50035
index only elements marked for indexation
jamal-khey Jul 29, 2024
3faccb1
fix syles
jamal-khey Jul 30, 2024
397d9c4
add delete index
jamal-khey Jul 31, 2024
4fb0ead
renaming endpoints
jamal-khey Jul 31, 2024
dff6051
add tests
jamal-khey Aug 1, 2024
f8761fa
fix old tests
jamal-khey Aug 5, 2024
82643ee
add tests for supervision controller
jamal-khey Aug 5, 2024
eab35af
Merge branch 'refs/heads/main' into jamalkhey/disable-auto-case-index…
jamal-khey Aug 5, 2024
66fafa8
clean conde
jamal-khey Aug 5, 2024
571a851
check styles
jamal-khey Aug 5, 2024
4abc90c
split test
jamal-khey Aug 5, 2024
d787751
split test
jamal-khey Aug 5, 2024
51bbe76
split test
jamal-khey Aug 5, 2024
11f42d1
assert queues are empty after each test
jamal-khey Aug 6, 2024
809b620
clean code
jamal-khey Aug 6, 2024
cb26803
impove test and coverage
jamal-khey Aug 6, 2024
5524154
fix review comments
jamal-khey Aug 14, 2024
9f80c67
fix review comments
jamal-khey Aug 16, 2024
d4f5b02
remove any ref to elements
jamal-khey Aug 19, 2024
08a7a2d
throw NOT_Found
jamal-khey Aug 19, 2024
e821fe6
extract getCaseMetaDataEntity into a function
jamal-khey Aug 20, 2024
0f911ae
Review
jamal-khey Aug 16, 2024
b9d7b67
Merge pull request #39 from powsybl/jamalkhey/disable-auto-case-index…
jamal-khey Aug 21, 2024
256056c
fix a bug affecting duplication of non indexed cases
jamal-khey Aug 21, 2024
7b716f1
fix review comments
jamal-khey Aug 22, 2024
80f06cb
Merge pull request #40 from powsybl/jamal-khey/fix-duplication-on-non…
jamal-khey Aug 22, 2024
2c2741e
Springboot 3.3.2 migration
Sep 2, 2024
c7be03f
change ws dependencies version
Sep 6, 2024
e1e9f74
Merge branch 'main' into springboot_3.3.2_migration
Sep 18, 2024
da27831
fix merge
Sep 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
<parent>
<groupId>com.powsybl</groupId>
<artifactId>powsybl-parent-ws</artifactId>
<version>17</version>
<version>20</version>
<relativePath/>
</parent>

Expand Down Expand Up @@ -48,7 +48,7 @@

<mockito-inline.version>3.11.1</mockito-inline.version>
<log4j2-mock-version>0.0.2</log4j2-mock-version>
<powsybl-ws-dependencies.version>2.9.0</powsybl-ws-dependencies.version>
<powsybl-ws-dependencies.version>2.14.0</powsybl-ws-dependencies.version>
<testcontainers.version>1.18.3</testcontainers.version>
</properties>

Expand Down Expand Up @@ -143,10 +143,6 @@
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-autoconfigure</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<!-- jpa, crud repository -->
<dependency>
<groupId>org.springframework.boot</groupId>
Expand Down Expand Up @@ -224,6 +220,16 @@
<artifactId>log4j-over-slf4j</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
<scope>runtime</scope>
</dependency>
<!-- Test dependencies -->
<dependency>
<groupId>com.google.guava</groupId>
Expand Down
24 changes: 9 additions & 15 deletions src/main/java/com/powsybl/caseserver/CaseController.java
Original file line number Diff line number Diff line change
Expand Up @@ -123,9 +123,10 @@ public ResponseEntity<byte[]> downloadCase(@PathVariable("caseUuid") UUID caseUu
public ResponseEntity<byte[]> exportCase(
@PathVariable UUID caseUuid,
@RequestParam String format,
@RequestParam(value = "fileName", required = false) String fileName,
@RequestBody(required = false) Map<String, Object> formatParameters) throws IOException {
LOGGER.debug("exportCase request received with parameter caseUuid = {}", caseUuid);
return caseService.exportCase(caseUuid, format, formatParameters).map(networkInfos -> {
return caseService.exportCase(caseUuid, format, fileName, formatParameters).map(networkInfos -> {
var headers = new HttpHeaders();
headers.setContentDisposition(
ContentDisposition.builder("attachment")
Expand All @@ -151,22 +152,23 @@ public ResponseEntity<Boolean> exists(@PathVariable("caseUuid") UUID caseUuid) {
@Operation(summary = "import a case")
@SuppressWarnings("javasecurity:S5145")
public ResponseEntity<UUID> importCase(@RequestParam("file") MultipartFile file,
@RequestParam(value = "withExpiration", required = false, defaultValue = "false") boolean withExpiration) {
@RequestParam(value = "withExpiration", required = false, defaultValue = "false") boolean withExpiration,
@RequestParam(value = "withIndexation", required = false, defaultValue = "false") boolean withIndexation) {
LOGGER.debug("importCase request received with file = {}", file.getName());
UUID caseUuid = caseService.importCase(file, withExpiration);
UUID caseUuid = caseService.importCase(file, withExpiration, withIndexation);
return ResponseEntity.ok().body(caseUuid);
}

@PostMapping(value = "/cases")
@PostMapping(value = "/cases", params = "duplicateFrom")
@Operation(summary = "create a case from an existing one")
@ApiResponses(value = {@ApiResponse(responseCode = "200", description = "The case has been duplicated"),
@ApiResponse(responseCode = "404", description = "Source case not found"),
@ApiResponse(responseCode = "500", description = "An error occurred during the case file duplication")})
public ResponseEntity<UUID> duplicateCase(
@RequestParam("duplicateFrom") UUID sourceCaseUuid,
@RequestParam("duplicateFrom") UUID caseId,
@RequestParam(value = "withExpiration", required = false, defaultValue = "false") boolean withExpiration) {
LOGGER.debug("duplicateCase request received with parameter sourceCaseUuid = {}", sourceCaseUuid);
UUID newCaseUuid = caseService.duplicateCase(sourceCaseUuid, withExpiration);
LOGGER.debug("duplicateCase request received with parameter sourceCaseUuid = {}", caseId);
UUID newCaseUuid = caseService.duplicateCase(caseId, withExpiration);
return ResponseEntity.ok().body(newCaseUuid);
}

Expand Down Expand Up @@ -205,14 +207,6 @@ public ResponseEntity<List<CaseInfos>> searchCases(@RequestParam(value = "q") St
return ResponseEntity.ok().body(cases);
}

@PostMapping(value = "/cases/reindex-all")
@Operation(summary = "reindex all cases")
public ResponseEntity<Void> reindexAllCases() {
LOGGER.debug("reindex all cases request received");
caseService.reindexAllCases();
return ResponseEntity.ok().build();
}

@GetMapping(value = "/cases/metadata")
@Operation(summary = "Get cases Metadata")
public ResponseEntity<List<CaseInfos>> getMetadata(@RequestParam("ids") List<UUID> ids) {
Expand Down
77 changes: 44 additions & 33 deletions src/main/java/com/powsybl/caseserver/CaseService.java
Original file line number Diff line number Diff line change
Expand Up @@ -15,13 +15,13 @@
import com.powsybl.caseserver.repository.CaseMetadataEntity;
import com.powsybl.caseserver.repository.CaseMetadataRepository;
import com.powsybl.commons.datasource.DataSource;
import com.powsybl.commons.datasource.DataSourceUtil;
import com.powsybl.commons.datasource.MemDataSource;
import com.powsybl.computation.ComputationManager;
import com.powsybl.computation.local.LocalComputationManager;
import com.powsybl.iidm.network.Exporter;
import com.powsybl.iidm.network.Importer;
import com.powsybl.iidm.network.Network;
import org.apache.commons.io.FilenameUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
Expand All @@ -44,22 +44,16 @@
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import java.time.LocalDateTime;
import java.time.ZoneOffset;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import java.util.Properties;
import java.util.UUID;
import java.time.Instant;
import java.time.temporal.ChronoUnit;
import java.util.*;
import java.util.stream.Collectors;
import java.util.stream.Stream;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;

import static com.powsybl.caseserver.CaseException.createDirectoryNotFound;
import static com.powsybl.caseserver.dto.CaseInfos.*;

/**
* @author Abdelsalem Hedhili <abdelsalem.hedhili at rte-france.com>
Expand Down Expand Up @@ -121,7 +115,7 @@ public List<CaseInfos> getCases(Path directory) {

private CaseInfos getCaseInfos(Path file) {
try {
return createInfos(file.getFileName().toString(), UUID.fromString(file.getParent().getFileName().toString()), getFormat(file));
return createInfos(file, UUID.fromString(file.getParent().getFileName().toString()));
} catch (Exception e) {
LOGGER.error("Error processing file {}: {}", file.getFileName(), e.getMessage(), e);
return null;
Expand Down Expand Up @@ -179,7 +173,7 @@ boolean caseExists(UUID caseName) {
return Files.exists(caseFile) && Files.isRegularFile(caseFile);
}

UUID importCase(MultipartFile mpf, boolean withExpiration) {
UUID importCase(MultipartFile mpf, boolean withExpiration, boolean withIndexation) {
checkStorageInitialization();

UUID caseUuid = UUID.randomUUID();
Expand Down Expand Up @@ -214,9 +208,11 @@ UUID importCase(MultipartFile mpf, boolean withExpiration) {
throw e;
}

createCaseMetadataEntity(caseUuid, withExpiration);
createCaseMetadataEntity(caseUuid, withExpiration, withIndexation);
CaseInfos caseInfos = createInfos(caseFile.getFileName().toString(), caseUuid, importer.getFormat());
caseInfosService.addCaseInfos(caseInfos);
if (withIndexation) {
caseInfosService.addCaseInfos(caseInfos);
}
sendImportMessage(caseInfos.createMessage());
return caseUuid;
}
Expand All @@ -235,10 +231,13 @@ UUID duplicateCase(UUID sourceCaseUuid, boolean withExpiration) {
newCaseFile = newCaseUuidDirectory.resolve(existingCaseFile.getFileName());
Files.copy(existingCaseFile, newCaseFile, StandardCopyOption.COPY_ATTRIBUTES);

CaseInfos existingCaseInfos = caseInfosService.getCaseInfosByUuid(sourceCaseUuid.toString()).orElseThrow();
CaseInfos caseInfos = createInfos(existingCaseInfos.getName(), newCaseUuid, existingCaseInfos.getFormat());
caseInfosService.addCaseInfos(caseInfos);
createCaseMetadataEntity(newCaseUuid, withExpiration);
CaseMetadataEntity existingCase = getCaseMetaDataEntity(sourceCaseUuid);
CaseInfos caseInfos = createInfos(newCaseFile, newCaseUuid);
if (existingCase.isIndexed()) {
caseInfosService.addCaseInfos(caseInfos);
}

createCaseMetadataEntity(newCaseUuid, withExpiration, existingCase.isIndexed());

sendImportMessage(caseInfos.createMessage());
return newCaseUuid;
Expand All @@ -248,12 +247,28 @@ UUID duplicateCase(UUID sourceCaseUuid, boolean withExpiration) {
}
}

private void createCaseMetadataEntity(UUID newCaseUuid, boolean withExpiration) {
LocalDateTime expirationTime = null;
private CaseInfos createInfos(Path caseFile, UUID caseUuid) {
return createInfos(caseFile.getFileName().toString(), caseUuid, getFormat(caseFile));
}

private CaseMetadataEntity getCaseMetaDataEntity(UUID caseUuid) {
return caseMetadataRepository.findById(caseUuid).orElseThrow(() -> new ResponseStatusException(HttpStatus.NOT_FOUND, "case " + caseUuid + " not found"));
}

private void createCaseMetadataEntity(UUID newCaseUuid, boolean withExpiration, boolean withIndexation) {
Instant expirationTime = null;
if (withExpiration) {
expirationTime = LocalDateTime.now(ZoneOffset.UTC).plusHours(1);
expirationTime = Instant.now().plus(1, ChronoUnit.HOURS);
}
caseMetadataRepository.save(new CaseMetadataEntity(newCaseUuid, expirationTime));
caseMetadataRepository.save(new CaseMetadataEntity(newCaseUuid, expirationTime, withIndexation));
}

public List<CaseInfos> getCasesToReindex() {
Set<UUID> casesToReindex = caseMetadataRepository.findAllByIndexedTrue()
.stream()
.map(CaseMetadataEntity::getId)
.collect(Collectors.toSet());
return getCases(getStorageRootDir()).stream().filter(c -> casesToReindex.contains(c.getUuid())).toList();
}

CaseInfos createInfos(String fileBaseName, UUID caseUuid, String format) {
Expand All @@ -269,7 +284,7 @@ CaseInfos createInfos(String fileBaseName, UUID caseUuid, String format) {

@Transactional
public void disableCaseExpiration(UUID caseUuid) {
CaseMetadataEntity caseMetadataEntity = caseMetadataRepository.findById(caseUuid).orElseThrow(() -> new ResponseStatusException(HttpStatus.NOT_FOUND, "case " + caseUuid + " not found"));
CaseMetadataEntity caseMetadataEntity = getCaseMetaDataEntity(caseUuid);
caseMetadataEntity.setExpirationDate(null);
}

Expand Down Expand Up @@ -373,10 +388,6 @@ private void sendImportMessage(Message<String> message) {
caseInfosPublisher.send("publishCaseImport-out-0", message);
}

public void reindexAllCases() {
caseInfosService.recreateAllCaseInfos(getCases(getStorageRootDir()));
}

public List<CaseInfos> getMetadata(List<UUID> ids) {
List<CaseInfos> cases = new ArrayList<>();
ids.forEach(caseUuid -> {
Expand Down Expand Up @@ -408,7 +419,7 @@ Optional<byte[]> getCaseBytes(UUID caseUuid) {
return Optional.empty();
}

public Optional<ExportCaseInfos> exportCase(UUID caseUuid, String format, Map<String, Object> formatParameters) throws IOException {
public Optional<ExportCaseInfos> exportCase(UUID caseUuid, String format, String fileName, Map<String, Object> formatParameters) throws IOException {
if (!Exporter.getFormats().contains(format)) {
throw CaseException.createUnsupportedFormat(format);
}
Expand All @@ -426,17 +437,17 @@ public Optional<ExportCaseInfos> exportCase(UUID caseUuid, String format, Map<St
network.write(format, exportProperties, memDataSource);

var listNames = memDataSource.listNames(".*");
String networkName = FilenameUtils.removeExtension(getCaseName(caseUuid));
String fileOrNetworkName = fileName != null ? fileName : DataSourceUtil.getBaseName(getCaseName(caseUuid));
byte[] networkData;
if (listNames.size() == 1) {
String extension = listNames.iterator().next();
networkName += extension;
fileOrNetworkName += extension;
networkData = memDataSource.getData(extension);
} else {
networkName += ".zip";
fileOrNetworkName += ".zip";
networkData = createZipFile(listNames, memDataSource);
}
return Optional.of(new ExportCaseInfos(networkName, networkData));
return Optional.of(new ExportCaseInfos(fileOrNetworkName, networkData));
} else {
return Optional.empty();
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,7 @@
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Service;

import java.time.LocalDateTime;
import java.time.ZoneOffset;
import java.time.Instant;

/**
* @author Abdelsalem Hedhili <abdelsalem.hedhili at rte-france.com>
Expand All @@ -34,10 +33,10 @@ public ScheduledCaseCleaner(CaseMetadataRepository caseMetadataRepository, CaseS

@Scheduled(cron = "${cleaning-cases-cron}", zone = "UTC")
public void deleteExpiredCases() {
LocalDateTime localDateTime = LocalDateTime.now(ZoneOffset.UTC);
LOGGER.info("Cleaning cases cron starting execution at {}", localDateTime);
Instant now = Instant.now();
LOGGER.info("Cleaning cases cron starting execution at {}", now);
caseMetadataRepository.findAll().stream().filter(caseMetadataEntity -> caseMetadataEntity.getExpirationDate() != null)
.filter(caseMetadataEntity -> localDateTime.isAfter(caseMetadataEntity.getExpirationDate()))
.filter(caseMetadataEntity -> now.isAfter(caseMetadataEntity.getExpirationDate()))
.forEach(caseMetadataEntity -> {
caseService.deleteCase(caseMetadataEntity.getId());
caseMetadataRepository.deleteById(caseMetadataEntity.getId());
Expand Down
82 changes: 82 additions & 0 deletions src/main/java/com/powsybl/caseserver/SupervisionController.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
/**
* Copyright (c) 2024, RTE (http://www.rte-france.com)
* This Source Code Form is subject to the terms of the Mozilla Public
* License, v. 2.0. If a copy of the MPL was not distributed with this
* file, You can obtain one at http://mozilla.org/MPL/2.0/.
*/
package com.powsybl.caseserver;

import com.powsybl.caseserver.elasticsearch.CaseInfosService;
import com.powsybl.caseserver.services.SupervisionService;
import io.swagger.v3.oas.annotations.Operation;
import io.swagger.v3.oas.annotations.responses.ApiResponse;
import io.swagger.v3.oas.annotations.responses.ApiResponses;
import io.swagger.v3.oas.annotations.tags.Tag;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.data.elasticsearch.client.ClientConfiguration;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;

/**
* @author Jamal KHEYYAD <jamal.kheyyad at rte-international.com>
*/
@RestController
@RequestMapping(value = "/" + CaseConstants.API_VERSION + "/supervision")
@Tag(name = "case-server - Supervision")
public class SupervisionController {
private static final Logger LOGGER = LoggerFactory.getLogger(SupervisionController.class);

private final SupervisionService supervisionService;
private final CaseService caseService;
private final ClientConfiguration elasticsearchClientConfiguration;
private final CaseInfosService caseInfosService;

public SupervisionController(SupervisionService supervisionService, CaseService caseService, ClientConfiguration elasticsearchClientConfiguration, CaseInfosService caseInfosService) {
this.supervisionService = supervisionService;
this.caseService = caseService;
this.elasticsearchClientConfiguration = elasticsearchClientConfiguration;
this.caseInfosService = caseInfosService;
}

@GetMapping(value = "/elasticsearch-host")
@Operation(summary = "get the elasticsearch address")
@ApiResponses(value = {@ApiResponse(responseCode = "200", description = "the elasticsearch address")})
public ResponseEntity<String> getElasticsearchHost() {
String host = elasticsearchClientConfiguration.getEndpoints().get(0).getHostName()
+ ":"
+ elasticsearchClientConfiguration.getEndpoints().get(0).getPort();
return ResponseEntity.ok().contentType(MediaType.TEXT_PLAIN).body(host);
}

@GetMapping(value = "/cases/index-name")
@Operation(summary = "get the indexed cases index name")
@ApiResponses(value = {@ApiResponse(responseCode = "200", description = "Indexed directory cases index name")})
public ResponseEntity<String> getIndexedCasesFIndexName() {
return ResponseEntity.ok().contentType(MediaType.TEXT_PLAIN).body(caseInfosService.getDirectoryCasesIndexName());
}

@PostMapping(value = "/cases/reindex")
@Operation(summary = "reindex all cases")
public ResponseEntity<Void> reindexAllCases() {
LOGGER.debug("reindex all cases request received");
caseInfosService.recreateAllCaseInfos(caseService.getCasesToReindex());
return ResponseEntity.ok().build();
}

@DeleteMapping(value = "/cases/indexation")
@Operation(summary = "delete indexed cases")
@ApiResponses(value = {@ApiResponse(responseCode = "200", description = "all indexed cases have been deleted")})
public ResponseEntity<String> deleteIndexedCases() {
return ResponseEntity.ok().contentType(MediaType.TEXT_PLAIN).body(Long.toString(supervisionService.deleteIndexedCases()));
}

@GetMapping(value = "/cases/indexation-count")
@Operation(summary = "get indexed cases count")
@ApiResponses(value = {@ApiResponse(responseCode = "200", description = "Indexed cases count")})
public ResponseEntity<String> getIndexedCasesCount() {
return ResponseEntity.ok().contentType(MediaType.TEXT_PLAIN).body(Long.toString(supervisionService.getIndexedCasesCount()));
}

}
Loading
Loading