Skip to content

Commit

Permalink
Test lakefsfs with mock server (#6646)
Browse files Browse the repository at this point in the history
* Partial component test of lakeFSFS using mock lakeFS server

LakeFSFileSystemServerTest will replace LakeFSFileSystemTest once all tests
are translated.  It fakes actual lakeFS responses using a MockServer.  That
is a more complete test: test lakeFSFS versus the on-the-wire interface not
versus the generated SDK.  In particular the generated SDK will change soon
(#6424) to have a completely different API.  And Mockito cannot easily mock
the new API.

Fixes #6554.

* Change default response to 418 ("not a teapot")

lakeFSFS relies on 404s ("not found") for things such as directory marker
creation.  Without this, its behaviour in various states cannot be
simulated.  Also always returning a 404 makes it nearly impossible to
distinguish that no mock request matched.

* Add test{Delete,Create,Mkdirs,Open,ListStatus,Rename}*

Some testRename* tests still commented-out, we will do them separately so as
not to delay the new Java SDK.

* Split off tests that require S3 (MinIO) from others

Initial refactoring to allow LakeFSFileSystemPresignedModeTest and
LakeFSFileSystemSimpleModeTest to be clearly written.  This refactoring also
removes MinIO from _most_ tests in LakeFSFileSystemServerTest, which allows
these tests to be less brittle while running faster.

**MUST** bump Maven Surefire Plugin (the JUnit test runner) version to 3.x
from 2.x, to get rid of "No tests found matching Method..." failure messages
that appear for no apparent reason.

* Reduce logs (they were doubled) and configure S3A to avoid a warning

* Add simple & presigned tests for open, create

These are over MinIO, ensuring that presigned works at least there.  (MinIO
is similar enough to S3 on this, and indeed _probably_ any HTTP server
supporting GET and PUT would be enough to check presigned works...).

* Remove unused files

These now happen as parameters in LakeFSFileSystemServerS3Test.

* "expect" -> "mock"

Flavour of the day :-)

* [CR] Uncomment remaining tests, fix TODOne TODOs
  • Loading branch information
arielshaqed authored Oct 3, 2023
1 parent e216531 commit 6f69b3e
Show file tree
Hide file tree
Showing 12 changed files with 1,586 additions and 1,162 deletions.
27 changes: 20 additions & 7 deletions clients/hadoopfs/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ To export to S3:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.12.4</version>
<version>3.1.2</version>
<configuration>
<systemPropertyVariables>
<lakefs.access_mode>presigned</lakefs.access_mode>
Expand Down Expand Up @@ -231,7 +231,7 @@ To export to S3:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.12.4</version>
<version>3.1.2</version>
<configuration>
<excludes>
<exclude>${exclude.tests}</exclude>
Expand Down Expand Up @@ -368,11 +368,24 @@ To export to S3:
</dependency>

<dependency>
<groupId>org.mockito</groupId>
<!-- "inline" currently required in order to mock constructor calls. -->
<artifactId>mockito-inline</artifactId>
<version>3.10.0</version>
<scope>test</scope>
<groupId>org.mock-server</groupId>
<artifactId>mockserver-junit-rule-no-dependencies</artifactId>
<version>5.14.0</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.immutables</groupId>
<artifactId>value</artifactId>
<version>2.9.3</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>32.1.2-jre</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
1 change: 1 addition & 0 deletions clients/hadoopfs/src/main/java/io/lakefs/Constants.java
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ public class Constants {
public static final String ENDPOINT_KEY_SUFFIX = "endpoint";
public static final String LIST_AMOUNT_KEY_SUFFIX = "list.amount";
public static final String ACCESS_MODE_KEY_SUFFIX = "access.mode";
public static final String SESSION_ID = "session_id";

public static enum AccessMode {
SIMPLE,
Expand Down
5 changes: 5 additions & 0 deletions clients/hadoopfs/src/main/java/io/lakefs/LakeFSClient.java
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,11 @@ public LakeFSClient(String scheme, Configuration conf) throws IOException {
basicAuth.setUsername(accessKey);
basicAuth.setPassword(secretKey);

String sessionId = FSConfiguration.get(conf, scheme, Constants.SESSION_ID);
if (sessionId != null) {
apiClient.addDefaultCookie("sessionId", sessionId);
}

this.objectsApi = new ObjectsApi(apiClient);
this.stagingApi = new StagingApi(apiClient);
this.repositoriesApi = new RepositoriesApi(apiClient);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -421,6 +421,7 @@ private boolean renameFile(LakeFSFileStatus srcStatus, Path dst) throws IOExcept
LOG.debug("renameFile: dst {} exists and is a {}", dst, dstFileStatus.isDirectory() ? "directory" : "file");
if (dstFileStatus.isDirectory()) {
dst = buildObjPathOnExistingDestinationDir(srcStatus.getPath(), dst);
LOG.debug("renameFile: use {} to create dst {}", srcStatus.getPath(), dst);
}
} catch (FileNotFoundException e) {
LOG.debug("renameFile: dst does not exist, renaming src {} to a file called dst {}",
Expand Down
Loading

0 comments on commit 6f69b3e

Please sign in to comment.