Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test lakefsfs with mock server #6646

Merged
merged 9 commits into from
Oct 3, 2023
Merged

Test lakefsfs with mock server #6646

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
a8b8181
Partial component test of lakeFSFS using mock lakeFS server
arielshaqed Sep 10, 2023
7d23d81
Change default response to 418 ("not a teapot")
arielshaqed Sep 10, 2023
6ed5dc7
Add test{Delete,Create,Mkdirs,Open,ListStatus,Rename}*
arielshaqed Sep 27, 2023
f8372ad
Split off tests that require S3 (MinIO) from others
arielshaqed Oct 1, 2023
a7c8dbe
Reduce logs (they were doubled) and configure S3A to avoid a warning
arielshaqed Oct 1, 2023
e0b075d
Add simple & presigned tests for open, create
arielshaqed Oct 1, 2023
3806816
Remove unused files
arielshaqed Oct 2, 2023
ae4a9d2
"expect" -> "mock"
arielshaqed Oct 2, 2023
3cbb650
[CR] Uncomment remaining tests, fix TODOne TODOs
arielshaqed Oct 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 20 additions & 7 deletions clients/hadoopfs/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ To export to S3:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.12.4</version>
<version>3.1.2</version>
<configuration>
<systemPropertyVariables>
<lakefs.access_mode>presigned</lakefs.access_mode>
Expand Down Expand Up @@ -231,7 +231,7 @@ To export to S3:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.12.4</version>
<version>3.1.2</version>
<configuration>
<excludes>
<exclude>${exclude.tests}</exclude>
Expand Down Expand Up @@ -368,11 +368,24 @@ To export to S3:
</dependency>

<dependency>
<groupId>org.mockito</groupId>
<!-- "inline" currently required in order to mock constructor calls. -->
<artifactId>mockito-inline</artifactId>
<version>3.10.0</version>
<scope>test</scope>
<groupId>org.mock-server</groupId>
<artifactId>mockserver-junit-rule-no-dependencies</artifactId>
<version>5.14.0</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.immutables</groupId>
<artifactId>value</artifactId>
<version>2.9.3</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>32.1.2-jre</version>
<scope>test</scope>
</dependency>
</dependencies>
</project>
1 change: 1 addition & 0 deletions clients/hadoopfs/src/main/java/io/lakefs/Constants.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ public class Constants {
public static final String ENDPOINT_KEY_SUFFIX = "endpoint";
public static final String LIST_AMOUNT_KEY_SUFFIX = "list.amount";
public static final String ACCESS_MODE_KEY_SUFFIX = "access.mode";
public static final String SESSION_ID = "session_id";

public static enum AccessMode {
SIMPLE,
Expand Down
5 changes: 5 additions & 0 deletions clients/hadoopfs/src/main/java/io/lakefs/LakeFSClient.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,11 @@ public LakeFSClient(String scheme, Configuration conf) throws IOException {
basicAuth.setUsername(accessKey);
basicAuth.setPassword(secretKey);

String sessionId = FSConfiguration.get(conf, scheme, Constants.SESSION_ID);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the purpose of this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recommended by MockServer docs. You put a cookie that defines a session ID header on all requests. This lets you debug MockServer outputs.

Why? MockServer is a separate process, it logs a lot but doesn't even know which test is running. The session-ID cookie make it possible for a dev to correlate its output with the test run. I've found it very useful.

if (sessionId != null) {
apiClient.addDefaultCookie("sessionId", sessionId);
}

this.objectsApi = new ObjectsApi(apiClient);
this.stagingApi = new StagingApi(apiClient);
this.repositoriesApi = new RepositoriesApi(apiClient);
Expand Down
1 change: 1 addition & 0 deletions clients/hadoopfs/src/main/java/io/lakefs/LakeFSFileSystem.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -421,6 +421,7 @@ private boolean renameFile(LakeFSFileStatus srcStatus, Path dst) throws IOExcept
LOG.debug("renameFile: dst {} exists and is a {}", dst, dstFileStatus.isDirectory() ? "directory" : "file");
if (dstFileStatus.isDirectory()) {
dst = buildObjPathOnExistingDestinationDir(srcStatus.getPath(), dst);
LOG.debug("renameFile: use {} to create dst {}", srcStatus.getPath(), dst);
}
} catch (FileNotFoundException e) {
LOG.debug("renameFile: dst does not exist, renaming src {} to a file called dst {}",
Expand Down
Loading
Loading