Return token expiry with presigned URLs (#6337)

* Add "ExpiresAt" to S3 clients * Log expiry time of clients when fetching from cache * Add expiry time to presigned URLs to adaptors Only S3 works, the others just return a zero time.Time for "not implemented". * Add presigned URL expiry to API * Return expiry time on presigned URLs * Lint fixes * Report physical address expiry in `lakectl fs stat --pre-signed` Useful, and also lets me test that a deployment on lakeFS.Cloud staging works! >:-) * Select good golden file for `lakectl fs stat --pre-sign` test Some lakeFS block adaptors report expiry, others do not. Test lakectl using an appropriate golden file * [CR] Add TODO to support expiry on Azure & GCS Also fix tpyo.
treeverse · Aug 8, 2023 · accced2 · accced2
1 parent 1255b6e
commit accced2
Show file tree

Hide file tree

Showing 28 changed files with 338 additions and 51 deletions.
diff --git a/api/swagger.yml b/api/swagger.yml
@@ -280,6 +280,16 @@ components:
             The location of the object on the underlying object store.
             Formatted as a native URI with the object store type as scheme ("s3://...", "gs://...", etc.)
             Or, in the case of presign=true, will be an HTTP URL to be consumed via regular HTTP GET
+        physical_address_expiry:
+          type: integer
+          format: int64
+          description: |
+            If present and nonzero, physical_address is a presigned URL and
+            will expire at this Unix Epoch time.  This will be shorter than
+            the presigned URL lifetime if an authentication token is about
+            to expire.
+
+            This field is *optional*.
         checksum:
           type: string
         size_bytes:
@@ -1139,7 +1149,16 @@ components:
           type: string
           nullable: true
           description: if presign=true is passed in the request, this field will contain a presigned URL to use when uploading
+        presigned_url_expiry:
+          type: integer
+          format: int64
+          description: |
+            If present and nonzero, physical_address is a presigned URL and
+            will expire at this Unix Epoch time.  This will be shorter than
+            the presigned URL lifetime if an authentication token is about
+            to expire.
 
+            This field is *optional*.
 
     StagingMetadata:
       type: object

diff --git a/clients/java/api/openapi.yaml b/clients/java/api/openapi.yaml
diff --git a/clients/java/docs/ObjectStats.md b/clients/java/docs/ObjectStats.md
diff --git a/clients/java/docs/StagingLocation.md b/clients/java/docs/StagingLocation.md
diff --git a/clients/java/src/main/java/io/lakefs/clients/api/model/ObjectStats.java b/clients/java/src/main/java/io/lakefs/clients/api/model/ObjectStats.java
diff --git a/clients/java/src/main/java/io/lakefs/clients/api/model/StagingLocation.java b/clients/java/src/main/java/io/lakefs/clients/api/model/StagingLocation.java
diff --git a/clients/java/src/test/java/io/lakefs/clients/api/model/ObjectStatsTest.java b/clients/java/src/test/java/io/lakefs/clients/api/model/ObjectStatsTest.java
diff --git a/clients/java/src/test/java/io/lakefs/clients/api/model/StagingLocationTest.java b/clients/java/src/test/java/io/lakefs/clients/api/model/StagingLocationTest.java
diff --git a/clients/python/docs/ObjectStats.md b/clients/python/docs/ObjectStats.md
diff --git a/clients/python/docs/StagingApi.md b/clients/python/docs/StagingApi.md
diff --git a/clients/python/docs/StagingLocation.md b/clients/python/docs/StagingLocation.md
diff --git a/clients/python/lakefs_client/model/object_stats.py b/clients/python/lakefs_client/model/object_stats.py
diff --git a/clients/python/lakefs_client/model/staging_location.py b/clients/python/lakefs_client/model/staging_location.py