Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IO Implementation using Go CDK #176

Merged
merged 14 commits into from
Dec 16, 2024
Merged

IO Implementation using Go CDK #176

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
7baec1f
IO Implementation using Go CDK
loicalleyne Oct 16, 2024
9e89fa9
fix comment
loicalleyne Oct 16, 2024
69c1e2a
refactor CreateBlobFileIO
loicalleyne Oct 17, 2024
b89431d
line endings to LF
loicalleyne Nov 14, 2024
77e216a
remove deprecated endpoint resolver
dwilson1988 Dec 1, 2024
015f893
unexport createBlobFileIO
dwilson1988 Dec 11, 2024
251985c
remove unused urlToBucketPath
loicalleyne Dec 11, 2024
9301166
reinit go.mod go.sum
loicalleyne Dec 12, 2024
b3bb876
run go mod tidy to update deps
zeroshade Dec 12, 2024
f38e790
clean-up and fixing buckets
zeroshade Dec 13, 2024
bdac9d4
somehow lost the license in go.mod
zeroshade Dec 13, 2024
39c7baa
fix spacing due to upgraded dependency
zeroshade Dec 13, 2024
0b37479
missed a close for the parquet reader
zeroshade Dec 13, 2024
6861e09
add config option to force s3 virtual addressing
zeroshade Dec 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 19 additions & 20 deletions cmd/iceberg/output_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,13 +63,13 @@ var testArgs = []struct {
"snapshot-log": [ ],
"metadata-log": [ ],
"refs": { }
}`,
`Table format version | 2
Metadata location |
}`,
`Table format version | 2
Metadata location |
Table UUID | 9c12d441-03fe-4693-9a96-a0705ddf69c1
Last updated | 1602638573590
Sort Order | 0: []
Partition Spec | []
Last updated | 1602638573590
Sort Order | 0: []
Partition Spec | []

Current Schema, id=0
└──1: x: required long
Expand All @@ -79,7 +79,7 @@ Current Snapshot |
Snapshots

Properties
key | value
key | value
----------------------------------
read.split.target.size | 134217728

Expand Down Expand Up @@ -145,18 +145,18 @@ read.split.target.size | 134217728
],
"metadata-log": [{"metadata-file": "s3://bucket/.../v1.json", "timestamp-ms": 1515100}],
"refs": {"test": {"snapshot-id": 3051729675574597004, "type": "tag", "max-ref-age-ms": 10000000}}
}`,
`Table format version | 2
Metadata location |
}`,
`Table format version | 2
Metadata location |
Table UUID | 9c12d441-03fe-4693-9a96-a0705ddf69c1
Last updated | 1602638573590
Sort Order | 3: [
| 2 asc nulls-first
| bucket[4](3) desc nulls-last
| ]
Partition Spec | [
| 1000: x: identity(1)
| ]
Last updated | 1602638573590
Sort Order | 3: [
| 2 asc nulls-first
| bucket[4](3) desc nulls-last
| ]
Partition Spec | [
| 1000: x: identity(1)
| ]

Current Schema, id=1
├──1: x: required long
Expand All @@ -170,14 +170,13 @@ Snapshots
└──Snapshot 3055729675574597004, schema 1: s3://a/b/2.avro

Properties
key | value
key | value
----------------------------------
read.split.target.size | 134217728

`},
}


func TestDescribeTable(t *testing.T) {
var buf bytes.Buffer
pterm.SetDefaultOutput(&buf)
Expand Down
71 changes: 50 additions & 21 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,33 +24,42 @@ toolchain go1.23.2
require (
github.com/apache/arrow-go/v18 v18.0.1-0.20241029153821-f0c5d9939d3f
github.com/aws/aws-sdk-go-v2 v1.32.6
github.com/aws/aws-sdk-go-v2/config v1.28.5
github.com/aws/aws-sdk-go-v2/credentials v1.17.46
github.com/aws/aws-sdk-go-v2/service/glue v1.102.0
github.com/aws/aws-sdk-go-v2/config v1.28.6
github.com/aws/aws-sdk-go-v2/credentials v1.17.47
github.com/aws/aws-sdk-go-v2/service/glue v1.103.0
github.com/aws/aws-sdk-go-v2/service/s3 v1.71.0
github.com/aws/smithy-go v1.22.1
github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815
github.com/google/uuid v1.6.0
github.com/hamba/avro/v2 v2.27.0
github.com/pterm/pterm v0.12.79
github.com/pterm/pterm v0.12.80
github.com/stretchr/testify v1.10.0
github.com/substrait-io/substrait-go v1.2.0
github.com/twmb/murmur3 v1.1.8
github.com/wolfeidau/s3iofs v1.5.2
gocloud.dev v0.40.0
golang.org/x/sync v0.10.0
google.golang.org/api v0.211.0
gopkg.in/yaml.v3 v3.0.1
)

require (
atomicgo.dev/cursor v0.2.0 // indirect
atomicgo.dev/keyboard v0.2.9 // indirect
atomicgo.dev/schedule v0.1.0 // indirect
cloud.google.com/go v0.116.0 // indirect
cloud.google.com/go/auth v0.12.1 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.6 // indirect
cloud.google.com/go/compute/metadata v0.5.2 // indirect
cloud.google.com/go/iam v1.2.2 // indirect
cloud.google.com/go/storage v1.43.0 // indirect
github.com/JohnCGriffin/overflow v0.0.0-20211019200055-46fa312c352c // indirect
github.com/alecthomas/participle/v2 v2.1.0 // indirect
github.com/andybalholm/brotli v1.1.1 // indirect
github.com/apache/thrift v0.21.0 // indirect
github.com/aws/aws-sdk-go v1.55.5 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.7 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.20 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.21 // indirect
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.11 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.25 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.25 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1 // indirect
Expand All @@ -59,26 +68,35 @@ require (
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.4.6 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.6 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.18.6 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.24.6 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.5 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.33.1 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.24.7 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.6 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.33.2 // indirect
github.com/containerd/console v1.0.3 // indirect
github.com/creasty/defaults v1.8.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/fatih/color v1.15.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/goccy/go-json v0.10.3 // indirect
github.com/goccy/go-yaml v1.11.0 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/flatbuffers v24.3.25+incompatible // indirect
github.com/google/s2a-go v0.1.8 // indirect
github.com/google/wire v0.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.4 // indirect
github.com/googleapis/gax-go/v2 v2.14.0 // indirect
github.com/gookit/color v1.5.4 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/asmfmt v1.3.2 // indirect
github.com/klauspost/compress v1.17.11 // indirect
github.com/klauspost/cpuid/v2 v2.2.8 // indirect
github.com/klauspost/cpuid/v2 v2.2.9 // indirect
github.com/lithammer/fuzzysearch v1.1.8 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.19 // indirect
github.com/mattn/go-runewidth v0.0.15 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.16 // indirect
github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8 // indirect
github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
Expand All @@ -91,15 +109,26 @@ require (
github.com/substrait-io/substrait v0.57.1 // indirect
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
github.com/zeebo/xxh3 v1.0.2 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.54.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.54.0 // indirect
go.opentelemetry.io/otel v1.29.0 // indirect
go.opentelemetry.io/otel/metric v1.29.0 // indirect
go.opentelemetry.io/otel/trace v1.29.0 // indirect
golang.org/x/crypto v0.30.0 // indirect
golang.org/x/exp v0.0.0-20240909161429-701f63a606c0 // indirect
golang.org/x/mod v0.21.0 // indirect
golang.org/x/net v0.30.0 // indirect
golang.org/x/sys v0.26.0 // indirect
golang.org/x/term v0.25.0 // indirect
golang.org/x/text v0.19.0 // indirect
golang.org/x/tools v0.26.0 // indirect
golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 // indirect
golang.org/x/mod v0.22.0 // indirect
golang.org/x/net v0.32.0 // indirect
golang.org/x/oauth2 v0.24.0 // indirect
golang.org/x/sys v0.28.0 // indirect
golang.org/x/term v0.27.0 // indirect
golang.org/x/text v0.21.0 // indirect
golang.org/x/time v0.8.0 // indirect
golang.org/x/tools v0.28.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
google.golang.org/genproto v0.0.0-20241113202542-65e8d215514f // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20241118233622-e639e219e697 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20241206012308-a4fef0638583 // indirect
google.golang.org/grpc v1.67.1 // indirect
google.golang.org/protobuf v1.35.1 // indirect
google.golang.org/protobuf v1.35.2 // indirect
)
Loading
Loading