v4.1.1

🎉 New features

• Adds dynamic wait based on rate-limit response header by @cmgrote in #1166
• Adds SnowflakeDynamicTable as possible parentType for columns by @ErnestoLoma in #1132
• Adds get operation to retrieve selective relationships and attributes by @cmgrote in #1170
• Adds support for Anaplan assets by @prateekrai-atlan in #1177
• Adds four new AWS connectors by @rupeshatlan in #1179

🐞 Bug fixes

• Modifies workflow findByType to use regular expression to avoid false positive results by @ErnestoLoma in #1183

📦 Packages

• Fixes typedef toolkit to generate appropriate AttributeDefs on RelationshipDefs by @cmgrote in #1174
• Allows OpenAPI spec loader to support loading multiple JSON files by @pavanmanishd in #1113
• Adds interchangeable Excel and CSV options for export packages by @cmgrote in #1181
• Changes Python package rendering to support Open Telemetry logging by @ErnestoLoma in #1182

New Contributors

• @pavanmanishd made their first contribution in #1113

Full Changelog: v4.1.0...v4.1.1

v4.1.0

🎉 New features

• Adds logging of underlying causes for back-end errors by @cmgrote in #1098
• Adds new connector BIGID by @rupeshatlan in #1112

📦 Packages

• Adds OpenTelemetry logging for all Kotlin-based packages by @cmgrote in #1114
• Adds an uncaught exception handler so any unexpected errors are caught and logged by @cmgrote
• Fixes cyclical relationship detection in asset import by @cmgrote in #1107
• Fixes persistent connection cache population by @cmgrote in #1110
• Improves logging on any CSV decoding errors by @cmgrote in #1109
• Updates packages to use latest version of image by @ErnestoLoma in #1131

🥗 QOL improvements

• Parallelizes package tests by @cmgrote in #1117

New Contributors

• @rupeshatlan made their first contribution in #1112

Full Changelog: v4.0.3...v4.1.0

v4.0.3

📦 Packages

• Fixes bug where URL-based ingestion of OpenAPI spec would fail by @cmgrote in #1097
• Changes level to DEBUG for logger_root in Python packages by @ErnestoLoma in #1094

Full Changelog: v4.0.2...v4.0.3

v4.0.2

🐞 Bug fixes

• Fixes potential NPE in user listing iteration by @cmgrote in #1087
• Fixes bug that attempted to process non-existent input files by @cmgrote in #1095

Full Changelog: v4.0.1...v4.0.2

v4.0.1

This is essentially a rerelease of 4.0.0, so full release notes included here for simplicity:

⛑️ Breaking changes

• Removes a number of static objects, in particular the "default" Atlan client that was accessible through Atlan.getDefaultClient() by @cmgrote in #1072. Every operation that interacts with Atlan (ultimately sending it an API request) must now be explicitly passed an AtlanClient. This change was made for two primary reasons:
  1. It makes explicit which operations will interact with Atlan (those that require an AtlanClient to be passed) vs those that only operate on information locally in-memory (those that do not require an AtlanClient to be passed).
  2. It removes concurrency limitations and risks with shared static resources across the entire JVM (including all threads therein). This applies not only to the default client but also to some of the caches that were being automatically managed.
• Packages developed using this release of the package toolkit will now need to make use of a broader PackageContext object that similarly encapsulates the client and any extra caches specific to packages.

🧪 Experimental

• Adds limited ability to configure an SSO provider by @cmgrote in #1069

🐞 Bug fixes

• Fixes validations when creating a README by @cmgrote in #1059

📦 Packages

• Adds multi-pass handling of cyclical relationships to asset import package by @cmgrote in #1061
• Fixes Python outputs generated by package toolkit by @ErnestoLoma in #1068
• Adds the ability in the package toolkit to set default values into the configuration objects by defining a fallback for any inputs in the package.pkl by @cmgrote in #1072
• Fixes impersonation within packages by @cmgrote in #1084
• Adds further type-checking on input CSVs by @cmgrote in #1086
• Fixes potential NPE in user iteration listing by @cmgrote in #1087

🥗 QOL improvements

• Migrates bulk streaming of search to use search_after by @cmgrote in #1060

Full Changelog: v3.1.2...v4.0.1

AtlanClient changes

As mentioned in the breaking changes, this release introduces a complete revamp of the way the AtlanClient is managed and passed around to operations that actually call Atlan APIs behind-the-scenes.

Resource-isolated

Previously the client itself and some caches were managed statically across the entire JVM. This could cause non-obvious and very difficult reproduce issues when heavy concurrency exists. With this release, all caches are managed within the AtlanClient directly, and there is no longer a JVM-wide static client. Instead, you can create a new client and have it "live" only for as long as you need it:

try (AtlanClient client = new AtlanClient()) {
    // Do anything you need against Atlan within this block
}
// Once you exit the block, the client and all its internal caches will be removed

Must be explicitly passed

Furthermore, any operation that actually interacts with Atlan directly (ultimately via its APIs behind-the-scenes) must now be explicitly passed such a client:

GlossaryTerm term = GlossaryTerm.findByName(client, "A", "B");
term.trimToRequired().userDescription("Hi there!").build().save(client);

This means there is an extensive set of changes to many of the methods across the SDK — if you're planning to adopt this release (and higher), rather than reading a huge list of such operations it'll probably be simplest to update your dependency to this latest release in your codebase and then look for any compilation breakages. The explicit AtlanClient is almost always expected as the new first parameter in any methods that now require it.

Package toolkit

Introduction of PackageContext

In the same way we have resource-isolated the client and its caches for the SDK, we have resource-isolated the context of a running package and its further caches in this release. The full context needed for running a package is now bundled into an object called a PackageContext, which can be created from a package's configuration using Utils.initializeContext() and passing it te configuration. Where you previously may have just used this pattern:

@JvmStatic
fun main(args: Array<String>) {
    val config = Utils.setPackageOps<OpenAPISpecLoaderCfg>()
    // Receive inputs from the config to decide what to do in your logic
}

You should now use this pattern:

@JvmStatic
fun main(args: Array<String>) {
    Utils.initializeContext<OpenAPISpecLoaderCfg>().use { ctx ->
        // Receive inputs from the config to decide what to do in your logic
        val url = ctx.config.specUrl // config items will be available nested under ctx.config
        val client = ctx.client // the AtlanClient will be nested under ctx.client
        val termCache = ctx.termCache // a number of lazy-load-managed caches are also available in the context
    }
}

v4.0.0

⛑️ Breaking changes

• Removes a number of static objects, in particular the "default" Atlan client that was accessible through Atlan.getDefaultClient() by @cmgrote in #1072. Every operation that interacts with Atlan (ultimately sending it an API request) must now be explicitly passed an AtlanClient. This change was made for two primary reasons:
  1. It makes explicit which operations will interact with Atlan (those that require an AtlanClient to be passed) vs those that only operate on information locally in-memory (those that do not require an AtlanClient to be passed).
  2. It removes concurrency limitations and risks with shared static resources across the entire JVM (including all threads therein). This applies not only to the default client but also to some of the caches that were being automatically managed.
• Packages developed using this release of the package toolkit will now need to make use of a broader PackageContext object that similarly encapsulates the client and any extra caches specific to packages.

🧪 Experimental

• Adds limited ability to configure an SSO provider by @cmgrote in #1069

🐞 Bug fixes

• Fixes validations when creating a README by @cmgrote in #1059

📦 Packages

• Adds multi-pass handling of cyclical relationships to asset import package by @cmgrote in #1061
• Fixes Python outputs generated by package toolkit by @ErnestoLoma in #1068
• Adds the ability in the package toolkit to set default values into the configuration objects by defining a fallback for any inputs in the package.pkl by @cmgrote in #1072

🥗 QOL improvements

• Migrates bulk streaming of search to use search_after by @cmgrote in #1060

Full Changelog: v3.1.2...v4.0.0

AtlanClient changes

As mentioned in the breaking changes, this release introduces a complete revamp of the way the AtlanClient is managed and passed around to operations that actually call Atlan APIs behind-the-scenes.

Resource-isolated

Previously the client itself and some caches were managed statically across the entire JVM. This could cause non-obvious and very difficult reproduce issues when heavy concurrency exists. With this release, all caches are managed within the AtlanClient directly, and there is no longer a JVM-wide static client. Instead, you can create a new client and have it "live" only for as long as you need it:

try (AtlanClient client = new AtlanClient()) {
    // Do anything you need against Atlan within this block
}
// Once you exit the block, the client and all its internal caches will be removed

Must be explicitly passed

Furthermore, any operation that actually interacts with Atlan directly (ultimately via its APIs behind-the-scenes) must now be explicitly passed such a client:

GlossaryTerm term = GlossaryTerm.findByName(client, "A", "B");
term.trimToRequired().userDescription("Hi there!").build().save(client);

This means there is an extensive set of changes to many of the methods across the SDK — if you're planning to adopt this release (and higher), rather than reading a huge list of such operations it'll probably be simplest to update your dependency to this latest release in your codebase and then look for any compilation breakages. The explicit AtlanClient is almost always expected as the new first parameter in any methods that now require it.

Package toolkit

Introduction of PackageContext

In the same way we have resource-isolated the client and its caches for the SDK, we have resource-isolated the context of a running package and its further caches in this release. The full context needed for running a package is now bundled into an object called a PackageContext, which can be created from a package's configuration using Utils.initializeContext() and passing it te configuration. Where you previously may have just used this pattern:

@JvmStatic
fun main(args: Array<String>) {
    val config = Utils.setPackageOps<OpenAPISpecLoaderCfg>()
    // Receive inputs from the config to decide what to do in your logic
}

You should now use this pattern:

@JvmStatic
fun main(args: Array<String>) {
    Utils.initializeContext<OpenAPISpecLoaderCfg>().use { ctx ->
        // Receive inputs from the config to decide what to do in your logic
        val url = ctx.config.specUrl // config items will be available nested under ctx.config
        val client = ctx.client // the AtlanClient will be nested under ctx.client
        val termCache = ctx.termCache // a number of lazy-load-managed caches are also available in the context
    }
}

v3.1.2

🐞 Bug fixes

• Fixes bug that caused full delta calculation in relational and cube assets builders to miss net-new assets by @cmgrote in #1055

Full Changelog: v3.1.1...v3.1.2

v3.1.1

🎉 New features

• Adds immediateNeighbors to FluentLineage to simplify lineage retrieval by @pavanmanish-atlan in #1048

🐞 Bug fixes

• Fixes bug with automated token exchange on expired API token retries by @cmgrote in #1043

📦 Packages

• Corrects downloads for GCS and ADLS deltas by @cmgrote in #1041
• Adds full delta calculation logic for relational and cube assets builders by @cmgrote in #1044
• Prevents renaming of connections as part of the enrichment migrator by @ErnestoLoma in #1049

New Contributors

• @pavanmanish-atlan made their first contribution in #1048

Full Changelog: v3.0.1...v3.1.0

v3.0.1

🎉 New features

• Adds username and extra (non-sensitive) fields to credential response by @cmgrote in #1026

🧪 Experimental

• Adds application typedefs and further data model typedef enhancements by @cmgrote in #1035

🐞 Bug fixes

• Corrects typo that caused CategoryCache to be loaded incorrectly by @cmgrote in #1030
• Avoids page overruns in edge cases where total results is very nearly the limit before query rewriting needs to happen by @cmgrote in #1033

📦 Packages

• Adds case insensitivity and table view agnosticism options to enrichment migrator by @ErnestoLoma in #1032
• Fixes bug where a purpose with no tag could cause admin export to fail by @cmgrote in #1036

Full Changelog: v3.0.0...v3.0.1

v3.0.0

🎉 New features

• Adds off-heap caching to reduce memory footprint by @cmgrote in #984
• Parses precision, scale, max length and raw definition from SQL dataType by @cmgrote in #1006
• Adds new connector types by @Aryamanz29 in #1005

🧪 Experimental

• Adds a new Application type with creator and unit tests by @prateekrai-atlan in #1008

⛑️ Breaking changes

• Previously, there were at least 2-3 separate approaches used for caching across the SDK and custom package toolkits, despite all of them relying purely on on-heap memory. In this release caches have been revamped:
  • to be more consistent (same operations across caches)
  • to move them off-heap (with disk spilling)
• We have tried to minimize the impact of this change, but if you are using these caches directly (or related custom package runtime toolkit functionality) you may be impacted. See the end of the release notes for further details.
• The custom package testing toolkit's interface has also been changed slightly:
  • The PackageTest class now requires a "tag" parameter, that is used in all generated names for that test to make them more immediately identifiable.
  • Rather than calling the setup() method with your custom package's configuration and then directly invoking your custom package's code (typically via its main() method), you should now use the runCustomPackage() method — which takes both the configuration and your custom package's execution method as parameters. This ensures your custom package is executed with appropriate isolation from any others that may be tested in parallel.

🐞 Bug fixes

• Fixes an issue in asset batching related to case-insensitivity handling by @ErnestoLoma in #990

📦 Packages

• Fixes a dependency issue on Google Cloud by @ErnestoLoma in #974
• Fixes container name used on Azure by @ErnestoLoma in #1013
• Passes through failOnErrors in relational assets builder, to allow continuation beyond failures by @cmgrote in #994
• Asset import improvements by @cmgrote in #1014
• Extends off-heap caching to all custom package asset caches by @cmgrote in #988
• Fixes generalization-specialization relationship attribute names for data model assets by @Aryamanz29 in #975

🥗 QOL improvements

• Extends off-heap caching to asset batch processing tracking by @cmgrote in #988

Full Changelog: v2.3.2...v3.0.0

Caching changes

As mentioned in the breaking changes, this release introduces a complete revamp of the way caching is done across both the SDK and custom package toolkits.

Moved off-heap

Previously the caching was done entirely in-memory, on-heap. This naturally limited its scale — to the point where we had begun to observe OutOfMemory errors in some cases. With this release, all caches are moved off-heap and managed by an embedded, persistent key-value store that still offers high performance. (Note that this does mean some temporary files will be created locally to manage the caches — these are created using Files.createTempDirectory(), so the specific location will depend on how the JVM manages this on your system.)

Consistency changes

We've consolidated 2-3 different interfaces for caching that had evolved independently during the lifecycle of the SDK and package toolkit to date, resolving that tech debt so that all can directly benefit from the move to off-heap.

• Each cache now manages the following for each object in it:
  • an ID (always a UUID)
  • a human-readable name
  • an optional secondary ID (such as the hashed-string internal representation for tags and custom metadata)
  • the actual object that was cached
• Retrieval now uses these options almost exclusively, for consistency. For example, instead of .getCustomMetadataDef() on the custom metadata cache, you would now just use .getByName().
• Prior operations like lookupAssetByIdentity and lookupAssetByGuid in some cache variations have been made consistent, to lookupByIdentity and lookupById.
• Wherever possible the caches are now implemented using Java generics typing, so should return strong types without needing casting (rather than generic types like Asset).

Necessary interface changes

Various places that used to make use of collections (Collection<Asset>, List<Asset>, etc) now instead either directly make use of the new OffHeapAssetCache or of Stream<Asset> — to no longer require everything in-memory on the heap in order to be able to process it. All caches are now generally Closeable, too, to allow them to self-manage cleaning up any temporary files they use for persistence.

• Utils.updateConnectionCache() in the custom package runtime toolkit (now uses OffHeapAssetCache)
• PersistentConnectionCache.deleteAssets() and .addAssets() in the custom package runtime toolkit (now process Stream<Asset>)
• AssetGenerator.cacheCreated() interface (now processes Stream<Asset>)
• ImportResults.Details internal tracking of created, updated, restored and skipped assets (now use OffHeapAssetCache)
• AssetRemover.deleteAssets() in the custom package runtime toolkit (now uses OffHeapAssetCache)
• DeltaProcessor.run() in the custom package runtime toolkit (now uses OffHeapAssetCache)
• AtlanClient itself is now Closeable, so it can clean up its own managed caches
• AssetBatch and ParallelBatch no longer offer fluent-builder creation, but only overloaded constructors, and both now track their processing of created, updated, restored and skipped assets in OffHeapAssetCaches, and are now Closeable so they can clean up their own managed caches.