Add support for Versions Catalogs `gradle/libs.versions.toml` #509

jmfayard · 2022-02-25T19:47:29Z

Hello & bonjour,

What?

Resolves #333

Add support for Versions Catalogs gradle/libs.versions.toml in all our tasks (refreshVersions, refreshVersionsMigrate, refreshVersionsCleanup, refreshVersionsCatalog)

Why?

First class support for Gradle versions catalogs, removing fear of plugin lock-in for our users.

Testing?

You can test it in your own project, because I published a snapshot version on Sonatype

Setup refreshVersions "0.41.0-SNAPSHOT" as in https://jmfayard.github.io/refreshVersions/setup/#if-you-want-to-use-a-snapshot-version

Then:

# Versions Catalog are stable in Gradle 7.4
$ ./gradlew wrapper --gradle-version 7.4

# Generate the versions catalog
$ ./gradlew -refresh-dependencies refreshVersionsCatalog --versions --all
        modified:   versions.properties
        modified:   gradle/libs.versions.toml

# Migrate your build.gradle(.kts)
$ ./gradlew refreshVersionsMigrate --toml
        modified:   /Users/jmfayard/tignum/backend-tignum-x/build.gradle.kts

# Show dependency updates in gradle/libs.versions.toml
$ ./gradlew refreshVersions
        modified:   versions.properties
        modified:   gradle/libs.versions.toml

How?

If your project is using Gradle 7.0, you can use instead [Gradle versions catalog](https://docs.gradle.org/current/userguide/platforms.html) ```bash $ ./gradlew refreshVersionsCatalog > Task :refreshVersionsCatalog new file: gradle/libs.versions.toml modified: settings.gradle.kts modified: versions.properties ``` The generated versions catalog looks like this: ```toml ## Generated by $ ./gradlew refreshVersionsCatalog [libraries] browser = "androidx.browser:browser:_" cardview = "androidx.cardview:cardview:_" okhttp = "com.squareup.okhttp3:okhttp:_" junit = "junit:junit:_" ``` Like with buildSrcLibs, you get type-safe accessors available in the IDE: <img width="949" alt="refreshVersions_–_build_gradle_kts__sample-kotlin_" src="https://user-images.githubusercontent.com/459464/128611606-4ce90b01-475d-402a-b223-1757a1c51deb.png">

# Conflicts: # sample-kotlin/build.gradle.kts # sample-kotlin/gradle/wrapper/gradle-wrapper.properties # sample-kotlin/versions.properties

yogurtearl · 2022-02-25T23:57:53Z

Cool! Tried out the snapshot. :)

it inlines all the versions into [libraries], but also leaves the old versions unmodified under [versions]

[versions]
slf4j = "1.7.30"
log4j2 = "2.17.1"
kotlinLogging = "2.0.10"

[libraries]
slf4j-api = { module = "org.slf4j:slf4j-api", version.ref = "slf4j" }
slf4j-log4j12 = { module = "org.apache.logging.log4j:log4j-slf4j-impl", version.ref = "log4j2" }
log4j2 = { module = "org.apache.logging.log4j:log4j-core", version.ref = "log4j2" }
kotlin-logging = { module = "io.github.microutils:kotlin-logging-jvm", version.ref = "kotlinLogging" }

[bundles]
kotlinSlf4jLog4j2 = ["slf4j-api", "slf4j-log4j12", "log4j2", "kotlin-logging"]

if I run ./gradlew refreshVersionsCatalog --versions --all --no-configuration-cache

it outputs something like this:

[versions]
slf4j = "1.7.29"
log4j2 = "2.17.1"
kotlinLogging = "2.0.9"

[libraries]

kotlin-logging-jvm = "io.github.microutils:kotlin-logging-jvm:2.0.10"

log4j-core = "org.apache.logging.log4j:log4j-core:2.17.1"

log4j-slf4j-impl = "org.apache.logging.log4j:log4j-slf4j-impl:2.17.1"

slf4j-api = "org.slf4j:slf4j-api:1.7.30"

[bundles]
kotlinSlf4jLog4j2 = ["slf4j-api", "slf4j-log4j12", "log4j2", "kotlin-logging"]

Now we have duplicated versions and the kotlinSlf4jLog4j2 bundle reference a slf4j-log4j12 library that no longer exists.
Also, the next build fails with:

* What went wrong:
org.gradle.api.InvalidUserDataException: Invalid catalog definition:
  - Problem: In version catalog libs, a bundle with name 'kotlinSlf4jLog4j2' declares a dependency on 'slf4j.log4j12' which doesn't exist.
    
    Reason: Bundles can only contain references to existing library aliases.
    
    Possible solutions:
      1. Make sure that the library alias 'slf4j.log4j12' is declared.
      2. Remove 'slf4j.log4j12' from bundle 'kotlinSlf4jLog4j2'.
    
    Please refer to https://docs.gradle.org/7.4/userguide/version_catalog_problems.html#undefined_alias_reference for more details about this problem.

Maybe it should remove the [versions] section and update the bundles with the new names?

yogurtearl · 2022-02-25T23:58:50Z

with something like this:

[versions]
kotlin = "1.6.0"

[plugins]
kotlin-gradle = { id = "org.jetbrains.kotlin.jvm", version.ref = "kotlin" }

it seems to just ignore the [plugins] block.

yogurtearl · 2022-02-25T23:59:36Z

Also, it doesn't seem to work nicely with Configuration cache.
https://docs.gradle.org/7.4/userguide/configuration_cache.html#config_cache:requirements:use_project_during_execution

invocation of Task.project📋 at execution time is unsupported.[ ?](https://docs.gradle.org/7.4/userguide/configuration_cache.html#config_cache:requirements:use_project_during_execution)
⌄task:refreshVersionsCatalog📋 of type de.fayard.refreshVersions.RefreshVersionsCatalogTask📋
⌄exception stack trace 📋
org.gradle.api.InvalidUserCodeException: Invocation of 'Task.project' by task ':refreshVersionsCatalog' at execution time is unsupported.
	at org.gradle.configurationcache.initialization.DefaultConfigurationCacheProblemsListener.onTaskExecutionAccessProblem(ConfigurationCacheProblemsListener.kt:74)
	at org.gradle.configurationcache.initialization.DefaultConfigurationCacheProblemsListener.onProjectAccess(ConfigurationCacheProblemsListener.kt:53)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.event.DefaultListenerManager$ListenerDetails.dispatch(DefaultListenerManager.java:464)
	at org.gradle.internal.event.DefaultListenerManager$ListenerDetails.dispatch(DefaultListenerManager.java:446)
	at org.gradle.internal.event.AbstractBroadcastDispatch.dispatch(AbstractBroadcastDispatch.java:61)
	at org.gradle.internal.event.DefaultListenerManager$EventBroadcast$ListenerDispatch.dispatch(DefaultListenerManager.java:434)
	at org.gradle.internal.event.DefaultListenerManager$EventBroadcast.dispatch(DefaultListenerManager.java:221)
	at org.gradle.internal.event.DefaultListenerManager$EventBroadcast.dispatch(DefaultListenerManager.java:192)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at com.sun.proxy.$Proxy170.onProjectAccess(Unknown Source)
	at org.gradle.api.internal.AbstractTask.notifyProjectAccess(AbstractTask.java:1053)
	at org.gradle.api.internal.AbstractTask.getProject(AbstractTask.java:237)
	at org.gradle.api.DefaultTask.getProject(DefaultTask.java:59)
	at de.fayard.refreshVersions.RefreshVersionsCatalogTask.refreshVersionsCatalogAction(RefreshVersionsCatalogTask.kt:46)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)

yogurtearl · 2022-02-26T00:15:26Z

If I have something like this, it seems to just remove these libraries altogether.
These are only used in my buildSrc dir.

[libraries]
kotlin-gradle = { module = "org.jetbrains.kotlin:kotlin-gradle-plugin", version.ref = "kotlin" }
detekt-gradle = { module = "io.gitlab.arturbosch.detekt:detekt-gradle-plugin", version.ref = "detekt" }

yogurtearl · 2022-02-26T00:18:43Z

I have this in buildSrc/settings.gradle.kts to enable using the same version catalog in buildSrc and in my build:

dependencyResolutionManagement {
    versionCatalogs {
        create("libs") {
            from(files("../gradle/libs.versions.toml"))
        }
    }
}

and I use some of the [libraries] in buildSrc/build.gradle.kts

after I run
./gradlew refreshVersionsCatalog --versions --all --no-configuration-cache

when I run:
./gradlew refreshVersionsMigrate --no-configuration-cache

if fails because buildSrc/build.gradle.kts is referencing the old library names.

jmfayard · 2022-02-26T17:39:27Z

@yogurtearl thanks a lot for trying out the snapshot!

I initially thought the task ./gradlew refreshVersionsCatalog as a one-off thing that you use when you don't have versions catalog and then never use again. So yes, currently the libraries section is rewriting from scratch and the other are ignored. Which indeed create a build error next time if you already use the versions catalog. I will improve this.

Thanks for your reports on the configuration cache, I will have a look.

Now, we are testing with realistic data when it comes to the versions retrieved from the repos, and this will help ensure we are going in a code path that does the proper filtering. Also, we are now testing for the migration into the version catalog of dependencies that don't have a version specified (e.g. because specified by a BoM). Yes, the tests are currently failing.

The freshly introduced overload will be testable without having to perform network calls, which means we will be able to use static, and fake data. Most importantly, the new overload doesn't depend on global mutable state, but only from the arguments it has been passed.

We no longer allocate on each call. This commit also removes the unused parameter of the hasHardcodedVersion function which always had the default value of emptySet().

LouisCAD · 2022-08-08T14:01:49Z

Quick update:

I am a bit late on the ETA I provided, but we're almost done. Here's the proof that we're making progress:

As you can see, we're now filtering the available versions properly, and we support usage of versions.properties and versions catalog together.

Note that we currently don't support non-default catalogs yet.

Two things left blocking this PR:

Migration of version-less dependencies
Disabling the feature by default while we continue testing remaining edge cases thoroughly

Just one line to fix it, unbelievable!

This avoids unnecessary calculations

The problem was that the RefreshVersionsExtension was directly touching RefreshVersionsConfigHolder, whose instance is reused by Gradle across runs. Now, the rejectVersionIf predicate defaults to null, is taken each time the plugin is applied to be set on the RefreshVersionsConfigHolder, so if it's been removed, it's set back to null as it should.

In case someone relies on the reset behavior later (though current code doesn't), the behavior will be correct and will avoid potential memory leaks.

LouisCAD · 2022-08-09T18:57:44Z

I did further testing, and after my latest commits, it doesn't seem to introduce regressions, so I think we will keep the flag enabled by default.

…introduce test to ensure it's never empty, and properly formatted.

This optimization will make upgrading refreshVersions slightly faster when there's no potential dependency notations that have been removed since last run.

LouisCAD · 2022-08-09T23:50:17Z

I don't know why the CI is failing, since it's giving 404 Not Found errors when I try to look…

I reported the issue to GitHub support.

On my machine the check task runs fine.

I was ready to merge this.

LouisCAD · 2022-08-10T03:04:02Z

GitHub Actions got fixed, and I added the missing opt-in that is not required as the Gradle 7.5.1 update updated Kotlin.

Now is time to merge!

imashnake0 · 2022-08-10T03:16:00Z

pog

jmfayard added 18 commits August 7, 2021 21:26

Merge remote-tracking branch 'origin/main' into refreshVersionsCatalog

771db0c

# Conflicts: # sample-kotlin/build.gradle.kts # sample-kotlin/gradle/wrapper/gradle-wrapper.properties # sample-kotlin/versions.properties

refreshVersionsCatalog works again

12b57e6

Versions Catalog are stable in Gradle 7.4

7fd805a

versions catalog: use kebab-case

1cff52c

versions catalog: filter out built-in dependencies

9b7402d

sample-kotlin

f4b3f04

pr: self-review

0cdc912

ci: fix test

6515c49

./gradlew refreshVersionsMigrate completed

92b393d

Polish refreshVersionsCatalog

a3bae6e

feat(VersionsCatalog): parse TOML lines

7728fd2

feat(VersionsCatalog): Implement Toml Updater

e28ff07

feat(VersionsCatalog): Enable in task refreshVersions

f40b9e1

feat(VersionsCatalog): refreshVersionsCatalog --versions --all

5ded7bd

ci: fix test

ed6a4a9

tests: move tests to the right package

9d084a8

pr: self-review

499f1b4

jmfayard mentioned this pull request Feb 25, 2022

Add support for Gradle 7.4 Version Catalogs #333

Closed

feat: dependencies from versions catalog are not hard-coded

cb4ff90

jmfayard changed the base branch from refreshVersionsCatalog to main February 26, 2022 17:42

jmfayard mentioned this pull request Feb 26, 2022

Migrate instantly to gradle versions catalog with refreshVersions #504

Closed

6 tasks

jmfayard added 2 commits February 26, 2022 21:16

refactor: helpers for TomlLine

9732622

feat(refreshVersionsCatalog): add plugins

3f6450b

jmfayard linked an issue Aug 6, 2022 that may be closed by this pull request

Caveat: refreshVersions might show available updates in versions catalog for dependencies you don't actually use #571

Closed

LouisCAD added 9 commits August 7, 2022 00:52

Fix plural of local property

250e81c

Replace unclear acronym lambda param name with default

076df12

Skip getting the version catalog when unsupported

aa1f4d5

Optimize the fun Dependency.isManageableVersion function

dc87d7b

We no longer allocate on each call. This commit also removes the unused parameter of the hasHardcodedVersion function which always had the default value of emptySet().

Fix version filtering and unused version entries flagging

cb2e924

Add missing dash in KDoc

c85d1dd

Rename TomlUpdater to VersionsCatalogUpdater (and its test file)

cbbc2c8

LouisCAD added 6 commits August 9, 2022 16:46

Fix catalog migration for version-less dependencies

a3ccc1a

Just one line to fix it, unbelievable!

Don't compute versionKey for version-less dependencies

7a1ef79

This avoids unnecessary calculations

Reset RefreshVersionsConfigHolder prior to its initialization

daa2b2d

In case someone relies on the reset behavior later (though current code doesn't), the behavior will be correct and will avoid potential memory leaks.

Don't ignore dependencies used just in versionFor

f5d02d4

Transform TODO call into TODO comment

1d518e8

LouisCAD added 4 commits August 10, 2022 01:23

Fill back version-to-removals-revision-mapping.txt and…

4517729

…introduce test to ensure it's never empty, and properly formatted.

Gradle 7.5.1

cb1988b

Don't walk the Gradle files if removals history is empty

be49222

This optimization will make upgrading refreshVersions slightly faster when there's no potential dependency notations that have been removed since last run.

Use built-in dependency notations for JUnit

1d67eb7

Add required opt-in for ExperimentalTime in tests

2e31ae9

LouisCAD merged commit 0f0d63f into main Aug 10, 2022

LouisCAD deleted the update-toml branch August 10, 2022 03:04

LouisCAD restored the update-toml branch August 11, 2022 08:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Versions Catalogs `gradle/libs.versions.toml` #509

Add support for Versions Catalogs `gradle/libs.versions.toml` #509

jmfayard commented Feb 25, 2022 •

edited by LouisCAD

Loading

yogurtearl commented Feb 25, 2022

yogurtearl commented Feb 25, 2022

yogurtearl commented Feb 25, 2022

yogurtearl commented Feb 26, 2022

yogurtearl commented Feb 26, 2022

jmfayard commented Feb 26, 2022 •

edited by LouisCAD

Loading

LouisCAD commented Aug 8, 2022

LouisCAD commented Aug 9, 2022

LouisCAD commented Aug 9, 2022

LouisCAD commented Aug 10, 2022

imashnake0 commented Aug 10, 2022

Add support for Versions Catalogs gradle/libs.versions.toml #509

Add support for Versions Catalogs gradle/libs.versions.toml #509

Conversation

jmfayard commented Feb 25, 2022 • edited by LouisCAD Loading

What?

Why?

Testing?

How?

yogurtearl commented Feb 25, 2022

yogurtearl commented Feb 25, 2022

yogurtearl commented Feb 25, 2022

yogurtearl commented Feb 26, 2022

yogurtearl commented Feb 26, 2022

jmfayard commented Feb 26, 2022 • edited by LouisCAD Loading

LouisCAD commented Aug 8, 2022

LouisCAD commented Aug 9, 2022

LouisCAD commented Aug 9, 2022

LouisCAD commented Aug 10, 2022

imashnake0 commented Aug 10, 2022

Add support for Versions Catalogs `gradle/libs.versions.toml` #509

Add support for Versions Catalogs `gradle/libs.versions.toml` #509

jmfayard commented Feb 25, 2022 •

edited by LouisCAD

Loading

jmfayard commented Feb 26, 2022 •

edited by LouisCAD

Loading