Support protoc Kotlin code generation #44552

PhilKes · 2024-11-17T16:50:56Z

This PR adds support to generate Kotlin DSLs/Extensions for protoc via adding the kotlin_out parameter to the protoc command:

By default, the Kotlin code generation is enabled if the dependency io.quarkus:quarkus-kotlin is present
It can be explicetely en-/disabled via the quarkus.generate-code.grpc.kotlin.generate property

To test it I reused the kotlin-grpc-project integration-test to use the generated DSL for HelloMsg.
Now currently when running the KotlinGRPCProjectBuildTest, compileKotlin fails with:

e: file://.../quarkus/integration-tests/gradle/target/classes/kotlin-grpc-project/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:15:17 Annotation argument must be a compile-time constant.
e: file://.../quarkus/integration-tests/gradle/target/classes/kotlin-grpc-project/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:15:37 Unresolved reference 'kotlin'.
e: file://.../quarkus/integration-tests/gradle/target/classes/kotlin-grpc-project/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:16:24 Unresolved reference 'kotlin'.

HelloMsgKt.kt

// Generated by the protocol buffer compiler. DO NOT EDIT!
// source: hello.proto

// Generated files should ignore deprecation warnings
@file:Suppress("DEPRECATION")
package io.quarkus.example;

@kotlin.jvm.JvmName("-initializehelloMsg")
public inline fun helloMsg(block: io.quarkus.example.HelloMsgKt.Dsl.() -> kotlin.Unit): io.quarkus.example.HelloMsg =
io.quarkus.example.HelloMsgKt.Dsl._create(io.quarkus.example.HelloMsg.newBuilder()).apply { block() }._build()
/**
* Protobuf type `io.quarkus.example.HelloMsg`
*/
public object HelloMsgKt {
@kotlin.OptIn(com.google.protobuf.kotlin.OnlyForUseByGeneratedProtoCode::class)
@com.google.protobuf.kotlin.ProtoDslMarker
public class Dsl private constructor(
  private val _builder: io.quarkus.example.HelloMsg.Builder
) {
  public companion object {
    @kotlin.jvm.JvmSynthetic
    @kotlin.PublishedApi
    internal fun _create(builder: io.quarkus.example.HelloMsg.Builder): Dsl = Dsl(builder)
  }

  @kotlin.jvm.JvmSynthetic
  @kotlin.PublishedApi
  internal fun _build(): io.quarkus.example.HelloMsg = _builder.build()

  /**
   * `string message = 1;`
   */
  public var message: kotlin.String
    @JvmName("getMessage")
    get() = _builder.getMessage()
    @JvmName("setMessage")
    set(value) {
      _builder.setMessage(value)
    }
  /**
   * `string message = 1;`
   */
  public fun clearMessage() {
    _builder.clearMessage()
  }

  /**
   * `.google.protobuf.Timestamp date_time = 2;`
   */
  public var dateTime: com.google.protobuf.Timestamp
    @JvmName("getDateTime")
    get() = _builder.getDateTime()
    @JvmName("setDateTime")
    set(value) {
      _builder.setDateTime(value)
    }
  /**
   * `.google.protobuf.Timestamp date_time = 2;`
   */
  public fun clearDateTime() {
    _builder.clearDateTime()
  }
  /**
   * `.google.protobuf.Timestamp date_time = 2;`
   * @return Whether the dateTime field is set.
   */
  public fun hasDateTime(): kotlin.Boolean {
    return _builder.hasDateTime()
  }

  /**
   * `.io.quarkus.example.HelloMsg.Status status = 3;`
   */
  public var status: io.quarkus.example.HelloMsg.Status
    @JvmName("getStatus")
    get() = _builder.getStatus()
    @JvmName("setStatus")
    set(value) {
      _builder.setStatus(value)
    }
  public var statusValue: kotlin.Int
    @JvmName("getStatusValue")
    get() = _builder.getStatusValue()
    @JvmName("setStatusValue")
    set(value) {
      _builder.setStatusValue(value)
    }
  /**
   * `.io.quarkus.example.HelloMsg.Status status = 3;`
   */
  public fun clearStatus() {
    _builder.clearStatus()
  }
}
}
@kotlin.jvm.JvmSynthetic
public inline fun io.quarkus.example.HelloMsg.copy(block: `io.quarkus.example`.HelloMsgKt.Dsl.() -> kotlin.Unit): io.quarkus.example.HelloMsg =
`io.quarkus.example`.HelloMsgKt.Dsl._create(this.toBuilder()).apply { block() }._build()

public val io.quarkus.example.HelloMsgOrBuilder.dateTimeOrNull: com.google.protobuf.Timestamp?
get() = if (hasDateTime()) getDateTime() else null

I am guessing this is because the generated kotlin code should not be in build/classes/java but rather build/classes/kotlin?
But if we provide a different output path for the kotlin_out than for java_out we would need a way to add this extra path also to the compile sources, which is done in CodeGenerator.init(), right?

github-actions · 2024-11-17T17:18:17Z

🎊 PR Preview 3dff5a4 has been successfully built and deployed to https://quarkus-pr-main-44552-preview.surge.sh/version/main/guides/

Images of blog posts older than 3 months are not available.
Newsletters older than 3 months are not available.

gsmet

This looks promising, thanks for the PR.

I suggested some minor doc changes and more importantly it seems some Gradle tests are failing and I think that's the important bit:

2024-11-17T17:13:44.2059374Z Options for KOTLIN DAEMON: IncrementalCompilationOptions(super=CompilationOptions(compilerMode=INCREMENTAL_COMPILER, targetPlatform=JVM, reportCategories=[0, 3], reportSeverity=2, requestedCompilationResults=[0], kotlinScriptExtensions=[]), areFileChangesKnown=false, modifiedFiles=null, deletedFiles=null, classpathChanges=NotAvailableForNonIncrementalRun, workingDir=/home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/cacheable, multiModuleICSettings=MultiModuleICSettings(buildHistoryFile=/home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/local-state/build-history.bin, useModuleDetection=false), usePreciseJavaTracking=true, icFeatures=IncrementalCompilationFeatures(withAbiSnapshot=false, preciseCompilationResultsBackup=true, keepIncrementalCompilationCachesInMemory=true, enableUnsafeIncrementalCompilationForMultiplatform=false), outputFiles=[/home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/kotlin/main, /home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/cacheable, /home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/local-state])
2024-11-17T17:13:44.2061194Z e: file:///home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:15:17 Annotation argument must be a compile-time constant.
2024-11-17T17:13:44.2063191Z e: file:///home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:15:37 Unresolved reference 'kotlin'.
2024-11-17T17:13:44.2065030Z e: file:///home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:16:24 Unresolved reference 'kotlin'.
2024-11-17T17:13:44.2065363Z Finished executing kotlin compiler using DAEMON strategy

docs/src/main/asciidoc/grpc-generation-reference.adoc

gsmet · 2024-11-18T10:36:51Z

@cdsap I'm not sure if it's related to this change but I have seen the following in the logs and it doesn't look like something we would want:

2024-11-17T17:13:44.2065823Z 2 problems were found storing the configuration cache.
2024-11-17T17:13:44.2068318Z - Task :quarkusDev of type io.quarkus.gradle.tasks.QuarkusDev: cannot serialize object of type 'org.gradle.api.internal.artifacts.configurations.DefaultUnlockedConfiguration', a subtype of 'org.gradle.api.artifacts.Configuration', as these are not supported with the configuration cache.
2024-11-17T17:13:44.2069077Z See https://docs.gradle.org/8.9/userguide/configuration_cache.html#config_cache:requirements:disallowed_types
2024-11-17T17:13:44.2071428Z - Task :quarkusDev of type io.quarkus.gradle.tasks.QuarkusDev: cannot serialize object of type 'org.gradle.api.internal.tasks.DefaultSourceSet', a subtype of 'org.gradle.api.tasks.SourceSet', as these are not supported with the configuration cache.
2024-11-17T17:13:44.2072168Z See https://docs.gradle.org/8.9/userguide/configuration_cache.html#config_cache:requirements:disallowed_types

cdsap · 2024-11-19T00:16:40Z

@gsmet I tested with a version without the configuration cache changes (3.15.1), I'm observing the same log:
https://ge.solutions-team.gradle.com/s/g33okmapudxeg/console-log?page=1#L73

Note that I'm executing the build like: ./gradlew clean quarkusDev --configuration-cache --no-build-cache

Will confirm with the team

PhilKes · 2024-11-19T17:44:56Z

This looks promising, thanks for the PR.

I suggested some minor doc changes and more importantly it seems some Gradle tests are failing and I think that's the important bit:

2024-11-17T17:13:44.2059374Z Options for KOTLIN DAEMON: IncrementalCompilationOptions(super=CompilationOptions(compilerMode=INCREMENTAL_COMPILER, targetPlatform=JVM, reportCategories=[0, 3], reportSeverity=2, requestedCompilationResults=[0], kotlinScriptExtensions=[]), areFileChangesKnown=false, modifiedFiles=null, deletedFiles=null, classpathChanges=NotAvailableForNonIncrementalRun, workingDir=/home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/cacheable, multiModuleICSettings=MultiModuleICSettings(buildHistoryFile=/home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/local-state/build-history.bin, useModuleDetection=false), usePreciseJavaTracking=true, icFeatures=IncrementalCompilationFeatures(withAbiSnapshot=false, preciseCompilationResultsBackup=true, keepIncrementalCompilationCachesInMemory=true, enableUnsafeIncrementalCompilationForMultiplatform=false), outputFiles=[/home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/kotlin/main, /home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/cacheable, /home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/kotlin/compileKotlin/local-state])
2024-11-17T17:13:44.2061194Z e: file:///home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:15:17 Annotation argument must be a compile-time constant.
2024-11-17T17:13:44.2063191Z e: file:///home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:15:37 Unresolved reference 'kotlin'.
2024-11-17T17:13:44.2065030Z e: file:///home/runner/work/quarkus/quarkus/integration-tests/gradle/target/kotlin-grpc-project12669661681070306965/build/classes/java/quarkus-generated-sources/grpc/io/quarkus/example/HelloMsgKt.kt:16:24 Unresolved reference 'kotlin'.
2024-11-17T17:13:44.2065363Z Finished executing kotlin compiler using DAEMON strategy

Thanks for the feedback. I am aware of the failing tests, as I described in the PR description I am not sure what the problem is since I dont have much experienced with mixed java/kotlin generated sources. Do the generated kotlin sources have to be in another folder than the generated java code? (see my original PR description)

quarkus-bot · 2024-11-19T18:18:18Z

Status for workflow `Quarkus Documentation CI`

This is the status report for running Quarkus Documentation CI on commit 8c47257.

✅ The latest workflow run for the pull request has completed successfully.

It should be safe to merge provided you have a look at the other checks in the summary.

Warning

There are other workflow runs running, you probably need to wait for their status before merging.

quarkus-bot · 2024-11-19T18:59:45Z

Status for workflow `Quarkus CI`

This is the status report for running Quarkus CI on commit 8c47257.

Failing Jobs

Status	Name	Step	Failures	Logs	Raw logs	Build scan
✖	Gradle Tests - JDK 17	`Build`	Failures	Logs	Raw logs	🔍
✖	Gradle Tests - JDK 17 Windows	`Build`	Failures	Logs	Raw logs	🔍

Full information is available in the Build summary check run.
You can consult the Develocity build scans.

Failures

⚙️ Gradle Tests - JDK 17 #

- Failing: integration-tests/gradle

📦 integration-tests/gradle

✖ io.quarkus.gradle.KotlinGRPCProjectBuildTest.testBasicMultiModuleBuild - History - More details - Source on GitHub

java.lang.AssertionError: Gradle build failed with exit code 1
	at app//io.quarkus.gradle.QuarkusGradleWrapperTestBase.runGradleWrapper(QuarkusGradleWrapperTestBase.java:140)
	at app//io.quarkus.gradle.QuarkusGradleWrapperTestBase.runGradleWrapper(QuarkusGradleWrapperTestBase.java:57)
	at app//io.quarkus.gradle.QuarkusGradleWrapperTestBase.runGradleWrapper(QuarkusGradleWrapperTestBase.java:52)
	at app//io.quarkus.gradle.KotlinGRPCProjectBuildTest.testBasicMultiModuleBuild(KotlinGRPCProjectBuildTest.java:14)
	at [email protected]/java.lang.reflect.Method.invoke(Method.java:569)
	at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)
	at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)

✖ io.quarkus.gradle.devmode.KotlinProjectWithCompilerArgsDevModeTest.main - History - More details - Source on GitHub

java.lang.AssertionError: 
Expecting actual:
  "BROKEN: quarkusDev mode has terminated"
to contain:
  "[Banana!]!" 
	at app//io.quarkus.gradle.devmode.KotlinProjectWithCompilerArgsDevModeTest.testDevMode(KotlinProjectWithCompilerArgsDevModeTest.java:17)
	at app//io.quarkus.gradle.devmode.QuarkusDevGradleTestBase.main(QuarkusDevGradleTestBase.java:60)
	at [email protected]/java.lang.reflect.Method.invoke(Method.java:569)

⚙️ Gradle Tests - JDK 17 Windows #

- Failing: integration-tests/gradle

📦 integration-tests/gradle

✖ io.quarkus.gradle.KotlinGRPCProjectBuildTest.testBasicMultiModuleBuild - History - More details - Source on GitHub

java.lang.AssertionError: Gradle build failed with exit code 1
	at app//io.quarkus.gradle.QuarkusGradleWrapperTestBase.runGradleWrapper(QuarkusGradleWrapperTestBase.java:140)
	at app//io.quarkus.gradle.QuarkusGradleWrapperTestBase.runGradleWrapper(QuarkusGradleWrapperTestBase.java:57)
	at app//io.quarkus.gradle.QuarkusGradleWrapperTestBase.runGradleWrapper(QuarkusGradleWrapperTestBase.java:52)
	at app//io.quarkus.gradle.KotlinGRPCProjectBuildTest.testBasicMultiModuleBuild(KotlinGRPCProjectBuildTest.java:14)
	at [email protected]/java.lang.reflect.Method.invoke(Method.java:569)
	at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)
	at [email protected]/java.util.ArrayList.forEach(ArrayList.java:1511)

✖ io.quarkus.gradle.devmode.KotlinProjectWithCompilerArgsDevModeTest.main - History - More details - Source on GitHub

java.lang.AssertionError: 
Expecting actual:
  "BROKEN: quarkusDev mode has terminated"
to contain:
  "[Banana!]!" 
	at app//io.quarkus.gradle.devmode.KotlinProjectWithCompilerArgsDevModeTest.testDevMode(KotlinProjectWithCompilerArgsDevModeTest.java:17)
	at app//io.quarkus.gradle.devmode.QuarkusDevGradleTestBase.main(QuarkusDevGradleTestBase.java:60)
	at [email protected]/java.lang.reflect.Method.invoke(Method.java:569)

quarkus-bot bot added area/documentation area/gradle Gradle area/grpc gRPC labels Nov 17, 2024

This comment has been minimized.

Sign in to view

gsmet requested changes Nov 18, 2024

View reviewed changes

docs/src/main/asciidoc/grpc-generation-reference.adoc Outdated Show resolved Hide resolved

docs/src/main/asciidoc/grpc-generation-reference.adoc Outdated Show resolved Hide resolved

Support protoc Kotlin code generation

8c47257

PhilKes force-pushed the grpc-generate-kotlin branch from 3227bb7 to 8c47257 Compare November 19, 2024 17:41

PhilKes requested a review from gsmet November 19, 2024 17:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support protoc Kotlin code generation #44552

Support protoc Kotlin code generation #44552

PhilKes commented Nov 17, 2024 •

edited

Loading

github-actions bot commented Nov 17, 2024 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

gsmet left a comment

gsmet commented Nov 18, 2024 •

edited

Loading

cdsap commented Nov 19, 2024

PhilKes commented Nov 19, 2024

quarkus-bot bot commented Nov 19, 2024

quarkus-bot bot commented Nov 19, 2024 •

edited by github-actions bot

Loading

Support protoc Kotlin code generation #44552

Are you sure you want to change the base?

Support protoc Kotlin code generation #44552

Conversation

PhilKes commented Nov 17, 2024 • edited Loading

github-actions bot commented Nov 17, 2024 • edited Loading

This comment has been minimized.

This comment has been minimized.

gsmet left a comment

Choose a reason for hiding this comment

gsmet commented Nov 18, 2024 • edited Loading

cdsap commented Nov 19, 2024

PhilKes commented Nov 19, 2024

quarkus-bot bot commented Nov 19, 2024

Status for workflow Quarkus Documentation CI

quarkus-bot bot commented Nov 19, 2024 • edited by github-actions bot Loading

Status for workflow Quarkus CI

Failing Jobs

Failures

⚙️ Gradle Tests - JDK 17 #

📦 integration-tests/gradle

⚙️ Gradle Tests - JDK 17 Windows #

📦 integration-tests/gradle

PhilKes commented Nov 17, 2024 •

edited

Loading

github-actions bot commented Nov 17, 2024 •

edited

Loading

gsmet commented Nov 18, 2024 •

edited

Loading

Status for workflow `Quarkus Documentation CI`

quarkus-bot bot commented Nov 19, 2024 •

edited by github-actions bot

Loading

Status for workflow `Quarkus CI`