Airbyte OSS Client Credentials authentication fails

[amardatar]

Client Credentials authentication for this SDK does not appear to be working with OSS Airbyte at the moment. There was a similar issue recently that mentioned this - #111 - although it has been closed without resolution.

The issue seems to stem from data being sent as form-encoded data, which is not accepted by the OSS Airbyte server. Specifically here, the request is made as data=payload instead of data=json.dumps(payload) (the underlying requests library defaults to form-encoded, see here).

When attempting to connect via the SDK, I get the following logs from the airbyte-server pod which point to the issue:

[default-nioEventLoopGroup-1-6] ERROR i.a.c.s.s.AirbyteHttpRequestFieldExtractor(contentToJson):78 - Failed to parse content as JSON: grant_type=client_credentials&client_id=<redacted>&client_secret=<redacted>
java.lang.RuntimeException: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'grant_type': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 11]
 at io.airbyte.commons.json.Jsons.deserialize(Jsons.java:163)
 at io.airbyte.commons.server.support.AirbyteHttpRequestFieldExtractor.contentToJson(AirbyteHttpRequestFieldExtractor.java:75)
 at io.airbyte.commons.server.support.AuthorizationServerHandler.updateHeaders(AuthorizationServerHandler.java:64)
 at io.airbyte.commons.server.support.AuthorizationServerHandler.channelRead(AuthorizationServerHandler.java:48)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
 at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:324)
 at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
 at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
 at io.airbyte.commons.server.support.AuthorizationServerHandler.channelRead(AuthorizationServerHandler.java:52)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
 at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
 at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'grant_type': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
 at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 11]
 at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:2572)
 at com.fasterxml.jackson.core.JsonParser._constructReadException(JsonParser.java:2598)
 at com.fasterxml.jackson.core.JsonParser._constructReadException(JsonParser.java:2606)
 at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:765)
 at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:3018)
 at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:2052)
 at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:780)
 at com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4934)
 at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3280)
 at io.airbyte.commons.json.Jsons.deserialize(Jsons.java:161)
 ... 42 common frames omitted

Attempting to connect manually (ie not using the SDK) with a JSON-encoded body does work, however it also highlights that the OSS response only contains the key access_token, which means an error would inevitably be thrown here in any case. The code further down also expects (but doesn't fail on) an expires_in property which is also not available in the OSS response. The Airbyte docs indicate a token timeout of 3 minutes, which should possibly be respected in this case? Although from testing I'm not sure if that timeout actually applies. After testing - it looks like there's an endpoint /api/v1/applications/token which doesn't return token_type or expires_in, however /api/public/v1/applications/token does. I originally used the first one as that is what's mentioned in the Airbyte docs - should the docs be updated to refer to the second endpoint?

---

Impact: This issue seems to be the root cause of an issue for version 4.0.0 of the Airbyte Provider for Airflow. There are some issues in the Airflow repository which mention this, for example apache/airflow#42520. At this stage, it seems the updated version of the Airbyte Provider can't be used with OSS Airflow.

I'm happy to contribute to a fix, however I haven't used speakeasy so unclear where changes should be made to fix the issue; and also unsure if the preference would be to update this SDK, or to update the OSS API to allow for form-data and return the extra properties in the response.