Merge pull request #1212 from disneystreaming/0.18-changelog

0.18 changelog progress

CHANGELOG.md

@@ -2,6 +2,43 @@
 
 ## Behavioural changes
 
+The default timestamp format in Json serialisation is now `EPOCH_SECONDS`. This change is motivated by a desire to align with AWS and to improve
+our compatibility with their tooling. Timestamps shapes (or members pointing to timestamp shapes) now will need to be annotated with `@timestampFormat("DATE_TIME")`
+in order to retrieve the previous behaviour.
+
+## Significant rewrite of the abstractions.
+
+The abstractions that power smithy4s have been overhauled to facilitate integration with other protocols than simpleRestJson and other http libraries than http4s.
+Many levels of the library has been impacted in significant ways, which is likely to break a great many third-party integrations. The amount of breaking
+changes is too large to list exhaustively, therefore only a highlight is provided in this changelog.
+
+* `smithy4s.schema.Field` is no longer a GADT differentiating from required/optional fields. There is now a `smithy4s.schema.Schema.OptionSchema` GADT member instead, which was required to support some traits.
+* `smithy4s.schema.Schema.UnionSchema` now references an ordinal function, as opposed to the previous dispatch function.
+* `smithy4s.Endpoint` now contains a `smithy4s.schema.OperationSchema`, which is a construct gathering all schemas related to an operation.
+* `smithy4s.Service` now allows to get an ordinal value out of a reified operation, thus making it easier to dispatch it to the correct handler.
+* `smithy4s.Service` now contains some methods for instantiation of services from an endpoint compilers.
+* Two new packages in `core` have appeared : `smithy4s.server` and `smithy4s.client`, each containing protocol-agnostic constructs that aim at taking care of some of the complexity of integrating libraries/protocols with smithy4s.
+* A `smithy4s.capability.MonadThrowLike` and `smith4s.capability.Zipper` types have been created, unlocking the writing of generic functions that benefits integrations
+with various third-party libraries.
+* `smithy4s.http.HttpRequest` and `smithy4s.http.HttpResponse` types have been created.
+* `smithy4s.http.HttpUnaryClientCodecs` and `smithy4s.http.HttpUnaryServerCodecs` are new constructs that aim at facilitating the integration of http-protocols. In particular, they take care of a fair amount of complexity related to handling `smithy.api#http*` traits (including the reconciliation of data coming from http metadata and http bodies).
+* Overall, the amount of code in the `smithy4s-http4s` module has drastically diminished, as the constructs necessary for the generalisation of the http-related logic have been created. We (maintainers) love http4s, and are not planning on publicly maintaining any other integration, but we are responsible for other integrations in our work ecosystem. Therefore, generalising what we can makes our jobs easier, but also should allow for third parties to have an easier time integrating their http-libraries of choice with Smithy4s.
+
+### Highlight : schema partitioning
+
+The most ground-breaking change of 0.18, which is crucial for how things are now implemented, is the addition of a `smithy4s.schema.SchemaPartition` utility that allow to split schemas into sub-schemas that each take care of the subset of the data. This mechanism allows to completely decouple the (de)serialisation of http bodies from the decoding of http metadata. This means, for instance, that JSON serialisation no longer has to be aware of traits such as `httpHeader`, `httpQuery`, `httpLabel`. This greatly facilitates the integration of other serialisation technologies (XML, URL Form, etc) as these no longer has to contain convoluted logic related to which fields should be skipped during (de)-serialisation.
+
+As a result, the **smithy4s-json** module has been rewritten. In particular, the code it contains is now held in the `smithy4s.json` package, since it is no longer coupled with http-semantics. The `smithy4s.json.Json` object has also been created to provide high-level methods facilitating the encoding/decoding of generated types into json, which is helpful for a number of usecases that fall out of the server/client bindings.
+
+## Features
+
+### AWS SDK support.
+
+Smithy4s' coverage of the AWS protocols has increased drastically. Now, the vast majority of services and operations are supported. This does mean that Smithy4s can effectively be used as a cross-platform AWS SDK, delegating to `http4s` for transport.
+
+
+This however comes with a caveat, please refer yourself to the relevant [documentation page](https://disneystreaming.github.io/smithy4s/docs/protocols/aws/aws).
+
 ### Mill
 
 The `mill` plugin is build for version `0.11.0`. The changes to the API are solely results of this migration.
@@ -34,7 +71,8 @@ See https://github.com/disneystreaming/smithy4s/pull/912
 
 ### smithy4s.Blob
 
-`smithy4s.ByteArray` has been deprecated in favor of `smithy4s.Blob`.
+`smithy4s.ByteArray` has been deprecated in favor of `smithy4s.Blob`. This new type is more flexible, in that it can be backed by byte arrays and byte buffers alike.
+Additionally, it allows for O(1) concatenation. This change is motivated by a desire to ease integration with third party libraries whilst reducing the need of copies of binary data.
 
 ### Smithy4s Optics Instances
 
@@ -54,6 +92,22 @@ Added convenient methods for working with unions including projectors for each u
 
 See https://github.com/disneystreaming/smithy4s/pull/1144
 
+### Sparse collections
+
+The `sparse` trait is now supported, allowing for the modelling of collections with null values. Its presence leads to the code-generation of `List[Option[A]]` and `Map[String, Option[A]]`.
+
+See https://github.com/disneystreaming/smithy4s/pull/993
+
+### Xml support
+
+The `smithy4s-xml` now exists, containing utilities to parse XML blobs into the generated data classes, and render XML from the generated data classes. This serde logic abides by the rules described in the the official [smithy documentation](https://smithy.io/2.0/spec/protocol-traits.html?highlight=xml#xml-bindings).
+
+### application/x-www-form-urlencoded support
+
+The `smithy4s-core` now contains utilities to parse [application/x-www-form-urlencoded](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/POST) payloads into the generated data classes, and render those payloads same payloads the generated data classes. This encoding allows for a few customisation, driven by [alloy traits](https://github.com/disneystreaming/alloy#alloyurlformflattened).
+
+See https://github.com/disneystreaming/smithy4s/pull/1113
+
 # 0.17.20
 
 * Add empty line separating generated case classes from their companion objects in [#1175](https://github.com/disneystreaming/smithy4s/pull/1175)

modules/core/src/smithy4s/http/UrlForm.scala

@@ -35,7 +35,7 @@ import java.nio.charset.StandardCharsets
 import scala.collection.immutable.BitSet
 import scala.collection.mutable
 
-private[smithy4s] final case class UrlForm(values: List[UrlForm.FormData]) {
+final case class UrlForm(values: List[UrlForm.FormData]) {
 
   def render: String = {
     val builder = new mutable.StringBuilder
@@ -50,7 +50,7 @@ private[smithy4s] final case class UrlForm(values: List[UrlForm.FormData]) {
   }
 }
 
-private[smithy4s] object UrlForm {
+object UrlForm {
 
   final case class FormData(path: PayloadPath, maybeValue: Option[String]) {
 
@@ -81,7 +81,9 @@ private[smithy4s] object UrlForm {
   }
 
   // This is based on http4s' own equivalent, but simplified for our use case.
-  def parse(urlFormString: String): Either[UrlFormDecodeError, UrlForm] = {
+  private[smithy4s] def parse(
+      urlFormString: String
+  ): Either[UrlFormDecodeError, UrlForm] = {
     val inputBuffer = CharBuffer.wrap(urlFormString)
     val encodedTermBuilder = new StringBuilder(capacity = 32)
     val outputBuilder = List.newBuilder[UrlForm.FormData]

modules/docs/markdown/03-protocols/03-aws/01-aws.md

@@ -2,16 +2,18 @@
 sidebar_label: AWS
 ---
 
-**WARNING: THIS IS EXPERIMENTAL, DO NOT NOT EXPECT PRODUCTION READINESS**
+**WARNING: READ THE FOLLOWING, AND USE WITH CAUTION**
 
-Smithy4s provides functions to create AWS clients from generated code. At the time of writing this, smithy4s is only able to derive clients for AWS services.
+Smithy4s provides functions to create AWS clients from generated code. As of 0.18, Smithy4s supports (at least partially) all AWS protocols that are publicly documented.
 
-At the time of writing this, Smithy4s supports a subset of the [protocols](https://awslabs.github.io/smithy/1.0/spec/aws/index.html?highlight=aws%20protocols#aws-protocols) that AWS uses in their services.
+Our implementation of the AWS protocols is tested against the official [compliance-tests](https://github.com/smithy-lang/smithy/tree/main/smithy-aws-protocol-tests/model), which gives us a reasonable level of confidence that most of the (de)serialisation logic is correct involved when communicating with AWS is correct. Our implementation of the AWS signature algorithm.
+(which allows AWS to authenticate requests) is tested against the Java implementation used by the official AWS SDK.
 
-The supported protocols are :
+### What is missing ?
 
-* AWS Json 1.0
-* AWS Json 1.1
+* streaming operations (such as S3 `putObject`, `getObject`, or Kinesis' `subscribeToShard`) are currently unsupported.
+* [service-specific customisations](https://smithy.io/2.0/aws/customizations/index.html)  are currently unsupported.
+* **users should not use smithy4s to talk to AWS S3**
 
 ### Where to find the specs ?
 
@@ -81,5 +83,5 @@ Below you'll find a generated summary of the maven coordinates for the AWS speci
 that the version of the spec might not be the latest one. Refer yourself to [this repo](https://github.com/disneystreaming/aws-sdk-smithy-specs) to get the latest version of the specs.
 
 ```scala mdoc:passthrough
-docs.AwsServiceList.renderServiceList()
+smithy4s.aws.docs.AwsServiceList.renderServiceList()
 ```

modules/docs/markdown/05-design/02-schemas.md

@@ -306,4 +306,118 @@ Regarding the `underlyingSchema` value in the companion object of `IntList`, you
 
 ### Enumerations
 
-**TODO** (waiting for smithy 2.0 which changes the syntax)
+Smithy allows for two types of enumerations : string and integer enumerations.
+
+Additionally, smithy4s supports specifying whether an operation is open or closed. An open enumeration allows
+for holding unknown values, whereas a closed one is strictly limited to a set of specified values. This brings
+the total number of possible "flavours" of enumerations to 4, which is reified via a `smithy4s.schema.EnumTag` ADT
+that comprises 4 different cases : one for each combination between `[open, closed]` and `[int, string]`.
+
+Enumerations are typically modelled as Algebraic Data types. Each case of an enumeration is associated with both a String and Int value. In the case of `intEnum`, the string value is the name of the case. In the case of a normal (string) `enum`, the integer value is the index of the case in the list.
+
+Additionally, each enumeration case holds its own hints.
+
+#### Closed enumerations
+
+Given this smithy code :
+
+```kotlin
+namespace example
+
+intEnum Numbers {
+  ONE = 1
+  TWO = 2
+}
+```
+
+The corresponding generated Scala-code is :
+
+```scala
+sealed abstract class Numbers(_value: String, _name: String, _intValue: Int, _hints: Hints) extends Enumeration.Value {
+  override type EnumType = Numbers
+  override val value: String = _value
+  override val name: String = _name
+  override val intValue: Int = _intValue
+  override val hints: Hints = _hints
+  override def enumeration: Enumeration[EnumType] = Numbers
+  @inline final def widen: Numbers = this
+}
+object Numbers extends Enumeration[Numbers] with ShapeTag.Companion[Numbers] {
+  val id: ShapeId = ShapeId("smithy4s.example", "Numbers")
+
+  val hints: Hints = Hints.empty
+
+  case object ONE extends Numbers("ONE", "ONE", 1, Hints())
+  case object TWO extends Numbers("TWO", "TWO", 2, Hints())
+
+  val values: List[Numbers] = List(
+    ONE,
+    TWO,
+  )
+  val tag: EnumTag[Numbers] = EnumTag.ClosedIntEnum
+  implicit val schema: Schema[Numbers] = enumeration(tag, values).withId(id).addHints(hints)
+}
+```
+
+#### Open enumeration
+
+Given this smithy code :
+
+```kotlin
+namespace example
+
+use alloy#openEnum
+
+@openEnum
+intEnum OpenNums {
+  ONE = 1
+  TWO = 2
+}
+```
+
+The corresponding generated Scala-code is :
+
+```scala
+package smithy4s.example
+
+import smithy4s.Enumeration
+import smithy4s.Hints
+import smithy4s.Schema
+import smithy4s.ShapeId
+import smithy4s.ShapeTag
+import smithy4s.schema.EnumTag
+import smithy4s.schema.Schema.enumeration
+
+sealed abstract class OpenNums(_value: String, _name: String, _intValue: Int, _hints: Hints) extends Enumeration.Value {
+  override type EnumType = OpenNums
+  override val value: String = _value
+  override val name: String = _name
+  override val intValue: Int = _intValue
+  override val hints: Hints = _hints
+  override def enumeration: Enumeration[EnumType] = OpenNums
+  @inline final def widen: OpenNums = this
+}
+object OpenNums extends Enumeration[OpenNums] with ShapeTag.Companion[OpenNums] {
+  val id: ShapeId = ShapeId("smithy4s.example", "OpenNums")
+
+  val hints: Hints = Hints(
+    alloy.OpenEnum(),
+  )
+
+  case object ONE extends OpenNums("ONE", "ONE", 1, Hints())
+  case object TWO extends OpenNums("TWO", "TWO", 2, Hints())
+  final case class $Unknown(int: Int) extends OpenNums("$Unknown", "$Unknown", int, Hints.empty)
+
+  val $unknown: Int => OpenNums = $Unknown(_)
+
+  val values: List[OpenNums] = List(
+    ONE,
+    TWO,
+  )
+  val tag: EnumTag[OpenNums] = EnumTag.OpenIntEnum($unknown)
+  implicit val schema: Schema[OpenNums] = enumeration(tag, values).withId(id).addHints(hints)
+}
+```
+
+As you can see, the main difference between the two is the presence of an `final case class $Unknown` ADT member
+in the open enumeration, which allows to capture values that are not defined in the specification.

modules/docs/markdown/05-design/03-services.md

@@ -134,7 +134,7 @@ trait Service[Final[_[_, _, _, _, _]]] {
 }
 ```
 
-Implementations of such interfaces are code-generated. This implies that any smithy `Service` shape gets translated as a finally-encoded interface, but also as an initially-encoded `GADT`
+Implementations of such interfaces are typically code-generated. This implies that any smithy `Service` shape gets translated as a finally-encoded interface, but also as an initially-encoded `GADT`
 
 ## The high-level philosophy of Smithy4s
 
@@ -180,20 +180,28 @@ The flows described above are merely conceptual, and do not account for the opti
 The `smithy4s.Endpoint` abstraction ties a specific operation to the various schemas that are tied to it.
 
 ```scala
-trait Endpoint[Initial[_, _, _, _, _], I, E, O, SI, SO] {
-  def shapeId: ShapeId
-  def hints: Hints
-  def input: Schema[I]
-  def output: Schema[O]
-  def streamedInput: StreamingSchema[SI]
-  def streamedOutput: StreamingSchema[SO]
-  def wrap(input: I): Initial[I, E, O, SI, SO]
-
-  def errorable: Option[Errorable[E]]
+trait Endpoint[Op[_, _, _, _, _], I, E, O, SI, SO] {
+  def schema: OperationSchema[I, E, O, SI, SO]
+  def wrap(input: I): Op[I, E, O, SI, SO]
 }
 ```
 
-Endpoints are not type-classes. Instead, the `Endpoint` trait is extended by the companion object of each member of the `GADT` forming the initial-encoding of the service interface. So, going back to our `KVStore`, the corresponding sealed-trait would look like this :
+where `smithy4s.schema.OperationSchema` is a product of all schemas involved in an specific operation.
+
+```scala
+final case class OperationSchema[I, E, O, SI, SO](
+    id: ShapeId,
+    hints: Hints,
+    input: Schema[I],
+    error: Option[ErrorSchema[E]],
+    output: Schema[O],
+    streamedInput: Option[StreamingSchema[SI]],
+    streamedOutput: Option[StreamingSchema[SO]]
+) {
+
+```
+
+Endpoints are not type-classes. Instead, an `Endpoint` instance is provided by the companion object of each member of the `GADT` forming the initial-encoding of the service interface. So, going back to our `KVStore`, the corresponding sealed-trait would look like this :
 
 ```scala
 sealed trait KVStoreOp[Input, Error, Output, StreamedInput, StreamedOutput]
@@ -203,13 +211,19 @@ and the `put` operation would look like :
 
 ```scala
 case class Put(input: PutRequest) extends KVStoreOp[PutRequest, PutError, PutResult, Nothing, Nothing]
-object Put extends Endpoint[KVStoreOp, PutRequest, PutError, PutResult, Nothing, Nothing] with Errorable[PutError]{
+object Put extends Endpoint[KVStoreOp, PutRequest, PutError, PutResult, Nothing, Nothing] {
   val input = PutRequest.input
-  val output = PutRequest.input
+  val output = PutRequest.schema
   val streamedInput = SteamingSchema.nothing
   val streamedOutput = StreamingSchema.nothing
   val errorable: Option[Errorable[PutResult]] = this
   // ...
+  val schema: OperationSchema[PutRequest, PutError, PutResult, Nothing, Nothing] =
+    Schema.operation(ShapeId("namespace", "Put"))
+      .withInput(PutRequest.schema)
+      .withError(PutError.errorSchema)
+      .withOutput(PutResult.schema)
+    def wrap(input: PutRequest) = Put(input)
 }
 ```
 
@@ -219,14 +233,14 @@ As stated previously, Smithy4s generates a coproduct type for each operation, wh
 
 As a result, it is important for Smithy4s to expose functions that generically enable the filtering of throwables against the `Error` type parameter of an operation, so that interpreters can intercept errors and apply the correct encoding (dictated via `Schema`) before communicating them back to the caller over the wire. Conversely, it is important to expose a function that allows to go from the generic `Error` type parameter to `Throwable`, so that errors received via low-level communication channels can be turned into `Throwable` at the client call site, in order to populate the relevant error channel when exposing mono-functor semantics.
 
-Therefore, when a smithy operation has `errors` defined, the corresponding `smithy4s.Endpoint` also extends the `Errorable` interface, which looks like this :
+Therefore, when a smithy operation has `errors` defined, the corresponding `smithy4s.schema.OperationSchema` references a `smithy4s.schema.ErrorSchema`, which looks like this :
 
 ```scala
-trait Errorable[E] {
-  def error: UnionSchema[E]
-  def liftError(throwable: Throwable): Option[E]
-  def unliftError(e: E): Throwable
-}
+case class ErrorSchema[E] private[smithy4s] (
+    schema: Schema[E],
+    liftError: Throwable => Option[E],
+    unliftError: E => Throwable
+)
 ```
 
 ## Services and endpoints
@@ -276,13 +290,21 @@ Each `@http` occurrence get translated to a scala value in the `Hints` associate
 Therefore, the `Service` abstraction needs to be enriched with the following methods :
 
 ```scala
-trait Service[Final[_[_, _, _, _, _]], Initial[_, _, _, _, _]] {
+trait Service[Final[_[_, _, _, _, _]]] {
+
+  type Initial[_, _, _, _, _]
+
   // ...
 
   // useful for server-side
-  def endpoints: List[Endpoint[Initial, _, _, _, _, _]]
+  def endpoints: IndexedSeq[Endpoint[Initial, _, _, _, _, _]]
   // useful for client-side
-  def endpoint[I, E, O, SI, SO](op: Initial[I, E, O, SI, SO]): (I, Endpoint[Initial, I, E, O, SI, SO])
+
+  // provides the index of the endpoint associated to the operation
+  def ordinal[I, E, O, SI, SO](op: Initial[I, E, O, SI, SO]): Int
+  // extracts the input value out of a reified operation
+  def input[I, E, O, SI, SO](op: Operation[I, E, O, SI, SO]): I
+
 }
 ```