Write a comparison to Kluent, AssertK, Strikt, Atrium, Kotest, other Kotlin assertion libraries?

[cpovirk]

They have various Kotlin-specific advantages. It would be good to understand what those are. Maybe someday we can provide some of them ourselves (though we haven't followed through on our Kotlin ambitions much yet).

[edit: EventFahrplan/EventFahrplan#141 (comment) was interested in this.]

[edit: One particular thing we should look into is whether those libraries support Kotlin multiplatform (KMP). I've heard that Kotest supports it, and it sounds like Kluent has at least some support.]

[cpovirk]

Dumping a couple scraps of information:

First, on multiplatform:

KMP support

• AssertK?
• Atrium?
• Kotest?

No KMP support

• Kluent until they release 2.0?
• Strikt?

Second, a series (still in progress) that compares Kotlin assertion libraries:

• introduction
• AssertJ as baseline
• AssertK
• Strikt
• Kotest

[cpovirk]

I should note that I've been ignoring HamKrest for some of the reasons discussed in https://truth.dev/comparison#vs-hamcrest. Mainly, I still worry about its less effective autocompletion. My second possible concern would be that the failure messages may still be complex, but it's possible that that's been improved, and I haven't checked. The one concern that doesn't carry over is that, as the HamKrest site notes, generics for Matcher types work better in Kotlin than in Java: "Kotlin's type system means that developers don't have to worry about getting the variance of generic signatures right." [edit: Eh, that may still be optimistic, given that part of the challenge with Hamcrest seems to have been the conflict between "I want to assert that this Foo has bar = "baz"" and "I want to assert that this object is a Foo which has bar = "baz"." But maybe HamKrest has scoped itself down to cover only the former??]

On that note: I got worried because I saw Kotest assertions written in a Hamcrest style: message should contain("foo"), with the expected poor autocompletion. However, Kotest also supports message shouldContain "foo": The Matcher style is merely an additional option, similar to what's provided by AssertJ and probably numerous Kotlin libraries.

[cpovirk]

It's also worth noting (though veering slightly off-topic) that the built-in kotlin.test provides a richer set of assertion functions than some of us are used to from the JUnit 4 world (and probably even the JUnit 5 world, since the JUnit 5 developers are trying not to go too far down the road of providing assertions).

[cpovirk]

More scattered thoughts from looking into parts of the "KMP support" libraries:

Assertions as extension methods

I confirmed that assertions are implemented with extension methods. (For AssertK and Atrium, you define the extension on Assert<Foo> or Expect<Foo>; for Kotest, you define it on Foo.) This is what I had assumed that Kotlin libraries would do, and it unlocks some fun like assertThat(futureOfOptionalOfMyProto).result().value().longitude().isWithin(...).of(...) that Java can't support.

[edit: I should note that Kotest does seem to nudge people toward the Hamcrest-style Matcher approach, even as it obviously still supports the more autocomplete-friendly style on top of that.]

Failure messages

The only one that immediately concerned me was the Kotest failure message...

java.lang.AssertionError: Map should contain key x

...with no information about the map contents. On the one hand, this is easily fixable. On the other hand, Kotest has been around a while, and no one has fixed it yet. And if the easiest way to produce a failure message is the way that omits the value under test, then I imagine this isn't the only such failure message in Kotest, and authors of custom assertions are likely to fall into the same trap.

At the opposite side of the spectrum, we have these Atrium examples. I applaud the inclusion of lots of information, but several lines with information like (kotlin.Int <1234789>) is a bit much for a simple "What does this Iterable contain?" assertion.

Docs

AssertK and Kotest look pretty reasonable at first glance. Atrium is kind of scary:

val <E, T> IterableLikeContains.EntryPointStep<E, T, InAnyOrderSearchBehaviour>.only: IterableLikeContains.EntryPointStep<E, T, InAnyOrderOnlySearchBehaviour>
Defines that the constraint "only the specified entries exist in the IterableLike" shall be applied to this sophisticated contains IterableLike assertion.

But to be fair, the bigger question is what the experience of using these is like in an IDE, since an IDE is (from what I hear) basically a necessity for writing significant Kotlin already.

Evolution

I wonder if the biggest concern is going to be how the various libraries are maintained over time.

First, of course, we want them simply to be maintained :) The 3 I've been looking at have all been around at least a few years, so that's encouraging. All have one dominant contributor, but it's most extreme with Atrium (vs. AssertK, Kotest). We might also look to current popularity. Based just on GitHub watches/stars/forks, I see Kotest well above AssertK and Atrium. It's worth noting that Kotest offers more than just assertions, so that doesn't mean that Kotest assertions are getting a ton of specific attention, just that the project is (hopefully) less likely to be abandoned.

Another thing to watch for is breaking changes. Yes, we have been known to make breaking changes, we see value in making them judiciously, and we have tools to keep up with them. Still, we might not want to build our own extensions on an ecosystem that is changing a lot, since our users might not be in the same position.

• Atrium seems very much pre-1.0, so it may be a little early to adopt it unless you're up for that. (Conversely, if you're looking for a project to deeply influence, it might be a good pick for the same reason!)
• AssertK is also pre-1.0 but seems much less in flux.
• Kotest has had some multiple major releases (and has removed APIs in minor releases, like 3.3.0), but my impression is that it may be the most stable at this point. [edit: Or maybe not: I see a report that it relies on experimental/unstable APIs and thus breaks during Kotlin upgrades—though it sounds like the owners are trying to avoid that nowadays.]

I wish people didn't have a method named "containsAll"

...given our experience with that method. But AssertK and Kotest both do, and Atrium has a varargs contains method that sounds like it will lead to the same mistakes :(

[cpovirk]

Anyway:

• This thread is not nominally about "What should I do if I want something like Truth but for KMP?"
• I have written zero lines of code using any of the assertion libraries here.

So no one should take the following too seriously. But if you have no idea which to pick and you're going to try one first, I'm kind of liking my early impression of AssertK:

• Atrium looks like it's still in flux, its APIs look a bit complex, and it looks committed to some failure messages that are too detailed for my taste.
• Kotest looks more stable, but the "Map should contain key x" failure message concerns me, and I worry that we'll see autocomplete-unfriendly Hamcrest-style Matcher objects mixed in with the extension functions.

AssertK, while probably still a bit in flux itself and arguably lacking the API coverage of Kotest (but that claim seems out of date, as I see assertions for Optional and for floating-point numbers), looks likely to be stable "enough" and looks most likely to be helpful when your tests fail and you really need the help.

As a more conservative alternative, it sounds like the built-in kotlin.test assertions will get you pretty far, too, with failure messages that likely contain what you need to know but without AssertK's nicer formatting.

I'd be happy to hear about others' hands-on experiences, especially if there are features that could translate well to Truth or important features that just can't translate to Truth.

[cpovirk]

One thing that's not entirely clear to me is how well the various libraries support collection assertions:

AssertK:

• doesn't have containsExactly on a Map (though that sounds easy enough to add)
• doesn't have containsExactlyElementsIn-style (i.e., Iterable-accepting) methods (aside from containsSubList, an unusual case) -- though there looks to be a relevant feature request that lays out some of the design questions

Kotest:

• also essentially doesn't have containsExactlyElementsIn-style methods, with exceptions for unusual cases shouldContainInOrder, shouldStartWith, shouldEndWith, and shouldContainAnyOf

Atrium probably offers a lot ("sophisticated contains assertions"), though I noted my worry about complexity.

Anyway: Maybe the theory for the containsExactlyElementsIn family is that you just use isEqualTo/shouldBe? I wonder if people find that option as easily, though. I also wonder if the failure messages are as good as the failure messages for collection-specific assertions.

[yogurtearl]

more assertion libs to compare with:

• kotlin-power-assert
• truthish

[yogurtearl]

I like the assertAll {} block that AssertK offers. It reports all assertion failures, not just the first one that fails.

[astubbs]

and it unlocks some fun like assertThat(futureOfOptionalOfMyProto).result().value().longitude().isWithin(...).of(...) that Java can't support.

FYI that's one of the awesome things that truth-generator unlocks for Java 🤗

E.g.: confluentinc/parallel-consumer@0f993dd

assertWithMessage("The only incomplete record now is offset zero, which we are blocked on")
  .that(partitionState)
  .getAllIncompleteOffsets()
  .containsExactlyElementsIn(blockedOffsets);

assertTruth(employee).hasBoss().hasCard().hasName().ignoringTrailingWhiteSpace().equals("Tom");

[phellipealexandre]

Hey everyone, I did an analysis a while ago comparing some of the assertion libraries that could be used in Kotlin projects. Browsing on GitHub I was surprised to see that this is exactly what people in this thread are looking for 😄

Here is the repository link. I was not as granular as @cpovirk in some topics, but hope you can find something useful there.

[robstoll]

@cpovirk thanks for the write-up, gives me a chance to see where we can improve Atrium further :)

but several lines with information like (kotlin.Int <1234789>)

That's planned (I even started two or even three years ago but don't have enough time for Atrium in the moment unfortunately and as you noted, most contributions are written on my keyboard) => robstoll/atrium#295

I wish people didn't have a method named "containsAll"

Atrium's expect(list).toContain(1,2,3) is a shortcut for expect(list).toContain.atLeast(1).values(1,2,3) as it doesn't mention anything about All nor about its order or similar, I think it is pretty safe to use as shortcut-name and should not re-introduce the same mistake. Also the error report is clear about it IMO:

I expected subject: [1, 2, 2, 4]        (java.util.Arrays.ArrayList <1234789>)
◆ to contain, in any order: 
  ⚬ an element which equals: 3        (kotlin.Int <1234789>)
      » but no such element was found

Or do you think Atirum did the same mistake?

In any case, I am very open to collaborate where it makes sense. Atrium is built up by several modules and it would for instance be possible to re-use the reporting mechanism.

[cpovirk]

Hi, @robstoll (and @vlsi, whose username I see here and there pretty often :)).

My fear with such methods isn't so much the failures as it is the unintentional successes: If I write expect(primes).toContain(2, 3, 5, 7), then the assertion would pass with a list of [1, 2, 3, 4, 5, 6, 7, 8, 9], and I might never realize it.

I suspect you're right that "toContain" is less likely to be misunderstood than a "toContainAll" method would be: Without the word "all," users are less likely to assume that the values they pass are expected to be "all" the contents. I still wouldn't be surprised to see it misused occasionally, since a user in a hurry may latch onto any varargs "contains" method as "an assertion about the [entire] contents." But I'm speculating; I don't have anything like the very clear data we had from back when Truth had a containsAll method. I'd probably still go with "toContainAtLeast" if I were writing an Atrium-style library from scratch, but without data, I don't have a strong case for moving off an existing name. (If you were interested in investigating, maybe you could browse some existing Atrium users' code. But I admit that I'm not rushing off to do so, so I could understand if you don't plan to, either :))

[robstoll]

If I write expect(primes).toContain(2, 3, 5, 7), then the assertion would pass with a list of [1, 2, 3, 4, 5, 6, 7, 8, 9], and I might never realize it.

I see your fear and I think it is important to prevent users from such pitfalls. In this particular case I think it is safe to use the name as it has the same semantic as Collection.contains and I believe most developers are familiar with it.