Skip to content

Commit

Permalink
Merge pull request #9 from VirtusLab/iskra-rebranding
Browse files Browse the repository at this point in the history
Rename typed-frames to iskra
  • Loading branch information
prolativ authored Aug 30, 2022
2 parents d0fb4ba + d9a2c49 commit acd4924
Showing 1 changed file with 14 additions and 14 deletions.
28 changes: 14 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,49 +1,49 @@
# TypedFrames
# Iskra

TypedFrames is a Scala 3 wrapper around Apache Spark API which allows writing typesafe and boilerplate-free but still efficient Spark code.
Iskra is a Scala 3 wrapper around Apache Spark API which allows writing typesafe and boilerplate-free but still efficient Spark code.

## How is it possible to write Spark applications in Scala 3?

Starting from the release of 3.2.0, Spark is cross-compiled also for Scala 2.13, which opens a way to using Spark from Scala 3 code, as Scala 3 projects can depend on Scala 2.13 artifacts.

However, one might run into problems when trying to call a method requiring an implicit instance of Spark's `Encoder` type. Derivation of instances of `Encoder` relies on presence of a `TypeTag` for a given type. However `TypeTag`s are not generated by Scala 3 compiler anymore (and there are no plans to support this) so instances of `Encoder` cannot be automatically synthesized in most cases.

TypedFrames tries to work around this problem by using its own encoders (unrelated to Spark's `Encoder` type) generated using Scala 3's new metaprogramming API.
Iskra tries to work around this problem by using its own encoders (unrelated to Spark's `Encoder` type) generated using Scala 3's new metaprogramming API.

## How does TypedFrames make things typesafe and efficient at the same time?
## How does Iskra make things typesafe and efficient at the same time?

TypedFrames provides thin (but strongly typed) wrappers around `DataFrame`s, which track types and names of columns at compile time but let Catalyst perform all of its optimizations at runtime.

TypedFrames uses structural types rather than case classes as data models, which gives us a lot of flexibility (no need to explicitly define a new case class when a column is added/removed/renamed!) but we still get compilation errors when we try to refer to a column which doesn't exist or can't be used in a given context.
Iskra uses structural types rather than case classes as data models, which gives us a lot of flexibility (no need to explicitly define a new case class when a column is added/removed/renamed!) but we still get compilation errors when we try to refer to a column which doesn't exist or can't be used in a given context.

## Usage

:warning: This library is in its early stage of development - the syntax and type hierarchy might still change,
the coverage of Spark's API is far from being complete and more tests are needed.

1) Add TypedFrames as a dependency to your project, e.g.
1) Add Iskra as a dependency to your project, e.g.

* in a file compiled with Scala CLI:
```scala
//> using lib "org.virtuslab::typed-frames:0.0.1"
//> using lib "org.virtuslab::iskra:0.0.1"
```

* when starting Scala CLI REPL:
```shell
scala-cli repl --dep org.virtuslab::typed-frames:0.0.1
scala-cli repl --dep org.virtuslab::iskra:0.0.1
```

* in `build.sbt` in an sbt project:
```scala
libraryDependencies += "org.virtuslab" %% "typed-frames" % "0.0.1"
libraryDependencies += "org.virtuslab" %% "iskra" % "0.0.1"
```

TypedFrames is built with Scala 3.1.3 so it's compatible with Scala 3.1.x and newer minor releases (starting from 3.2.0-RC1 you'll get code completions for names of columns in REPL and Metals!).
TypedFrames transitively depends on Spark 3.2.0.
Iskra is built with Scala 3.1.3 so it's compatible with Scala 3.1.x and newer minor releases (starting from 3.2.0-RC1 you'll get code completions for names of columns in REPL and Metals!).
Iskra transitively depends on Spark 3.2.0.

2) Import the basic definitions from the API
```scala
import org.virtuslab.typedframes.api.*
import org.virtuslab.iskra.api.*
```

3) Get a Spark session, e.g.
Expand Down Expand Up @@ -87,9 +87,9 @@ foos.innerJoin(bars).on($.foos.barId === $.bars.id).select(...)
```
* As you might have noticed above, the aliases for `foos` and `bars` were automatically inferred

6) For reference look at the [examples](src/test/example/) and the [API docs](https://virtuslab.github.io/typed-frames/)
6) For reference look at the [examples](src/test/example/) and the [API docs](https://virtuslab.github.io/iskra/)

## Local development


This project is built using [scala-cli](https://scala-cli.virtuslab.org/) so just use the traditional commands with `.` as root like `scala-cli compile .` or `scala-cli test .`.
This project is built using [scala-cli](https://scala-cli.virtuslab.org/) so just use the traditional commands with `.` as root like `scala-cli compile .` or `scala-cli test .`.

0 comments on commit acd4924

Please sign in to comment.