Adding new IBIS backend

[Nicoretti]

Dear Team & Maintainers,

I am working at @exasol which is considering implementing an IBIS backend for the EXASOL database.

📑 Note:
If implemented it very likely would be a SQLA based Backend (Expression Generating Backend).

While going through the IBIS documentation, a few questions arose,
which may also be relevant for other backend implementors so bringing them as in a
GitHub discussion seemed like a good choice to me.

Questions

Is there specific documentation on "how to implement an IBIS Backend"?

Don't get me wrong, at first glance it does not seem too complicated based on the existing Backends,
still I wanted to make sure I didn't miss it in case there are things like "best practices"
and "known pitfalls".

What is the project's take on upstreaming new backends?

I understand that upstreaming code takes extra efforts on both sides.
Nonetheless when looking at your current code base and pypi it looks like you integrated
most backends directly into your code base by now.

📑 Note:
As far as I know, EXASOL would be willing to invest the extra effort for contributing to upstream
if that aligns with your mode of operation.

Based on your preferred mode of contribution, I would have an extra question for you:

In case upstream contributions are preferred

Do you have additional expectations except the "obvious" ones like using your code style(s) & checks,
passing your test suite(s) etc.?

e.g.:

• Maintenance "beyond bug fixing"
• Code Coverage Percentages (>92%)
• A contact to reach out to, beyond the implementor
• ...

In case external plugins are preferred

As far as I could see, "external" backends can't easily make use of the IBIS test suite. Therefore I was wondering if you
have some requirements which you would consider essential for every IBIS Backend to support etc.

Appreciate your efforts, thanks in advance!

best
Nico

[cpcloud]

@Nicoretti Welcome 👋🏻 !

Thanks for opening the discussion.

There's a short blog post on the topic here.

I'll reiterate the steps in more detail here so the information remains in a single place for the purposes of our discussion here 🙂

But first, let me try and answer your questions.

Is there specific documentation on "how to implement an IBIS Backend"?

😅 Unfortunately no.

Part of the reason is that most of the work to introduce a new backend is typically done by people working on ibis full-time and at this point I think most of those poeple have added at least one new backend.

We're definitely not averse to having such documentation and I think we're probably at a point where the process is repeatable enough to be documented.

What is the project's take on upstreaming new backends?

We prefer if a backend exists in the ibis-project/ibis repository.

We tried to have third party backends in the past and it proved quite difficult to maintain backwards compatibility because our internals change quite a bit right now. We expect to stabilize at some point (hopefully over the next year), but for now (and probably for the foreseeable future) we'd like new backends to live in the main repository.

As far as I know, EXASOL would be willing to invest the extra effort for contributing to upstream
if that aligns with your mode of operation.

That is really great to hear and it most definitely aligns with our mode of operation.

Do you have additional expectations except the "obvious" ones like using your code style(s) & checks,
passing your test suite(s) etc.?

Hopefully not too many:

• Maintenance "beyond bug fixing"

We would definitely appreciate bug fixes, and ideally some community engagement where it makes sense. For example, if an exasol user has a question about the exasol backend chiming in to help them out would be most welcome.

Another area for contribution with respect to the specific backend is keeping up with new features in the exasol product and rolling those into ibis as soon as you'd like them to be available.

• Code Coverage Percentages (>92%)

We don't have any strict requirements on code coverage, and we have coverage annotations set up for PRs so you can see what lines are not covered when you make the PR.

I think our requirements here are pretty loose: avoiding adding a lot of untested code 🙂

We can help with any ambiguities in PR review.

• A contact to reach out to, beyond the implementor

We would really really like it if there was a person or set of people that we could contact directly about the exasol backend for questions, especially if after doing some due diligence looking at documentation and scouring the internet we are unable to find the answer.

Whew, okay all that out of the way let's talk implementation!

Implementation

Here are the evaluation steps for adding a new backend.

Running the backend

• What is the backend's architecture?

• If it's a client-server architecture, ideally it can be deployed locally using docker.

Looks like there's an official docker image for exasol, which is great! ✅

It also looks like it's actively maintained. Even better ✅✅

Python Client API

• Is there a SQLAlchemy dialect?

Two for two! Looks like there's an actively maintained official SQLAlchemy dialect as well.

Where to Start?

• Start by copying one of the existing sqlalchemy backends. Postgres, DuckDB, and Snowflake are good places to start.
• If you're up for it, try to roll in the exasol docker setup to our docker-compose.yml.
• Finally--and this is the most tedious part--get the backend test suite passing using this command:

pytest -m exasol

This part will take some collaboration because it can be very very annoying to get all the markers done and correct the first time around. So far, this has always been a one-time cost for adding the initial implementation of a new backend and I would expect it to be the same for exasol.

There are a bunch of details here to work out and I'm sure you'll have questions, but let's work all that out in a pull request!

[Nicoretti]

Hi @cpcloud,

Thanks for the warm welcome and the detailed response! (Appreciate it ❤️ )
I think it should be enough to get most people started.

---

📑 Side Note (regarding Exasol):

Our current internal feedback and discussion, indicates that we will first stabilize the websocket based sqla dialect.
Once the stabilization is done, we will address the optional dependencies in our sqla dialect before we start with the IBIS backend.

So to keep this thread clean I think we can pick up further Exasol related questions and discussion in a future pull request.

best
Nico

[cpcloud]

Nice!

If you're up for it, I don't think the optional dependencies issue needs to be resolved to open a draft PR if only to get this all kicked off as soon as possible.