libsql: attach databases from other namespaces as readonly

[psarna]

With this proof-of-concept patch, other namespaces hosted on the same sqld machine are now attached in readonly mode, so that users can read from other databases when connected to a particular one.

Example:
Database1:

→  create table users(id, email);
→  insert into users values (1, '[email protected]');
→  insert into users values (2, '[email protected]');

Database2:

→  create table numbers(number, description);
→  insert into numbers values (1, 'one'), (2, 'two');
→  select * from numbers join another.users on number = id;

→  insert into another.users values (3, '[email protected]');
Error: failed to execute SQL: insert into another.users values (3, '[email protected]');
SQLite error: attempt to write a readonly database

[psarna]

Assumption in this PoC is that all namespaces in a single sqld instance belong to a single user, so they're visible. That assumption is not going to hold long-term, with our multitenancy plans, and then we'd have to only attach databases that a particular user is allowed to see

[MarinPostma]

This is not going to work for multiple reasons:

• There is no reason to think that the same set of namespaces are replicated the same way across clusters; therefore, the set of databases attached is not deterministic across sqld instances
• The attached databases are not tied to an active namespace, and there should not be active connections to inactive namespaces
• There are no consistency guarantees across the attached databases since they all replicate at different paces

And I don't think that this is a good idea either:

• It breaks the namespace isolation guarantee
• If it worked, it would force all namespaces to be loaded all the time, which would be a big perf trap
• It makes creating connections a lot more expensive since we need to open many db files (it also sends the number of open FDs through the roof)

[psarna]

All valid points, but we had users asking how to access another database of theirs, so that they can read from it. Maybe we should just consider an API that allows you to dynamically attach another database from a different namespace to either all future connections, or one particular connection (the latter e.g. by overriding the ATTACH statement).

[psarna]

Also,

It breaks the namespace isolation guarantee

How? And did we specify the isolation guarantees somewhere? In general if we have multiple users using namespaces then we don't want them to know about each other indeed, but shareable namespaces of a single user can be a worthy feature, if doable

[psarna]

And one more - what's an active and inactive namespace? Where's this defined?

[MarinPostma]

An active namespace is a namespace that is loaded in memory, and whose life-cycle is managed by the namespace store

[psarna]

Downgrading to draft, since it should have been marked draft in the first place. Also, some preliminary conclusions:

• Each ATTACH creates a readonly connection to a different namespace that sqld doesn't explicitly track, so we need to be careful about that; it's readonly so I consider it safe to just do it and error-out if the namespace gets destroyed while attached, but we need to keep that in mind.
• Once we allow multiple users to coexist in one sqld instance, we have to be very careful to only allow attaching databases that a user has permissions to read.
• We can only attach databases that have primaries in the same sqld instance, and that will get trickier once we get heterogeneous multitenancy - perhaps some additional platform work to ensure that users can specify they want multiple primaries to be colocated in a single sqld instance.
• We need some kind of API to only attach databases if a user explicitly asks for it, otherwise it will create too many connections and open too many files every time a request comes. We can consider overriding how the ATTACH SQL statement is interpreted, our query_analysis parses statements anyway