Make PyO3 bindings an optional feature

[kc611]

As titled,

This PR makes use of cargo --features flag to optionally compile the python bindings when required. Without the usage of --features python-bindings, cargo will only compile the pure rust functions.

Note:: The last function create_fsm_index_end_to_end cannot be internalized right away as a pure Rust function because there is usage of PyDict within the function logic, but that should be easy enough to resolve. The main issue is that if we use the logic that we are using for the rest of the functions (wrapping the Python bindings around a *_internal Rust version) we'd have to convert the huge states_to_token_subsets dictionary from Rust HashMaps to PyDict wherein it could've been constructed as PyDict in the first place, as it is being done right now. This would be a regression in terms of performance.

• [ ] Make a Rust version of create_fsm_index_end_to_end so that it's available in the pure Rust package without python bindings. Create Rust-only index construction function #9
• Check: Confirm that the pure Rust package (without Python bindings feature) doesn't use PyO3.

[brandonwillard]

Note:: The last function create_fsm_index_end_to_end cannot be internalized right away as a pure Rust function because there is usage of PyDict within the function logic, but that should be easy enough to resolve. The main issue is that if we use the logic that we are using for the rest of the functions (wrapping the Python bindings around a *_internal Rust version) we'd have to convert the huge states_to_token_subsets dictionary from Rust HashMaps to PyDict wherein it could've been constructed as PyDict in the first place, as it is being done right now. This would be a regression in terms of performance.

• Make a Rust version of create_fsm_index_end_to_end so that it's available in the pure Rust package without python bindings.

I'm fine moving forward with this and addressing the Rust-only version of states_to_token_subsets in a follow-up.

We might want to wait until #15 is merged so that we can rebase this and then merge it.

The merge-base changed after approval.

[ErikKaum]

Looks good 👍

One small (but important thing) is to have the rlib in there as well, otherwise I think it can't be imported by rust projects:
crate-type = ["cdylib", "rlib"]

I also when running cargo build --features python-bindings, I get a compile error:

...
            "__Py_NoneStruct", referenced from:
                pyo3_ffi::object::Py_None::hb5c6297611a670d5 in outlines_core_rs.ax4ei8tw6kqx8u3n4jsaoie49.rcgu.o
                pyo3_ffi::object::Py_None::h0a3a434a6d719957 in libpyo3-f9af37ea37f46715.rlib[15](pyo3-f9af37ea37f46715.pyo3.a12b04b38ef4efd4-cgu.12.rcgu.o)
            "__Py_TrueStruct", referenced from:
                pyo3_ffi::boolobject::Py_True::h325d151b7e5600ac in libpyo3-f9af37ea37f46715.rlib[6](pyo3-f9af37ea37f46715.pyo3.a12b04b38ef4efd4-cgu.03.rcgu.o)
          ld: symbol(s) not found for architecture arm64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)
          

error: could not compile `outlines-core-rs` (lib) due to 1 previous error

But I'm pretty sure this is something off on my system. So probably nothing that should block merging this PR.

[brandonwillard]

One small (but important thing) is to have the rlib in there as well, otherwise I think it can't be imported by rust projects:
crate-type = ["cdylib", "rlib"]

Added.

[ErikKaum]

Awesome 🚀