Simplify the code in python/outlines-core

[rlouf]

The requirements are similar to those of #1, except we also need to write Python bindings. Where the interface lies is yet to be discussed; for now I suggest implementing the guides in this library.

• Remove the logic that is unrelated to structured generation (Clean the repository #3)
• Use adapt_tokenizer instead of TransformersTokenizer so we don't have to keep the models directory
• Flatten the library
• Remove .readthedoc.yml

[brandonwillard]

After #52, outlines-core no longer has tokenizer support, aside from the two copies of TransformerTokenizer in the test and benchmark code. What's the plan wrt. this?

If the plan is to use adapt_tokenizer to patch transformers tokenizers, it's not clear how that's an improvement over a custom tokenizer wrapper classes and a conditional transformers dependency, for example. In general, we could move TransformerTokenizer back to outlines-core and make transformers optional, then outlines-core will be usable with llama-based tokenizers and we won't need two copies for testing.

[rlouf]

I agree, having integrated outlines-core in different libraries I think we should move TransformersTokenizer back to outlines-core; otherwise we have to duplicate this code wherever outlines-core is used.

[rlouf]

Broke this down in #80 and #81