Is there a possibility to work with data without download it

[Asma-droid]

Hello,

I would like to know if there is a possibility to handle data (PDF, word, txt) without download it ! I need to give juste the URL.

My use case is to use moodle documents and to create a QA chatbot from this documents. I would'nt like to download data as it already exists in moodle.

Thanks

[TuanaCelik]

Hey @Asma-droid - I just converted the issue to a Discussion here :)
You don't necessarily have to download the data to your local machine to use it in a Haystack pipeline. But you do have to write it to a document store. Haystack uses DocumentStores to retrieve the documents/data that you want to use in its pipelines. So to write your already existent documents to a DocumentStore you might effectively have to download them first, although you won't have to keep them downloaded once you've written them to a document store.
Unfortunately, Haystack can't directly connect to your moodle data, so you would have to find a way to get that data and put it into one of the DocumentStores. The good news is there are many helper functions that might be useful for you here.

I suggest taking a look at the following resources:

• The DocumentStores: https://docs.haystack.deepset.ai/docs/document_store
• Take a look at the first tutorial here. You'll see a code block where the data is pulled from an s3 bucket with a fetch_archive_from_http and written into a DocumentStore: https://haystack.deepset.ai/tutorials/01_basic_qa_pipeline
• I don't know how moodle stores/provides the data you have but it might be that you can use the same helper function to fetch your documents and write them to a document store

Hope this helps! Let me know :)