Uploaded file processing and document indexing is quite quite slow

[ravikhunt]

Hey, Team
Is there any way to improve the document indexing and processing file performance?
it's taking so much time to process and index the file if not get an error
Someone can estimate the time to take for the processing file based on file type or file size number of chunks created etc

for example
1MB - PDF - 30 minutes
1mb - CSV - 15 minutes
Something like this

OR is there any way that we can improve that process by changing the Azure services configurations on the portal?

[dayland]

You can scale up the App Service Plans behind the Azure Functions and Embeddings App Service. See here... https://github.com/microsoft/PubSec-Info-Assistant/blob/main/docs/deployment/autoscale_sku.md

There are also various delays and retries in the logic that help to keep it running at a minimal cost. Many of these can be adjusted in App Service Configuration values.. more details on the settings can be found at https://github.com/microsoft/PubSec-Info-Assistant/blob/main/docs/features/document_pre_processing.md#function--app-service-configuration