Deleting uploaded files/index data

[lon-tierney]

@dayland can you please share steps to clear out index/uploaded doc data from an installation?

[gitBenJo]

@dayland can you please share steps to clear out index/uploaded doc data from an installation?

Could this be added to the UX pls, e.g. select File from Upload Status list -> Delete, then trigger Search re-index?

[dayland]

This will be a feature in the future after v1.0 is released. It is a bit more complicated as you will see below.

[dayland]

@lon-tierney, Removing/Clearing files from the system can be done in a few ways depending on your needs.

Removing ALL data from Azure Storage and Search Index only (leaving logs intact)

As we don't yet have an automated way to remove files from the system, some manual steps are required:

1. Use Azure Storage Explorer or the Azure Portal to remove all files from "upload" and "contents" containers in Azure Storage Account
2. Delete the "all-files-index" (Gamma) or "vector-index" (vNext-Dev)
3. Rerun make deploy-search-indexes from VSCode

This process cleans up the files and clears and recreates the index. Then once you upload new files they will be processed and indexed normally.

Remove ALL Data from Azure Storage, Search Index, and Logs

To do this follow Steps 1 and 2 above, then also complete these additional steps manually:

3. Delete the "statusdb" container in Azure CosmosDB
4. Rerun make deploy

This will recreate both the Azure CosmosDB container and the search index. Then new files uploaded will process as normal and all logs in the UX will reflect only newly processed files.

[gitBenJo]

got it, thanks @lon-tierney & @dayland ... fyi -> @stu-ball let's give this a try sometime

[stu-ball]

Thanks @gitBenJo - that's exactly the process we followed last time :)

[codylittle]

@dayland is this still the preferred steps for removing content in the Delta release?

[dayland]

What has changed in 0.4-Delta

The overall pattern of how we index content has changed a lot in 0.4-Delta.

In 0.3-Gamma, we simply wrote out JSON files to Azure Blob storage and then depended on an Azure AI Search Indexer, Skillset, Datasource, and Index combination to detect the files in blob storage and enrich the index with OOTB skillsets. This had some useful features like the ability to detect deleted files in the blob storage container and clean up the index, but also came with a lot of uncontrollable errors in the OOTB skillsets.

In 0.4-Delta, we have removed the Azure AI Search Indexer, Skillsets, and Datasource leaving only a single index. And rather than depending on the Indexer to come around on a schedule and detect new files, we are directly inserting data into the index via a REST API call. We have also replaced the OOTB skillsets with enrichment REST API calls in our Azure Functions and Azure App services.

What does this mean for deletion?

Removing some file(s) from the index

As we don't yet have an automated way to remove files from the system, some manual steps are required:

1. Make a REST API call to the Azure AI Search index. You need to provide a document key, which is the Base64 encoded version of the Blob path of the document you want to delete. You can find a working example of this logic in our functional test code at run_tests.py in the cleanup_after_test() function.
2. Manually delete the chunk JSON files in the contents container in Azure Storage
3. (Optional) If you want to clear the document from the Manage Content section of the UX, you will need to delete the matching records from the Azure CosmosDB statusdb/statuscontainer container where the file_path attribute matches the files you deleted above.

Removing ALL data from Azure Storage and Search Index only (leaving logs intact)

1. Use Azure Storage Explorer or the Azure Portal to remove all files from "upload" and "contents" containers in Azure Storage Account
2. Delete "vector-index" index in Azure AI Search
3. Rerun make deploy-search-indexes from VSCode

This process cleans up the files and clears and recreates the index. Then once you upload new files they will be processed and indexed normally.

Remove ALL Data from Azure Storage, Search Index, and Logs

To do this follow Steps 1 and 2 above, then also complete these additional steps manually:

3. Delete the "statusdb" container in Azure CosmosDB
4. Rerun make deploy

This will recreate both the Azure CosmosDB container and the search index. Then new files uploaded will process as normal and all logs in the UX will reflect only newly processed files.

[helpsystems-mushkevych]

I have extended approach from the cleanup_after_test() and made it available at:
https://gist.github.com/helpsystems-mushkevych/0c21964e5efc0b0c3d97c02adc2afb1f

[ravikhunt]

Hi
Removing some file(s) from the index there is a second "Manually delete the chunk JSON files in the contents container in Azure Storage"
but when I tried the same things with the run_tests.py file, actually it's deleting the JSON files from the content container but not from the upload container

[AThomsen]

What about updating content? If I re-upload a document with updated contents, will it overwrite all chunks and index-documents using the filename as key?

[dayland]

If you upload the same file over again, it will simply overwrite the record in the index and update with any new chunks. It will not duplicate as it is keyed off the file name.

[gitBenJo]

ok good to know ... I think this logic carries through, but best to confirm - if the new version is shorter, more concise, perhaps a synopsis, or some sections are removed ... the old chunks from the prior version relating to those sections don't persist?

[AThomsen]

But if the new document version is shorter, resulting in less chunks, will any remaining chunks from the old version still be in the index?

[dayland]

Let us confirm and get back to you. I don't know without some testing. We checked that it overwrite existing chunks, but didn't check for "less chunks" from the same file.

[ArpitaisAn0maly]

uploading same document with less chunks does not remove remaining chunk from old version. I confirmed it. This is expected behaviour. Our index structure is push type. Deletion of existing chunk can only happen if we push it. when you reupload same doc with less chunk it will overwrite chunks which are present in both version but chunks which were only present in old version won't get deleted automatically.