Cannot attach table when running a garbage solution

[alifirat]

Hi guys,

I would like to share with you an issue faced when we applied on own garbage solution. In terms of code, I've checked the query you're running and we have almost the same solution so this issue is more a warning for you.

What happens:

1. We run our own garbage solution in the staging cluster
2. We restart clickhouse
3. Then we get the following error:

Application: DB::Exception: The specified key does not exist.: while reading key: mybucket/mom/rhkhwuhsjjrecuikuwpxpszpqivnq, from bucket: mybucket-eu-west-1-staging: Cannot attach table `default`.`mytable` from metadata file /data/disk_00/clickhouse/store/34e/34ea9ee7-ae4f-44e5-b584-d31c12cc62d6/mytable.sql from query ATTACH TABLE default.mytable UUID 'e92a9435-3b9d-4727-a643-af89fb549ac6' (`sign` Int8, ...)

At this time, we don't know exactly yet what happens to raise a such error.

[aalexfvk]

Hello!
It looks like, your garbage solution deleted something that it shouldn't. One of the points to mitigate this possibility, we use a default guard-period (24h) and deleted only S3 objects older it and not found on all replicas.
Nevertheless, our solution is not 100% production ready, we continue working on it (e.g. support of freezed data) and will pay attention on the case you mentioned.

[Scartiloffista]

and not found on all replicas.

how do you ensure this? by looking at system.remote_data_paths of all the replicas or do you have more checks in place?

we also experienced some broken permissions on the local copy of the S3 data, FYI

[aalexfvk]

Now, we use only clusterAllReplicas('{cluster_name}', system.remote_data_paths) for requesting all replicas.