Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Circus Train should also sync the external schema for Avro Tables when replication mode is METADATA_UPDATE #144

Open
abhimanyugupta07 opened this issue Sep 9, 2019 · 1 comment

Comments

@abhimanyugupta07
Copy link
Member

abhimanyugupta07 commented Sep 9, 2019

As a user of CT,

I want that the Circus Train should also sync the external schema for an Avro Table when the replication mode is either METADATA_UPDATE or METADATA_MIRROR.

Context
At the moment, when the replication mode is METADATA_UPDATE or METADATA_MIRROR, the table location is not needed to be provided in the config file which means that the null check on https://github.com/HotelsDotCom/circus-train/blob/master/circus-train-avro/src/main/java/com/hotels/bdp/circustrain/avro/transformation/AbstractAvroSerDeTransformation.java#L50 returns empty and the Avro Transform is not triggered which in turn will not copy the schema file to replica.

This was discovered while working on: #131

Related PR: #141

Possible Solution
CT can detect if the table location is not provided and in the Avro transform, it can get the table location from the target HMS.

@patduin
Copy link
Contributor

patduin commented Sep 9, 2019

This is not needed for METADATA_MIRROR as that is an exact metadata copy of the table so all locations will be the same

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants