-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
support except table #74
base: dev
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add a regression test for the except tables, and cover the below situations on the except tables:
- normal insert
- add/drop partition
- drop table
- alter/truncate table
- light schema change on the table
Src base.Spec `json:"src,required"` | ||
Dest base.Spec `json:"dest,required"` | ||
SkipError bool `json:"skip_error"` | ||
ExceptTable string `json:"except_table"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Define it to an array might be better.
ExceptTable string `json:"except_table"` | |
ExceptTables []string `json:"except_tables"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Then run strings.TrimSpace(exceptTable)
before converting jobContext
to Job
.
dest base.Spec | ||
db storage.DB | ||
skipError bool | ||
exceptTable string |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
exceptTable string | |
exceptTables []string |
IDest base.Specer `json:"-"` | ||
destMeta Metaer `json:"-"` | ||
SkipError bool `json:"skip_error"` | ||
ExceptTable string `json:"except_table"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ExceptTable string `json:"except_table"` | |
ExceptTables []string `json:"except_tables"` |
} | ||
} | ||
|
||
func IsExceptTable(exceptTables string, targetTable string) bool { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Define it as a method of Job
.
It likes:
func (j *Job) isExceptTable(targetTable string) bool {
backupTableList = append(backupTableList, table.Name) | ||
} | ||
log.Infof("fullsync table is : %s", backupTableList) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There may be thousands of tables here, so it is not suitable for printing logs.
@@ -918,6 +959,14 @@ func (j *Job) handleDropPartition(binlog *festruct.TBinlog) error { | |||
if j.SyncType == TableSync { | |||
destTableName = j.Dest.Table | |||
} else if j.SyncType == DBSync { | |||
// use sourceTableName to judge if table is except | |||
var sourceTableName string | |||
sourceTableName, err = j.srcMeta.GetTableNameById(dropPartition.TableId) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
GetTableNameById
is a high-cost operation, it needs to connect doris FE and execute 'SHOW TABLE'.
One possible solution is to query the table ID of the except tables before adding the job to JobManager, see Job::FirstRun
for details.
post is
You could specify except_table, that tables will not sync。
BTW, except_table could be "table1, table2,table3", separate by comma