Fix: MAP value converter for data via debezium #2156

priyanshi-yb · 2025-01-06T13:18:30Z

Describe the changes in this pull request

Fix hstore datatype value converter (MAP) to convert values properly from debezium in debezium exporter.
Report the store datatype in Unsupported datatype for live migration with fall-forward/fall-back as gRPC connector doesn't support it yet.
https://yugabyte.atlassian.net/browse/DB-14753

Describe if there are any user-facing changes

No

How was this pull request tested?

end-to-end tests

Does your PR have changes that can cause upgrade issues?

Component	Breaking changes?
MetaDB	No
Name registry json	No
Data File Descriptor Json	No
Export Snapshot Status Json	No
Import Data State	No
Export Status Json	No
Data .sql files of tables	No
Export and import data queue	No
Schema Dump	No
AssessmentDB	No
Sizing DB	No
Migration Assessment Report Json	No
Callhome Json	No
YugabyteD Tables	No
TargetDB Metadata Tables	No

yb-voyager/src/tgtdb/suites/yugabytedbSuite.go

…um.data.Json converter to convert it properly for complex cases as well

makalaaneesh · 2025-01-16T08:26:15Z

...r-voyagerexporter/src/main/java/io/debezium/server/ybexporter/DebeziumRecordTransformer.java

+            case MAP:
+	            StringBuilder mapString = new StringBuilder();
+                for (Map.Entry<String, String> entry : ((HashMap<String, String>) fieldValue).entrySet()) {
+                    String key = entry.getKey();


let's add debug logs before and after our transformation for the key and value.

makalaaneesh · 2025-01-16T08:27:03Z

migtests/scripts/run-test.sh

@@ -136,6 +136,14 @@ main() {
 	    fi
 	fi

+	if [ "${TEST_DIR}" = "${TESTS_DIR}/pg/datatypes" ]; then
+		cat ${EXPORT_DIR}/data/hstore_example_data.sql


this is just for debugging? remove?

makalaaneesh · 2025-01-16T08:30:28Z

migtests/tests/pg/datatypes/validate

@@ -26,6 +26,7 @@ EXPECTED_ROW_COUNT = {
 	'datetime_type2': 2, 
 	'null_and_default' :2,
 	'decimal_types': 3,
+	'hstore_example': 13,


let's also add data validations (for each row), for source-target, source-source-replica, target-source-replica(ignore if this is not supported).

You can use get_distinct_values_of_column_of_table

makalaaneesh · 2025-01-16T08:32:30Z

yb-voyager/src/tgtdb/suites/yugabytedbSuite.go

-			transformedMapValue = transformedMapValue + fmt.Sprintf("\"%s\"=>\"%s\",", key, value)
-		}
-		return fmt.Sprintf("'%s'", transformedMapValue[:len(transformedMapValue)-1]), nil //remove last comma and add quotes
+	"MAP": func(columnValue string, formatIfRequired bool, dbzmSchema *schemareg.ColumnSchema) (string, error) {


what about postgressuite? Do we re-use yugabytedbSuite?

we have some implementation for "MAP" in oracleSuite as well. I don't imagine that is useful, since we won't have MAP coming from oracle source at all, so remove it?

what about postgressuite? Do we re-use yugabytedbSuite?

yes we use the same for both

we have some implementation for "MAP" in oracleSuite as well. I don't imagine that is useful, since we won't have MAP coming from oracle source at all, so remove it?

yeah ok

makalaaneesh · 2025-01-20T12:37:59Z

migtests/tests/pg/datatypes/validateAfterChanges

@@ -130,6 +130,14 @@ def migration_completed_checks(tgt):
 	expected_distinct_values = EXPECTED_DISNTICT_VALUES['v5']
 	tgt.assert_distinct_values_of_col("datatypes2", "v5", "public", 
 		None, expected_distinct_values = expected_distinct_values)
+
+	print("hstore_example:")


This would run for all of target, source-replica, source(in case of fb), right?
how is it passing for fb, if YB cdc connector does not support it?

yes it runs on all target, source, and source-replica.
For the fall-forward/fall-back workflows we still run with the hstore_example table and snapshot data is still present in all the DBs but the CDC events are not added for the target_delta i.e. after cutover to target. So only CDC events from target for hstore are not getting tested.

makalaaneesh

LGTM

sanyamsinghal · 2025-01-20T13:45:32Z

...r-voyagerexporter/src/main/java/io/debezium/server/ybexporter/DebeziumRecordTransformer.java

+                    mapString.append("\"");
+                    mapString.append(key);
+                    mapString.append("\"");
+                    mapString.append(" => ");
+                    mapString.append("\"");
+                    mapString.append(val);
+                    mapString.append("\"");
+                    mapString.append(",");


@priyanshi-yb, do we have something like fmt.Sprintf() in Java to use here?

+1 that would be a better. No idea why I wrote it like this initially 😀

There is this String.format() which looks similar to fmt.Sprintf() can fix that in a follow up PR.

priyanshi-yb added 5 commits January 6, 2025 18:47

Fix: MAP value converter for debezium

1ccdb8e

add cdc events for live

78ba6c5

move hstore to PostgreToYBConverter

2f5b411

try

a952d74

schema log

7ed4582

priyanshi-yb force-pushed the priyanshi/fix-hstore branch from 94f78ec to 7ed4582 Compare January 7, 2025 17:51

priyanshi-yb added 5 commits January 8, 2025 11:15

default hstore mode

1a56540

json hstore mode

16399c4

cleanup

891cddf

cleanup

d2433c3

cleanup

a72b633

priyanshi-yb requested a review from makalaaneesh January 8, 2025 07:37

priyanshi-yb marked this pull request as ready for review January 8, 2025 07:37

makalaaneesh reviewed Jan 8, 2025

View reviewed changes

yb-voyager/src/tgtdb/suites/yugabytedbSuite.go Outdated Show resolved Hide resolved

priyanshi-yb added 2 commits January 8, 2025 21:20

changed the hstore handling mode to json and handling it in io.debezi…

2849530

…um.data.Json converter to convert it properly for complex cases as well

try java map

0a22864

priyanshi-yb force-pushed the priyanshi/fix-hstore branch from 8186ed4 to 0a22864 Compare January 8, 2025 16:13

fix reports

2c9e33b

priyanshi-yb force-pushed the priyanshi/fix-hstore branch from 60744f0 to 2c9e33b Compare January 8, 2025 16:44

priyanshi-yb added 2 commits January 8, 2025 22:19

add all datatype tests

73bffd0

fix compilation

b013864

priyanshi-yb force-pushed the priyanshi/fix-hstore branch from 17d7f9a to b013864 Compare January 8, 2025 17:30

priyanshi-yb added 4 commits January 9, 2025 00:07

fix run test to cat only in specific case

69c393f

fix

dd4e212

Fix java hashmap approach to escape " -> \"

b894c51

fix

c996e46

priyanshi-yb force-pushed the priyanshi/fix-hstore branch from 9c10882 to c996e46 Compare January 9, 2025 15:28

change the inserts to be proper

ffbe291

priyanshi-yb force-pushed the priyanshi/fix-hstore branch from 7025d2c to ffbe291 Compare January 10, 2025 16:04

priyanshi-yb added 2 commits January 12, 2025 18:15

fix expected file

24d9530

review comments

099f1c1

priyanshi-yb force-pushed the priyanshi/fix-hstore branch from c3d9a8c to 7c6f2c1 Compare January 15, 2025 15:35

fix unit test/expected file

15f04d6

priyanshi-yb force-pushed the priyanshi/fix-hstore branch from 7c6f2c1 to 15f04d6 Compare January 15, 2025 16:36

priyanshi-yb requested review from makalaaneesh January 16, 2025 05:58

priyanshi-yb added 2 commits January 16, 2025 13:41

add tests of reporting

d12e8ed

Merge branch 'main' into priyanshi/fix-hstore

d4265dc

makalaaneesh reviewed Jan 16, 2025

View reviewed changes

priyanshi-yb added 7 commits January 16, 2025 17:03

review comments

d746bd1

fix tests

4ce41f2

fix build

4f779a3

add --yes to import-data

1830dad

live-migration validateAfterchanges fix

88d0cf5

fix sample-is

03e2b96

clean up

cb27941

makalaaneesh reviewed Jan 20, 2025

View reviewed changes

makalaaneesh approved these changes Jan 20, 2025

View reviewed changes

priyanshi-yb merged commit ae2a72f into main Jan 20, 2025
67 checks passed

priyanshi-yb deleted the priyanshi/fix-hstore branch January 20, 2025 13:40

sanyamsinghal reviewed Jan 20, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix: MAP value converter for data via debezium #2156

Fix: MAP value converter for data via debezium #2156

priyanshi-yb commented Jan 6, 2025 •

edited

Loading

makalaaneesh Jan 16, 2025

makalaaneesh Jan 16, 2025

priyanshi-yb Jan 16, 2025

makalaaneesh Jan 16, 2025

makalaaneesh Jan 16, 2025

priyanshi-yb Jan 16, 2025

makalaaneesh Jan 20, 2025

priyanshi-yb Jan 20, 2025 •

edited

Loading

makalaaneesh left a comment

sanyamsinghal Jan 20, 2025

makalaaneesh Jan 20, 2025 •

edited

Loading

priyanshi-yb Jan 20, 2025 •

edited

Loading

Fix: MAP value converter for data via debezium #2156

Fix: MAP value converter for data via debezium #2156

Conversation

priyanshi-yb commented Jan 6, 2025 • edited Loading

Describe the changes in this pull request

Describe if there are any user-facing changes

How was this pull request tested?

Does your PR have changes that can cause upgrade issues?

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

priyanshi-yb Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

makalaaneesh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

makalaaneesh Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

priyanshi-yb Jan 20, 2025 • edited Loading

Choose a reason for hiding this comment

priyanshi-yb commented Jan 6, 2025 •

edited

Loading

priyanshi-yb Jan 20, 2025 •

edited

Loading

makalaaneesh Jan 20, 2025 •

edited

Loading

priyanshi-yb Jan 20, 2025 •

edited

Loading