Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Review TDVT results differences between JDBC & ODBC connectors #25

Open
hab6 opened this issue Aug 16, 2023 · 5 comments
Open

Review TDVT results differences between JDBC & ODBC connectors #25

hab6 opened this issue Aug 16, 2023 · 5 comments
Assignees

Comments

@hab6
Copy link
Collaborator

hab6 commented Aug 16, 2023

Testing of JDBC and ODBC tacos built with recent changes to the connector code produces results that are very close, although not identical, between the two tacos.

Connector Passed Failed Total Tests Total Time
JDBC 819 46 865 50.13s
ODBC 823 42 865 25.78s

We need to investigate and compare the results, ensuring at a minimum that all of the tests passing for the ODBC connector are also passing for the JDBC connector.

  1. error/pass counts - priority get ensure all passing ODBC tests also pass for JDBC
  2. performance - potentially a new/separate issue - may need to pull in additional team members once we have a handle on this.
@hab6 hab6 self-assigned this Aug 16, 2023
@hab6 hab6 mentioned this issue Aug 16, 2023
@clach04
Copy link
Member

clach04 commented Aug 16, 2023

@hab6 I've opened #27 which maybe relevant, specifically the "Changes / TODO notes that need follow up" section.

@clach04
Copy link
Member

clach04 commented Aug 16, 2023

Those timings are a concern, 🐢 100% slower 😱

@hab6
Copy link
Collaborator Author

hab6 commented Aug 17, 2023

For a quick sanity check, I ran the TDVT suite a few times with the JDBC and ODBC connectors of release 1.0.2. The ODBC connector was more than twice as fast as the JDBC connector.

JDBC ODBC
50.98s
49.47s
49.39s
24.78s
22.18s
21.39s
Avg
49.95
Avg
22.78

Investigating TDVT results with these priorities:

  1. Error/pass counts
  2. Performance

@hab6
Copy link
Collaborator Author

hab6 commented Aug 17, 2023

These are the 4 TDVT tests that fail when using the Actian JDBC connector and yet do NOT fail when using the Actian ODBC connector.

TestName TestPath Test Case
date.datename.sow.week setup.date.datename.sow.week.txt DATENAME('week', [date2], 'sunday')
date.datename.sow.week setup.date.datename.sow.week.txt DATENAME('week', [datetime0], 'sunday')
date.datename.week setup.date.datename.week.txt DATENAME('week', [date2])
date.datename.week setup.date.datename.week.txt DATENAME('week', [datetime0])

Test Set: expression.standard.mahds
TDSName: cast_calcs.mahds
GitHub Location of Failing Tests: https://github.com/tableau/connector-plugin-sdk/tree/master/tdvt/tdvt/exprtests/standard

@hab6
Copy link
Collaborator Author

hab6 commented Aug 28, 2023

Test Results with the latest TDVT suite

JDBC Connector v1.0.3(dev)

Target System Passed (Rate) Failed Total Tests Avg Time
Ingres 11.2 local Windows 823 (95%) 45 868 51s
Ingres 11.2 remote Ubuntu 820 (94%) 48 868 2m 35s
Vector 6.3 remote Ubuntu 831 (96%) 37 868 1m 25s

ODBC Connector v1.0.3(dev)

Target System Passed (Rate) Failed Total Tests Avg Time
Ingres 11.2 local Windows 823 (95%) 45 868 26s
Ingres 11.2 remote Ubuntu 818 (94%) 50 868 9m
Vector 6.3 remote Ubuntu 831 (96%) 37 868 4m

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants