Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](runtime filter) Fix runtime filter producers #44294

Merged
merged 1 commit into from
Nov 20, 2024

Conversation

Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Nov 19, 2024

What problem does this PR solve?

Problem Summary:

A runtime filter producer may have multiple targets some of which are managed in global mgr and others are managed in local mgr. To process it, producer will be shared by both of global mgr and local mgr. In this PR, a producer will be always created by a local mgr and we can always find it by a queryCtx's RF mgr.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.02% (9900/26036)
Line Coverage: 29.20% (82800/283544)
Region Coverage: 28.34% (42522/150055)
Branch Coverage: 24.91% (21564/86584)
Coverage Report: http://coverage.selectdb-in.cc/coverage/45ec850edc7b002659a6735efb5af41d15700bce_45ec850edc7b002659a6735efb5af41d15700bce/report/index.html

@Gabriel39
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.02% (9900/26036)
Line Coverage: 29.20% (82807/283542)
Region Coverage: 28.34% (42522/150053)
Branch Coverage: 24.91% (21569/86582)
Coverage Report: http://coverage.selectdb-in.cc/coverage/728a1f01f0638d2d09143fe5ad7254c070f2cbe1_728a1f01f0638d2d09143fe5ad7254c070f2cbe1/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 40181 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 728a1f01f0638d2d09143fe5ad7254c070f2cbe1, data reload: false

------ Round 1 ----------------------------------
q1	17603	7595	7344	7344
q2	2045	177	173	173
q3	10533	1178	1257	1178
q4	10239	687	773	687
q5	7654	3275	2771	2771
q6	246	152	151	151
q7	991	638	622	622
q8	9264	1860	1878	1860
q9	6680	6473	6455	6455
q10	7014	2274	2352	2274
q11	454	261	255	255
q12	409	221	223	221
q13	17767	3065	3014	3014
q14	254	212	218	212
q15	577	537	526	526
q16	650	585	573	573
q17	991	534	538	534
q18	7511	6672	6650	6650
q19	1348	968	968	968
q20	471	183	190	183
q21	4099	3289	3215	3215
q22	389	327	315	315
Total cold run time: 107189 ms
Total hot run time: 40181 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7311	7271	7317	7271
q2	324	239	228	228
q3	2962	2790	2788	2788
q4	2015	1676	1778	1676
q5	5412	5498	5427	5427
q6	223	140	138	138
q7	2176	1731	1728	1728
q8	3294	3443	3457	3443
q9	8630	8591	8607	8591
q10	3510	3488	3455	3455
q11	595	508	495	495
q12	802	603	582	582
q13	11087	3059	3057	3057
q14	287	277	266	266
q15	569	522	525	522
q16	664	626	632	626
q17	1827	1591	1590	1590
q18	8120	7578	7434	7434
q19	1689	1503	1533	1503
q20	2127	1893	1852	1852
q21	5467	5491	5215	5215
q22	610	554	554	554
Total cold run time: 69701 ms
Total hot run time: 58441 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193502 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 728a1f01f0638d2d09143fe5ad7254c070f2cbe1, data reload: false

query1	976	393	397	393
query2	6542	2189	2134	2134
query3	6724	214	217	214
query4	33589	23663	23562	23562
query5	4365	474	452	452
query6	278	184	191	184
query7	4627	304	312	304
query8	297	241	235	235
query9	9512	2804	2779	2779
query10	464	256	261	256
query11	18182	15412	15308	15308
query12	167	105	104	104
query13	1678	425	434	425
query14	10893	7767	7269	7269
query15	294	184	179	179
query16	7980	504	470	470
query17	1795	583	563	563
query18	2038	300	288	288
query19	364	155	148	148
query20	119	110	111	110
query21	200	101	102	101
query22	5160	4866	4856	4856
query23	35031	34320	34169	34169
query24	11262	2475	2543	2475
query25	696	399	396	396
query26	1727	147	149	147
query27	2809	276	289	276
query28	8085	2487	2474	2474
query29	982	410	414	410
query30	304	150	154	150
query31	1030	808	829	808
query32	97	55	58	55
query33	783	300	272	272
query34	965	510	543	510
query35	879	741	733	733
query36	1111	945	981	945
query37	141	80	75	75
query38	4439	4260	4222	4222
query39	1468	1445	1434	1434
query40	281	100	102	100
query41	48	44	45	44
query42	107	98	99	98
query43	552	516	496	496
query44	1241	830	823	823
query45	187	165	165	165
query46	1149	704	706	704
query47	1949	1876	1827	1827
query48	449	316	340	316
query49	1244	386	390	386
query50	828	378	389	378
query51	7271	7223	7133	7133
query52	101	87	88	87
query53	266	184	182	182
query54	1217	404	421	404
query55	83	81	77	77
query56	257	250	232	232
query57	1363	1235	1230	1230
query58	242	219	225	219
query59	3353	3134	3063	3063
query60	281	257	248	248
query61	112	110	112	110
query62	924	667	673	667
query63	213	186	190	186
query64	5115	676	664	664
query65	3301	3208	3245	3208
query66	1418	321	321	321
query67	16341	15973	15935	15935
query68	5019	562	568	562
query69	435	261	255	255
query70	1222	1156	1123	1123
query71	437	252	260	252
query72	6425	4045	4040	4040
query73	788	363	372	363
query74	10694	9098	9181	9098
query75	3501	2662	2687	2662
query76	3110	1153	1131	1131
query77	544	292	280	280
query78	9999	9276	9233	9233
query79	1092	616	603	603
query80	633	453	437	437
query81	499	234	232	232
query82	213	120	123	120
query83	175	159	156	156
query84	242	79	77	77
query85	1071	447	299	299
query86	306	311	294	294
query87	4826	4742	4675	4675
query88	3693	2259	2193	2193
query89	392	301	298	298
query90	1964	193	185	185
query91	141	103	102	102
query92	60	55	54	54
query93	1068	545	548	545
query94	942	366	301	301
query95	343	251	257	251
query96	634	287	284	284
query97	2834	2700	2715	2700
query98	209	201	202	201
query99	1539	1314	1308	1308
Total cold run time: 302996 ms
Total hot run time: 193502 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.21 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 728a1f01f0638d2d09143fe5ad7254c070f2cbe1, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.11	0.10
query5	0.42	0.41	0.40
query6	1.16	0.66	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.51	0.51
query10	0.56	0.57	0.56
query11	0.14	0.11	0.11
query12	0.14	0.11	0.12
query13	0.61	0.60	0.58
query14	2.73	2.72	2.70
query15	0.91	0.83	0.82
query16	0.38	0.39	0.39
query17	1.08	1.05	1.07
query18	0.19	0.20	0.19
query19	1.96	1.89	2.02
query20	0.02	0.02	0.01
query21	15.35	0.59	0.58
query22	2.94	1.97	1.81
query23	16.80	0.97	0.89
query24	2.79	1.50	1.33
query25	0.33	0.10	0.10
query26	0.48	0.13	0.13
query27	0.04	0.04	0.04
query28	10.24	1.09	1.08
query29	12.52	3.23	3.18
query30	0.24	0.06	0.06
query31	2.86	0.38	0.39
query32	3.26	0.45	0.48
query33	2.97	3.10	3.09
query34	16.88	4.52	4.51
query35	4.54	4.57	4.53
query36	0.67	0.51	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.29 s
Total hot run time: 33.21 s

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 20, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

@Gabriel39 Gabriel39 merged commit eef9400 into apache:master Nov 20, 2024
28 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.3-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants