Added GENERATE_ARRAY function #12293

aadilkhalifa · 2024-01-21T10:39:14Z

Pull request for the issue: #11786

Adding GENERATE_ARRAY function to allow generation of array of values with start, end and step value (optional).
Supported data types are int, long, float and double.
Currently does not support multi-stage query engine.

Eg. SELECT GENERATE_ARRAY(1, 5, 1) from table limit 1;

xiangfu0 · 2024-01-21T19:04:58Z

Thanks @aadilkhalifa for raising the PR!

Can you add some integration tests under org.apache.pinot.integration.tests.custom.ArrayTest

codecov-commenter · 2024-01-21T19:38:34Z

Codecov Report

Attention: Patch coverage is 41.93548% with 126 lines in your changes missing coverage. Please review.

Project coverage is 62.09%. Comparing base (59551e4) to head (cf17495).
Report is 738 commits behind head on master.

Files	Patch %	Lines
...sform/function/GenerateArrayTransformFunction.java	46.90%	84 Missing and 19 partials ⚠️
...e/pinot/common/function/scalar/ArrayFunctions.java	0.00%	20 Missing ⚠️
...r/transform/function/TransformFunctionFactory.java	0.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #12293      +/-   ##
============================================
+ Coverage     61.75%   62.09%   +0.34%     
+ Complexity      207      198       -9     
============================================
  Files          2436     2559     +123     
  Lines        133233   141154    +7921     
  Branches      20636    21915    +1279     
============================================
+ Hits          82274    87646    +5372     
- Misses        44911    46873    +1962     
- Partials       6048     6635     +587

Flag	Coverage Δ
custom-integration1	`<0.01% <0.00%> (-0.01%)`	⬇️
integration	`<0.01% <0.00%> (-0.01%)`	⬇️
integration1	`<0.01% <0.00%> (-0.01%)`	⬇️
integration2	`0.00% <0.00%> (ø)`
java-11	`62.05% <41.93%> (+0.34%)`	⬆️
java-21	`61.96% <41.93%> (+0.33%)`	⬆️
skip-bytebuffers-false	`62.07% <41.93%> (+0.32%)`	⬆️
skip-bytebuffers-true	`61.94% <41.93%> (+34.21%)`	⬆️
temurin	`62.09% <41.93%> (+0.34%)`	⬆️
unittests	`62.08% <41.93%> (+0.34%)`	⬆️
unittests1	`46.69% <41.93%> (-0.20%)`	⬇️
unittests2	`27.62% <0.00%> (-0.11%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

walterddr · 2024-01-23T03:06:43Z

pinot-common/src/test/java/org/apache/pinot/common/function/FunctionDefinitionRegistryTest.java

@@ -43,7 +43,7 @@ public class FunctionDefinitionRegistryTest {
      // Geo functions are defined in pinot-core
      "geotoh3",
      // ArrayToMV and ArrayValueConstructor are placeholder functions without implementation
-      "arraytomv", "arrayvalueconstructor",
+      "arraytomv", "arrayvalueconstructor", "arraygenerate",


can we also update the comment ^

I've updated the comment.

walterddr · 2024-01-23T03:07:41Z

pinot-common/src/main/java/org/apache/pinot/common/function/TransformFunctionType.java

@@ -271,6 +271,7 @@ public enum TransformFunctionType {
      OperandTypes.family(ImmutableList.of(SqlTypeFamily.ARRAY)), "vector_norm"),

  ARRAY_VALUE_CONSTRUCTOR("arrayValueConstructor", "array_value_constructor"),
+  ARRAY_GENERATE("arrayGenerate", "array_generate"),


what's the goal we want to achieve here? ARRAY_GENERATE unlike ARRAY_VALUE_CONSTRUCTOR is not a SqlStdOperatorTable entry so it will not work with multi-stage simply by adding this alone

I've removed this.

removing this entry will make your transform function implementation unable to register (thus unable to run / dead code).

i would suggest wait until #12302 to be checked in b/c the type system implication is complex for ARRAY_GEN
also another note to reviewer --> histogram does support a weird array gen. let's see if we can merge the 2

aadilkhalifa · 2024-01-27T07:26:50Z

Thanks @aadilkhalifa for raising the PR!

Can you add some integration tests under org.apache.pinot.integration.tests.custom.ArrayTest

@xiangfu0 I've added the integration tests.

Jackie-Jiang

Shall we keep the same name as BQ: generateArray. For the scalar function version, we can do generateIntArray, generateLongArray etc.

xiangfu0 · 2024-03-14T18:54:13Z

Shall we keep the same name as BQ: generateArray. For the scalar function version, we can do generateIntArray, generateLongArray etc.

+1, @aadilkhalifa can you update the function names?

aadilkhalifa · 2024-03-21T08:01:13Z

@Jackie-Jiang @xiangfu0 I've updated the names.

npawar · 2024-05-23T00:41:12Z

can this be merged and issue closed?

Jackie-Jiang · 2024-05-23T22:02:49Z

...n/java/org/apache/pinot/core/operator/transform/function/GenerateArrayTransformFunction.java

+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.roaringbitmap.RoaringBitmap;
+
+public class GenerateArrayTransformFunction implements TransformFunction {


@xiangfu0 I don't think we need a TransformFunction for this. It always only takes literal, thus always being resolved on the broker side

@aadilkhalifa Please take a look at this comment. Do you see this TransformFunction being used?

@Jackie-Jiang yes it is being called

aadilkhalifa · 2024-07-10T16:16:52Z

can this be merged and issue closed?

@xiangfu0 Can this be merged?

aadilkhalifa force-pushed the array-generate branch 3 times, most recently from 5f3c103 to 736db6f Compare January 21, 2024 10:52

aadilkhalifa changed the title ~~[DRAFT] Added ARRAY_GENERATE function~~ Added ARRAY_GENERATE function Jan 21, 2024

aadilkhalifa force-pushed the array-generate branch from 736db6f to 765b1f8 Compare January 21, 2024 11:04

walterddr reviewed Jan 23, 2024

View reviewed changes

aadilkhalifa force-pushed the array-generate branch from 765b1f8 to ef93fe0 Compare January 27, 2024 07:23

aadilkhalifa force-pushed the array-generate branch from ef93fe0 to 3924d74 Compare January 27, 2024 09:06

walterddr added the feature label Feb 2, 2024

Jackie-Jiang reviewed Mar 12, 2024

View reviewed changes

aadilkhalifa force-pushed the array-generate branch 2 times, most recently from d4498fb to 488a2ee Compare March 21, 2024 08:00

aadilkhalifa changed the title ~~Added ARRAY_GENERATE function~~ Added GENERATE_ARRAY function Mar 21, 2024

aadilkhalifa force-pushed the array-generate branch from 488a2ee to 63fe61b Compare March 22, 2024 06:14

xiangfu0 self-assigned this Apr 9, 2024

xiangfu0 approved these changes Apr 14, 2024

View reviewed changes

Jackie-Jiang reviewed May 23, 2024

View reviewed changes

aadilkhalifa force-pushed the array-generate branch 2 times, most recently from 7d23fb7 to 6dc75af Compare June 18, 2024 09:22

aadilkhalifa force-pushed the array-generate branch from 6dc75af to e1131e7 Compare July 10, 2024 14:17

Added SHOW TABLES and DESCRIBE table support

cf17495

aadilkhalifa force-pushed the array-generate branch from e1131e7 to cf17495 Compare July 10, 2024 15:11

Jackie-Jiang merged commit 3015b2e into apache:master Jul 15, 2024
20 checks passed

rajagopr pushed a commit to rajagopr/pinot that referenced this pull request Jul 17, 2024

Added SHOW TABLES and DESCRIBE table support (apache#12293)

6fcfd01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added GENERATE_ARRAY function #12293

Added GENERATE_ARRAY function #12293

aadilkhalifa commented Jan 21, 2024 •

edited

Loading

xiangfu0 commented Jan 21, 2024 •

edited

Loading

codecov-commenter commented Jan 21, 2024 •

edited

Loading

walterddr Jan 23, 2024

aadilkhalifa Jan 27, 2024

walterddr Jan 23, 2024

aadilkhalifa Jan 27, 2024

walterddr Jan 28, 2024

aadilkhalifa Jan 29, 2024

aadilkhalifa commented Jan 27, 2024

Jackie-Jiang left a comment

xiangfu0 commented Mar 14, 2024

aadilkhalifa commented Mar 21, 2024

npawar commented May 23, 2024

Jackie-Jiang May 23, 2024 •

edited

Loading

Jackie-Jiang Jul 11, 2024

aadilkhalifa Jul 12, 2024

aadilkhalifa commented Jul 10, 2024

Added GENERATE_ARRAY function #12293

Added GENERATE_ARRAY function #12293

Conversation

aadilkhalifa commented Jan 21, 2024 • edited Loading

xiangfu0 commented Jan 21, 2024 • edited Loading

codecov-commenter commented Jan 21, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aadilkhalifa commented Jan 27, 2024

Jackie-Jiang left a comment

Choose a reason for hiding this comment

xiangfu0 commented Mar 14, 2024

aadilkhalifa commented Mar 21, 2024

npawar commented May 23, 2024

Jackie-Jiang May 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aadilkhalifa commented Jul 10, 2024

aadilkhalifa commented Jan 21, 2024 •

edited

Loading

xiangfu0 commented Jan 21, 2024 •

edited

Loading

codecov-commenter commented Jan 21, 2024 •

edited

Loading

Jackie-Jiang May 23, 2024 •

edited

Loading