Create Job API

[vmmusings]

Description

• Datasource extraction from query will be handled in next PR.
• Documentation : https://github.com/vamsi-amazon/sql/blob/create-job-api/docs/user/interfaces/asyncqueryinterface.rst
• In the current PR, job APIs will be disabled if user doesn't configure plugins.query.executionengine.spark.config config. Once the customer updates the setting, user needs a domain restart to enable Job APIs.[Suggestions are welcome]
• Integration tests are not added, as we need emr serverless set up.

Sample Request Responses.

Job Creation API

curl --location 'http://ec2-54-70-149-21.us-west-2.compute.amazonaws.com:9200/_plugins/_query/_jobs' \
--header 'Content-Type: application/json' \
--data '{
    "query" : "select * from default.http_logs limit 10"
}'

{
  "jobId": "00fd796ut1a7eg0q"
}

Job Result API:

curl --location --request GET 'http://ec2-54-70-149-21.us-west-2.compute.amazonaws.com:9200/_plugins/_query/_jobs/00fd796ut1a7eg0q/result' \
--header 'Content-Type: application/json' \
--data '{
   "query" : "select * from default.http_logs limit 1"
}'

{"status":"RUNNING"}

Issues Resolved

Check List

• New functionality includes testing.
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented.
  • New functionality has javadoc added
  • New functionality has user manual doc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Merging #2070 (f96a1a5) into feature/job-apis (a7af359) will increase coverage by 0.21%.
The diff coverage is 94.27%.

@@                  Coverage Diff                   @@
##             feature/job-apis    #2070      +/-   ##
======================================================
+ Coverage               96.36%   96.57%   +0.21%     
- Complexity               4643     4696      +53     
======================================================
  Files                     425      433       +8     
  Lines                   12106    12387     +281     
  Branches                  832      847      +15     
======================================================
+ Hits                    11666    11963     +297     
+ Misses                    431      413      -18     
- Partials                    9       11       +2     

Flag	Coverage Δ
sql-engine	96.57% <94.27%> (+0.21%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...org/opensearch/sql/spark/client/EmrClientImpl.java	100.00% <ø> (ø)
...earch/sql/spark/data/constants/SparkConstants.java	0.00% <ø> (ø)
...sql/spark/rest/RestAsyncQueryManagementAction.java	0.00% <0.00%> (ø)
.../sql/spark/rest/model/CreateAsyncQueryRequest.java	0.00% <0.00%> (ø)
...transport/model/CancelAsyncQueryActionRequest.java	0.00% <ø> (ø)
...ransport/model/CancelAsyncQueryActionResponse.java	0.00% <ø> (ø)
...ransport/model/CreateAsyncQueryActionResponse.java	0.00% <ø> (ø)
...nsport/model/GetAsyncQueryResultActionRequest.java	0.00% <ø> (ø)
...sport/model/GetAsyncQueryResultActionResponse.java	0.00% <ø> (ø)
.../spark/asyncquery/model/AsyncQueryJobMetadata.java	85.18% <85.18%> (ø)
... and 18 more

[YANG-DB]

@vamsi-amazon can u plz add a swagger file for a consolidated view and documentation for this API ?

[penghuo]

the construct of SparkExecutionEngine should independent of JobExecutorService.

[vmmusings]

SparkExecutionEngineConfig contains applicationId which can change and it is picked up runtime during query execution. Probably I need to split up the settings.

[vmmusings]

@vamsi-amazon can u plz add a swagger file for a consolidated view and documentation for this API ?

@YANG-DB will address this in a separate task.