esrally.exceptions.SystemSetupError: Received a protocol error. Are you sure you're using the correct scheme (HTTP or HTTPS)?

[ziyunxiao]

Rally version (get with esrally --version):
esrally 2.10.0

Invoked command:
esrally race
--track=geonames
--target-hosts=es1.mydomain.com:443
--client-options="use_ssl:true,verify_certs:false,basic_auth_user:'benchmark',basic_auth_password:'${ELASTICSEARCH_PASSWORD}'"
--pipeline=benchmark-only
--test-mode

Configuration file (located in ~/.rally/rally.ini)):

[meta]
config.version = 17

[system]
env.name = local

[node]
root.dir = /home2/rsong/.rally/benchmarks
src.root.dir = /home2/rsong/.rally/benchmarks/src

[source]
remote.repo.url = https://github.com/elastic/elasticsearch.git
elasticsearch.src.subdir = elasticsearch

[benchmarks]
local.dataset.cache = /home2/rsong/.rally/benchmarks/data

[reporting]
datastore.type = in-memory
datastore.host =
datastore.port =
datastore.secure = False
datastore.user =
datastore.password =


[tracks]
default.url = https://github.com/elastic/rally-tracks

[teams]
default.url = https://github.com/elastic/rally-teams

[defaults]
preserve_benchmark_candidate = false

[distributions]
release.cache = true

JVM version:
openjdk 11.0.21 2023-10-17
OpenJDK Runtime Environment (build 11.0.21+9-post-Ubuntu-0ubuntu120.04)
OpenJDK 64-Bit Server VM (build 11.0.21+9-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)

OS version:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal

Description of the problem including expected versus actual behavior:
When try to run esrally race against existing Elasticsearch cluster I got error "esrally.exceptions.SystemSetupError: Received a protocol error. Are you sure you're using the correct scheme (HTTP or HTTPS)?"

Steps to reproduce:

1. curl -u "benchmark:$ELASTICSEARCH_PASSWORD" -k https://es1.mydomain.com/ I use this command to verify that cluster is running and the username and password works
2. Issue command

esrally race \
    --track=geonames \
    --target-hosts=es1.mydomain.com:443 \
    --client-options="use_ssl:true,verify_certs:false,basic_auth_user:'benchmark',basic_auth_password:'${ELASTICSEARCH_PASSWORD}'" \
    --pipeline=benchmark-only \
    --test-mode

Provide logs (if relevant):

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[INFO] Race id is [f3e0b144-6d8b-453a-b2b3-e3139485e26f]
[ERROR] Cannot race. Traceback (most recent call last):
  File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/esrally/client/factory.py", line 289, in wait_for_rest_layer
    es.cluster.health(wait_for_nodes=f">={expected_node_count}")
  File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped
    return api(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/elasticsearch/_sync/client/cluster.py", line 464, in health
    return self.perform_request(  # type: ignore[return-value]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/elasticsearch/_sync/client/_base.py", line 390, in perform_request
    return self._client.perform_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/esrally/client/synchronous.py", line 195, in perform_request
    meta, resp_body = self.transport.perform_request(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/elastic_transport/_transport.py", line 328, in perform_request
    meta, raw_data = node.perform_request(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/elastic_transport/_node/_http_urllib3.py", line 197, in perform_request
    raise err from None
elastic_transport.ConnectionError: Connection error caused by: ConnectionError(Connection error caused by: ProtocolError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/esrally/actor.py", line 92, in guard
    return f(self, msg, sender)
           ^^^^^^^^^^^^^^^^^^^^
  File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/esrally/racecontrol.py", line 111, in receiveMsg_Setup
    self.coordinator.setup(sources=msg.sources)
  File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/esrally/racecontrol.py", line 194, in setup
    ) = client.factory.cluster_distribution_version(hosts, client_options)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/esrally/client/factory.py", line 364, in cluster_distribution_version
    wait_for_rest_layer(es)
  File "/data/data-science/rsong/bia-gke1-prod/elasticsearch/benchmark/.venv/lib/python3.11/site-packages/esrally/client/factory.py", line 307, in wait_for_rest_layer
    raise exceptions.SystemSetupError(
esrally.exceptions.SystemSetupError: Received a protocol error. Are you sure you're using the correct scheme (HTTP or HTTPS)?


Getting further help:
*********************
* Check the log files in /home2/rsong/.rally/logs for errors.
* Read the documentation at https://esrally.readthedocs.io/en/2.10.0/.
* Ask a question on the forum at https://discuss.elastic.co/tags/c/elastic-stack/elasticsearch/rally.
* Raise an issue at https://github.com/elastic/rally/issues and include the log files in /home2/rsong/.rally/logs.

-------------------------------
[INFO] FAILURE (took 4 seconds)

Describe the feature:
Expecting to be able to run race when ElasticSearch is configured behind a load balancer with TLS enabled.