Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Align pool_maxsize for different connection pool implementations. #535

Merged
merged 4 commits into from
Oct 12, 2023

Conversation

dblock
Copy link
Member

@dblock dblock commented Oct 12, 2023

Description

  • Enable pool_maxsize for both Urllib3HttpConnection and RequestsHttpConnection pool implementations.
  • Document how and when to switch between connection pools.
  • Remove RequestsHttpConnection from the documentation when used as a default.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@codecov
Copy link

codecov bot commented Oct 12, 2023

Codecov Report

Merging #535 (b3a5ed5) into main (84ac172) will increase coverage by 0.05%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main     #535      +/-   ##
==========================================
+ Coverage   70.72%   70.77%   +0.05%     
==========================================
  Files          83       83              
  Lines        7852     7857       +5     
==========================================
+ Hits         5553     5561       +8     
+ Misses       2299     2296       -3     
Files Coverage Δ
opensearchpy/connection/http_urllib3.py 86.02% <100.00%> (+0.30%) ⬆️
opensearchpy/transport.py 92.16% <100.00%> (+0.14%) ⬆️

... and 1 file with indirect coverage changes

@dblock dblock force-pushed the pool-maxsize branch 3 times, most recently from 36fb442 to 4c1f049 Compare October 12, 2023 19:28
guides/connection_classes.md Outdated Show resolved Hide resolved
use_ssl = True,
verify_certs = False,
ssl_show_warn = False,
connection_class = Urllib3HttpConnection
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we make Urllib3HttpConnection default in code unless a connection_class is explicitly passed in?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's already default here:

DEFAULT_CONNECTION_CLASS = Urllib3HttpConnection

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I missed it, so only the documentation was a bit misleading.

Signed-off-by: dblock <[email protected]>
Copy link
Collaborator

@VachaShah VachaShah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @dblock!

@navneet1v
Copy link

@dblock can we add the recommendation somewhere in doc on what HTTPClient to use? with the benchmarks you have run.

@dblock
Copy link
Member Author

dblock commented Oct 12, 2023

@dblock can we add the recommendation somewhere in doc on what HTTPClient to use? with the benchmarks you have run.

In this PR I had added "We recommend you use the default, unless your application is standardized on requests." LMK if you think it should say something else.

@dblock dblock merged commit 7a638cd into opensearch-project:main Oct 12, 2023
54 checks passed
@dblock dblock deleted the pool-maxsize branch October 12, 2023 22:55
Djcarrillo6 added a commit to Djcarrillo6/opensearch-py that referenced this pull request Oct 14, 2023
Signed-off-by: Djcarrillo6 <[email protected]>

Updated CHANGELOG

Signed-off-by: Djcarrillo6 <[email protected]>

Updated CHANGELOG & link to sample.

Signed-off-by: Djcarrillo6 <[email protected]>

updated changelog (opensearch-project#522)

Signed-off-by: saimedhi <[email protected]>

Bump version to 2.3.2 (opensearch-project#524)

Signed-off-by: saimedhi <[email protected]>

Fix: typos. (opensearch-project#526)

* Fix: typo.

Signed-off-by: dblock <[email protected]>

* Fix: typo.

Signed-off-by: dblock <[email protected]>

* Fixed its.

Signed-off-by: dblock <[email protected]>

* Added Visual Code settings to .gitignore.

Signed-off-by: dblock <[email protected]>

* Added loop type for async client.

Signed-off-by: dblock <[email protected]>

---------

Signed-off-by: dblock <[email protected]>

Modified generator to generate api deprecation warnings (opensearch-project#527)

Signed-off-by: saimedhi <[email protected]>

Generate cat client from API specs (opensearch-project#529)

Signed-off-by: saimedhi <[email protected]>

Generate cluster client from API specs (opensearch-project#530)

Signed-off-by: saimedhi <[email protected]>

Added new guide & sample module for using index templates. (opensearch-project#531)

Added index_template guide and sample

Signed-off-by: Djcarrillo6 <[email protected]>

Removed EOL Python3.5 & bumped urllib3 version to patch security vulnerability (opensearch-project#533)

Updated CHANGELOG with pull #

Updated CHANGELOG with pull #

Updated CHANGELOG removed section.

Updated CHANGELOG removed section again

Signed-off-by: Djcarrillo6 <[email protected]>

Align pool_maxsize for different connection pool implementations. (opensearch-project#535)

* Align pool_maxsize for different connection pool implementations.

Signed-off-by: dblock <[email protected]>

* Document connection classes and settings.

Signed-off-by: dblock <[email protected]>

* Undo change in async for backwards compatibility.

Signed-off-by: dblock <[email protected]>

* Fix: typo.

Signed-off-by: dblock <[email protected]>

---------

Signed-off-by: dblock <[email protected]>

Add micro benchmarks. (opensearch-project#537)

* Align pool_maxsize for different connection pool implementations.

Signed-off-by: dblock <[email protected]>

* Added benchmarks.

Signed-off-by: dblock <[email protected]>

* Multi-threaded vs. async benchmarks.

Signed-off-by: dblock <[email protected]>

* Set pool size to the number of threads.

Signed-off-by: dblock <[email protected]>

* Added sync/async benchmark.

Signed-off-by: dblock <[email protected]>

* Report client-side latency.

Signed-off-by: dblock <[email protected]>

* Various updates to benchmarks, demonstrating threading improves throughput.

Signed-off-by: dblock <[email protected]>

* Bench info.

Signed-off-by: dblock <[email protected]>

* Fixup format.

Signed-off-by: dblock <[email protected]>

* Undo async maxsize.

Signed-off-by: dblock <[email protected]>

* Moved benchmarks folder.

Signed-off-by: dblock <[email protected]>

* Updated documentation and project description.

Signed-off-by: dblock <[email protected]>

---------

Signed-off-by: dblock <[email protected]>
roma2023 pushed a commit to roma2023/opensearch-py that referenced this pull request Dec 28, 2023
…ensearch-project#535)

* Align pool_maxsize for different connection pool implementations.

Signed-off-by: dblock <[email protected]>

* Document connection classes and settings.

Signed-off-by: dblock <[email protected]>

* Undo change in async for backwards compatibility.

Signed-off-by: dblock <[email protected]>

* Fix: typo.

Signed-off-by: dblock <[email protected]>

---------

Signed-off-by: dblock <[email protected]>
Signed-off-by: roma2023 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants