RTC Query taking forever

[MartinBarkerPhilips]

Hello, I have an RTC Query URL that If I open in my browser loads very quickly and only shows 302 items (web display is paginated, shows "1 - 50 of 302" and 'previous' / 'next' buttons)

If I query for the URL with an empty string for returned properties:

query_client.runSavedQueryByUrl(query_url, returned_properties="")

it takes 28 / 30 minutes to return a list of all 302 work items.

I'm running on version 6.0 of this package because my RTC client is an older version, and auth only worked with version 6.0 and some added code to prevent 'SSL: DH_KEY_TOO_SMALL' errors

 requests.packages.urllib3.disable_warnings()
 requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS += ':HIGH:!DH:!aNULL'
 try:
     requests.packages.urllib3.contrib.pyopenssl.util.ssl_.DEFAULT_CIPHERS += ':HIGH:!DH:!aNULL'
 except AttributeError:
     # no pyopenssl support used / needed / available
     pass

Would this query run faster on the current version? Or is there some way I could run the query faster, such as breaking it up into multiple smaller queries and running those in unison?

My script is really hindered by these long query times.

Thanks

[dixudx]

If I query for the URL with an empty string for returned properties:
query_client.runSavedQueryByUrl(query_url, returned_properties="")
it takes 28 / 30 minutes to return a list of all 302 work items.

@MartinBarkerPhilips I'd suggest setting a non-empty string for returned properties, such as returned_properties="dc:title,dc:identifier,rtc_cm:state,rtc_cm:ownedBy".

#148 introduced multi-processing which can speed up the performance. This feature has been shipped with v0.8.0.

A speed-up of approximately 43% for a runSavedQueryByID call with 82 entries.

---

I'm running on version 6.0 of this package because my RTC client is an older version, and auth only worked with version 6.0

I've cherry-picked commits from #148 to a new branch release-0.6, which was checkout from official release 0.6.0. It will be convenient to stick with older versions.

You can install it with,

pip install --force-reinstall git+https://github.com/dixudx/[email protected]

[MartinBarkerPhilips]

Thank you for your response I appreciate the linked and mentioned new branch, I will try using it and let you know how it goes.
Currently working on an RTC to ADS migration script, will make it public once it's finished

[MartinBarkerPhilips]

@dixudx

C:\Users\320178017\Documents\repos>python
Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from rtcclient.utils import setup_basic_logging
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\320178017\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\rtcclient\__init__.py", line 31, in <module>
    from rtcclient.client import RTCClient
  File "C:\Users\320178017\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\rtcclient\client.py", line 1372
    while True:
    ^
IndentationError: expected an indented block after 'with' statement on line 1371
>>>

There is some indentation issue that happens when I run the command from rtcclient.utils import setup_basic_logging
I'm looking into trying to fix it, seems to just be indentation issue on line 1371 of client.py

[MartinBarkerPhilips]

i believe I fixed the indentation issue to the best of my ability, it doesnt throw an error but Im not 100% sure if my solution is the correct intended indentation order.

Either way my local version of the repo with the indentation fix + 'ssl too small' err fix, I get an error when trying to run

ISD_Project_Area = rtc_client.getProjectArea(projectarea_name="ULT")

err:

    raise exception.NotFound("No ProjectArea named %s" % projectarea_name)
rtcclient.exception.NotFound: No ProjectArea named RTC-ADS

Even though this line has worked fine in version0.6

[MartinBarkerPhilips]

Here is my repo fork:
https://github.com/MartinBarkerPhilips/rtcclient
started with your release-0.6 branch and made these changes:

• indentation error fix to client.py
• added fix for 'ssl too small' err in base.py
• added additional logging to getProjectArea() to try and solve why it no longer works

[MartinBarkerPhilips]

I still can't figure out why getProjectArea() no longer works, when It's receiving the exact same auth credentials / project name string as before. That error is blocking my progress using this repo branch

[MartinBarkerPhilips]

master doesnt work for me because im running an older version of rtc, and the custom release-0.6 branch had some file indentation formatting errors last time I tried to run it

[dixudx]

@MartinBarkerPhilips I've cherry-picked multi-thread patches to v0.6.0.

Please install with

pip install --force-reinstall git+https://github.com/dixudx/[email protected]

[MartinBarkerPhilips]

@dixudx running this command results in an error:

 pip install --force-reinstall git+https://github.com/dixudx/[email protected]
Collecting git+https://github.com/dixudx/[email protected]
  Cloning https://github.com/dixudx/rtcclient.git (to revision release-0.6) to c:\users\320178017\appdata\local\temp\pip-req-build-qut9uccb
  Running command git clone --filter=blob:none --quiet https://github.com/dixudx/rtcclient.git 'C:\Users\320178017\AppData\Local\Temp\pip-req-build-qut9uccb'
  Running command git checkout -b release-0.6 --track origin/release-0.6
  branch 'release-0.6' set up to track 'origin/release-0.6'.
  Switched to a new branch 'release-0.6'
  Resolved https://github.com/dixudx/rtcclient.git to commit 10876a0111d9c1fad83aba638f0e3b5c28170988
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [26 lines of output]
      C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1264.0_x64__qbz5n2kfra8p0\lib\site-packages\setuptools\dist.py:717: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
        warnings.warn(
      C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1264.0_x64__qbz5n2kfra8p0\lib\site-packages\setuptools\dist.py:717: UserWarning: Usage of dash-separated 'home-page' will not be supported in future versions. Please use the underscore name 'home_page' instead
        warnings.warn(
      C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.1264.0_x64__qbz5n2kfra8p0\lib\site-packages\setuptools\dist.py:51: DistDeprecationWarning: Do not call this function        warnings.warn("Do not call this function", DistDeprecationWarning)
      ERROR:root:Error parsing
      Traceback (most recent call last):
        File "c:\users\320178017\appdata\local\temp\pip-req-build-qut9uccb\.eggs\pbr-0.11.1-py3.10.egg\pbr\core.py", line 109, in pbr
          attrs = util.cfg_to_args(path)
        File "c:\users\320178017\appdata\local\temp\pip-req-build-qut9uccb\.eggs\pbr-0.11.1-py3.10.egg\pbr\util.py", line 243, in cfg_to_args
          pbr.hooks.setup_hook(config)
        File "c:\users\320178017\appdata\local\temp\pip-req-build-qut9uccb\.eggs\pbr-0.11.1-py3.10.egg\pbr\hooks\__init__.py", line 25, in setup_hook
          metadata_config.run()
        File "c:\users\320178017\appdata\local\temp\pip-req-build-qut9uccb\.eggs\pbr-0.11.1-py3.10.egg\pbr\hooks\base.py", line 27, in run
          self.hook()
        File "c:\users\320178017\appdata\local\temp\pip-req-build-qut9uccb\.eggs\pbr-0.11.1-py3.10.egg\pbr\hooks\metadata.py", line 25, in hook
          self.config['version'] = packaging.get_version(
        File "c:\users\320178017\appdata\local\temp\pip-req-build-qut9uccb\.eggs\pbr-0.11.1-py3.10.egg\pbr\packaging.py", line 645, in get_version
          version = _get_version_from_git(pre_version)
        File "c:\users\320178017\appdata\local\temp\pip-req-build-qut9uccb\.eggs\pbr-0.11.1-py3.10.egg\pbr\packaging.py", line 587, in _get_version_from_git
          result = _get_version_from_git_target(git_dir, target_version)
        File "c:\users\320178017\appdata\local\temp\pip-req-build-qut9uccb\.eggs\pbr-0.11.1-py3.10.egg\pbr\packaging.py", line 546, in _get_version_from_git_target
          raise ValueError(
      ValueError: git history requires a target version of pbr.version.SemanticVersion(0.6.1), but target version is pbr.version.SemanticVersion(0.6.0)
      error in setup command: Error parsing C:\Users\320178017\AppData\Local\Temp\pip-req-build-qut9uccb\setup.cfg: ValueError: git history requires a target version of pbr.version.SemanticVersion(0.6.1), but target version is pbr.version.SemanticVersion(0.6.0)
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[MartinBarkerPhilips]

tried to fix by installing with pip3 and upgrading my python tools, but trying to run the command again results in different error:

ValueError: git history requires a target version of pbr.version.SemanticVersion(0.6.1), but target version is pbr.version.SemanticVersion(0.6.0)
      error in setup command: Error parsing C:\Users\320178017\AppData\Local\Temp\pip-req-build-c54jy_ka\setup.cfg: ValueError: git history requires a target version of pbr.version.SemanticVersion(0.6.1), but target version is pbr.version.SemanticVersion(0.6.0)
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

[MartinBarkerPhilips]

Tried to clone repo, checkout to release-0.6 branch, and pip install -e the folder locally but got same error.

i made a fix , had to change
version = 0.6.0 to version = 0.6.1 in setup.cfg

[MartinBarkerPhilips]

after cloning [email protected] and making setup.cfg version fix, as well as adding:

        requests.packages.urllib3.disable_warnings()
        requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS += ':HIGH:!DH:!aNULL'
        try:
            requests.packages.urllib3.contrib.pyopenssl.util.ssl_.DEFAULT_CIPHERS += ':HIGH:!DH:!aNULL'
        except AttributeError:
            # no pyopenssl support used / needed / available
            pass

to base.py so that the package works with my companies older version of rtc, the release-0.6 branch worked.

I've been using rtcclient for a project I've been working on, an rtc to ads migration script. I created some custom rtc functions for my use. i want to release my script publicly so maybe there could be some branch / version of this repo I use that has support for my older rtc instance and my custom functions?
( i can add my functions to master branch as well, maybe I could add my ssl code fix to release-0.6 as well and use that branch/version? )

[MartinBarkerPhilips]

Facing a problem where when I run a query from a URL I only get at max 34 responses.
Is there a different way to run rtc queries with multithreading? If I try this method:

from datetime import datetime
import time

import RTC
rtc_client = RTC.rtcclient
query_client = rtc_client.query

print('Beginning RTC query')


start_time = time.time()      

query_url="https://rtcus1.ta.philips.com/ccm/web/projects/ULT#action=com.ibm.team.workitem.runSavedQuery&id=_ztYxxxxxxxxxxxxxxxxxMA"

returned_prop="dc:title,dc:identifier,rtc_cm:state,rtc_cm:ownedBy"

query_results = query_client.runSavedQueryByUrl(
    query_url,
    returned_prop
    )
    returned_properties='dc:type,dc:subject,dc:title,dc:description,dc:identifier,rtc_cm:ownedBy,rtc_cm:modifiedBy,rtc_cm:state,rtc_cm:targeted_release,oslc_cm:priority,dc:type,rtc_cm:plannedFor'
    )


duration=time.time() - start_time

print('Found '+ str(len(query_results)) + ' results in ' + str(round(duration, 2)) +' seconds.')
print('done')     

my query url is for 200+ results, but only ever returns 34 in my query_results list

[MartinBarkerPhilips]

tried to follow the instructions in the docs https://rtcclient.readthedocs.io/en/0.6.0/quickstart.html#query-workitems-by-saved-query-url

[dixudx]

i made a fix , had to change
version = 0.6.0 to version = 0.6.1 in setup.cfg

Good job. I've updated this change to branch release-0.6 as well.

[dixudx]

i want to release my script publicly so maybe there could be some branch / version of this repo I use that has support for my older rtc instance and my custom functions?
( i can add my functions to master branch as well, maybe I could add my ssl code fix to release-0.6 as well and use that branch/version? )

Sure. We can add this to master branch, and cherry-pick to release-0.6 later. @MartinBarkerPhilips

[dixudx]

Facing a problem where when I run a query from a URL I only get at max 34 responses.
Is there a different way to run rtc queries with multithreading?

Both runSavedQuery and runSavedQueryByUrl are ok.
Move to the discussions to #158 .