Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix test_get_yarn_spark_resource_config after AWS_REGION was added into properties #77

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

pavel-balas
Copy link

Issue #, if available:

The pull/75 change introduced AWS_REGION variables, but they are not reflected in unit test

% pipenv run python -m pytest -s -vv test/unit
...
E         Left contains 2 more items:
E         {'spark.executorEnv.AWS_REGION': 'None',
E          'spark.yarn.appMasterEnv.AWS_REGION': 'None'}
...
FAILED test/unit/test_bootstrapper.py::test_get_yarn_spark_resource_config - assert {'spark.default.parallelism': '8',\n 'spark.driver.defaultJavaOptions': "-XX:OnOutOfMemoryError='kill -9 %p' "\n          ...

Description of changes:

I don't like just update assertion to have string "None", so I'm patching the os variable to have a reasonable value and then asserting that the value made it to the spark properties.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@pavel-balas pavel-balas changed the title Update test_get_yarn_spark_resource_config after AWS_REGION as added into properties Update test_get_yarn_spark_resource_config after AWS_REGION was added into properties Feb 15, 2022
@pavel-balas pavel-balas changed the title Update test_get_yarn_spark_resource_config after AWS_REGION was added into properties fix test_get_yarn_spark_resource_config after AWS_REGION was added into properties Feb 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant