Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Python3.12 #3266

Merged
merged 8 commits into from
May 9, 2024
Merged

Support Python3.12 #3266

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
62b0e52
add py312
Dec 20, 2023
0369cfb
import boto when python<=311
Dec 20, 2023
2a9da0c
fix removed assert
Dec 20, 2023
e7130f8
fix removed assert
Dec 20, 2023
b8733d4
skip boto test for py312
Dec 20, 2023
1628cfe
Merge branch 'master' into add-python312-support
ukky17 May 8, 2024
4323995
Update README.rst
ukky17 May 8, 2024
3a66f28
Update README.rst
ukky17 May 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .github/workflows/pythonbuild.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ jobs:
tox-env: py310-core
- python-version: "3.11"
tox-env: py311-core
- python-version: "3.12"
tox-env: py312-core

steps:
- uses: actions/checkout@v2
Expand Down Expand Up @@ -92,6 +94,8 @@ jobs:
tox-env: py310-postgres
- python-version: "3.11"
tox-env: py311-postgres
- python-version: "3.12"
tox-env: py312-postgres

steps:
- uses: actions/checkout@v2
Expand Down Expand Up @@ -145,6 +149,8 @@ jobs:
tox-env: py310-aws
- python-version: "3.11"
tox-env: py311-aws
- python-version: "3.12"
tox-env: py312-aws

- python-version: "3.6"
tox-env: py36-unixsocket
Expand All @@ -164,6 +170,9 @@ jobs:
- python-version: "3.11"
tox-env: py311-unixsocket
OVERRIDE_SKIP_CI_TESTS: True
- python-version: "3.12"
tox-env: py312-unixsocket
OVERRIDE_SKIP_CI_TESTS: True

- python-version: "3.6"
tox-env: py36-apache
Expand All @@ -177,6 +186,8 @@ jobs:
tox-env: py310-apache
- python-version: "3.11"
tox-env: py311-apache
- python-version: "3.12"
tox-env: py312-apache

- python-version: "3.6"
tox-env: py36-azureblob
Expand All @@ -190,6 +201,8 @@ jobs:
tox-env: py310-azureblob
- python-version: "3.11"
tox-env: py311-azureblob
- python-version: "3.12"
tox-env: py312-azureblob


- python-version: 3.9
Expand Down
2 changes: 1 addition & 1 deletion README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
:target: https://luigi.readthedocs.io/en/stable/?badge=stable
:alt: Documentation Status

Luigi is a Python (3.6, 3.7, 3.8, 3.9, 3.10, 3.11 tested) package that helps you build complex
Luigi is a Python (3.6, 3.7, 3.8, 3.9, 3.10, 3.11, 3.12 tested) package that helps you build complex
pipelines of batch jobs. It handles dependency resolution, workflow management,
visualization, handling failures, command line integration, and much more.

Expand Down
1 change: 1 addition & 0 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,7 @@ def get_static_files(path):
'Programming Language :: Python :: 3.9',
'Programming Language :: Python :: 3.10',
'Programming Language :: Python :: 3.11',
'Programming Language :: Python :: 3.12',
'Topic :: System :: Monitoring',
],
)
2 changes: 1 addition & 1 deletion test/cmdline_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -362,7 +362,7 @@ def test_cmd_line_params_are_available_for_execution_summary(self):
print(stdout)
print(stderr)

self.assertNotEquals(returncode, 1)
self.assertNotEqual(returncode, 1)
self.assertFalse(b'required_test_param' in stderr)


Expand Down
8 changes: 4 additions & 4 deletions test/contrib/external_daily_snapshot_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,18 +32,18 @@ class ExternalDailySnapshotTest(unittest.TestCase):
def test_latest(self):
MockTarget('data-xyz-zebra-Congo-2012-01-01').open('w').close()
d = DataDump.latest(date=datetime.date(2012, 1, 10), param='xyz')
self.assertEquals(d.date, datetime.date(2012, 1, 1))
self.assertEqual(d.date, datetime.date(2012, 1, 1))

def test_latest_not_exists(self):
MockTarget('data-abc-zebra-Congo-2012-01-01').open('w').close()
d = DataDump.latest(date=datetime.date(2012, 1, 11), param='abc', lookback=5)
self.assertEquals(d.date, datetime.date(2012, 1, 7))
self.assertEqual(d.date, datetime.date(2012, 1, 7))

def test_deterministic(self):
MockTarget('data-pqr-zebra-Congo-2012-01-01').open('w').close()
d = DataDump.latest(date=datetime.date(2012, 1, 10), param='pqr', a='zebra', aa='Congo')
self.assertEquals(d.date, datetime.date(2012, 1, 1))
self.assertEqual(d.date, datetime.date(2012, 1, 1))

MockTarget('data-pqr-zebra-Congo-2012-01-05').open('w').close()
d = DataDump.latest(date=datetime.date(2012, 1, 10), param='pqr', aa='Congo', a='zebra')
self.assertEquals(d.date, datetime.date(2012, 1, 1)) # Should still be the same
self.assertEqual(d.date, datetime.date(2012, 1, 1)) # Should still be the same
2 changes: 1 addition & 1 deletion test/contrib/hadoop_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,7 @@ def test_run_2(test_case):
job = WordFreqJob(use_hdfs=test_case.use_hdfs)
luigi.build([job], local_scheduler=True)
c = read_wordcount_output(job.output())
test_case.assertAlmostEquals(float(c['jk']), 6.0 / 33.0)
test_case.assertAlmostEqual(float(c['jk']), 6.0 / 33.0)

@staticmethod
def test_map_only(test_case):
Expand Down
4 changes: 2 additions & 2 deletions test/contrib/postgres_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,11 +132,11 @@ def test_bulk_complete(self, mock_connect):

def test_override_port(self):
output = DummyPostgresQueryWithPort(date=datetime.datetime(1991, 3, 24)).output()
self.assertEquals(output.port, 1234)
self.assertEqual(output.port, 1234)

def test_port_encoded_in_host(self):
output = DummyPostgresQueryWithPortEncodedInHost(date=datetime.datetime(1991, 3, 24)).output()
self.assertEquals(output.port, '1234')
self.assertEqual(output.port, '1234')


@pytest.mark.postgres
Expand Down
12 changes: 7 additions & 5 deletions test/contrib/s3_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@
import tempfile

import boto3
from boto.s3 import key
if sys.version_info[:2] <= (3, 11):
from boto.s3 import key
from botocore.exceptions import ClientError
from mock import patch

Expand Down Expand Up @@ -107,6 +108,7 @@ def test_read_no_file_sse(self):
t = self.create_target(encrypt_key=True)
self.assertRaises(FileNotFoundException, t.open)

@unittest.skipIf(tuple(sys.version_info) >= (3, 12), "boto is not supported on Python 3.12+")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the better thing to do here would be to make the s3 module compatible with boto3

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can the s3 module be improved in another PR? Ubuntu 24 came out with Python 3.12 as default, and the lack of support in Luigi will block updates for many users (although fortunately, the changes only affect tests, so it's not critical).

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the need for 3.12 Luigi support, i'm ok getting this PR out the door.

@ukky17 , if you can resolve the merge conflict, i can merge if tests still pass.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dlstadther I have resolved the conflict. Please review again and merge if it looks good to you.

def test_read_iterator_long(self):
# write a file that is 5X the boto buffersize
# to test line buffering
Expand Down Expand Up @@ -342,7 +344,7 @@ def test_get(self):
s3_client.get('s3://mybucket/putMe', tmp_file_path)
with open(tmp_file_path, 'r') as f:
content = f.read()
self.assertEquals(content, self.tempFileContents.decode("utf-8"))
self.assertEqual(content, self.tempFileContents.decode("utf-8"))
tmp_file.close()

def test_get_as_bytes(self):
Expand All @@ -352,7 +354,7 @@ def test_get_as_bytes(self):

contents = s3_client.get_as_bytes('s3://mybucket/putMe')

self.assertEquals(contents, self.tempFileContents)
self.assertEqual(contents, self.tempFileContents)

def test_get_as_string(self):
create_bucket()
Expand All @@ -361,7 +363,7 @@ def test_get_as_string(self):

contents = s3_client.get_as_string('s3://mybucket/putMe2')

self.assertEquals(contents, self.tempFileContents.decode('utf-8'))
self.assertEqual(contents, self.tempFileContents.decode('utf-8'))

def test_get_as_string_latin1(self):
create_bucket()
Expand All @@ -370,7 +372,7 @@ def test_get_as_string_latin1(self):

contents = s3_client.get_as_string('s3://mybucket/putMe3', encoding='ISO-8859-1')

self.assertEquals(contents, self.tempFileContents.decode('ISO-8859-1'))
self.assertEqual(contents, self.tempFileContents.decode('ISO-8859-1'))

def test_get_key(self):
create_bucket()
Expand Down
4 changes: 2 additions & 2 deletions test/interface_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,9 +204,9 @@ class CoreConfigTest(LuigiTestCase):

@with_config({})
def test_parallel_scheduling_processes_default(self):
self.assertEquals(0, core().parallel_scheduling_processes)
self.assertEqual(0, core().parallel_scheduling_processes)

@with_config({'core': {'parallel-scheduling-processes': '1234'}})
def test_parallel_scheduling_processes(self):
from luigi.interface import core
self.assertEquals(1234, core().parallel_scheduling_processes)
self.assertEqual(1234, core().parallel_scheduling_processes)
2 changes: 1 addition & 1 deletion test/task_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -577,4 +577,4 @@ def __init_subclass__(cls, x, **kwargs):

class Receiver(ReceivesClassKwargs, x=1):
pass
self.assertEquals(Receiver.x, 1)
self.assertEqual(Receiver.x, 1)
2 changes: 1 addition & 1 deletion tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[tox]
envlist = py{35,36,37,38,39,310,311}-{cdh,hdp,core,contrib,apache,aws,gcloud,postgres,unixsocket,azureblob,dropbox}, visualiser, docs, flake8
envlist = py{35,36,37,38,39,310,311,312}-{cdh,hdp,core,contrib,apache,aws,gcloud,postgres,unixsocket,azureblob,dropbox}, visualiser, docs, flake8
skipsdist = True

[pytest]
Expand Down
Loading