Improve import time of various stdlib modules

[AlexWaygood]

Feature or enhancement

Proposal:

As noted in https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173, typing isn't the slowest stdlib module in terms of import time, but neither is it one of the quickest. We should speed it up, if possible.

Links to previous discussion of this feature:

https://discuss.python.org/t/deferred-computation-evalution-for-toplevels-imports-and-dataclasses/34173

Linked PRs

• gh-109653: typing.py: improve import time by creating soft-deprecated members on demand #109651
• gh-109653: Improve enum import time by avoiding import of functools #109789
• gh-109653: Remove unused imports in the Lib/ directory #109803
• gh-109653: Avoid a top-level import of types in functools #109804
• gh-109653: Just import recursive_repr in dataclasses #109822
• gh-109653: Improve the import time of email.utils #109824
• GH-109653: Defer import of importlib.metadata._adapters #109829
• gh-109653: Fix py312 regression in the import time of random #110221
• [3.12] gh-109653: Fix py312 regression in the import time of random (GH-110221) #110247
• gh-109653: Defer importing warnings in several modules #110286
• gh-109653: Improve import time of logging by lazy loading traceback #112995
• gh-109653: Speedup import of threading module #114509
• gh-109653: Improve import time of importlib.metadata / email.utils #114664
• gh-109653: Reduce import overhead of uuid module on Linux #115160
• gh-109653: What's new: Note improved import times for several stdlib modules in Python 3.13 #118697

[AlexWaygood]

Retitling this issue to reflect a change of scope: part of the reason for the slow import time of typing is the slow import time of some of the modules it imports. There may be low-hanging fruit in some other stdlib modules as well.

[hugovk]

An idea: review the output of a linter for "unused" imports.

Some may be false positives due to needing an import's side effects, but I expect there will be some that are no longer needed.

[itamaro]

Some may be false positives due to needing an import's side effects ...

How do you feel about making an intentional effort to get rid of such reliance on side effects in the stdlib?
Explicit is better than implicit :)

[AlexWaygood]

An idea: review the output of a linter for "unused" imports.

FWIW, I can only find one in the stdlib that I'm 100% confident with, using ruff/pycln:

cpython/Lib/asyncio/tasks.py

Line 20 in e8be0c9

import warnings

I think this is because Victor went through them all fairly recently, in #105411. There are a fair few in Lib/test/, though. I'll file a PR in a bit.

[sobolevn]

This is also interesting:

• dataclasses copies recursive_repr to avoid a dependency:

  cpython/Lib/dataclasses.py

  Lines 248 to 266 in 51863b7

  	# This function's logic is copied from "recursive_repr" function in
  	# reprlib module to avoid dependency.
  	def _recursive_repr(user_function):
  	# Decorator to make a repr function return "..." for a recursive
  	# call.
  	repr_running = set()
  	@functools.wraps(user_function)
  	def wrapper(self):
  	key = id(self), _thread.get_ident()
  	if key in repr_running:
  	return '...'
  	repr_running.add(key)
  	try:
  	result = user_function(self)
  	finally:
  	repr_running.discard(key)
  	return result
  	return wrapper

• But, it imports functools:

  cpython/Lib/dataclasses.py

  Line 7 in 51863b7

  import functools

• And it imports reprlib.recursive_repr:

  cpython/Lib/functools.py

  Line 20 in 51863b7

  from reprlib import recursive_repr

[sobolevn]

One more interesting case:

• functools says that import types is deferred to singledispatch():

  cpython/Lib/functools.py

  Line 19 in 51863b7

  # import types, weakref # Deferred to single_dispatch()

• But several lines below types module is imported:

  cpython/Lib/functools.py

  Line 22 in 51863b7

  from types import GenericAlias

[AlexWaygood]

@sobolevn, the situation with dataclasses and reprlib you point out in #109653 (comment) is indeed interesting. From my local experiments, it does seem like the import time of dataclasses is unaffected if dataclasses is altered so that it just imports recursive_repr from reprlib rather than reimplementing that logic. The import time for functools also doesn't really change if it vendors the reprlib code instead of importing it; it seems like that import is really cheap.

So, should we just go for cleaner code in dataclasses, and import recursive_repr from reprlib? Maybe, but there are other reasons than just import time to avoid unnecessary dependencies between different stdlib modules. I think it's not the biggest issue in the world; we can probably leave the code as it is. I'll defer to the dataclasses experts on whether or not they feel like changing this.

[hugovk]

Great to see these improvements!

Let's add a brief overview in What's New in 3.13, at least before beta (2024-05-07).

[layday]

I believe there are futher improvements to be made here - how would you like to handle these? For example, importlib.metadata imports zipfile at the top for a function that won't be called in the vast majority of cases. It also imports importlib.abc, which in turn imports importlib.resources, to subclass an ABC with a single, non-abstract method - I assume redefining the method in importlib.metadata would be harmless.

[AlexWaygood]

I believe there are futher improvements to be made here - how would you like to handle these? For example, importlib.metadata imports zipfile at the top for a function that won't be called in the vast majority of cases. It also imports importlib.abc, which in turn imports importlib.resources, to subclass an ABC with a single, non-abstract method - I assume redefining the method in importlib.metadata would be harmless.

Sure -- I felt like the low-hanging fruit had been tackled here, and this issue hadn't seen much activity in quite a while, so I felt like it was a good time to close. But more improvements are obviously always welcome. Maybe open a new issue if you have some more targeted improvements you want to propose?

[ofek]

Please link the new issue here so we all can start following that as well!

[layday]

#118761