gh-118761: Improve import time of mimetypes

[hugovk]

Makes import time 11 to 16 times as fast. Measured with a PGO and LTO non-debug build on macOS.

• The slowest import in mimetypes is urllib.parse (taking 5,174 of mimetypes 5,448 μs = 95%).

• After deferring urllib.parse, import time is 480 μs. That's 11.35 times as fast.

• We could stop here. The other imports are easy enough to defer as well and have some benefit when running with -S to not import site on initialisation: 7,540 μs -> 469 μs = 16.08 times as fast.

-X importtime

python.exe

total import time: 0.010s -> 0.006s -> 0.005s

main	defer urllib.parse	defer rest

mimetypes import time: 0.005s -> 0.000s -> 0.000s

main	defer urllib.parse	defer rest

python.exe -S

total import time: 0.013s -> 0.009s -> 0.006s

main	defer urllib.parse	defer rest

mimetypes import time: 0.013s -> 0.004s -> 0.001s

main	defer urllib.parse	defer rest

hyperfine

python.exe: 24.2 ms -> 11.2 ms

main

❯ hyperfine --warmup 8 "./python.exe -c 'import mimetypes'"
Benchmark 1: ./python.exe -c 'import mimetypes'
  Time (mean ± σ):      24.2 ms ±  11.3 ms    [User: 15.2 ms, System: 5.7 ms]
  Range (min … max):    16.2 ms …  59.6 ms    133 runs

PR

❯ hyperfine --warmup 8 "./python.exe -c 'import mimetypes'"
Benchmark 1: ./python.exe -c 'import mimetypes'
  Time (mean ± σ):      11.2 ms ±   0.7 ms    [User: 8.1 ms, System: 2.6 ms]
  Range (min … max):    10.3 ms …  14.2 ms    170 runs

python.exe -S: 15.1 ms -> 8.4 ms

main

❯ hyperfine --warmup 8 "./python.exe -S -c 'import mimetypes'"
Benchmark 1: ./python.exe -S -c 'import mimetypes'
  Time (mean ± σ):      15.1 ms ±   0.7 ms    [User: 11.4 ms, System: 3.1 ms]
  Range (min … max):    14.3 ms …  19.2 ms    133 runs

PR

❯ hyperfine --warmup 8 "./python.exe -S -c 'import mimetypes'"
Benchmark 1: ./python.exe -S -c 'import mimetypes'
  Time (mean ± σ):       8.4 ms ±   0.4 ms    [User: 5.7 ms, System: 2.2 ms]
  Range (min … max):     7.9 ms …  12.3 ms    216 runs

• Issue: Improve import time of various stdlib modules #118761

[asvetlov]

Import time improvement looks great!
My question is: how do these import mod lines in a function body affect the function execution time itself?
IIRC import statement accuires a lock, isn't it?

[danielhollas]

Nice! 🎉

[danielhollas]

I wonder if lazy import of os module is worth it here, it is needed three times and the improvement is small and only in a special case.

[vstinner]

Python always imports os and sys at startup. Moving import sys makes sense, it's only used by _main(). But I'm not sure about moving os.

[danielhollas]

In the OP, @hugovk provides timings for when the python is run without the site.py module (python -S), in which case os and sys are not loaded I believe?

[hugovk]

Yes, it makes a difference with python -S, compare the last three images above.

But I'm fine reverting any of these, because I expect with site.py is by far the most usual thing.

[hugovk]

Import time improvement looks great! My question is: how do these import mod lines in a function body affect the function execution time itself? IIRC import statement accuires a lock, isn't it?

They do affect execution time but in a much smaller way. Let's time these scripts that call a function a million times:

# 1.py
import urllib.parse


def thing():
    urllib.parse.urlparse("https://example.com")


for _ in range(1_000_000):
    thing()

# 2.py
def thing():
    import urllib.parse

    urllib.parse.urlparse("https://example.com")


for _ in range(1_000_000):
    thing()

Importing outside the loop is 1.06 times faster, but this is with a million loops:

❯ hyperfine --warmup 1 "python3.14 1.py" "python3.14 2.py"
Benchmark 1: python3.14 1.py
  Time (mean ± σ):      1.403 s ±  0.014 s    [User: 1.380 s, System: 0.019 s]
  Range (min … max):    1.392 s …  1.441 s    10 runs

Benchmark 2: python3.14 2.py
  Time (mean ± σ):      1.486 s ±  0.006 s    [User: 1.464 s, System: 0.020 s]
  Range (min … max):    1.477 s …  1.498 s    10 runs

Summary
  python3.14 1.py ran
    1.06 ± 0.01 times faster than python3.14 2.py

Looping 1,000 times shows no difference:

❯ hyperfine --warmup 3 "python3.14 1.py" "python3.14 2.py"
Benchmark 1: python3.14 1.py
  Time (mean ± σ):      17.5 ms ±   1.5 ms    [User: 14.1 ms, System: 2.8 ms]
  Range (min … max):    16.7 ms …  34.1 ms    152 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 2: python3.14 2.py
  Time (mean ± σ):      17.5 ms ±   1.0 ms    [User: 14.1 ms, System: 2.7 ms]
  Range (min … max):    16.9 ms …  28.2 ms    150 runs

  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Summary
  python3.14 2.py ran
    1.00 ± 0.10 times faster than python3.14 1.py

[asvetlov]

6% slower in the worst case and no difference in a more realistic scenario sounds good to me.
Thanks!

[vstinner]

LGTM.

[hugovk]

Thanks for the reviews!