Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid listing code path directories #7711

Merged
merged 1 commit into from
Oct 23, 2023

Conversation

josevalim
Copy link
Contributor

If a code path directory has too many .beam files,
then list_dir/1 becomes a quite expensive operation.
With this commit, code:where_is_file/2 and
code:which/1 attempt to directly find a file,
instead of listing the directory.

If a code path directory has too many .beam files,
then `list_dir/1` becomes a quite expensive operation.
With this commit, code:where_is_file/2 and
code:which/1 attempt to directly find a file,
instead of listing the directory.
@github-actions
Copy link
Contributor

github-actions bot commented Oct 2, 2023

CT Test Results

       2 files       65 suites   1h 3m 39s ⏱️
1 508 tests 1 256 ✔️ 252 💤 0
1 700 runs  1 399 ✔️ 301 💤 0

Results for commit 20f3c11.

♻️ This comment has been updated with latest results.

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

// Erlang/OTP Github Action Bot

@josevalim
Copy link
Contributor Author

josevalim commented Oct 2, 2023

It is unclear why the previous code was written with list_dir/1 operations. I have blamed the code and it goes back to R13, so I was not able to learn much.

The proposed code should be faster in the huge majority of cases. That's because read_file_info/1 will only succeed when it finds a file (returning error most of the time) while list_dir will consistently list several entries per path. Here is some benchmark on top of kernel's ebin:

~/OSS/otp/lib/kernel/ebin[jv-which-read-file-info %]$ cat foo.exs
Mix.install [:benchee]
{:ok, p} = :file.get_cwd()

Benchee.run(
  %{
    "list_dir" => fn -> :erl_prim_loader.list_dir(p) end,
    "read_file_info" => fn -> :erl_prim_loader.read_file_info(p ++ ~c"unknown") end
  },
  time: 10,
  memory_time: 2
)
~/OSS/otp/lib/kernel/ebin[jv-which-read-file-info %]$ elixir foo.exs
Operating System: macOS
CPU Information: Apple M1 Max
Number of Available Cores: 10
Available memory: 32 GB
Elixir 1.16.0-dev
Erlang 26.0.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 10 s
memory time: 2 s
reduction time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 28 s

Benchmarking list_dir ...
Benchmarking read_file_info ...

Name                     ips        average  deviation         median         99th %
read_file_info      116.50 K        8.58 μs   ±243.74%        7.38 μs       37.96 μs
list_dir              7.75 K      128.96 μs    ±47.73%      106.17 μs      265.74 μs

Comparison:
read_file_info      116.50 K
list_dir              7.75 K - 15.02x slower +120.38 μs

**All measurements for memory usage were the same**

@frazze-jobb frazze-jobb self-assigned this Oct 4, 2023
@frazze-jobb frazze-jobb added team:VM Assigned to OTP team VM testing currently being tested, tag is used by OTP internal CI labels Oct 4, 2023
@frazze-jobb frazze-jobb merged commit 95de100 into erlang:master Oct 23, 2023
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
team:VM Assigned to OTP team VM testing currently being tested, tag is used by OTP internal CI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants