Significant slow down finding spago.yaml config when building

[OwenGraves]

On changing from versions [email protected] to [email protected] there is now a significant slow down after the Gathering all the spago configs in the tree... step when running a spago build:

Before 0.93.30, this step would be nearly instant instead of taking several minutes:

The spago.yaml is in the same directory as when running the command spago build

[Blugatroff]

Have you tried using a newer version of spago?
There is a series of patches after 0.93.29 which aimed to improve the performance of finding spago.yamls. Afterwards, many performance regressions were discovered and eventually fixed.
Probably (hopefully) your problem is caused by one of those already fixed regressions, but maybe not.

For the interested reader, these are all the issues and PRs which changed and fixed the globbing code since 0.93.29

Release of 0.93.29
#1182
#1209
#1210
#1222
Release of 0.93.30
#1231
Release of 0.93.32
#1234
#1236
#1242
Release of 0.93.34 and 0.93.35
#1244
#1251
Release of 0.93.37

[OwenGraves]

Yes, there is no improvement with [email protected] or [email protected], the top screenshot is actually from 0.93.40. The issue seems to have started between 0.93.29 and 0.93.30 though

[f-f]

@OwenGraves do you have a repro at hand that I could have a look at? Having a codebase to poke would really help to see where the slowdown is.

[OwenGraves]

@OwenGraves do you have a repro at hand that I could have a look at? Having a codebase to poke would really help to see where the slowdown is.

Unfortunately the only repo where I see this occur is private, but I think the slowdown is from navigating a large .spago directory adjacent to the spago.yaml. Removing a bunch of dependencies and extraPackages and then deleting the .spago directory seems to allow spago build to quickly go through this step

[ysangkok]

Ok, I have a reproducer, though it is not as extreme as the screenshot shows.

Spago.yaml:

package:
  name: app
  build:
    strict: true
  bundle:
    module: App
    outfile: "dist/assets/Main.js"
  dependencies:
    - console
    - effect
    - prelude
    - a
    - b
    - c
    - d
    - e
    - f
    - g
    - h
    - i
    - j
    - k
    - l
    - m
    - n
    - o
workspace:
  packageSet:
    registry: 46.2.0
  extraPackages:
    a:
      git: https://github.com/ysangkok/multi-proj.git
      ref: &lib-ref 676016965c3e47a158951439f325c006d9008734
      subdir: a
    b:
      git: https://github.com/ysangkok/multi-proj.git
      ref: *lib-ref
      subdir: b
    c:
      git: https://github.com/ysangkok/multi-proj.git
      ref: *lib-ref
      subdir: c
    d:
      git: https://github.com/ysangkok/multi-proj.git
      ref: *lib-ref
      subdir: d
    e:
      git: https://github.com/ysangkok/multi-proj.git
      ref: *lib-ref
      subdir: e
    f:
      git: https://github.com/ysangkok/multi-proj.git
      ref: *lib-ref
      subdir: f
    g:
      git: https://github.com/ysangkok/multi-proj.git
      ref: *lib-ref
      subdir: g
    h:
      git: https://github.com/ysangkok/multi-proj.git
      ref: *lib-ref
      subdir: h
    i:
      git: https://github.com/ysangkok/multi-proj.git
      ref: *lib-ref
      subdir: i
    j:
      git: https://github.com/ysangkok/multi-proj.git
      ref: *lib-ref
      subdir: j
    k:
      git: https://github.com/ysangkok/multi-proj.git
      ref: *lib-ref
      subdir: k
    l:
      git: https://github.com/ysangkok/multi-proj.git
      ref: *lib-ref
      subdir: l
    m:
      git: https://github.com/ysangkok/multi-proj.git
      ref: *lib-ref
      subdir: m
    n:
      git: https://github.com/ysangkok/multi-proj.git
      ref: *lib-ref
      subdir: n
    o:
      git: https://github.com/ysangkok/multi-proj.git
      ref: *lib-ref
      subdir: o

With an existing cache, this takes more than 1100ms to find spago configs:

[     120ms] Gathering all the spago configs in the tree...
[    1285ms] Found packages at these paths:

Even though the files have to be read and parsed, this seems like too long, since PyYAML can parse/dump these files in less than 10% of the time it takes Spago:

$ time (find ../multi-proj -name spago.yaml | python3 <(echo -e 'import sys, yaml\nfor i in sys.stdin.readlines():\n  with open(i.strip()) as f:\n    print(yaml.load(f, Loader=yaml.CLoader))'))
0.02s

And with 0.93.29, this build finishes in 450ms. So that indicates that this is the same issue that OP is experiencing.

[f-f]

@ysangkok I can't repro with your example, on my machine a spago build using your config only takes 6ms to gather all the spago configs in the tree.

In any case I just released 0.93.41 with a few more fixes related to this (#1293 might help), @OwenGraves @ysangkok could you try that and see if it addresses your issue?

[OwenGraves]

0.93.41 fixes this, thank you!