Performance bug: Many OPA processes spawned during editing

[Octogonapus]

When editing a rego file and using operations like copy/pasting variable names around, or when using undo or redo, many opa processes are spawned (anywhere between 3 and 16 on my machine). This uses a large (or 100% depending) amount of CPU and a large amount of memory, lagging both vscode and my system. This occurs when editing small files (50 lines or less) in a small project (300 lines total).

I am not sure how to debug this further, please let me know if there is anything you want me to try.

linux 5.19.6-200.fc36.x86_64
vscode v1.71.0
vscode-opa v0.12.1
opa v0.43.0

[anderseknert]

Hey @Octogonapus ! And thanks for reporting this. That's interesting 🤔 The commands launched by this plug-in should generally be very short lived... can you see which commands (i.e opa eval, opa parse, etc) are hanging, and which flags were passed to them?

[Octogonapus]

For my test, I opened a rego file and copy-pasted a line a few times within the file. These processes were spawned:

salmon    143254  214  2.0 4581060 1330948 ?     Sl   17:07   0:02 opa oracle find-definition --stdin-buffer --bundle=file:///home/salmon/Documents/a/b /home/salmon/Documents/a/c/d/e.rego:638
salmon    143292  191  0.9 3633676 603488 ?      Sl   17:07   0:01 opa oracle find-definition --stdin-buffer --bundle=file:///home/salmon/Documents/a/b /home/salmon/Documents/a/c/d/e.rego:636
salmon    143331  179  0.8 2957156 559956 ?      Sl   17:07   0:01 opa oracle find-definition --stdin-buffer --bundle=file:///home/salmon/Documents/a/b /home/salmon/Documents/a/c/d/e.rego:650
salmon    143372  165  0.9 2884192 632044 ?      Sl   17:07   0:01 opa oracle find-definition --stdin-buffer --bundle=file:///home/salmon/Documents/a/b /home/salmon/Documents/a/c/d/e.rego:642

I have had to redact folder/file names. I don't know why --bundle=file:///home/salmon/Documents/a/b is present; that doesn't make any sense to me. opa.bundleMode was set to its default value (presumably true); this appears to be the relevant feature flag. The feature flag explains why --bundle is present at all, but it doesn't explain the choice of the b directory. That folder, b, is first directory in my workspace; maybe that's why? It is quite large. If OPA is trying to search through it on many parallel processes, that would absolutely lag my system.

Treating the workspace as a bundle could be a useful feature if there was indexing in place to avoid searching it repeatedly. b also isn't my workspace, it's just one directory in my workspace.

I tried setting "opa.bundleMode": false and repeating the test. opa processes are no longer spawned, which fixes most of the lag. There is still some lag when pasting, though. It's not clear to me why. Maybe the language server is lagging?

[anderseknert]

Thank you! That's useful. That command indeed hangs "forever" unless input is provided:

opa oracle find-definition --stdin-buffer --bundle . policy.rego:1

I suppose the thing to investigate in the scope of this plugin would be why there's no input passed to the command.
In the scope of OPA, I think it would make sense to have a default timeout of a few seconds for this command, as it is not meant to run for a longer duration of time.

[Octogonapus]

I think the problem here is that OPA is trying to read the contents of all the files in the bundle. E.g.

opa oracle find-definition --stdin-buffer --bundle=file:///home/salmon/Documents/a/b /home/salmon/Documents/a/c/d/e.rego:638
error: bundle /home/salmon/Documents/a/b: bundle file 'foo.mp4' exceeded max size (1073741824 bytes)

There are many such files in that bundle location. It's not feasible to load everything each time I make a code change. Is there a way to:

1. Configure caching so that the extension loads this data once?
2. Configure a file filter so that only rego files are loaded?

[anderseknert]

Hmm, the "easy" way around it would probably be to just not load a workspace including lots of large binary files. Most OPA commands will allow you to specify an --ignore flag, but it doesn't look like it's there for the oracle command, which itself isn't really exposed "publicly" as an available command... it's AFAIK currently only used for this purpose. While it would be simple to add an --ignore option, it would perhaps also be somewhat confusing to provide a config option for an undocumented command 🤔

Is keeping the large files outside of the policy workspace ruled out as an alternative?

[Octogonapus]

For now, I can make edits in a new workspace containing just the Rego code, though it would be helpful if I could continue using one large workspace as I am always working in multiple projects. Some of these projects do require having these large files in the workspace.

[anderseknert]

That makes sense. I know some work is being prepared for open-policy-agent/opa#4041 ... which would likely change a lot for the better, for VS Code integration and elsewhere... but it'll be a while, I'm sure.

[pauly4it]

Chiming in to add my experience with a different use case. For context, I'm mainly using the OPA extension for code highlighting and use the OPA CLI for any testing, formatting, or evaluation so that I can specify the exact OPA version for those commands (using the OPA asdf plugin for version management).

I've run into this performance issue recently after adding several hundred additional Rego files to a git repo, which I open as a workspace in VS Code. The performance issues cause VS Code to freeze for up to a couple minutes and also cause Mac OS to freeze during that time. In my case, the workaround of opening only a sub-folder of Rego files as a VS Code workspace would make working with the policy library overall more difficult.

[pauly4it]

Some additional findings from the last few days after disabling the OPA extension. In the past few months I had been seeing fork: Resource temporarily unavailable errors when running various terminal commands. In particular, I saw this issue when running programs installed with homebrew (e.g., go or asdf installed with homebrew). Running a simple command like go test would result in hundreds of fork: Resource temporarily unavailable errors before the command would actually execute. The root cause of that error is that processes were not able to fork because the max number of process threads was reached.

Since disabling the OPA VSC extension, those errors are no longer occurring, which would make sense if the OPA VSC extension is creating many new processes.

[anderseknert]

Should probably be made more robust here, but some minor tweaks to the OPA command should help too: open-policy-agent/opa#5678

[anderseknert]

Is anyone still seeing this with the latest version(s) of OPA?

[ikunduraci]

I was having issues until i came across this thread.. i've set opa.bundleMode: false and opa.checkOnSave: false which seems to have fixed the 10+ opa processes that were being spawned.

[anderseknert]

@ikunduraci were you till seeing that using the latest version of OPA? That includes some fixes in the command called by this plugin which hopefully should remedy that issue.

[anderseknert]

@pauly4it same question to you :)

[ikunduraci]

I was on 0.46.1, just updated to 0.52.0.. will post back if i see the same behavior. Thanks for your help

[anderseknert]

Haven't heard back, so I'll assume this is no longer an issue unless someone tells me it is :) Closing for now.

[dolevf]

I'm actually seeing this behavior in 0.64.1 version and VS code extension version 0.13.4

dfarhi           11287  33.6  0.2 35182484  36036   ??  R    11:33AM   0:01.27 /Users/dfarhi/opa oracle find-definition --stdin-buffer --bundle=file:///Users/dfarhi/Documents/Code 
dfarhi           11316  31.0  0.2 35181584  26116   ??  R    11:33AM   0:00.45 /Users/dfarhi/opa oracle find-definition --stdin-buffer --bundle=file:///Users/dfarhi/Documents/Code 
dfarhi           11333  30.9  0.1 35189088  22356   ??  R    11:33AM   0:00.17 /Users/dfarhi/opa oracle find-definition --stdin-buffer --bundle=file:///Users/dfarhi/Documents/Code 
dfarhi           11284  30.9  0.2 35182360  35648   ??  R    11:33AM   0:01.26 /Users/dfarhi/opa oracle find-definition --stdin-buffer --bundle=file:///Users/dfarhi/Documents/Code 
dfarhi           11281  26.6  0.2 35173360  37868   ??  R    11:33AM   0:01.39 /Users/dfarhi/opa oracle find-definition --stdin-buffer --bundle=file:///Users/dfarhi/Documents/Code 
dfarhi           11263  25.2  0.2 35181848  37876   ??  R    11:33AM   0:01.42 /Users/dfarhi/opa oracle find-definition --stdin-buffer --bundle=file:///Users/dfarhi/Documents/Code 
dfarhi           11299  21.9  0.2 35198684  31960   ??  R    11:33AM   0:00.91 /Users/dfarhi/opa oracle find-definition --stdin-buffer --bundle=file:///Users/dfarhi/Documents/Code 
dfarhi           11239  21.6  1.2 35341336 202720   ??  R    11:33AM   0:02.42 /Users/dfarhi/opa oracle find-definition --stdin-buffer --bundle=file:///Users/dfarhi/Documents/Code 
dfarhi           11313  21.2  0.2 35198944  31560   ??  R    11:33AM   0:00.84 /Users/dfarhi/opa oracle find-definition --stdin-buffer --bundle=file:///Users/dfarhi/Documents/Code 

[anderseknert]

Hi Dolev 👋 Could you bump the extension to v0.14.0? If you have Regal installed, and I assume you do ;) that is now being used as the provider for the find definition feature.

(we might need to look into why this feature might hang though, but we'll do that in that project)

[dolevf]

Just did - issue hasn't returned!