The [opa eval] command is unable to access Rego policies located on other drives on windows #6922

pckvcode · 2024-08-12T08:45:06Z

The [opa eval] command is unable to access Rego policies located on other drives in Windows. For more details, see #6910 and open-policy-agent/conftest#979.

OPA should be capable of accessing Rego policies from specified paths including drives (e.g., c:\a\b\c.rego).

The code has been updated to allow OPA to load Rego policies from paths with drives. Now, OPA can load Rego policies from paths(e.g., /a/b/c/example.rego), path with URLs(e.g., file:///path/to/file.json), and path with drives(e.g., C:\a\b\c\example.rego).

Fixes #6910

Why the changes in this PR are needed?

The opa eval command encountered an issue where it could not load Rego policies from different drives on Windows. The error message was:

CreateFile \\Users\\pck\\AppData\\Local\\Temp\\temp_dir_12342318055089\\example.rego: The system cannot find the path specified

It fixes: #6910 and open-policy-agent/conftest#979

What are the changes in this PR?

Code Update:

The code has been modified to allow OPA to load Rego policies from paths that include drive letters. Now, OPA supports loading Rego policies from:

Existing Inputs:
- Paths (e.g., /a/b/c/example.rego)
- Paths with URLs (e.g., file:///path/to/file.json)
- Paths with drive letters and URLs (e.g., "C:file:///C:/a/b/c")
New Input:
- Paths with drive letters (e.g., C:\a\b\c\example.rego)

Documentation Reference:

According to the official documentation, the expected values for the --data flag in the opa eval command are:

A simple path (e.g., /a/b/c/example.rego)
A path with a URL (e.g., file:///path/to/file.json)

Ideally, specifying a path like C:\a\b\c\example.rego should work, but it was failing. The code was modified to support paths with drive letters by ensuring that the prefix is only separated when its length is greater than 1.

Notes to assist PR review:

Steps to Reproduce the issue:

Place the OPA binary on the D drive.
Store the Rego policies on the C drive.
Attempt to evaluate the policies using the opa eval command with the policies located on the C drive. The above error will occur. For more details, refer to this GitHub issue.

Further comments:

[opa eval] command before fix. (Failed: opa in D drive and Failed to load rego policies from C drive.)

[opa eval] command after fix. (Success: opa in D drive and Successfully load rego policies from C drive.)

Unit Testcase pass screenshot. Function tested

@boranx Can you please review this change and share your comments? Thanks in advance

…ther drives in Windows. For more details, see open-policy-agent#6910 and open-policy-agent/conftest#979. OPA should be capable of accessing Rego policies from specified paths including drives (e.g., c:\a\b\c.rego). The code has been updated to allow OPA to load Rego policies from paths with drives. Now, OPA can load Rego policies from paths, URLs, and drives. Fixes open-policy-agent#6910 Signed-off-by: pckvcode <[email protected]>

philipaconrad · 2024-08-12T16:52:30Z

The new test appears to be causing failures on non-Windows platforms (linux, darwin). We may need to either split the test off for Windows, or find some other way to OS-gate it.

netlify · 2024-08-21T10:42:45Z

✅ Deploy Preview for openpolicyagent ready!

Name	Link
🔨 Latest commit	`9601e7e`
🔍 Latest deploy log	https://app.netlify.com/sites/openpolicyagent/deploys/66c6d425d695500008ad8b6a
😎 Deploy Preview	https://deploy-preview-6922--openpolicyagent.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

…nput sources: Existing Inputs: 1. Absolute file paths (e.g., /a/b/c/example.rego) 2. Paths with URLs (e.g., file:///path/to/file.json) 3. Paths with drive letters and URLs (e.g., "C:file:///C:/a/b/c") New Inputs: 1. Paths with drive letters (e.g., C:\a\b\c\example.rego) This requirement ensures that Opa can handle a variety of input formats, including local file paths, remote URLs, and even mixed formats that include both drive letters and URLs. open-policy-agent#6910 Signed-off-by: pckvcode <[email protected]>

pckvcode · 2024-08-22T06:08:51Z

@philipaconrad
I fixed it now. Can you please approve the workflow and review the changes?

I got to know we are supporting these inputs.

Paths (e.g., /a/b/c/example.rego)
Paths with URLs (e.g., file:///path/to/file.json)
Paths with drive letters and URLs (e.g., "C:file:///C:/a/b/c")

Added the support for below

Paths with drive letters (e.g., C:\a\b\c\example.rego)

My comments:

C:file:///C:/a/b/c is not a right url format but we are supporting it. no where mentioned in the documentation https://www.openpolicyagent.org/docs/latest/cli/#examples
Made code changes to support all the path types mentioned above.

boranx

IMO, intuitively, I'd say we can support the following

Paths (e.g., /a/b/c/example.rego)
Paths with URLs (e.g., file:///path/to/file.json)

but I'm not sure if Paths with drive letters and URLs (e.g., "C:file:///C:/a/b/c") is a valid format, as I haven't seen such thing before 🤔

My 2 cents is to add a unit test for each one of those cases, as supported vs not supported

pckvcode · 2024-08-23T08:46:35Z

I tested with the latest OPA Windows AMD64 binary. Here are the results:

curl -O https://github.com/open-policy-agent/opa/releases/download/v0.67.1/opa_windows_amd64.exe

Note:

In all these cases, I kept the binary on the Windows D drive.

Succeeded for these paths:

Paths with URLs (e.g., file:///path/to/file.json)
Paths with drive letters and URLs (e.g., "C:file:///C:/a/b/c")
Standard paths (e.g., /a/b/c/example.rego)

Commands used:

.\opa_windows_amd64.exe eval --input "C:\Users\pck\Punith\TestFiles\input.json" --data "file:///C:/Users/pck/Punith/TestFiles/rego_policies/example.rego" "data.example.authz.allow"
.\opa_windows_amd64.exe eval --input "C:\Users\pck\Punith\TestFiles\input.json" --data "C:file:///C:/Users/pck/Punith/TestFiles/rego_policies/example.rego" "data.example.authz.allow"
# able to access from D drive since the binary in D drive
.\opa_windows_amd64.exe eval --input "C:\Users\pck\Punith\TestFiles\input.json" --data "D:/punith/opa-bug/TestFiles/rego_policies/example.rego" "data.example.authz.allow"

Failed for this path:

Paths with drive letters (e.g., C:\a\b\c\example.rego)

# Fails since the binary is in D drive and loading policies from C drive
.\opa_windows_amd64.exe eval --input "C:\Users\pck\Punith\TestFiles\input.json" --data "C:\Users\pck\Punith\TestFiles\rego_policies\example.rego" "data.example.authz.allow"

I then tested with my code changes, and it worked for the following paths:

Paths with URLs (e.g., file:///path/to/file.json)
Paths with drive letters and URLs (e.g., "C:file:///C:/a/b/c")
Standard paths (e.g., /a/b/c/example.rego)
Paths with drive letters (e.g., C:\a\b\c\example.rego)

Commands used:

.\opa-new.exe eval --input "C:\Users\pck\Punith\TestFiles\input.json" --data "file:///C:/Users/pck/Punith/TestFiles/rego_policies/example.rego" "data.example.authz.allow"
.\opa-new.exe eval --input "C:\Users\pck\Punith\TestFiles\input.json" --data "C:file:///C:/Users/pck/Punith/TestFiles/rego_policies/example.rego" "data.example.authz.allow"
.\opa-new.exe eval --input "D:/punith/opa-bug/TestFiles/rego_policies/example.rego" "data.example.authz.allow"
.\opa-new.exe eval --input "C:\Users\pck\Punith\TestFiles\input.json" --data "C:\Users\pck\Punith\TestFiles\rego_policies\example.rego" "data.example.authz.allow"

I also added a unit test case to verify loading files from paths with drive letters (e.g., C:\a\b\c\example.rego).

…clude a Windows drive letter (e.g., C:\a\b\c\example.rego). When tests are run on Windows, temporary directories are created on the C drive (e.g., C:\Users\<username>\AppData\Local\Temp). Fixes: open-policy-agent#6910 Signed-off-by: pckvcode <[email protected]>

Signed-off-by: pckvcode <[email protected]>

johanfylling

Just a few questions.

I don't have a windows machine to test on, but I would expect the path file:///c:/path/to/my/file/data.json to work. Have you tested this format?

johanfylling · 2024-08-26T12:01:00Z

loader/loader.go

@@ -582,7 +582,9 @@ func SplitPrefix(path string) ([]string, string) {
 	}
 	parts := strings.SplitN(path, ":", 2)
 	if len(parts) == 2 && len(parts[0]) > 0 {
-		return strings.Split(parts[0], "."), parts[1]
+		if strings.HasPrefix(parts[1], "file:///") || len(parts[0]) > 1 {


Why do we put significance on the path containing file:/// or the document prefix being more than one chars long for us to return those parts rather than the given path?

To add some context: I believe the prefix part is the prefix for which the contents of that file will be found under the data document during evaluation.

E.g.: given the following data file data.json:

{ "foo": 42 }

and we run:

$ opa eval -fpretty -d file:///path/to/my/file/data.json 'data = x'

we expect the output:

+------------+ | x | +------------+ | {"foo":42} | +------------+

But if we add the a: prefix to the file path, we expect this prefix to be added to the data document:

opa eval -fpretty -d a:file:///path/to/my/file/data.json 'data = x'

+------------------+ | x | +------------------+ | {"a":{"foo":42}} | +------------------+

johanfylling · 2024-08-26T12:23:29Z

loader/loader_test.go

@@ -879,6 +889,15 @@ func TestSplitPrefix(t *testing.T) {
 			wantParts: []string{"x", "y"},
 			wantPath:  "file:///c:/a/b/c",
 		},
+		{
+			input:    "c:/a/b/c",
+			wantPath: "c:/a/b/c",


This doesn't seem compatible with how data prefixes work in OPA, as we'd expect c to become the data prefix.

Is it advisable to always use a file URL(ex: file:///c:/path/to/my/file/data.json instead of c:/path/to/my/file/data.json), especially on Windows?
How does the system behave if we provide the path as c:/path/to/my/file/data.json? In this case, should c be interpreted as a prefix or as the drive letter?

In this image, we see that using c:/path/to/my/file/data.json has failed

pckvcode · 2024-08-27T13:06:14Z

Just a few questions.

I don't have a windows machine to test on, but I would expect the path file:///c:/path/to/my/file/data.json to work. Have you tested this format?

I tried using the path `file:///c:/path/to/my/file/data.json`, and it worked.
However, when I used `c:/path/to/my/file/data.json`, it failed.

Is it advisable to always use a file URL, especially on Windows?
How does the system behave if we provide the path as c:/path/to/my/file/data.json? In this case, should c be interpreted as a prefix or as the drive letter?

johanfylling · 2024-08-27T16:05:02Z

The current behavior is to interpret the c: in c:/path/to/my/file/data.json as the data prefix c. Modifying this behavior would be a breaking change, and given that there are no recommendations/requirements for the number of characters in a prefix, I don't think this is something we can safely change.

Given the verbosity of file-urls and that Windows is the only supported operating system that uses the drive letter syntax, I don't think we can say the usage of file-urls are a general recommendation; but definitely a requirement for specifying absolute paths on Windows.
The prefix:path syntax is an unfortunate overlap. If not already clarified in the docs, I think it warrants an mention.

Thank you for bringing this to our attention @pckvcode 😃! I'm closing this PR, but if you still want to contribute to the project with your findings, you are more than welcome to open a new PR with updates to the docs.

Conftest encounters errors on Windows when loading file paths that include drive letters (e.g., `C:/path/to/data.yaml`). Even when using a file URL (e.g., `file:///C:/path/to/data.yaml`), we still face issues. With these code changes, Conftest can now successfully load files using a file URL (e.g., `file:///C:/path/to/data.yaml`). We opted for file URLs instead of paths with drive letters (e.g., `C:/path/to/data.yaml`) because OPA does not support file paths with drive letters. For more details, see [this issue comment](open-policy-agent/opa#6922 (comment)). Resolves: open-policy-agent#979 Signed-off-by: Punith C K <[email protected]>

pckvcode force-pushed the access-rego-policies-from-other-windows-drives branch from b00c103 to 02dc039 Compare August 21, 2024 10:48

Merge branch 'main' into access-rego-policies-from-other-windows-drives

9601e7e

boranx reviewed Aug 22, 2024

View reviewed changes

pckvcode added 2 commits August 23, 2024 14:21

Fixing lint in loader_test.go

035070d

Signed-off-by: pckvcode <[email protected]>

johanfylling reviewed Aug 26, 2024

View reviewed changes

johanfylling closed this Aug 27, 2024

pckvcode mentioned this pull request Aug 30, 2024

Bug: Accessing Rego Policies from Different Drives in Windows open-policy-agent/conftest#979

Open

pckvcode mentioned this pull request Sep 3, 2024

fix: Conftest can now successfully load files using a file URL (e.g., file:///C:/path/to/data.yaml) on windows open-policy-agent/conftest#999

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

The [opa eval] command is unable to access Rego policies located on other drives on windows #6922

The [opa eval] command is unable to access Rego policies located on other drives on windows #6922

pckvcode commented Aug 12, 2024 •

edited

Loading

philipaconrad commented Aug 12, 2024

netlify bot commented Aug 21, 2024 •

edited

Loading

pckvcode commented Aug 22, 2024

boranx left a comment

pckvcode commented Aug 23, 2024

johanfylling left a comment

johanfylling Aug 26, 2024

johanfylling Aug 26, 2024

johanfylling Aug 26, 2024

pckvcode Aug 27, 2024

pckvcode commented Aug 27, 2024

johanfylling commented Aug 27, 2024

The [opa eval] command is unable to access Rego policies located on other drives on windows #6922

The [opa eval] command is unable to access Rego policies located on other drives on windows #6922

Conversation

pckvcode commented Aug 12, 2024 • edited Loading

Why the changes in this PR are needed?

What are the changes in this PR?

Code Update:

Documentation Reference:

Notes to assist PR review:

Steps to Reproduce the issue:

Further comments:

[opa eval] command before fix. (Failed: opa in D drive and Failed to load rego policies from C drive.)

[opa eval] command after fix. (Success: opa in D drive and Successfully load rego policies from C drive.)

Unit Testcase pass screenshot. Function tested

philipaconrad commented Aug 12, 2024

netlify bot commented Aug 21, 2024 • edited Loading

✅ Deploy Preview for openpolicyagent ready!

pckvcode commented Aug 22, 2024

boranx left a comment

Choose a reason for hiding this comment

pckvcode commented Aug 23, 2024

johanfylling left a comment

Choose a reason for hiding this comment

johanfylling Aug 26, 2024

Choose a reason for hiding this comment

johanfylling Aug 26, 2024

Choose a reason for hiding this comment

johanfylling Aug 26, 2024

Choose a reason for hiding this comment

pckvcode Aug 27, 2024

Choose a reason for hiding this comment

pckvcode commented Aug 27, 2024

I tried using the path file:///c:/path/to/my/file/data.json, and it worked. However, when I used c:/path/to/my/file/data.json, it failed.

johanfylling commented Aug 27, 2024

pckvcode commented Aug 12, 2024 •

edited

Loading

netlify bot commented Aug 21, 2024 •

edited

Loading

I tried using the path `file:///c:/path/to/my/file/data.json`, and it worked.
However, when I used `c:/path/to/my/file/data.json`, it failed.