Difficulties with duniverse and mdx

[jonludlam]

This seems like the most appropriate place for this error report as it's specific to a packaging issue in a duniverse, but if this needs to go somewhere else I'm happy to open issues wherever necessary.

I ran into a few issues last week when trying to create a monorepo in which to run mdx. A minimal repro is:

• dune-project

(lang dune 3.3)
(using mdx 0.3)
(name mdxtest)
(generate_opam_files true)
(package (name mdxtest) (depends mdx))

• dune

(mdx (files *.mld))

• test.mld

{@ocaml[
# print_endline "Hello, world";;
]}

With just these files, running dune runtest results in the expected output "Hello, world" being promoted into the test mld file. However, when turning this into a duniverse, we then get a different output:

mdx-bug-repro$ opam monorepo lock
==> Using 1 locally scanned package as the target.
[WARNING] Unknown variable "ocaml-system:version"
[WARNING] Unknown variable "ocaml-base-compiler:version"
[WARNING] Unknown variable "ocaml-variants:version"
==> Found 29 opam dependencies for the target package.
==> Querying opam database for their metadata and Dune compatibility.
==> Calculating exact pins for each of them.
==> Wrote lockfile with 18 entries to /Users/jonathanludlam/devel/tmp/mdx-bug-repro/mdxtest.opam.locked. You can now run opam monorepo pull to fetch their sources.
mdx-bug-repro$ opam monorepo pull
==> Using lockfile /Users/jonathanludlam/devel/tmp/mdx-bug-repro/mdxtest.opam.locked
Successfully pulled 18/18 repositories
mdx-bug-repro$ dune runtest
File "dune", line 1, characters 0-19:
1 | (mdx (files *.mld))
    ^^^^^^^^^^^^^^^^^^^
Fatal error: exception Fl_package_base.No_such_package("unix", "")

After some time I discovered that the reason this works in realworldocaml/book is that the mdx stanza includes a (packages mdx) line, which causes the findlib package to be installed before running mdx, causing the creation of the META files in _build/install/default/lib for the compiler-installed packages. Adding this gave another error though:

Warning: 'packages' was deprecated in version 0.2 of mdx extension to verify
code blocks in .md files.
Error: Multiple rules generated for _build/install/default/lib/bytes/META:
- duniverse/lib-findlib/site-lib-src/dune:6
- _build/default/duniverse/bytes/META.bytes:1
-> required by alias duniverse/lib-findlib/.findlib-files
-> required by alias duniverse/mdx/.mdx-files
-> required by _build/default/.mdx/test.mld.corrected
-> required by alias runtest in dune:1

so the META files for the bytes library are being installed by base-bytes now rather than from findlib. I fixed that by removing them from the findlib package. So this seems like a reasonable fix for now, and I can make a PR with this in if it's right. I'm also interested in the longer-term plan though, particularly OCaml 5.0 onwards. As OCaml 5.0 ships its own META files, it seems less divergent from upstream to use those in favour of creating new ones in the findlib package (AIUI findlib 1.9.6 has logic to not install META files on 5.0+). There seem to be a few options:

1. Continue installing META files with the findlib package - seems like we'll carry this patch forever if we do this.
2. Add the compiler's lib/ocaml dir as a findlib path - by design, or maybe happy coincidence, this will work right now, as the compiler ships all its meta files in subdirs of lib/ocaml. I think the plan is to move them to the top-level dir though, in which case we'll need to add the lib dir, which seems dangerous maybe (as it may bring other libraries into the path).
3. symlinks from the _build/install/default/lib dirs to the compiler package dirs
4. Something else?

Also, are there any good ideas for fixing the original problem of mdx in a duniverse? The error about missing "unix" is completely opaque and it took me ages to figure out what on earth was going wrong. How can this be improved?