Quarto engine: julia does not preview/render with Julia 1.11

[hurak]

Bug description

Even the simplest Quarto qmd document does not run (both preview and render fail) after Julia was upgraded from 1.10 to 1.11.

Steps to reproduce

The version 1.11 just installed. Within an empty directory I created (activated) a brand new Julia environment. I created this single qmd document and tried to preview/render it using Quarto.

Quarto v1.6.24

---
title: "Reproducing the error"
format: html
engine: julia
---

Reproducing the error when a Julia code is run in a Quarto document.

```{julia}
a = 1
```

Expected behavior

No response

Actual behavior

This is the error message I receive upon trying to run preview or render

➜  tmp_quarto_julia quarto render pokus.qmd
Starting julia control server process. This might take a while...
Julia server process started.
ERROR: Julia server returned error after receiving "run" command:
Failed to run notebook: /Users/hurak/Documents/julia/tmp_quarto_julia/pokus.qmd
ERROR: Remote exception from Malt.Worker on port 9503 with PID 18503:

InitError: ConcurrencyViolationError("deadlock detected in loading QuartoNotebookWorker -> QuartoNotebookWorker")
Stacktrace:
  [1] start_loading(modkey::Base.PkgId, build_id::UInt128, stalecheck::Bool)
    @ Base ./loading.jl:2048
  [2] __require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2295
  [3] #invoke_in_world#3
    @ ./essentials.jl:1088 [inlined]
  [4] invoke_in_world
    @ ./essentials.jl:1085 [inlined]
  [5] _require_prelocked(uuidkey::Base.PkgId, env::String)
    @ Base ./loading.jl:2287
  [6] macro expansion
    @ ./loading.jl:2226 [inlined]
  [7] macro expansion
    @ ./lock.jl:273 [inlined]
  [8] __require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2183
  [9] #invoke_in_world#3
    @ ./essentials.jl:1088 [inlined]
 [10] invoke_in_world
    @ ./essentials.jl:1085 [inlined]
 [11] require(into::Module, mod::Symbol)
    @ Base ./loading.jl:2176
 [12] top-level scope
    @ ~/.julia/packages/QuartoNotebookRunner/VUFgu/src/QuartoNotebookWorker/src/NotebookState.jl:44
 [13] eval
    @ ./boot.jl:430 [inlined]
 [14] define_notebook_module!(root::Module)
    @ QuartoNotebookWorker.NotebookState ~/.julia/packages/QuartoNotebookRunner/VUFgu/src/QuartoNotebookWorker/src/NotebookState.jl:51
 [15] define_notebook_module!
    @ ~/.julia/packages/QuartoNotebookRunner/VUFgu/src/QuartoNotebookWorker/src/NotebookState.jl:23 [inlined]
 [16] __init__()
    @ QuartoNotebookWorker.NotebookState ~/.julia/packages/QuartoNotebookRunner/VUFgu/src/QuartoNotebookWorker/src/NotebookState.jl:13
 [17] run_module_init(mod::Module, i::Int64)
    @ Base ./loading.jl:1336
 [18] register_restored_modules(sv::Core.SimpleVector, pkg::Base.PkgId, path::String)
    @ Base ./loading.jl:1324
 [19] _include_from_serialized(pkg::Base.PkgId, path::String, ocachepath::String, depmods::Vector{Any}, ignore_native::Nothing; register::Bool)
    @ Base ./loading.jl:1213
 [20] _include_from_serialized (repeats 2 times)
    @ ./loading.jl:1169 [inlined]
 [21] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt128, stalecheck::Bool; reasons::Dict{String, Int64}, DEPOT_PATH::Vector{String})
    @ Base ./loading.jl:1975
 [22] _require(pkg::Base.PkgId, env::Nothing)
    @ Base ./loading.jl:2435
 [23] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
    @ Base ./loading.jl:2300
 [24] #invoke_in_world#3
    @ ./essentials.jl:1088 [inlined]
 [25] invoke_in_world
    @ ./essentials.jl:1085 [inlined]
 [26] _require_prelocked
    @ ./loading.jl:2287 [inlined]
 [27] _require_prelocked
    @ ./loading.jl:2286 [inlined]
 [28] macro expansion
    @ ./lock.jl:273 [inlined]
 [29] require(uuidkey::Base.PkgId)
    @ Base ./loading.jl:2283
 [30] top-level scope
    @ ~/.julia/packages/QuartoNotebookRunner/VUFgu/src/worker.jl:7
 [31] eval
    @ ./boot.jl:430 [inlined]
 [32] (::var"#1#2"{Sockets.TCPSocket, UInt64, Bool, @Kwargs{}, Tuple{Module, Expr}, typeof(Core.eval)})()
    @ Main ~/.julia/packages/Malt/YJ2Ml/src/worker.jl:120
during initialization of module NotebookState
Stacktrace:
  [1] unwrap_worker_result(worker::Malt.Worker, result::Malt.WorkerResult)
    @ Malt ~/.julia/packages/Malt/YJ2Ml/src/Malt.jl:50
  [2] _wait_for_response(worker::Malt.Worker, msg_id::UInt64)
    @ Malt ~/.julia/packages/Malt/YJ2Ml/src/Malt.jl:325
  [3] _send_receive
    @ ~/.julia/packages/Malt/YJ2Ml/src/Malt.jl:336 [inlined]
  [4] #remote_call_fetch#41
    @ ~/.julia/packages/Malt/YJ2Ml/src/Malt.jl:406 [inlined]
  [5] remote_call_fetch
    @ ~/.julia/packages/Malt/YJ2Ml/src/Malt.jl:405 [inlined]
  [6] remote_eval_fetch
    @ ~/.julia/packages/Malt/YJ2Ml/src/Malt.jl:484 [inlined]
  [7] remote_eval_fetch
    @ ~/.julia/packages/Malt/YJ2Ml/src/Malt.jl:485 [inlined]
  [8] init!
    @ ~/.julia/packages/QuartoNotebookRunner/VUFgu/src/server.jl:97 [inlined]
  [9] QuartoNotebookRunner.File(path::String, options::Dict{String, Any})
    @ QuartoNotebookRunner ~/.julia/packages/QuartoNotebookRunner/VUFgu/src/server.jl:26
 [10] (::QuartoNotebookRunner.var"#41#44"{Bool, Dict{String, Any}, Server, String})()
    @ QuartoNotebookRunner ~/.julia/packages/QuartoNotebookRunner/VUFgu/src/server.jl:1116
 [11] lock(f::QuartoNotebookRunner.var"#41#44"{Bool, Dict{String, Any}, Server, String}, l::ReentrantLock)
    @ Base ./lock.jl:232
 [12] borrow_file!(f::QuartoNotebookRunner.var"#32#36"{Nothing, String, Bool, Dict{String, Any}, QuartoNotebookRunner.var"#chunk_callback#67"{TCPSocket}, Server}, server::Server, path::String; optionally_create::Bool, options::Dict{String, Any})
    @ QuartoNotebookRunner ~/.julia/packages/QuartoNotebookRunner/VUFgu/src/server.jl:1109
 [13] borrow_file!
    @ ~/.julia/packages/QuartoNotebookRunner/VUFgu/src/server.jl:1100 [inlined]
 [14] #run!#31
    @ ~/.julia/packages/QuartoNotebookRunner/VUFgu/src/server.jl:1069 [inlined]
 [15] _handle_response(socket::TCPSocket, notebooks::Server, request::@NamedTuple{type::String, content::Union{Dict{String, Any}, String}}, showprogress::Bool)
    @ QuartoNotebookRunner ~/.julia/packages/QuartoNotebookRunner/VUFgu/src/socket.jl:273
 [16] (::QuartoNotebookRunner.var"#59#66"{Bool, Base.RefValue{Bool}, Server, Base.UUID})()
    @ QuartoNotebookRunner ~/.julia/packages/QuartoNotebookRunner/VUFgu/src/socket.jl:201
ERROR: Internal julia server error

Stack trace:
    at writeJuliaCommand (file:///Applications/quarto/bin/quarto.js:41307:19)
    at eventLoopTick (ext:core/01_core.js:175:7)
    at async executeJulia (file:///Applications/quarto/bin/quarto.js:41201:22)
    at async Object.execute (file:///Applications/quarto/bin/quarto.js:40938:20)
    at async renderExecute (file:///Applications/quarto/bin/quarto.js:84987:27)
    at async renderFileInternal (file:///Applications/quarto/bin/quarto.js:85155:43)
    at async renderFiles (file:///Applications/quarto/bin/quarto.js:85023:17)
    at async render (file:///Applications/quarto/bin/quarto.js:89887:21)
    at async Command.actionHandler (file:///Applications/quarto/bin/quarto.js:90035:32)
    at async Command.execute (file:///Applications/quarto/bin/quarto.js:8070:13)

Your environment

• IDE: Verze: 1.94.1
• OS: Darwin arm64 24.0.0

Quarto check output

➜  tmp_quarto_julia quarto check
Quarto 1.6.24
[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.4.0: OK
      Dart Sass version 1.70.0: OK
      Deno version 1.46.3: OK
      Typst version 0.11.0: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 1.6.24
      Path: /Applications/quarto/bin

[✓] Checking tools....................OK
      TinyTeX: (not installed)
      Chromium: (not installed)

[✓] Checking LaTeX....................OK
      Using: Installation From Path
      Path: /Library/TeX/texbin
      Version: 2024

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.12.7
      Path: /opt/homebrew/opt/[email protected]/bin/python3.12
      Jupyter: (None)

      Jupyter is not available in this Python installation.
      Install with python3 -m pip install jupyter

[✓] Checking R installation...........(None)

      Unable to locate an installed version of R.
      Install R from https://cloud.r-project.org/

[cderv]

I can reproduce on my side too with same error.

It could be a problem in https://github.com/PumasAI/QuartoNotebookRunner.jl

@jkrumbiegel we need your help. Thank you !

[floswald]

I think the problem is that julia 1.11 changed the signature of the function Pkg.is_manifest_current() to Pkg.is_manifest_current(path::AbstractString)

in fact that's the error I get:

floswald@PTL11077 ~/g/CompEcon (main)> quarto preview /Users/floswald/git/CompEcon/absorbing-mc.qmd --no-browser --no-watch-i
nputs
Starting julia control server process. This might take a while...
ERROR: LoadError: MethodError: no method matching is_manifest_current()
The function `is_manifest_current` exists, but no method is defined for this combination of argument types.

Closest candidates are:
  is_manifest_current(::Pkg.Types.Context)
   @ Pkg ~/.julia/juliaup/julia-1.11.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.11/Pkg/src/API.jl:489
  is_manifest_current(::AbstractString)
   @ Pkg ~/.julia/juliaup/julia-1.11.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.11/Pkg/src/API.jl:490

Stacktrace:
 [1] top-level scope
   @ /Applications/quarto/share/julia/ensure_environment.jl:17
in expression starting at /Applications/quarto/share/julia/ensure_environment.jl:17
ERROR: Ensuring an updated julia server environment failed

I guess just changing to look for the manifest in the current dir could be a solution and say Pkg.is_manifest_current(".") instead

julia> Pkg.is_manifest_current(".")
true

[jkrumbiegel]

I think the problem is that julia 1.11 changed the signature of the function Pkg.is_manifest_current() to Pkg.is_manifest_current(path::AbstractString)

That was also a problem but I fixed that a while ago, you're probably on an older quarto version.

quarto-cli/src/resources/julia/ensure_environment.jl

Lines 17 to 21 in e51b07e

	is_manifest_current = @static if VERSION < v"1.11.0-DEV.1135"
	Pkg.is_manifest_current()
	else
	Pkg.is_manifest_current(dirname(Base.active_project()))
	end

The error here is different and tracked in PumasAI/QuartoNotebookRunner.jl#190

[floswald]

I'm on the latest quarto version (august 2024). I can see your commit is from early july. sure that ever got merged? or do I have to build quarto from source to get this? thanks

[jkrumbiegel]

The change is only on the 1.6 prerelease versions as far as I can see. You can download these under Github releases or here for example https://quarto.org/docs/download/

[floswald]

Arg...should have gotten that myself. Sorry.
Thanks so much for this new way of using Julia in quarto, it's a game changer for me!

[jkrumbiegel]

No worries :) glad it's useful!

[mcanouil]

To be clear, 1.6 is a pre-release and is not officially released as "stable" at this point.
The changelog for 1.6 https://prerelease.quarto.org/docs/download/prerelease.html

[RoyCCWang]

To be clear, 1.6 is a pre-release and is not officially released as "stable" at this point. The changelog for 1.6 https://prerelease.quarto.org/docs/download/prerelease.html

Under the "Highlights > Native Julia Engine" (section > bullet) from the Quarto 1.5 download webpage, it might help to mention only Julia versions v.1.10 and lower are supported as of right now. This is because Julia v1.11 was officially released just a few weeks ago, and mainstream Julia users who make this upgrade will encounter this issue when they try out the current Quarto v1.5.

[cscheid]

This is closed by #11100.