Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Turing model crashes without runtimeActivity #1623

Open
mhauru opened this issue Jul 10, 2024 · 10 comments
Open

Turing model crashes without runtimeActivity #1623

mhauru opened this issue Jul 10, 2024 · 10 comments
Assignees
@yebai @mhauru
Labels
more information needed

Comments

@mhauru
Copy link
Contributor

mhauru commented Jul 10, 2024 •
edited
Loading 

module MWE

import LogDensityProblems
using Turing: @model, MvNormal, LogDensityFunction

using Enzyme

@model function satellite_model_matrix(::Type{TV}=Matrix{Float64}) where {TV}
    @show TV
    P0 = vcat([0.1 0.0], [0.0 0.1])
    x = TV(undef, 2, 2)
    fill!(x, zero(eltype(x)))
    x[:, 2] ~ MvNormal(x[:, 1], P0)
    return nothing
end

model = satellite_model_matrix()

f = LogDensityFunction(model)
x = [1.0, 1.0]
Enzyme.autodiff(ReverseWithPrimal, LogDensityProblems.logdensity, Active, Const(f), Enzyme.Duplicated(x, zero(x)))

end
ERROR: Enzyme execution failed.
Mismatched activity for:   store {} addrspace(10)* %.fca.0.0.0.0.extract, {} addrspace(10)** %.fca.0.0.0.gep, align 8, !dbg !253, !noalias !240 const val:   %.fca.0.0.0.0.extract = extractvalue { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } } %0, 0, 0, 0, 0, !dbg !97
Type tree: {[-1]:Pointer, [-1,0]:Pointer, [-1,0,0]:Pointer, [-1,0,0,-1]:Integer, [-1,0,8]:Integer, [-1,0,9]:Integer, [-1,0,10]:Integer, [-1,0,11]:Integer, [-1,0,12]:Integer, [-1,0,13]:Integer, [-1,0,14]:Integer, [-1,0,15]:Integer, [-1,0,16]:Integer, [-1,0,17]:Integer, [-1,0,18]:Integer, [-1,0,19]:Integer, [-1,0,20]:Integer, [-1,0,21]:Integer, [-1,0,22]:Integer, [-1,0,23]:Integer, [-1,0,24]:Integer, [-1,0,25]:Integer, [-1,0,26]:Integer, [-1,0,27]:Integer, [-1,0,28]:Integer, [-1,0,29]:Integer, [-1,0,30]:Integer, [-1,0,31]:Integer, [-1,0,32]:Integer, [-1,0,33]:Integer, [-1,0,34]:Integer, [-1,0,35]:Integer, [-1,0,36]:Integer, [-1,0,37]:Integer, [-1,0,38]:Integer, [-1,0,39]:Integer, [-1,8]:Pointer, [-1,8,0]:Pointer, [-1,8,0,-1]:Integer, [-1,8,8]:Integer, [-1,8,9]:Integer, [-1,8,10]:Integer, [-1,8,11]:Integer, [-1,8,12]:Integer, [-1,8,13]:Integer, [-1,8,14]:Integer, [-1,8,15]:Integer, [-1,8,16]:Integer, [-1,8,17]:Integer, [-1,8,18]:Integer, [-1,8,19]:Integer, [-1,8,20]:Integer, [-1,8,21]:Integer, [-1,8,22]:Integer, [-1,8,23]:Integer, [-1,8,24]:Integer, [-1,8,25]:Integer, [-1,8,26]:Integer, [-1,8,27]:Integer, [-1,8,28]:Integer, [-1,8,29]:Integer, [-1,8,30]:Integer, [-1,8,31]:Integer, [-1,8,32]:Integer, [-1,8,33]:Integer, [-1,8,34]:Integer, [-1,8,35]:Integer, [-1,8,36]:Integer, [-1,8,37]:Integer, [-1,8,38]:Integer, [-1,8,39]:Integer, [-1,16]:Pointer, [-1,16,0]:Pointer, [-1,16,0,-1]:Integer, [-1,16,8]:Integer, [-1,16,9]:Integer, [-1,16,10]:Integer, [-1,16,11]:Integer, [-1,16,12]:Integer, [-1,16,13]:Integer, [-1,16,14]:Integer, [-1,16,15]:Integer, [-1,16,16]:Integer, [-1,16,17]:Integer, [-1,16,18]:Integer, [-1,16,19]:Integer, [-1,16,20]:Integer, [-1,16,21]:Integer, [-1,16,22]:Integer, [-1,16,23]:Integer, [-1,16,24]:Integer, [-1,16,25]:Integer, [-1,16,26]:Integer, [-1,16,27]:Integer, [-1,16,28]:Integer, [-1,16,29]:Integer, [-1,16,30]:Integer, [-1,16,31]:Integer, [-1,16,32]:Integer, [-1,16,33]:Integer, [-1,16,34]:Integer, [-1,16,35]:Integer, [-1,16,36]:Integer, [-1,16,37]:Integer, [-1,16,38]:Integer, [-1,16,39]:Integer, [-1,24]:Integer, [-1,25]:Integer, [-1,26]:Integer, [-1,27]:Integer, [-1,28]:Integer, [-1,29]:Integer, [-1,30]:Integer, [-1,31]:Integer, [-1,32]:Integer, [-1,33]:Integer, [-1,34]:Integer, [-1,35]:Integer, [-1,36]:Integer, [-1,37]:Integer, [-1,38]:Integer, [-1,39]:Integer, [-1,40]:Integer, [-1,41]:Integer, [-1,42]:Integer, [-1,43]:Integer, [-1,44]:Integer, [-1,45]:Integer, [-1,46]:Integer, [-1,47]:Integer, [-1,48]:Integer, [-1,49]:Integer, [-1,50]:Integer, [-1,51]:Integer, [-1,52]:Integer, [-1,53]:Integer, [-1,54]:Integer, [-1,55]:Integer, [-1,56]:Integer, [-1,57]:Integer, [-1,58]:Integer, [-1,59]:Integer, [-1,60]:Integer, [-1,61]:Integer, [-1,62]:Integer, [-1,63]:Integer}
 llvalue=  %.fca.0.0.0.0.extract = extractvalue { { [1 x [8 x {} addrspace(10)*]], {} addrspace(10)*, {} addrspace(10)* } } %0, 0, 0, 0, 0, !dbg !97
You may be using a constant variable as temporary storage for active memory (https://enzyme.mit.edu/julia/stable/faq/#Activity-of-temporary-storage). If not, please open an issue, and either rewrite this variable to not be conditionally active or use Enzyme.API.runtimeActivity!(true) as a workaround for now

Stacktrace:
 [1] _evaluate!!
   @ ~/.julia/packages/DynamicPPL/ACaKr/src/model.jl:967
 [2] evaluate_threadunsafe!!
   @ ~/.julia/packages/DynamicPPL/ACaKr/src/model.jl:941
 [3] evaluate!!
   @ ~/.julia/packages/DynamicPPL/ACaKr/src/model.jl:894
 [4] logdensity
   @ ~/.julia/packages/DynamicPPL/ACaKr/src/logdensityfunction.jl:100
 [5] logdensity
   @ ~/.julia/packages/DynamicPPL/ACaKr/src/logdensityfunction.jl:0

Stacktrace:
  [1] throwerr(cstr::Cstring)
    @ Enzyme.Compiler ~/.julia/dev/Enzyme/src/compiler.jl:1623
  [2] _evaluate!!
    @ ~/.julia/packages/DynamicPPL/ACaKr/src/model.jl:967 [inlined]
  [3] evaluate_threadunsafe!!
    @ ~/.julia/packages/DynamicPPL/ACaKr/src/model.jl:941 [inlined]
  [4] evaluate!!
    @ ~/.julia/packages/DynamicPPL/ACaKr/src/model.jl:894 [inlined]
  [5] logdensity
    @ ~/.julia/packages/DynamicPPL/ACaKr/src/logdensityfunction.jl:100 [inlined]
  [6] logdensity
    @ ~/.julia/packages/DynamicPPL/ACaKr/src/logdensityfunction.jl:0 [inlined]
  [7] diffejulia_logdensity_3364_inner_1wrap
    @ ~/.julia/packages/DynamicPPL/ACaKr/src/logdensityfunction.jl:0
  [8] macro expansion
    @ ~/.julia/dev/Enzyme/src/compiler.jl:6622 [inlined]
  [9] enzyme_call
    @ ~/.julia/dev/Enzyme/src/compiler.jl:6223 [inlined]
 [10] CombinedAdjointThunk
    @ ~/.julia/dev/Enzyme/src/compiler.jl:6100 [inlined]
 [11] autodiff
    @ ~/.julia/dev/Enzyme/src/Enzyme.jl:309 [inlined]
 [12] autodiff(::EnzymeCore.ReverseMode{true, EnzymeCore.FFIABI, false}, ::typeof(LogDensityProblems.logdensity), ::Type{EnzymeCore.Active}, ::EnzymeCore.Const{DynamicPPL.LogDensityFunction{…}}, ::EnzymeCore.Duplicated{Vector{…}})
    @ Enzyme ~/.julia/dev/Enzyme/src/Enzyme.jl:321
 [13] top-level scope
    @ REPL[1]:21
Some type information was truncated. Use `show(err)` to see complete types.

Works with runtimeActivity enabled. Enzyme_jll v0.0.133. First discussed in TuringLang/Turing.jl#1608.
@wsmoses
Copy link
Member

wsmoses commented Jul 10, 2024 via email

This was referenced Jul 24, 2024
Closed
Closed
@wsmoses
Copy link
Member

wsmoses commented Jul 21, 2024 •
edited
Loading

bump @mhauru @yebai

@yebai
Copy link

yebai commented Aug 21, 2024 •
edited
Loading

Can we work around it, assuming TuringLang/DynamicPPL.jl#643 (comment) was the cause?

@wsmoses
Copy link
Member

wsmoses commented Aug 21, 2024

The error message thrown says to use Enzyme runtimeactivry (which allows this to run iirc).

@yebai
Copy link

yebai commented Aug 21, 2024

Yes, the above model runs without issue with Enzyme.API.runtimeActivity!(true).

@wsmoses
Copy link
Member

wsmoses commented Sep 24, 2024

any updates on reduction here?

@yebai
Copy link

yebai commented Sep 25, 2024

This is not a Turing issue, nor an Enzyme issue. See JuliaLang/julia#55638

Also, this performance penalty is likely related to failed type inference for a customised interpreter: #1769

@wsmoses
Copy link
Member

wsmoses commented Sep 28, 2024

I don't think this issue relates to that. I think we just need a reduction for a minimal code here of why runtime activity is needed.

@yebai
Copy link

yebai commented Sep 28, 2024

I suggest we wait for JuliaLang/julia#55638 to be addressed before investing any further time in this.

@wsmoses
Copy link
Member

wsmoses commented Sep 28, 2024

I'm not confident this is due to that. Do you have evidence or a MWE which demonstrates that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yebai yebai

@mhauru mhauru

Labels
more information needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@wsmoses @yebai @mhauru