-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reverse gradient works with CPU + KA but errors out after ~minutes with error on CUDA + KA #2133
Comments
looks like the constructor rule for CuArray here https://github.com/roflmaostc/RadonKA.jl/blob/5960b7cd3d4d845a92e85ea1112cdf89f631b81c/src/radon.jl#L145C4-L145C40 isn't getting triggered for some reason. File an issue on CUDA.jl? (ideally with a self contained function we can add a test to make sure it doesn't pop up in the future) |
Now it got even worse using CUDA, Enzyme
function f(x, y)
y2 = similar(x, (size(y, 1),))
y2 .= y
return sum(x .* y2)
end
x = CUDA.rand(10)
y = 1:10
dx = make_zero(x)
@show f(x,y)
autodiff(Reverse, f, Active, Duplicated(x, dx), Const(y)) julia> include("/tmp/lel2.jl")
f(x, y) = 23.677639f0
julia: /workspace/srcdir/Enzyme/enzyme/Enzyme/GradientUtils.cpp:4412: DIFFE_TYPE GradientUtils::getDiffeType(llvm::Value*, bool) const: Assertion `unnecessaryValuesP' failed.
[583243] signal (6.-6): Aborted
in expression starting at /tmp/lel2.jl:15
pthread_kill at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x74dbba22881a)
__assert_fail at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
getDiffeType at /workspace/srcdir/Enzyme/enzyme/Enzyme/GradientUtils.cpp:4412
EnzymeGradientUtilsGetDiffeType at /home/felix/.julia/packages/Enzyme/RTS5U/src/api.jl:742
unknown function (ip: 0x74db5f12bba0)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
enzyme_custom_setup_args at /home/felix/.julia/packages/Enzyme/RTS5U/src/rules/customrules.jl:117
unknown function (ip: 0x74db5f11c0ca)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
aug_fwd_mi at /home/felix/.julia/packages/Enzyme/RTS5U/src/rules/customrules.jl:781 [inlined]
aug_fwd_mi at /home/felix/.julia/packages/Enzyme/RTS5U/src/rules/customrules.jl:774 [inlined]
has_aug_fwd_rule at /home/felix/.julia/packages/Enzyme/RTS5U/src/rules/customrules.jl:878 [inlined]
enzyme_custom_diffuse at /home/felix/.julia/packages/Enzyme/RTS5U/src/rules/customrules.jl:1602
unknown function (ip: 0x74db5f117b7f)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
enzyme_custom_diffuse_cfunc at /home/felix/.julia/packages/Enzyme/RTS5U/src/rules/llvmrules.jl:105
unknown function (ip: 0x74db68d374b7)
operator() at /workspace/srcdir/Enzyme/enzyme/Enzyme/CApi.cpp:383 [inlined]
__invoke_impl<bool, EnzymeRegisterDiffUseCallHandler(char*, CustomFunctionDiffUse)::<lambda(const llvm::CallInst*, const GradientUtils*, const llvm::Value*, bool, DerivativeMode, bool&)>&, const llvm::CallInst*, const GradientUtils*, const llvm::Value*, bool, DerivativeMode, bool&> at /opt/x86_64-linux-gnu/x86_64-linux-gnu/include/c++/10.2.0/bits/invoke.h:60 [inlined]
__invoke_r<bool, EnzymeRegisterDiffUseCallHandler(char*, CustomFunctionDiffUse)::<lambda(const llvm::CallInst*, const GradientUtils*, const llvm::Value*, bool, DerivativeMode, bool&)>&, const llvm::CallInst*, const GradientUtils*, const llvm::Value*, bool, DerivativeMode, bool&> at /opt/x86_64-linux-gnu/x86_64-linux-gnu/include/c++/10.2.0/bits/invoke.h:113 [inlined]
_M_invoke at /opt/x86_64-linux-gnu/x86_64-linux-gnu/include/c++/10.2.0/bits/std_function.h:291
operator() at /opt/x86_64-linux-gnu/x86_64-linux-gnu/include/c++/10.2.0/bits/std_function.h:622 [inlined]
is_use_directly_needed_in_reverse at /workspace/srcdir/Enzyme/enzyme/Enzyme/DifferentialUseAnalysis.cpp:464
is_value_needed_in_reverse<(QueryType)0> at /workspace/srcdir/Enzyme/enzyme/Enzyme/DifferentialUseAnalysis.h:419
is_value_needed_in_reverse<(QueryType)0> at /workspace/srcdir/Enzyme/enzyme/Enzyme/DifferentialUseAnalysis.h:243
is_value_needed_in_reverse<(QueryType)0> at /workspace/srcdir/Enzyme/enzyme/Enzyme/DifferentialUseAnalysis.h:243
is_value_needed_in_reverse<(QueryType)0> at /workspace/srcdir/Enzyme/enzyme/Enzyme/DifferentialUseAnalysis.h:243
computeForwardingProperties at /workspace/srcdir/Enzyme/enzyme/Enzyme/GradientUtils.cpp:8733
computeGuaranteedFrees at /workspace/srcdir/Enzyme/enzyme/Enzyme/GradientUtils.cpp:9276
CreateAugmentedPrimal at /workspace/srcdir/Enzyme/enzyme/Enzyme/EnzymeLogic.cpp:2344
recursivelyHandleSubfunction at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:5303
visitCallInst at /workspace/srcdir/Enzyme/enzyme/Enzyme/AdjointGenerator.h:6479
visit at /opt/x86_64-linux-gnu/x86_64-linux-gnu/sys-root/usr/local/include/llvm/IR/InstVisitor.h:111 [inlined]
CreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/EnzymeLogic.cpp:4305
EnzymeCreatePrimalAndGradient at /workspace/srcdir/Enzyme/enzyme/Enzyme/CApi.cpp:631
EnzymeCreatePrimalAndGradient at /home/felix/.julia/packages/Enzyme/RTS5U/src/api.jl:253
unknown function (ip: 0x74db5f115fae)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
enzyme! at /home/felix/.julia/packages/Enzyme/RTS5U/src/compiler.jl:4186
unknown function (ip: 0x74db5f114398)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
#codegen#19015 at /home/felix/.julia/packages/Enzyme/RTS5U/src/compiler.jl:7338
codegen at /home/felix/.julia/packages/Enzyme/RTS5U/src/compiler.jl:6146 [inlined]
_thunk at /home/felix/.julia/packages/Enzyme/RTS5U/src/compiler.jl:8468
_thunk at /home/felix/.julia/packages/Enzyme/RTS5U/src/compiler.jl:8468 [inlined]
cached_compilation at /home/felix/.julia/packages/Enzyme/RTS5U/src/compiler.jl:8509 [inlined]
thunkbase at /home/felix/.julia/packages/Enzyme/RTS5U/src/compiler.jl:8641
unknown function (ip: 0x74db5ff30600)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
#s2103#19072 at /home/felix/.julia/packages/Enzyme/RTS5U/src/compiler.jl:8778 [inlined]
#s2103#19072 at ./none:0
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
GeneratedFunctionStub at ./boot.jl:602
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_call_staged at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/method.c:540
ijl_code_for_staged at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/method.c:593
get_staged at ./compiler/utilities.jl:123
retrieve_code_info at ./compiler/utilities.jl:135 [inlined]
InferenceState at ./compiler/inferencestate.jl:430
typeinf_edge at ./compiler/typeinfer.jl:920
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2087
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_call at ./compiler/abstractinterpretation.jl:2162
abstract_call at ./compiler/abstractinterpretation.jl:2354
abstract_eval_call at ./compiler/abstractinterpretation.jl:2370
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2380
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2624
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2889
typeinf_local at ./compiler/abstractinterpretation.jl:3098
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3186
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_edge at ./compiler/typeinfer.jl:930
abstract_call_method at ./compiler/abstractinterpretation.jl:629
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:95
abstract_call_known at ./compiler/abstractinterpretation.jl:2087
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_apply at ./compiler/abstractinterpretation.jl:1612
abstract_call_known at ./compiler/abstractinterpretation.jl:2004
abstract_call at ./compiler/abstractinterpretation.jl:2169
abstract_call at ./compiler/abstractinterpretation.jl:2162
abstract_call at ./compiler/abstractinterpretation.jl:2354
abstract_eval_call at ./compiler/abstractinterpretation.jl:2370
abstract_eval_statement_expr at ./compiler/abstractinterpretation.jl:2380
abstract_eval_statement at ./compiler/abstractinterpretation.jl:2624
abstract_eval_basic_statement at ./compiler/abstractinterpretation.jl:2913
typeinf_local at ./compiler/abstractinterpretation.jl:3098
typeinf_nocycle at ./compiler/abstractinterpretation.jl:3186
_typeinf at ./compiler/typeinfer.jl:247
typeinf at ./compiler/typeinfer.jl:216
typeinf_ext at ./compiler/typeinfer.jl:1051
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1082
typeinf_ext_toplevel at ./compiler/typeinfer.jl:1078
jfptr_typeinf_ext_toplevel_45362.1 at /home/felix/.julia/juliaup/julia-1.10.6+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
jl_type_infer at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:394
jl_generate_fptr_impl at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/jitlayers.cpp:504
jl_compile_method_internal at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2481 [inlined]
jl_compile_method_internal at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2368
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2887 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_call at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:126
eval_value at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:223
eval_stmt_value at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:174 [inlined]
eval_body at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:617
jl_interpret_toplevel_thunk at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:775
jl_toplevel_eval_flex at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:385 [inlined]
include_string at ./loading.jl:2076
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
_include at ./loading.jl:2136
include at ./client.jl:494
unknown function (ip: 0x74dbb9a8af05)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
do_call at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:126
eval_value at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:223
eval_stmt_value at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:174 [inlined]
eval_body at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:617
jl_interpret_toplevel_thunk at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/interpreter.c:775
jl_toplevel_eval_flex at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/toplevel.c:934
jl_toplevel_eval_flex at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/toplevel.c:877
jl_toplevel_eval_flex at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/toplevel.c:877
jl_toplevel_eval_flex at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/toplevel.c:877
ijl_toplevel_eval_in at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/toplevel.c:985
eval at ./boot.jl:385 [inlined]
eval_user_input at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:150
repl_backend_loop at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:246
#start_repl_backend#46 at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:231
start_repl_backend at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:228
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
#run_repl#59 at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:389
run_repl at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/usr/share/julia/stdlib/v1.10/REPL/src/REPL.jl:375
jfptr_run_repl_91949.1 at /home/felix/.julia/juliaup/julia-1.10.6+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
#1013 at ./client.jl:437
jfptr_YY.1013_82918.1 at /home/felix/.julia/juliaup/julia-1.10.6+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
jl_f__call_latest at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/builtins.c:812
#invokelatest#2 at ./essentials.jl:892 [inlined]
invokelatest at ./essentials.jl:889 [inlined]
run_main_repl at ./client.jl:421
exec_options at ./client.jl:338
_start at ./client.jl:557
jfptr__start_82944.1 at /home/felix/.julia/juliaup/julia-1.10.6+0.x64.linux.gnu/lib/julia/sys.so (unknown line)
_jl_invoke at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:2895 [inlined]
ijl_apply_generic at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/gf.c:3077
jl_apply at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/julia.h:1982 [inlined]
true_main at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/jlapi.c:582
jl_repl_entrypoint at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/src/jlapi.c:731
main at /cache/build/builder-amdci5-5/julialang/julia-release-1-dot-10/cli/loader_exe.c:58
unknown function (ip: 0x74dbba22a1c9)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x4010b8)
Allocations: 196795431 (Pool: 196528539; Big: 266892); GC: 135
[1] 583243 IOT instruction (core dumped) julia
`` |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi,
the following is veeeery slow on CUDA and also errors out:
The text was updated successfully, but these errors were encountered: