StackOverflowError during type inference on PDMats 0.11.5

[arnaudh]

I'm not familiar with type inference so not sure what is going on, and whether this has more to do with KeyedDistributions than with PDMats. However I wasn't seeing the error with PDMats 0.11.4, but now seeing it with 0.11.5.

Here's my MWE below, happy to provide more information.

(mwe) pkg> st
      Status `~/src/fix-recursion-inference-PDMats/mwe/Project.toml`
  [31c24e10] Distributions v0.25.36
  [2576fb08] KeyedDistributions v0.1.4
  [90014a1f] PDMats v0.11.5

using Distributions
using KeyedDistributions
using PDMats

my_dist = KeyedDistribution(DiagNormal([0.8, 0.6], PDiagMat([1.0, 1.0])), (["a","b"],))

function my_func(dist::KeyedDistribution)
    return dist(["a"])
end

my_dist(["a"]) # runs, but prints a StackOverflowError() stack trace to stderr during type inference (see 1.log) (note from the REPL I get a segfault)

my_func(my_dist) # runs, but prints a StackOverflowError() stack trace to stderr during type inference (see 2.log)

[my_func(d) for d in [my_dist, my_dist]] # prints a StackOverflowError() stack trace to stderr during type inference, then errors with a StackOverflowError (see 3.log)

1.log
2.log
3.log

Here's the StackOverflowError during type inference (from 1.log, abriged with ... for the 3 lines that get repeated for a while):

Internal error: encountered unexpected error in runtime:
StackOverflowError()
subtype_ccheck at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
var_gt at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
forall_exists_subtype at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
subtype_ccheck at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
var_gt at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
forall_exists_subtype at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
...
subtype_ccheck at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
var_gt at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
forall_exists_subtype at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
subtype_ccheck at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
var_gt at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
forall_exists_subtype at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
subtype_ccheck at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
var_gt at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
subtype at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
forall_exists_subtype at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
subtype_ccheck at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
var_gt at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
forall_exists_equal at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
subtype at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
with_tvar at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
subtype at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
with_tvar at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
subtype at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
with_tvar at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
subtype at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
subtype_tuple_tail at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
subtype at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
with_tvar at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
subtype at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
with_tvar at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
subtype at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
with_tvar at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
subtype at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
with_tvar at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
subtype at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
forall_exists_subtype at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_subtype_env at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_type_intersection_env_s at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
ml_matches at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_matching_methods at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
_methods_by_ftype at ./reflection.jl:867 [inlined]
#findall#216 at ./compiler/methodtable.jl:57 [inlined]
findall##kw at ./compiler/methodtable.jl:54
jfptr_findallYY.YY.kw_13021.clone_1 at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
#218 at ./compiler/methodtable.jl:67
get! at ./iddict.jl:163 [inlined]
#findall#217 at ./compiler/methodtable.jl:66 [inlined]
findall##kw at ./compiler/methodtable.jl:65
jfptr_findallYY.YY.kw_13016.clone_1 at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:84
abstract_call_known at ./compiler/abstractinterpretation.jl:1033
abstract_call at ./compiler/abstractinterpretation.jl:1056
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1462
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call_known at ./compiler/abstractinterpretation.jl:1033
abstract_call at ./compiler/abstractinterpretation.jl:1056
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1462
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call_known at ./compiler/abstractinterpretation.jl:1033
abstract_call at ./compiler/abstractinterpretation.jl:1056
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1462
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_edge at ./compiler/typeinfer.jl:806
abstract_call_method at ./compiler/abstractinterpretation.jl:490
abstract_call_gf_by_type at ./compiler/abstractinterpretation.jl:143
abstract_call_known at ./compiler/abstractinterpretation.jl:1033
abstract_call at ./compiler/abstractinterpretation.jl:1056
abstract_call at ./compiler/abstractinterpretation.jl:1040
abstract_eval_statement at ./compiler/abstractinterpretation.jl:1167
typeinf_local at ./compiler/abstractinterpretation.jl:1462
typeinf_nocycle at ./compiler/abstractinterpretation.jl:1520
_typeinf at ./compiler/typeinfer.jl:214
typeinf at ./compiler/typeinfer.jl:209
typeinf_ext at ./compiler/typeinfer.jl:892
typeinf_ext_toplevel at ./compiler/typeinfer.jl:925
typeinf_ext_toplevel at ./compiler/typeinfer.jl:921
jfptr_typeinf_ext_toplevel_12207.clone_1 at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_type_infer at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_generate_fptr at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_compile_method_internal at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
do_call at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
eval_body at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_interpret_toplevel_thunk at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
top-level scope at /Users/arnaudhenry/src/fix-recursion-inference-PDMats/mwe/mwe.jl:11
jl_toplevel_eval_flex at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_toplevel_eval_flex at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
jl_toplevel_eval_in at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
eval at ./boot.jl:360 [inlined]
include_string at ./loading.jl:1094
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
_include at ./loading.jl:1148
include at ./Base.jl:386
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
exec_options at ./client.jl:285
_start at ./client.jl:485
jfptr__start_32022.clone_1 at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/sys.dylib (unknown line)
jl_apply_generic at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
true_main at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)
repl_entrypoint at /Applications/Julia-1.6.app/Contents/Resources/julia/lib/julia/libjulia-internal.1.6.dylib (unknown line)

[devmotion]

Maybe the same error as in invenia/KeyedDistributions.jl#24?

[iamed2]

Maybe the same error as in invenia/KeyedDistributions.jl#24?

Seems likely; this error also does not happen on Julia 1.7.

[devmotion]

Changing

PDMats.jl/src/pdiagmat.jl

Line 4 in 15fecad

struct PDiagMat{T<:Real,V<:AbstractVector{T}} <: AbstractPDMat{T}

to struct PDiagMat{T<:Real,V<:AbstractVector} <: AbstractMatrix{T} fixes the issue for me.

I assume that 1.6 (but apparently not Julia 1.7) has a problem with the type-inference failures of first(map(...)) in https://github.com/invenia/KeyedDistributions.jl/blob/e0a72a014d97810e2b25e211644ff0e288064be0/src/KeyedDistributions.jl#L66. I don't know how and why this happens though, and why this would cause problems with the callable syntax but not with standard indexing (invenia/KeyedDistributions.jl#24 (comment)).

[arnaudh]

Maybe the same error as in invenia/KeyedDistributions.jl#24?

Ah yes, sorry I should have looked there beforehand.

It seems there are different subtle changes that can trigger the type-inference failure, but PDMats is not at fault given it can also happen without PDMats.

Feel free to close this issue.

[devmotion]

PDMats is not at fault given it can also happen without PDMats.

Also the other (well, actually same) issue in KeyedDistributions is caused by the change of the type parameter in PDMats. I'd like to understand what's going on there and why it causes problems with Julia 1.6 since it is the correct and desired parameterization. However, maybe it would be easier to just apply the fix mentioned above - even though I think it should not be needed...