Add tape_type query given a parent compiler job context #1104

wsmoses · 2023-10-19T18:53:40Z

Still needs a test, but pinging @vchuravy for review/thoughts

wsmoses · 2023-11-20T13:58:03Z

@michel2323 @vchuravy re parent job/GPU things, if you want to try that out.

vchuravy · 2023-11-20T19:06:34Z

Looks about right.

codecov-commenter · 2023-11-21T01:49:48Z

Codecov Report

Attention: 49 lines in your changes are missing coverage. Please review.

Comparison is base (ba58bcb) 75.13% compared to head (8701270) 74.75%.
Report is 2 commits behind head on main.

Files	Patch %	Lines
src/Enzyme.jl	4.08%	47 Missing ⚠️
src/absint.jl	33.33%	2 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1104      +/-   ##
==========================================
- Coverage   75.13%   74.75%   -0.39%     
==========================================
  Files          35       35              
  Lines       10397    10442      +45     
==========================================
- Hits         7812     7806       -6     
- Misses       2585     2636      +51

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

wsmoses · 2023-12-05T17:03:21Z

@sriharikrishna

michel2323 · 2023-12-15T20:06:58Z

@wsmoses Trying to get GPUs in KA going with https://github.com/JuliaGPU/KernelAbstractions.jl/tree/enz_rev_gpu . I use the 'reverse_gpu.jl` script for development. Now, the compiler crashes.

In this PR I did this notable (uninformed) changes:

Uncomment your line world=... to use on fspec()
Changes key=hash((hash(job), parent_job) to key=hash((job, parent_job)). I'm not sure where tape_cache is used other than here. I guess it should also be used in the reverse.
tape_type() didn't return any type. So I return meta.TapeType.

I believe the GPU rules don't actually belong in KA but in the backends like CUDA.jl, but let's ignore that for the moment.

michel2323 · 2023-12-15T20:11:44Z

The log.
out.log

michel2323 · 2023-12-15T20:13:42Z

@vchuravy Does this make any sense: https://github.com/JuliaGPU/KernelAbstractions.jl/blob/d6c5a721068c7baab3a3d32db3d4c723de0c8cfe/ext/EnzymeExt.jl#L119-L120

vchuravy · 2023-12-18T09:02:11Z

Does this make any sense:

Yeah, you could probably use f and the unwrap the argument types, but identity is as good as any.

michel2323 · 2023-12-22T16:16:10Z

@wmoses Same error. Here is the updated log against latest main branches.
out2.log

michel2323 · 2024-02-06T17:31:15Z

autodiff_deferred_thunk doesn't go through GPUCompiler with CUDA. I commented back in the annotate(args..., but there seems to be more going on. Now it goes through this function, but the execution of forward fails.

src/Enzyme.jl

vchuravy · 2024-02-15T17:23:26Z

src/Enzyme.jl


 # output

 (7.26, 2.2, [3.3])
 ```
 """
-@inline function autodiff_deferred_thunk(::ReverseModeSplit{ReturnPrimal,ReturnShadow,Width,ModifiedBetweenT, RABI}, ::Type{FA}, ::Type{A}, args...) where {FA<:Annotation, A<:Annotation, ReturnPrimal,ReturnShadow,Width,ModifiedBetweenT, RABI<:ABI}
+@inline function autodiff_deferred_thunk(


Should we just delete this function? We could provide a helper that calls tape_type internally, but that seems less helpful.

Yeah I'm fine nuking this

Yeah I'm fine nuking this

Nuked and changed the API. Moving TapeType to second argument.

vchuravy · 2024-02-15T17:24:02Z

src/Enzyme.jl

    AugT = Compiler.AugmentedForwardThunk{Ptr{Cvoid}, FA, A2, TT, Val{width}, Val(ReturnPrimal), TapeType}
    @assert AugT == typeof(nondef[1])
    AdjT = Compiler.AdjointThunk{Ptr{Cvoid}, FA, A2, TT, Val{width}, TapeType}
    @assert AdjT == typeof(nondef[2])
    AugT(primal_ptr), AdjT(adjoint_ptr)
 end

+@inline function autodiff_deferred_thunk(


Maybe move TapeType after FA

autodiff_deferred_thunk is variadic. As long as we keep both methods I think this could lead to some trouble to distinguish between the two in a call. Or maybe not?

What is A in contrast to A2

michel2323 · 2024-02-22T19:44:48Z

Rebased and cleaned up. There are no tests yet for autodiff_deferred_thunk. Should I work on this before merging?

wsmoses · 2024-02-22T19:45:51Z

Yeah tests would be nice

…

On Thu, Feb 22, 2024 at 11:45 AM Michel Schanen ***@***.***> wrote: Rebased and cleaned up. There are no tests yet for autodiff_deferred_thunk. Should I work on this before merging? — Reply to this email directly, view it on GitHub <#1104 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJTUXBHRGDNOMWVEUFO5ZLYU6N3ZAVCNFSM6AAAAAA6HVPW46VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRQGE2DAMBRGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

michel2323 · 2024-02-23T18:23:07Z

Added a test. That test with CuArray fails, though, but I guess that's expected as it's not a kernel but uses broadcast.

michel2323 · 2024-02-26T17:59:55Z

@wsmoses What still needs to be done here?

test/runtests.jl

wsmoses

This looks good to me, except the Julia1.6 errors

@vchuravy any thoughts (maybe from the jit changes?)

wsmoses · 2024-02-29T21:00:05Z

@michel2323 bumping this, once 1.6 is fixed this should be good

michel2323 · 2024-03-04T18:30:32Z

@michel2323 bumping this, once 1.6 is fixed this should be good

@wsmoses Was indeed an easy fix. I merged back main. I hope I resolved the conflicts correctly. Might be worth not rebasing.

michel2323 · 2024-03-04T18:36:42Z

I don't know about the this error after the merge. Can you look into this?

Simple Complex tests: Test Failed at /home/runner/work/Enzyme.jl/Enzyme.jl/test/runtests.jl:354
  Expression: ((autodiff(ReverseHolomorphic, Base.inv, Active, Active(3.0 + 4.0im)))[1])[1] ≈ 0.0112 + 0.0384im
   Evaluated: 0.0112 - 0.038400000000000004im ≈ 0.0112 + 0.0384im

Edit: Seems to be an issue on main and the prebuilt Enzyme_jll.

wsmoses · 2024-03-04T23:40:31Z

LGTM. I can't actually approve this since I'm the one who made the PR. @vchuravy can you give it a once over (in particular the orcv1), approve, and let's merge!

wsmoses · 2024-03-04T23:41:03Z

and yeah @michel2323 that's a "we should jll bump soon, but I haven't gotten to, since at visit weekend, and there's some other complex number stuff that should also land"

michel2323 · 2024-03-04T23:44:25Z

LGTM. I can't actually approve this since I'm the one who made the PR. @vchuravy can you give it a once over (in particular the orcv1), approve, and let's merge!

He wrote the orcv1 stuff. That one change I did was for sure only a typo level mistake.

Edit: Actually, he wrote 95% of the PR.

wsmoses · 2024-03-04T23:45:17Z

Okay I'll just force merge it then.

wsmoses requested a review from vchuravy October 19, 2023 18:53

wsmoses force-pushed the tape_type_compilejob branch from 3ddc4a3 to fb74dc4 Compare November 21, 2023 01:39

michel2323 force-pushed the tape_type_compilejob branch from c5db69c to 57e884e Compare December 15, 2023 20:01

michel2323 force-pushed the tape_type_compilejob branch from 57e884e to 8bf56ea Compare December 21, 2023 20:41

wsmoses force-pushed the tape_type_compilejob branch 4 times, most recently from d4fdc35 to 8beef7a Compare January 24, 2024 22:54

michel2323 force-pushed the tape_type_compilejob branch from a0196f3 to 8083509 Compare January 29, 2024 21:01

vchuravy reviewed Feb 7, 2024

View reviewed changes

src/Enzyme.jl Outdated Show resolved Hide resolved

wsmoses commented Feb 12, 2024

View reviewed changes

src/Enzyme.jl Outdated Show resolved Hide resolved

michel2323 force-pushed the tape_type_compilejob branch from 8701270 to 1fc9b9b Compare February 14, 2024 16:46

wsmoses commented Feb 14, 2024

View reviewed changes

src/Enzyme.jl Outdated Show resolved Hide resolved

vchuravy reviewed Feb 15, 2024

View reviewed changes

wsmoses and others added 5 commits February 22, 2024 12:55

Add tape_type query given a parent compiler job context

1d21e71

Random fixes

fa72fb7

Fix for hash calculation

c1bb0db

fix var scope

1470c0e

Fix type cache type

c3b43b4

michel2323 force-pushed the tape_type_compilejob branch from ada6ed4 to 854b6c5 Compare February 22, 2024 18:55

michel2323 added 2 commits February 22, 2024 13:13

Deal with whitespaces in another PR

e9e9536

Fix

54d00a2

michel2323 added 2 commits February 23, 2024 11:48

Add docs

7e3dec9

Added tests for deferred and deferred thunk

eb50c37

michel2323 requested a review from vchuravy February 26, 2024 17:59

wsmoses commented Feb 26, 2024

View reviewed changes

test/runtests.jl Outdated Show resolved Hide resolved

Simplify test

faebdd1

wsmoses commented Feb 27, 2024

View reviewed changes

autodiff_deferred_thunk API

5ec9aae

michel2323 mentioned this pull request Feb 27, 2024

Add GPU reverse mode to EnzymeExt JuliaGPU/KernelAbstractions.jl#454

Merged

Fix 1.6 error

4711e88

michel2323 force-pushed the tape_type_compilejob branch from d2c78fd to 4711e88 Compare March 4, 2024 17:28

Merge branch 'main' into tape_type_compilejob

d47478b

michel2323 force-pushed the tape_type_compilejob branch from 87079ee to d47478b Compare March 4, 2024 18:26

wsmoses merged commit ead0dc5 into main Mar 4, 2024
17 of 44 checks passed

wsmoses deleted the tape_type_compilejob branch March 4, 2024 23:45

vchuravy mentioned this pull request Mar 5, 2024

Define compiler_job_from_backend #1328

Merged

vchuravy mentioned this pull request Apr 1, 2024

Add tape type mechanisms with parent_job capability #734

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tape_type query given a parent compiler job context #1104

Add tape_type query given a parent compiler job context #1104

wsmoses commented Oct 19, 2023

wsmoses commented Nov 20, 2023

vchuravy commented Nov 20, 2023

codecov-commenter commented Nov 21, 2023 •

edited

Loading

wsmoses commented Dec 5, 2023

michel2323 commented Dec 15, 2023 •

edited

Loading

michel2323 commented Dec 15, 2023

michel2323 commented Dec 15, 2023

vchuravy commented Dec 18, 2023

michel2323 commented Dec 22, 2023

michel2323 commented Feb 6, 2024

vchuravy Feb 15, 2024

wsmoses Feb 26, 2024

michel2323 Feb 27, 2024

vchuravy Feb 15, 2024

michel2323 Feb 22, 2024

wsmoses Feb 26, 2024

michel2323 commented Feb 22, 2024

wsmoses commented Feb 22, 2024 via email

michel2323 commented Feb 23, 2024

michel2323 commented Feb 26, 2024

wsmoses left a comment

wsmoses commented Feb 29, 2024

michel2323 commented Mar 4, 2024 •

edited

Loading

michel2323 commented Mar 4, 2024 •

edited

Loading

wsmoses commented Mar 4, 2024

wsmoses commented Mar 4, 2024

michel2323 commented Mar 4, 2024 •

edited

Loading

wsmoses commented Mar 4, 2024

Add tape_type query given a parent compiler job context #1104

Add tape_type query given a parent compiler job context #1104

Conversation

wsmoses commented Oct 19, 2023

wsmoses commented Nov 20, 2023

vchuravy commented Nov 20, 2023

codecov-commenter commented Nov 21, 2023 • edited Loading

Codecov Report

wsmoses commented Dec 5, 2023

michel2323 commented Dec 15, 2023 • edited Loading

michel2323 commented Dec 15, 2023

michel2323 commented Dec 15, 2023

vchuravy commented Dec 18, 2023

michel2323 commented Dec 22, 2023

michel2323 commented Feb 6, 2024

vchuravy Feb 15, 2024

Choose a reason for hiding this comment

wsmoses Feb 26, 2024

Choose a reason for hiding this comment

michel2323 Feb 27, 2024

Choose a reason for hiding this comment

vchuravy Feb 15, 2024

Choose a reason for hiding this comment

michel2323 Feb 22, 2024

Choose a reason for hiding this comment

wsmoses Feb 26, 2024

Choose a reason for hiding this comment

michel2323 commented Feb 22, 2024

wsmoses commented Feb 22, 2024 via email

michel2323 commented Feb 23, 2024

michel2323 commented Feb 26, 2024

wsmoses left a comment

Choose a reason for hiding this comment

wsmoses commented Feb 29, 2024

michel2323 commented Mar 4, 2024 • edited Loading

michel2323 commented Mar 4, 2024 • edited Loading

wsmoses commented Mar 4, 2024

wsmoses commented Mar 4, 2024

michel2323 commented Mar 4, 2024 • edited Loading

wsmoses commented Mar 4, 2024

codecov-commenter commented Nov 21, 2023 •

edited

Loading

michel2323 commented Dec 15, 2023 •

edited

Loading

michel2323 commented Mar 4, 2024 •

edited

Loading

michel2323 commented Mar 4, 2024 •

edited

Loading

michel2323 commented Mar 4, 2024 •

edited

Loading