Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

typechecking annottees inside a macroTransform crashes the compiler #67

Closed
matanox opened this issue Apr 29, 2015 · 10 comments
Closed

typechecking annottees inside a macroTransform crashes the compiler #67

matanox opened this issue Apr 29, 2015 · 10 comments

Comments

@matanox
Copy link

matanox commented Apr 29, 2015

Hi,

It seems if I typecheck the annottees: c.Expr[Any]* inside a macroTransform implementation, then eventually I crash compilation, whenever one or more case classes are being annotated. As if implying that typechecking has some mutable adverse effect, as the scala compiler seems to ultimately call paradise's org.scalamacros.paradise.typechecker, which crashes with the error copied below.

I type check the annottees as follows, as my macro needs to have type information availabe to it:
val typeCheckedAnnottees = annottees.map(a => c.typecheck(a.tree, silent = false))

I have yet to encounter the same crash eventually happening when the annottee is not a case class (e.g. a regular class or an object), and I wonder whether I really grasp the meaning of type checking inside a macroTransform, whether it is at all safe to do.

This is reproducible in my fork of the example project

May you please advise?

java.lang.AssertionError: assertion failed: 
  module TestPassword#7867 with maybeExpandeeCompanionCompleter for TestPassword#7867
     while compiling: /repos/sbt-example-paradise/core/src/main/scala/b.scala
        during phase: typer
     library version: version 2.11.6
    compiler version: version 2.11.6
  reconstructed args: .......ivy2/cache/org.scala-lang/scala-reflect/jars/scala-reflect-2.11.6.jar:/home/matan/.ivy2/cache/org.slf4j/slf4j-api/jars/slf4j-api-1.7.12.jar -deprecation -Xplugin:/home/matan/.ivy2/cache/org.scalamacros/paradise_2.11.6/jars/paradise_2.11.6-2.1.0-M5.jar -bootclasspath /usr/lib/jvm/java-7-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-7-oracle/jre/lib/rt.jar:/usr/lib/jvm/java-7-oracle/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-7-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-7-oracle/jre/classes:/home/matan/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.11.6.jar

  last tree to typer: term TestPassword
       tree position: <unknown>
              symbol: object TestPassword
   symbol definition: object TestPassword (a ModuleSymbol)
      symbol package: <empty>
       symbol owners: object TestPassword
           call site: method doB in class B1 in package <empty>

<Cannot read source file>
    at scala.tools.nsc.Global.assert(Global.scala:262)
    at org.scalamacros.paradise.typechecker.Namers$Namer$MaybeExpandeeCompleter.completeImpl(Namers.scala:323)
    at org.scalamacros.paradise.typechecker.Namers$Namer$MaybeExpandeeCompleter.completeImpl(Namers.scala:317)
    at scala.tools.nsc.typechecker.Namers$LockingTypeCompleter$class.complete(Namers.scala:1692)
    at org.scalamacros.paradise.typechecker.Namers$Namer$MaybeExpandeeCompleter.complete(Namers.scala:299)
    at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1488)
    at scala.reflect.internal.Symbols$Symbol.initialize(Symbols.scala:1633)
    at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5005)
    at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)
    at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)
    at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)
    at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)
    at scala.tools.nsc.typechecker.Typers$Typer.typedByValueExpr(Typers.scala:5452)
    at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedStat$1(Typers.scala:3046)
    at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$addSynthetics$1$1$$anonfun$apply$34.apply(Typers.scala:3108)
    at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$addSynthetics$1$1$$anonfun$apply$34.apply(Typers.scala:3107)
    at scala.Option$WithFilter.foreach(Option.scala:209)
    at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$addSynthetics$1$1.apply(Typers.scala:3107)
    at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$addSynthetics$1$1.apply(Typers.scala:3106)
    at scala.reflect.internal.Scopes$Scope.foreach(Scopes.scala:373)
    at scala.tools.nsc.typechecker.Typers$Typer.addSynthetics$1(Typers.scala:3106)
    at scala.tools.nsc.typechecker.Typers$Typer.typedStats(Typers.scala:3157)
    at scala.tools.nsc.typechecker.Typers$Typer.typedPackageDef$1(Typers.scala:5012)
    at scala.tools.nsc.typechecker.Typers$Typer.typedMemberDef$1(Typers.scala:5312)
    at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5359)
    at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5396)
    at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5423)
    at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5370)
    at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5374)
    at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5448)
    at scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3.apply(Analyzer.scala:102)
    at scala.tools.nsc.Global$GlobalPhase$$anonfun$applyPhase$1.apply$mcV$sp(Global.scala:441)
    at scala.tools.nsc.Global$GlobalPhase.withCurrentUnit(Global.scala:432)
    at scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:441)
    at scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3$$anonfun$run$1.apply(Analyzer.scala:94)
    at scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3$$anonfun$run$1.apply(Analyzer.scala:93)
    at scala.collection.Iterator$class.foreach(Iterator.scala:750)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1202)
    at scala.tools.nsc.typechecker.Analyzer$typerFactory$$anon$3.run(Analyzer.scala:93)
    at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1500)
    at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1487)
    at scala.tools.nsc.Global$Run.compileSources(Global.scala:1482)
    at scala.tools.nsc.Global$Run.compile(Global.scala:1580)
    at xsbt.CachedCompiler0.run(CompilerInterface.scala:116)
    at xsbt.CachedCompiler0.run(CompilerInterface.scala:95)
    at xsbt.CompilerInterface.run(CompilerInterface.scala:26)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:101)
    at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:47)
    at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:41)
    at sbt.compiler.MixedAnalyzingCompiler$$anonfun$compileScala$1$1.apply$mcV$sp(MixedAnalyzingCompiler.scala:51)
    at sbt.compiler.MixedAnalyzingCompiler$$anonfun$compileScala$1$1.apply(MixedAnalyzingCompiler.scala:51)
    at sbt.compiler.MixedAnalyzingCompiler$$anonfun$compileScala$1$1.apply(MixedAnalyzingCompiler.scala:51)
    at sbt.compiler.MixedAnalyzingCompiler.timed(MixedAnalyzingCompiler.scala:75)
    at sbt.compiler.MixedAnalyzingCompiler.compileScala$1(MixedAnalyzingCompiler.scala:50)
    at sbt.compiler.MixedAnalyzingCompiler.compile(MixedAnalyzingCompiler.scala:65)
    at sbt.compiler.IC$$anonfun$compileInternal$1.apply(IncrementalCompiler.scala:160)
    at sbt.compiler.IC$$anonfun$compileInternal$1.apply(IncrementalCompiler.scala:160)
    at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:66)
    at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:64)
    at sbt.inc.IncrementalCommon.cycle(IncrementalCommon.scala:31)
    at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:62)
    at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:61)
    at sbt.inc.Incremental$.manageClassfiles(Incremental.scala:89)
    at sbt.inc.Incremental$.compile(Incremental.scala:61)
    at sbt.inc.IncrementalCompile$.apply(Compile.scala:54)
    at sbt.compiler.IC$.compileInternal(IncrementalCompiler.scala:160)
    at sbt.compiler.IC$.incrementalCompile(IncrementalCompiler.scala:138)
    at sbt.Compiler$.compile(Compiler.scala:128)
    at sbt.Compiler$.compile(Compiler.scala:114)
    at sbt.Defaults$.sbt$Defaults$$compileIncrementalTaskImpl(Defaults.scala:806)
    at sbt.Defaults$$anonfun$compileIncrementalTask$1.apply(Defaults.scala:797)
    at sbt.Defaults$$anonfun$compileIncrementalTask$1.apply(Defaults.scala:795)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:235)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
[error] (core/compile:compileIncremental) java.lang.AssertionError: assertion failed: 
[error]   module TestPassword#7867 with maybeExpandeeCompanionCompleter for TestPassword#7867
[error]      while compiling: /repos/sbt-example-paradise/core/src/main/scala/b.scala
[error]         during phase: typer
[error]      library version: version 2.11.6
[error]     compiler version: version 2.11.6
[error]   reconstructed args: -classpath ....../.ivy2/cache/org.scalamacros/paradise_2.11.6/jars/paradise_2.11.6-2.1.0-M5.jar -bootclasspath /usr/lib/jvm/java-7-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-7-oracle/jre/lib/rt.jar:/usr/lib/jvm/java-7-oracle/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-7-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-7-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-7-oracle/jre/classes:/home/matan/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.11.6.jar
[error] 
[error]   last tree to typer: term TestPassword
[error]        tree position: <unknown>
[error]               symbol: object TestPassword
[error]    symbol definition: object TestPassword (a ModuleSymbol)
[error]       symbol package: <empty>
[error]        symbol owners: object TestPassword
[error]            call site: method doB in class B1 in package <empty>
[error] 
[error] <Cannot read source file>

I get this with every version of macro paradise I have tried for scala 2.11.

@matanox matanox changed the title typechecking annottees inside a macroTransform crashes annotated case class typechecking annottees inside a macroTransform crashes the compiler Apr 29, 2015
@xeno-by
Copy link
Member

xeno-by commented Apr 30, 2015

Unfortunately, c.typecheck inside macro annotations is pretty unreliable at our current level of technology. It often works, but sometimes it produces the unpleasant effects similar to what you've described.

Therefore, a typical approach to typechecking is (if possible) trying to move it to def macros (typecheck there has a much higher chance to work correctly). So, a macro annotation could expand into a bunch of calls to def macros, who, in their turn, would call c.typecheck and the collect necessary info. Do you think that this pattern would be implementable for your scenario?

@matanox
Copy link
Author

matanox commented Apr 30, 2015

Well on the face of it, I shouldn't really care if the macro annotation would expand to a more vanilla def macro, that somehow has the same level of access to the annotated entity's AST, and to type-checking it. Is it really that simple or am I missing something here? I wonder where does the difference arise from - do def macros expand at a later stage of compilation? I'd like to understand more before trying that....

@xeno-by
Copy link
Member

xeno-by commented May 8, 2015

c.typecheck relies on having internal compiler data structures called typer contexts to be set up properly. Among other things, typer contexts have the information about names in scope, so it's really fundamental to them to be in order to have typechecking working.

Macro annotations expand at a very sensitive point during typechecking, at which some of the contexts are already initialized and some contexts (notably, the very enclosing context of the annotated member being expanded) are in progress. Therefore, c.typecheck might lead to surprising and unexpected results.

Def macros expand when all the relevant contexts are already set up, so c.typecheck is much more stable at that point.

Therefore, if your macro annotation expands, say, a class into the same class with all its methods wrapped to calls to def macros, then calling c.typecheck within those macros will have a much higher chance of working correctly.

@matanox
Copy link
Author

matanox commented May 9, 2015

Thanks,

In my naive testing, everything works as intended in my macro annotations -
they produced all data they were intended to produce. Only the compilation
beyond the stage where they recorded all necessary information - failed.

Is there grounds to believe this would typically be the case - with scala
2.11 - that compilation as a whole will fail only after the annotation
macros have been successfully expanded? might it actually be the case that
type checking in the annotation macros only hampers later stages of
compilation? as per your last post - in my scenario only top level objects
are being annotated (i.e. an unested and unwrapped class or object in a
source file), so I'm not sure whether the enclosing context of the
annotated object really matters or not.

This can be fine. It is hopefully very simple to rerun compilation without
the annotations for the real compilation, e.g. by preprocessing the files
to remove the annotations and passing the result to a compilation task of
its own. So one could run two separate compilation tasks in sbt, one with
the annotations and one without, thus getting the job done in my scenario.

Thanks,
Matan

On Fri, May 8, 2015 at 2:27 PM, Eugene Burmako [email protected]
wrote:

c.typecheck relies on having internal compiler data structures called
typer contexts to be set up properly. Among other things, typer contexts
have the information about names in scope, so it's really fundamental to
them to be in order to have typechecking working.

Macro annotations expand at a very sensitive point during typechecking, at
which some of the contexts are already initialized and some contexts
(notably, the very enclosing context of the annotated member being
expanded) are in progress. Therefore, c.typecheck might lead to
surprising and unexpected results.

Def macros expand when all the relevant contexts are already set up, so
c.typecheck is much more stable at that point.

Therefore, if your macro annotation expands, say, a class into the same
class with all its methods wrapped to calls to def macros, then calling
c.typecheck within those macros will have a much higher chance of working
correctly.


Reply to this email directly or view it on GitHub
#67 (comment)
.

@xeno-by
Copy link
Member

xeno-by commented May 10, 2015

Well, since scalac uses a highly mutable architecture to implement signature computations and typechecking, c.typecheck may mutate some state at the point when it's not expected to, leading to spooky action at distance, starting from a few stack frames to even a few compiler phases later.

As for you idea, it would be really hard to separate macro annotation expansion from this particular crash, since the crash happens at typer, and typer is the phase when macro annotations expand (well, the reality is more complicated, but this approximation is adequate).

@xeno-by
Copy link
Member

xeno-by commented May 10, 2015

What I suggested in the email, is having

@macroAnnot
class C {
  def m1 = ...
  def m2 = ...
}

Expand into

class C {
  def m1 = macroDef { ... }
  def m2 = macroDef { ... }
}

Then, expansions of macroDef would be able to use c.typecheck without being afraid to corrupt internal typechecker state.

@matanox
Copy link
Author

matanox commented May 10, 2015

Okay, thought so. I'll try that when I come back to it, hopefully I can provide the AST from @macroAnnot as the content of the curly braces of macroDef. Thanks for your courtesy and the clarification...

@xeno-by
Copy link
Member

xeno-by commented May 11, 2015

Why do @macroAnnot and macroDef macros have to communicate? macroDef already has a subset of the AST automatically (its argument).

@matanox
Copy link
Author

matanox commented Jun 14, 2015

Okay I think I got it, coming back to this, and am working on this approach.

@xeno-by
Copy link
Member

xeno-by commented Nov 19, 2015

I've merged this issue into #75 and will close this issue now. If this is a blocker for you, please let me know, and we'll think of something.

@xeno-by xeno-by closed this as completed Nov 19, 2015
SethTisue pushed a commit to scalacommunitybuild/paradise that referenced this issue Nov 21, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants