Migrating from untyped macros

[stanch]

Hi,

Do you have any plans for migration from now deprecated untyped macros? I am not still sure if this is going to work, but maybe one could provide implicit views from Monad[A]/Functor[A] to A (e.g. with a @compileTimeOnly annotation), so that the code typechecks, and the macro would just disregard them, replacing with maps or binds. What do you think?

Nick

[aztek]

Hi,

It would certainly be great to get rid of untyped macros, but I can't think of how to do it right now.

To make your solution work, we would need to have implicit view for every instance of monad/functor separately, e.g. Option[A] => A (with the way things currently are) and somehow make it available only within context/workflow block so it won't mess with other code. Composition of functors/idioms won't work anymore (List[Option[A]] => A can't be resolved from List[A] => A and Option[A] => A). Also, powerful implicits like those might interfere with other implicits. For instance, Some("foo") + "bar" is valid Scala because of Predef.StringAdd implicit class and Option[String] => String simply wouldn't be called.

I'll keep trying to come up with something and, of course, all the ideas are highly appreciated.

[stanch]

To make your solution work, we would need to have implicit view for every instance of monad/functor separately, e.g. Option[A] => A (with the way things currently are) and somehow make it available only within context/workflow block so it won't mess with other code.

Well, providing the views for every functor is not that high a price to pay :) As for the messing, I believe @compileTimeOnly should keep them rather sane.

Composition of functors/idioms won't work anymore (List[Option[A]] => A can't be resolved from List[A] => A and Option[A] => A).

That’s true. Still can be done with an “explicit” implicit though, which is of course ugly.

For instance, Some("foo") + "bar" is valid Scala

This is an interesting matter. How do you know anyway—with the current implementation—that it’s not point(Some("foo").toString + "bar")? However, if part of the code typechecks like this and then fails typechecking somewhere midway because of wrong inferred types, things will break.

[stanch]

By the way, I just realized that the new macro annotations expand before type-checking! So conceptually a ported version could look like so:

@workflow[Option[_]] val x = Some(3) + 5

However, I just tried to pass the type argument through and it didn’t work quite well. It’s also unclear how to pass normal arguments, if we want @wokflow(zipList). @xeno-by, any thoughts on this?

[aztek]

How do you know anyway—with the current implementation—that it’s not point(Some("foo").toString + "bar")?

That's how rewriting works — we start with the most nested subexpression. Basically, the whole expression never gets to be typechecked as is (unless we walked through all of its subexpressions and left them untouched). In this case, Some("foo") is typechecked before Some("foo") + "bar", and rewritten with synthetic String argument to map.

[xeno-by]

What exactly didn't work?

[stanch]

@xeno-by

class Workflow[A] extends StaticAnnotation {
  def macroTransform(annottees: Any*) = macro WorkflowMacros.macroTransform[A]
}

object WorkflowMacros {
  def macroTransform[A: c.WeakTypeTag](c: MacroContext)(annottees: c.Expr[Any]*): c.Expr[Any] = {
    import c.universe._
    println(weakTypeOf[A])
    c.Expr[Any](Block(DefDef(Modifiers(), newTermName("x"), List(), List(), TypeTree(), Apply(Select(Literal(Constant(3)), newTermName("$plus")), List(Literal(Constant("asd"))))), Literal(Constant(()))))
  }
}

scala> import foo.Workflow
import foo.Workflow

scala> @Workflow[Int] val x = 4
x: Int = 4

So the annotation is not doing anything at all. Without the type arg it works fine:

class Workflow extends StaticAnnotation {
  def macroTransform(annottees: Any*) = macro WorkflowMacros.macroTransform
}

object WorkflowMacros {
  def macroTransform(c: MacroContext)(annottees: c.Expr[Any]*): c.Expr[Any] = {
    import c.universe._
    //println(weakTypeOf[A])
    c.Expr[Any](Block(DefDef(Modifiers(), newTermName("x"), List(), List(), TypeTree(), Apply(Select(Literal(Constant(3)), newTermName("$plus")), List(Literal(Constant("asd"))))), Literal(Constant(()))))
  }
}

scala>   import foo.Workflow
import foo.Workflow

scala> @Workflow val x = 4
x: String = 3asd

Another question is if we have a macro annotation with arguments (class Workflow(a: Int) extends StaticAnnotation), how do we access them in impls?

[aztek]

Another issue is that we need higher-kinded type as the argument. I seem to remember, that c.WeakTypeTag didn't work for that.

[stanch]

That's how rewriting works

I understand that, but what happens in this example?

workflow[Option] {
  Some(Success(42)) getOrElse "foobar"
}

is it option.map(x$1 ⇒ x$1.getOrElse("foobar"))(Some(Success(42)) or option.point(Some(Success(42)).getOrElse("foobar")) ? Maybe there’s no issue and I just need to sleep for a while :)

[aztek]

The first one. Method invocation is represented somewhat like Apply(Select('Some(Success(42))', 'getOrElse'), List("foobar")), so again, we start with the most nested subexpression: 42, then Success(42), then Some(Success(42)), that gets rewritten as synthetic x$1 : Success[Int] and getOrElse method will be taken from Try class, not Option.

Whereas

workflow[Option] {
  Some(Failure(new Exception)) getOrElse "foobar"
}

will produce Some("foobar").

[xeno-by]

@stanch

a) That's a bug: scalamacros/paradise#2. Probably would be easy to fix, but I don't have much time for that. If you need this fix badly, I can carry it out tomorrow. Otherwise, let's keep it on-demand.

2. c.prefix

[stanch]

@aztek It makes sense! Thanks for the explanation.

@xeno-by re 2) Lol now I remember that you suggested the same when we submitted https://issues.scala-lang.org/browse/SI-7736 :) So I’ll also try to get the type args with it and see how it goes.

Upd: even if I just put class Workflow[A] extends StaticAnnotation and never mention A again, the macro still does not expand.

[stanch]

Check this out!
stanch@506fcc7#L0R49
https://github.com/stanch/scala-workflow/blob/e0325cc641fd4ed952337a0aec797d2777a0247a/project/Build.scala

scala> @workflow(option) val x = Some("f") + Some("o")*2 + Some("bar")
x: Option[String] = Some(foobar)

However:

scala> @workflow(option) val x = Some(2) + 3
error: ambiguous reference to overloaded definition,
both method + in class Int of type (x: Char)Int
and  method + in class Int of type (x: Short)Int
match expected type ?
<console>:10: error: type mismatch;
 found   : Int(3)
 required: String
       @workflow(option) val x = Some(2) + 3
                                           ^

I am curious if this is an issue specific to how the tree is parsed/checked, e.g. https://github.com/stanch/scala-workflow/blob/506fcc75196406ce59ea5781630ea673768f8b50/core/src/main/scala/scala/workflow/package.scala#L195

[aztek]

Oh, cool!

I'm not sure about the error. Looks like the whole expression is typechecked at some point (you'll get the same error message without the annotation). Generated code is correct, so rewriter/typechecker worked well.

[stanch]

I think the first error happens somewhere in rewrite. The second one is just due to the failed annotation expansion. I will look into this today. We can also try to revive the context and $ duo in this fashion:

@context(option)
def foo(bar: Int) = {
  @$ val a = 4 // a = Some(4)
  a
}

Of course the annotations are not as good-looking as functions, and you can put them into fewer places. But on the other hand, I believe we can do

class async extends workflow(future)
@async val x = 5 // x = Future(5)

which is not bad!

Upd: The shown implementation of @async does not suffice because of how c.prefix works. I’m looking into it.

[stanch]

@xeno-by, after some investigation this looks very suspicious.

println(util.Try(c.typeCheck(q"4+")))
...
error: ambiguous reference to overloaded definition,
both method + in class Int of type (x: Char)Int
and  method + in class Int of type (x: Short)Int
match expected type ?
Success(4.<$plus: error>)

The Success part tells us that no exception was thrown. Could you check this? c.typeCheck(q"val x: String = 5") throws just fine.

[xeno-by]

Most likely a bug. Would be great if you could submit it. As a workaround,
after getting a success you could additionally check whether the resulting
tree is erroneous.

On Sunday, 1 September 2013, stanch [email protected] wrote:

@xeno-by, after some investigation this looks very suspicious.

println(s"typechecking ${scope wrapping tree.duplicate}")
println(util.Try(c.typeCheck(scope wrapping tree.duplicate)))
...
typechecking {
val arg$1: Int = $qmark$qmark$qmark;
arg$1.$plus
}
error: ambiguous reference to overloaded definition,
both method + in class Int of type (x: Char)Int
and method + in class Int of type (x: Short)Int
match expected type ?
Success({
private[this] val arg$1: Int = scala.this.Predef.???;
$iw.this.arg$1.<$plus: error>
})

The Success part tells us that no exception was thrown. Could you check
this? c.typeCheck(q"val x: String = 5") throws just fine.

—
Reply to this email directly or view it on GitHub.<
https://github.com/notifications/beacon/nHKVk3PF9lWIv8gP84yu9cLXl26slIG0NQ4V3nxnbsMOUVYPkqBZaUu54kC1tVcz.gif

<
http://sgmail.github.com/wf/open?upn=I9-2BAxytMHO4kqVosQHHNh2JYlg9HJJ3bQyQWssaMZjW-2BrCy5-2BzP9k2ApW0WnvSEnMs-2FjnZEoMoUoiQCNyE8T6tOVbw0ph7YBS2R-2FydwKqbRfJ75zEGpE-2BOVizZNKhkAQi2RliZzvreIn9QdzNF-2F6xsvUfjSpr5E89lwHCQHjWplwWDayYX3Zlo0eFx5SPHaZHuca4yeX4grLa7VJG7VXcA-3D-3D

[aztek]

Is that it https://issues.scala-lang.org/browse/SI-7461 ?

[stanch]

Is that it https://issues.scala-lang.org/browse/SI-7461 ?

Yes, with a difference that compilation does not fail.

@xeno-by should I submit a new one, or we reopen this one?

[xeno-by]

Dunno yet. Let me take a look first

[xeno-by]

@stanch The example from SI-7461 works for me, so maybe there's a problem with the particular tree that you're emitting. Could you provide a minimized reproduction and reopen the bug with it?

[stanch]

I know you are both watching that bug, but just in case—the reproduction is here https://github.com/stanch/paradise-bug.

And now to something completely different. I realized this morning that by macro-annotating the enclosing method and defining workflow, context and $ as stubs, we would probably be able to keep the original API untouched. The macro would collect all stubs and replace them with rewritten trees. @aztek what do you think? In general, do you plan to merge my port here or should we keep it as a separate repo with new docs?

[aztek]

@stanch Could ypu please put up some examples of macro-annotated workflows? Or, even better, revise some of the examples in the current specs.

I don't think anybody actually uses scala-workflow, so preserving API (even as simple as this one) shouldn't be a concern.

I don't want to abandon the current implementation completely, so I'd rather have two branches here — one with release version and another with old untyped macros. If you get anything useful with annotations, it's fine with me to merge it here. There're parts of the code independent to macro interface (like instances and composition classes, that I'll push soon), that should be easy to sync.

@xeno-by Anyway, do you think macro annotations will make their way to release? It's just it looks like we would switch untyped macros implementation with another untyped-at-heart implementation. Is it possible, that macro annotations will be eventually discarded for the same reasons untyped macros were?

[stanch]

I will put the examples in readme.md in my fork as soon as [1]—and hopefully [2]—is fixed. As for the usage, well, besides 90 stars that you already have I am actually planning to use it in https://github.com/stanch/macroid and dependent projects. Sometimes I even think it should be a SIP! workflow is way ahead of for, effectfully and F# computation expressions :)

I’ll be happy to submit a pull request with something useful, probably having a separate branch is indeed the best solution.

[1] https://issues.scala-lang.org/browse/SI-7461
[2] scalamacros/paradise#2

[xeno-by]

1 can be worked around, can't it?

[stanch]

@xeno-by Maybe it can. To maintain the code here [1], I need a way to get the error description from the erroneous tree. I did a quick search through auto-completion with no success...

[1] https://github.com/stanch/scala-workflow/blob/master/core/src/main/scala/scala/workflow/package.scala#L204

[xeno-by]

Well, that's impossible from what I know. I'll try to see what can be done this weekend.

[aztek]

@stanch To clear things up about the typeCheck(tree, scope) function:

• The purpose of the function is to decide whether or not tree is lifted to a current workflow (with bindings of local variables, forming a scope).
• It can basically result in three values: Success(<type>) (so later we decide if <type> corresponds to the type of the workflow), Success(EmptyTree) (when we know for a fact, that tree is not lifted) and Failure(e) (when there's a type error inside the lifted code, e.g. $(Some("foo") * Some("bar")); no error will be produced if the lifted code is correct).
• Peeking at fragments of error messages is a hack originating from all typechecking exceptions being of the same class.
• There're a few legitmate errors that we handle: typechecking uncurried functions ("follow this method with _'"), overloaded functions ("ambiguous reference"), package names ("package ... is not a value") and object constructors ("missing arguments for constructor", "too many arguments for constructor"). In all those cases the perfect strategy would be to perform a deeper analysis of the code to get the actual type, but so far it is assumed that it won't correspond to workflow type, so EmptyTree is returned.

[stanch]

@aztek Thanks! I kinda understood this, but your explanation makes it more clear.
I am working on my idea with stubs and would like to start a discussion on the future API.

The current API is:

• workflow
• context + $

It seems that for the future API we have the following options:

• Workflow annotations on ValDefs and possibly DefDefs

  @workflow(option) val x = Some(1) + Some(2) // val x = Some(3)

  This one seems quite useful, especially if we provide shortcuts for common things such as Option, Try, etc. The meaning is rather explicit, which is good.

• Method transformations that allow to use old API

  @enableWorkflow
  def x(a: Int, b: Int) = {
      val y = context[Option] { $(Some(2) + Some(3)) }
      val z = workflow[Option] { Some(4) + Some(5) }
  }

  As I mentioned before, this is based on @compileTimeOnly stubs.

• Same as above, but also providing default context

  @workflowContext[Option]
  def x(a: Int, b: Int) = {
      val y = $(Some(2) + Some(8)) // uses Option
      val z = workflow[Try] { 3 / 0 }
  }

Personally I think keeping everything would be a mess. So I suggest to select the most reasonable subset of the following (usage-wise):

• @workflow
• @enableWorkflow + workflow
• @enableWorkflow + context + $
• @workflowContext + $
• @workflowContext + workflow (i.e. @workflowContext also acts as @enableWorkflow)
• @workflowContext + context + $ (ditto)

I am leaning towards @workflow (for both ValDefs and DefDefs) and @workflowContext + $. This is more of a gut feeling, but the reasons are probably 1) @workflow and workflow is too much; 2) I doubt there will be two different contexts per average function—if there are, one can always use @workflow; 3) this keeps focus on annotations, since we have to deal with them anyway.

Please let me know what you guys think.

[stanch]

@xeno-by I just found a rather upsetting issue:

val x = Some(3)
@workflow(option) val y = "asd" + x

fails during annotation expansion with not found: value x. Do you have any ideas on how to overcome this?

[aztek]

Sorry for the confusion, by "you" I meant the programmer. My initial question was if it is possible not to expand both companions simultaneously when one of them is macro-annotated, but rather stick to moving expansion point. For this snippet a compilation error might be too harsh, because @foo macro might not affect object X at all and the whole code is safe. But than again, we can't determine that in general case, so compilation error it is. The error message could say that unannotated companion should precede annotated one. This also implies that companions can't be both macro-annotated at the same time, but that is probably an extremely rare use case anyway.

[stanch]

Sorry, I’m late to the party, but here’s a random thought: we could make a distinction between “whitebox” and “blackbox” macro-annotations. “blackbox” ones should promise that they do not mess with the declaration names and do not add or remove declarations. Now, everything should be visible for the typechecker, however, referencing whitebox-annotated stuff should produce an error. The tricky thing is of course how to restrict blackbox behavior without having to expand annotations. To the problems:

• 1b seems fine, that’s a price to pay for forward-referencing;
• 1c is gone;
• 2d is now not that bad, since no other annotation will be able to reference class X or object X.

2c is tough though :)

[stanch]

Also I think forward-referencing inside blocks/templates is bad and those who do it should feel bad!

[xeno-by]

Well, open recursion is what you get when you sign up for OOP :)

[xeno-by]

Can blackbox annotations change signatures of the annottees without changing their names? Could you also give an example of a code snippet that "references whitebox-annotated stuff" and an error that it produces?

[stanch]

1. Yes, they can. In case of ClassDefs and ModuleDefs, this would mean changing the type signatures of some of the members. The original type of the annottee, however, should not be available without performing the expansion. Thus, typechecking a blackbox-annotated definition could trigger a chain of expansions of fellow annottees, but they all would be of a blackbox kind (see 2), so what can go wrong :)
2. Consider

  @black val x = 8 + "Y.z"
  @white object Y { val z: Int = 4 }

where @black tries to eval "Y.z" and consequently typecheks Y.z. At this point, the Context should produce a compile-time error saying Illegal reference to a whitebox-annotated member “Y” in current scope.

[stanch]

What about a compiler plugin then? :) (I am no expert on these)

[xeno-by]

What about a compiler plugin?

[stanch]

@xeno-by I meant doing scala-workflow as one.

[xeno-by]

Without dirty hacks, compiler plugins can't change how typer works (with dirty hacks that's going to be roughly what an annotation-based solution is, I think). Analyzer plugins can do that, but they can only postprocess typechecking results, not change typechecking completely, like we need to do here.

[b-studios]

Hey everyone,
I just wanted to ask whether there is some progress regarding the scoping of the typeCheck method. I ran into the following usecase:

I want to analyse the type of a wrapped class (using the pimp my library pattern) in order to generate some methods.

implicit class FooWrapper(@myMacro self: Foo) {}

Currently I use the c.typeCheck(q"(null.asInstanceOf[$tpt])").tpe trick to get hands on the Type of the wrapped class, but this does neither work with Foo being a type alias, nor type-parametrized.

[xeno-by]

@b-studios So far there hasn't been much progress unfortunately - this is quite a tricky issue. Could you elaborate on your use case (a standalone github project would be ideal), so that we could possibly come up with a workaround?

[b-studios]

@xeno-by The usecase in short: I use the Type of the annotated parameter (Foo in the above example) to search for Java idiomatic getter and setter methods (getBar, setBar) in order to generate scala idiomatic ones (bar, bar_=).
I also published the code on github. In this test file line 81ff. are the examples which are currently not working. The type of the wrapped class is being resolved using Utils.toType.

Thanks alot

[xeno-by]

@b-studios Thanks for taking time to publish the reproduction! I'll try to take a look today/tomorrow.

[b-studios]

@xeno-by In order to save you some time I extracted the problem inside of a new branch. This way it is only two short files to take a look at: macrodefinition and usage

[xeno-by]

@b-studios The first problem is indeed scalamacros/paradise#14, and you'll have to reformulate the affected code at the moment. How much of a blocker is that to you?

The second problem, I think, could be successfully handled within the current framework. How about crafting a more elaborate tree for c.typeCheck, something like { class Dummy[T] { type Dummy123 = $tpt }; () }?

[xeno-by]

Also Happy New Year for everyone checking out this thread right now :)

[b-studios]

@xeno-by The type alias issue is not so much of a problem right now, since I currently have influence on the client code.

Regarding the type parameter problem: Great idea to bind the missing parameter in the crafted type tree. I will try to use this approach, but I fear it is only applicable if the binding instance of Tis known.

[xeno-by]

@b-studios Could you elaborate on potential problems with the proposed approach for handling type parameters?

[b-studios]

@xeno-by In the example the macro is applied in the context of the class definition where the type parameters are bound. This way one could query the type parameters and manually craft a tree - binding those type parameters in order to typeCheck the annotated type constructor (As you described above).

If the type arguments of the type constructor are bound outside of the annotated class definition it might be difficult to find the binding occurrence (one would have to reimplement the lookup of types). Please correct me, I just started using scala macros and I don't know how such a lookup could be implemented. I'm happy if I am just wrong with this assumption.

Since I perform the typecheck to retreive the Type of a tree, the above solution does not work anyway because it always returns Unit. So I came up with a dirty solution to be evaluated in the next days:

If the type tree is an AppliedTypeTree, typecheck null.isInstanceOf[$tpt[..$wildcards]] with the appropriate number of wildcards. This way I prevent access to the type variables and the "not found: type T" typecheck error.

[xeno-by]

@b-studios Oh wow, this is something that I didn't expect:

class X[T <: String] {
  class test[T <: Int] {
    implicit class XWrapper(@accessors self: P[T]) {}
  }
}

Prints members of String, not Int. Yes, that's a problem - I'll try to fix it.

[xeno-by]

@b-studios Yes, the type of the returned tree will be Unit, but if you inspect the types of its inner trees, then you'll be able to get to the type of Dummy123.

[xeno-by]

@b-studios I have fixed the problem with typecheck not seeing type parameters declared in enclosing classes. In the morning (in ~8-10 hours), if all the builds are green, I'll publish 2.0.0-M2 with the fix.

[xeno-by]

@b-studios 2.0.0-M2 is now published for 2.10.3 and 2.11.0-M7. Please let me know whether it solves the problem with type parameters.

[b-studios]

@xeno-by Sorry for the late message. Thank you alot! Your fix actually works perfectly for member annotations like

trait Foo[T] {
  @annotation def foo: T
}

I guess there is a reason for the constructor argument annotations not to work?

class Foo[T](@annotation foo: Bar[T])
// scala.reflect.macros.TypecheckException: not found: type T

but if you inspect the types of its inner trees, then you'll be able to get to the type of Dummy123

I tried this, but extracting Dummy123 from the results of typeCheck and asking for the type still yielded null. I have to investigate a little more, but sadly won't find the time in the next few days.

[xeno-by]

1. Annotations on type and value parameters of classes and methods expand the enclosing class/method, not just the member itself. Therefore T doesn't exist yet when the ctor param annotation expands. However you could grab it from the class being expanded and wrap the dummy as before.

2. Here's what I had in mind wrt Dummy123: https://gist.github.com/xeno-by/8255893

[b-studios]

@xeno-by Ad 2) I was using dummy123.tpe on the typechecked tree which always yielded <notype>. Switching to dummy123.symbol.typeSignature (as you did in your gist) works. Thanks a lot.