Consider adding dynamic/special variables to Hy

[gilch]

Lisp originally used dynamic variables, instead of the lexical variables used in Python. They're still the norm in Emacs Lisp, and useful enough sometimes that they're still available optionally in Common Lisp (as "special variables"). For those of you not familiar with Elisp or Common Lisp, they're basically Clojure's thread-local vars.

I also think that a dynamic let would be much easier to implement. The dynamic version doesn't need closures, and could be compiled into a with statement. After purging the broken lexical let from Hy hylang/hy#1056, we could replace it with a dynamic one that has the same semantics as Elisp's let (or Clojure's binding form).

[tuturto]

This sounds intriguing and useful. Recently I had to hack together something that sounds like this, but it's specifically only for functions and requires extra step for calling them: (call foo bar) instead of just (foo bar).

But I'm not completely sure if I understand (or even can think of) all possible cases for this. Variables with dynamic scope would only be available inside a let form that referes to them? Or would they be available somewhere else too? Could one define a function with defn, bind it to dynamically scoped variable and then call it later during program execution? How would the definition part and calling part look like in Hy code? Any idea (rough sketch is enough) what the resulting Python code would look like?

[gilch]

First, a quick demonstration of dynamic variables in Emacs Lisp, so we're on the same page:

ELISP> (defun greet ()
         (format "Hello, %s!" the-name))
greet
ELISP> (defvar the-name "World")
the-name
ELISP> (greet)

"Hello, World!"

nil
ELISP> (defun greet-alice ()
         (let ((the-name "Alice"))
           (greet)))
greet-alice
ELISP> (greet-alice)

"Hello, Alice!"

nil

This is not just a global, you can shadow an earlier binding with a later one, and it returns to its previous value:

ELISP> (defun greet-multiple ()
         (greet)
         (greet-alice)
         (let ((the-name "Bob"))
           (greet))
         (greet))

greet-multiple
ELISP> (greet-multiple)

"Hello, World!"

"Hello, Alice!"

"Hello, Bob!"

"Hello, World!"

nil
ELISP> (let ((the-name "Everyone"))
         (greet-multiple))

"Hello, Everyone!"

"Hello, Alice!"

"Hello, Bob!"

"Hello, Everyone!"

nil

How do we do this in Python? A naiive translation wouldn't work because Python doesn't have dynamic variables. We have to emulate them using Python's lexical variables. Here's a rough proof of concept:

from contextlib import contextmanager


_sentinel = object()
class DefDynamic:
    def __init__(self, value=None):
        self.bindings = [value]

    def __call__(self, value=_sentinel):
        if value is _sentinel:
            return self.bindings[-1]
        @contextmanager
        def manager():
            self.bindings.append(value)
            yield
            self.bindings.pop()
        return manager()

It's basically a stack object that plays nice with with. Now we can do something similar in Python:

>>> THE_NAME = DefDynamic("World")
>>> def greet():
    print("Hello, %s!" % THE_NAME())


>>> greet()
Hello, World!
>>> def greet_alice():
    with THE_NAME('Alice'):
        greet()


>>> greet_alice()
Hello, Alice!
>>> def greet_multiple():
    greet()
    greet_alice()
    with THE_NAME("Bob"):
        greet()
    greet()


>>> greet_multiple()
Hello, World!
Hello, Alice!
Hello, Bob!
Hello, World!
>>> with THE_NAME("Everyone"):
    greet_multiple()


Hello, Everyone!
Hello, Alice!
Hello, Bob!
Hello, Everyone!

Pretty good, but this version has one glaring problem:

ELISP> the-name
"World"

>>> THE_NAME
<__main__.DefDynamic object at 0x0000000003E3AF28>
>>> THE_NAME()
'World'

You have to call them to get the value! I don't think this is possible to fix in raw Python. But Hy is not quite Python. We could probably automate the call if we implemented symbol macros. But here's another option:

import builtins
class DynamicDict(dict):
    def __init__(self, *args, **kwargs):
        super().__init__(*args,**kwargs)
        self['_dynamic'] = {}

    def __getitem__(self, key):
        try:
            item = super().__getitem__(key)
            if isinstance(item,DefDynamic):
                item = item()
        except KeyError:
            item = builtins.__dict__[key]
        return item

    def __setitem__(self, key, value):
        super().__setitem__(key, value)
        if isinstance(value,DefDynamic):
            self['_dynamic'][key] = value

scope = DynamicDict()
scope.update(globals())
exec('''
Y = DefDynamic("Foo")
print("Y:", Y) # look Ma, no call!
def dynamic_greet():
    print("Hi", Y)  # Not here either
dynamic_greet()
def dynamic_greet_alice():
    with _dynamic['Y']("Alice"):
        dynamic_greet()
dynamic_greet_alice()
''', scope)

Y: Foo
Hi Foo
Hi Alice

How is this possible? I've customized dict to intercept reads to the global dict to call the function for us. It's not unusual to do this kind of thing with descriptors in Python classes, but you can't monkey patch the globals dict. I had to exec a string instead. This works because you can pass exec an arbitrary globals dict. So it doesn't work in raw Python, but Hy compiles to AST! There might be a way for Hy to use a customized globals dict like the above.

Despite mine epistle, I still consider this a rough sketch. There are other details that must be dealt with.

Emacs is single-threaded. The simple stack objects will get tangled if Python is threaded. This is easy to fix. We can take advice from Clojure and give each thread its own stack. This could be a dict with thread keys. (Maybe a weak dict to prevent leaks.)

There's also the question of modules. You may need to set a dynamic in a different namespace. How do you import these properly? Clojure's namespaces are a clue. How do you use a Hy dynamic if you import it in Python? It can't be quite as pretty, but as demonstrated, you can use calls.

I only tested it in CPython3. It should work in other implementations though.

There's also the question of what should happen when you yield from inside the with. This breaks the normal stack-based flow and might lead to surprising behavior. I don't know if there's anything similar in Clojure, Elisp, or Common Lisp. This was one of the main problems with our lexical let. But at least we don't have to implement closures.

If you mutate the DefDynamic object outside of a context manager, then the last context manager might not pop the same binding it pushed. Maybe it's enough to tell the user to not do that.

[gilch]

But I'm not completely sure if I understand (or even can think of) all possible cases for this. Variables with dynamic scope would only be available inside a let form that referes to them? Or would they be available somewhere else too? Could one define a function with defn, bind it to dynamically scoped variable and then call it later during program execution? How would the definition part and calling part look like in Hy code? Any idea (rough sketch is enough) what the resulting Python code would look like?

@tuturto "Inside" is in the dynamic sense, not the lexical one. Think stack frames, not code blocks. You could use the above DefDynamic as a decorator on a defn to make the function variable dynamic, or you could just put a lambda in the with statement.

# requires the custom globals dict

@DefDynamic
def foo():
    return 'did a foo'

def dofoo():
    print('doing a foo: ', foo())

def dodofoo():
    print('doing a dofoo')
    dofoo()

with _dynamic['foo'](lambda: 'foo for you too'):
    dodofoo()  # notice the shadowed form applies through an intermediate call

dofoo()

doing a dofoo
doing a foo: foo for you too
doing a foo: did a foo

I'm not sure what the Hy code to generate the above Python should look like. Actually it probably shouldn't generate exactly the above Python, because this is just a rough sketch with a number of problems.

We may want to rethink the def vs setv hylang/hy#911 and *foo* vs +foo+ hylang/hy#383 questions. In Clojure and Common Lisp, it is the dynamic variables that have the * form, not constants. Maybe we could have def make the DefDynamic objects, and setv be only for the normal lexical variables.

DefDynamic and DynamicDict could perhaps be called something else. They could also be made more efficient.

We need a way to access the DefDynamic object itself, similar to the way Clojure has #'/var to get the var object itself. The above example has a custom dict that puts it in a _dynamic global dict so the with form can get to it. Perhaps it could live in the appropriate HySymbol instead, but that might not work as well with namespaces. Maybe call the global _hy_dynamics or something.

I'm not sure if the form creating the with should be called let as in Elisp (giving us a sensible let), or binding as in Clojure to avoid confusion with the old lexical form.

Does that answer your questions?

[tuturto]

Thanks for writing this down. It clears up lots of questions that I had. I would call the new form binding instead of let I think. Would make sure that code written in the old Hy wouldn't accidentally use let without knowing that semantics have changed.

[hcarvalhoalves]

See https://pypi.python.org/pypi/xlocal to implement the equivalent of Clojure's with-bindings

[Kodiologist]

My impression is that a good-enough version of this could be implemented in a macro or context manager, without changes to Hy core.

[Zambito1]

@gilch The proof of concept that you wrote is semantically identical to the way that parameter objects work in Scheme. They must be called as a procedure to extract the value, rather than just using the identifier directly.

I adapted your Python example directly to Hy (not necessary for most probably, but it simplifies things in my project) and added two wrappers to give it a more similar API to Scheme:

(import contextmanager [contextlib])
(setv _sentinel (object))
(defclass Parameter []
  (defn __init__ [self [value None]]
    (setv self.bindings [value]))
  (defn __call__ [self [value _sentinel]]
    (if (is value _sentinel)
        (get self.bindings -1)
	(do
	  (defn [contextmanager] manager []
	    (self.bindings.append value)
	    (yield)
	    (self.bindings.pop))
	  (manager)))))

(defn make-parameter [obj] (Parameter obj))

(defmacro parameterize [bindings #* body]
  (if (= 1 (len bindings))
      `(with [~(get bindings 0)]
          ~@body)
      `(with [~(get bindings 0)]
          (parameterize ~(cut bindings 1 (len bindings)) ~@body))))

Usage:

(setv a (make-parameter 1))
(setv b (make-parameter 2))

(defn add-a-and-b [] 
    (+ (a) (b)))

(add-a-and-b) ; => 3
(parameterize [(a 5) (b 6)] (add-a-and-b)) ; => 11
(add-a-and-b) ; => 3

[Zambito1]

Actually I found a mismatch in the behavior vs Scheme:

(setv a (make-parameter 1))
(setv b (make-parameter 2))

(defn add-a-and-b [] 
    (+ (a) (b)))

(a 5)
(b 6)

(add-a-and-b) ; => 3. In Scheme this would be 11.

I would prefer for calling the parameter with a value to set the parameter to have a new value, but I don't really understand how the contextmanager works well enough to do that myself.

[Kodiologist]

You're making the bindings and then throwing them away. The intended use is like this:

(with [_ (a 5)  _ (b 6)]
  (print (add-a-and-b)))  ; => 11