Skip to content

dsl

Jump to bottom Edit New page
hadley edited this page Jun 18, 2013 · 8 revisions

Domain specific languages

The combination of first class environments and lexical scoping gives us a powerful toolkit for creating embedded domain specific languages (DSLs) in R. Embedded DSLs take advantage of a host language's parsing and execution framework, but adjust the semantics somewhat to make them more suitable for a specific task.

R already has a simple and popular DSL built in: the formula specification, which offers a succinct way of describing the relationship between predictors and the response. Other examples of DSLs include ggplot2 (for visualisation), and plyr (for data manipulation). Another package that makes extensive use of these ideas is dplyr, which provides to_sql() to converts R expressions into SQL:

library(dplyr)
to_sql(sin(x) + tan(y))
# <SQL> SIN(x) + TAN(y)
to_sql(x < 5 & !(y >= 5))
# <SQL> x < 5.0 AND NOT((y >= 5.0))
to_sql(first %like% "Had*")
# <SQL> first LIKE 'Had*'
to_sql(first %in% c("John", "Roger", "Robert"))
# <SQL> first IN ('John', 'Roger', 'Robert')
to_sql(like == 7)
# <SQL> "like" = 7.0

Once you have read this chapter, you might want to study the source code for dplyr. An important part of the overall structure of the package is partial_eval() which helps manage expressions where some of the components refer to variables in the database and some refer to local R objects. You could use very similar ideas if you needed to translate small R expressions into other languages, like javascript or python. Converting complete R programs would be extremely difficult, but often being able to communicate a simple description of computation between languages is very useful.

R is well suited for hosting DSLs because the combination of a small amount of computing on the language and constructing special evaluation environments is very powerful. Creating new DSLs in R uses many techniques that you've learned about elsewhere in the book, including:

  • scoping rules
  • creating and manipulating functions
  • computing on the language
  • S3 basics

This chapter will develop two simple, but useful, DSLs, one for generating HTML, and one for turning R mathematical expressions into a form suitable for inclusion in latex.

DSLs are a very large topic, and this chapter will only scratch the surface, focussing on important techniques and not so much on how you might come up with the language in the first place. If you're interested in learning more, I highly recommend Domain Specific Languages by Martin Fowler: it discusses many options for creating a DSL and provides many examples of different languages.

HTML

HTML is the language that underlies the majority of the web. It is a special case of SGML, and similar (but not identical) to XML. HTML looks like this:

<body>
  <h1 id = 'first'>A heading</h1>
  <p>Some text &amp; <b>some bold text.</b></p>
  <img src = 'myimg.png' width = '100' height = '100' />
</body>

Even if you've never seen HTML before, hopefully you can see the key component of the structure: HTML is composed of tags that look like <tag></tag>. Tags can be contained inside other tags and intermingled with text. Generally, html ignores whitespace: an sequence of whitespace is equivalent to a single space. You could put the previous example all on online and it would still display the same in the browser:

<body><h1 id='first'>A heading</h1><p>Some text &amp; <b>some bold
text.</b></p><img src = 'myimg.png' width = '100' height = '100' />
</body>

However, like R code, you usually want to indent HTML to make it more obvious to see the structure.

There are over 100 html tags, but to illustrate HTML we're going to focus on just a few:

  • <body>: the top-level tag that all content is enclosed within
  • <h1>: creates a heading-1, the top level heading
  • <p>: creates a paragraph
  • <b>: emboldens text
  • <img>: embeds an image

(you probably guessed what these did already!)

Tags can also have named attributes that look like <tag a = "a" b = "b"></tags>. Tag values should always be enclosed in either single or double quotes. Two important attributes used on just about every tag are id and class. These are used in conjunction with CSS (cascading style sheets) in order to control the style of the document.

Some tags, like <img>, can't have any content. These are called void tags and have a slightly different syntax: instead of writing <img></img> you write <img />. Since they have no content, attributes are more imporant, and img has three that are used for almost every image: src (where the image lives), width and height.

Because < and > have special meanings in html, you can't write them directly. Instead you have to use the html escapes &gt; and &lt;. And since those escapes use &, you also have to escape it with &amp; if you want a literal ampersand.

Goal

Our goal is to make it easy to generate html from R. To give a concrete example, we want to generate the following html:

<body>
  <h1 id = 'first'>A heading</h1>
  <p>Some text &amp; <b>some bold text.</b></p>
  <img src = 'myimg.png' width = '100' height = '100' />
</body>

using code that looks as similar to the html as possible. We will work our way up to the following DSL:

with_html(body(
  h1("A heading", id = "first"),
  p("Some text &", b("some bold text.")),
  img(src = "myimg.png", width = 100, height = 100)
))

Note that the nesting of function calls is the same as the nesting of tags, unnamed arguments become the content of the tag, and named arguments become the attributes. Because tags and text are clearly distinct in this api, we can automatically escape & and other special characters.

Escaping

Escaping is so fundamental we're going to start with it. We first start by creating a way of escaping the characters that have special meaning for html, while making sure we don't end up double-escaping at any point. The easiest way to do this is to create an S3 class that allows us to distinguish between regular text (that needs escaping) and html (that doesn't).

We then write an escape method that leaves html unchanged and escapes the special characters (&, <, >) in ordinary text. We also add a method for lists for convenience

html <- function(x) structure(x, class = "html")
print.html <- function(x, ...) cat("<HTML> ", x, "\n", sep = "")
escape <- function(x) UseMethod("escape")
escape.html <- function(x) x
escape.character <- function(x) {
  x <- gsub("&", "&amp;", x)
  x <- gsub("<", "&lt;", x)
  x <- gsub(">", "&gt;", x)
  
  html(x)
}
escape.list <- function(x) {
  lapply(x, escape)
}

# Now we check that it works
escape("This is some text.")
# <HTML> This is some text.
escape("x > 1 & y < 2")
# <HTML> x &gt; 1 &amp; y &lt; 2

# Double escaping is not a problem
escape(escape("This is some text. 1 > 2"))
# <HTML> This is some text. 1 &gt; 2

# And text we know is html doesn't get escaped.
escape(html("<hr />"))
# <HTML> <hr />

Escaping is an important component for any dsl.

Basic tag functions

Next we'll write a few simple tag functions and then figure out how to generalise for all possible html tags. Let's start with <p>. HTML tags can have both attributes (e.g. id, or class) and children (like <b> or <i>). We need some way of separating these in the function call: since attributes are named values and children don't have names, it seems natural to separate using named vs. unnamed arguments. Then a call to p() might look like:

p("Some text.", b("some bold text"), class = "mypara")

We could list all the possible attributes of the p tag in the function definition, but that's hard because there are so many, and it's possible to use custom attributes Instead we'll just use ... and separate the components based on whether or they are named. To do this correctly, we need to be aware of a "feature" of names():

names(c(a = 1, b = 2))
# [1] "a" "b"
names(c(a = 1, 2))
# [1] "a" ""
names(c(1, 2))
# NULL

With this in mind we create two helper functions to extract the named and unnamed components of a vector:

named <- function(x) {
  if (is.null(names(x))) return(NULL)
  x[names(x) != ""]
}
unnamed <- function(x) {
  if (is.null(names(x))) return(x)
  x[names(x) == ""]
}

We can now create our p() function. There's one new function here: html_attributes(). This takes a list of name-value pairs and creates the correct html attributes specification from them. It's a little complicated (to deal with some idiosyncracies of html that I haven't mentioned), not that important and doesn't introduce any new ideas, so I won't discuss it here, but it's included at the end of the chapter.

source("code/html-attributes.r")
p <- function(...) {
  args <- list(...)
  attribs <- html_attributes(named(args))
  children <- unlist(escape(unnamed(args)))
  
  html(paste0(
    "<p", attribs, ">", 
    paste(children, collapse = ""), 
    "</p>"
  ))
}

p("Some text")
# <HTML> <p>Some text</p>
p("Some text", id = "myid")
# <HTML> <p id = 'myid'>Some text</p>
p("Some text", image = NULL)
# <HTML> <p image>Some text</p>
p("Some text", class = "important", "data-value" = 10)
# <HTML> <p class = 'important' data-value = '10'>Some text</p>

Tag functions

With this definition of p() it's pretty easy to see what will change for different tags: we just need to replace "p" with a variable. We'll use a closure to make it easy to generate a tag function given a tag name:

tag <- function(tag) {
  force(tag)
  function(...) {
    args <- list(...)
    attribs <- html_attributes(named(args))
    children <- unlist(escape(unnamed(args)))
    
    html(paste0(
      "<", tag, attribs, ">", 
      paste(children, collapse = ""), 
      "</", tag, ">"
    ))
  }
}

(We're forcing the evaluation tag with the expectation we'll be calling this function from a loop later on - that avoids potential bugs caused by lazy evaluation.)

Now we can run our earlier example:

p <- tag("p")
b <- tag("b")
i <- tag("i")
p("Some text.", b("Some bold text"), i("Some italic text"), 
  class = "mypara")
# <HTML> <p class = 'mypara'>Some text.<b>Some bold text</b><i>Some italic text</i></p>

Before we continue to generate functions for every possible html tag, we need a variant of tag() for void tags. It can be very similar to tag(), but needs to throw an error if there are any unnamed tags, and the tag itself looks slightly different:

void_tag <- function(tag) {
  force(tag)
  function(...) {
    args <- list(...)
    if (length(unnamed(args)) > 0) {
      stop("Tag ", tag, " can not have children", call. = FALSE)
    }
    attribs <- html_attributes(named(args))
    
    html(paste0("<", tag, attribs, " />"))
  }
}

img <- void_tag("img")
img(src = "myimage.png", width = 100, height = 100)
# <HTML> <img src = 'myimage.png' width = '100' height = '100' />

Processing all tags

Next we need a list of all the html tags:

tags <- c("a", "abbr", "address", "article", "aside", "audio", "b", 
  "bdi", "bdo", "blockquote", "body", "button", "canvas", "caption", 
  "cite", "code", "colgroup", "data", "datalist", "dd", "del", 
  "details", "dfn", "div", "dl", "dt", "em", "eventsource", 
  "fieldset", "figcaption", "figure", "footer", "form", "h1", "h2", 
  "h3", "h4", "h5", "h6", "head", "header", "hgroup", "html", "i", 
  "iframe", "ins", "kbd", "label", "legend", "li", "mark", "map", 
  "menu", "meter", "nav", "noscript", "object", "ol", "optgroup", 
  "option", "output", "p", "pre", "progress", "q", "ruby", "rp", 
  "rt", "s", "samp", "script", "section", "select", "small", "span", 
  "strong", "style", "sub", "summary", "sup", "table", "tbody", 
  "td", "textarea", "tfoot", "th", "thead", "time", "title", "tr", 
  "u", "ul", "var", "video") 

void_tags <- c("area", "base", "br", "col", "command", "embed", 
  "hr", "img", "input", "keygen", "link", "meta", "param", "source", 
  "track", "wbr")

If you look at this list carefully, you'll see there are quite a few tags that have the same name as base R functions (body, col, q, source, sub, summary, table), and others that clash with popular packages (e.g. map). That implies we don't want to make all the functions available (in either the global environment or a package environment) by default. Instead, we'll put them in a list, and add some additional code to make it easy to use them when desired. First we make a named list:

tag_fs <- c(
  setNames(lapply(tags, tag), tags), 
  setNames(lapply(void_tags, void_tag), void_tags)
)

This gives us a way to call tag functions explicitly, but is a little verbose:

tag_fs$p("Some text.", tag_fs$b("Some bold text"), 
  tag_fs$i("Some italic text"))
# <HTML> <p>Some text.<b>Some bold text</b><i>Some italic text</i></p>

Then we finish off our HTML DSL by creating a function that allows us to evaluate code in the context of that list:

with_html <- function(code) {
  eval(substitute(code), tag_fs)
}

This gives us a succinct API which allows us to write html when we need it without cluttering up the namespace when we don't. Inside with_html if you want to access the R function overridden by an html tag of the same name, you can use the full package::function specification.

with_html(body(
  h1("A heading", id = "first"),
  p("Some text &", b("some bold text.")),
  img(src = "myimg.png", width = 100, height = 100)
))
# <HTML> <body><h1 id = 'first'>A heading</h1><p>Some text &amp;<b>some bold text.</b></p><img src = 'myimg.png' width = '100' height = '100' /></body>

Exercises

  • The escaping rules for <script> and <style> tags are different: you don't want to escape angle brackets or ampersands, but you do want to escape </. Adapt the code above to follow these rules.

  • The use of ... for all functions has some big downsides: there's no input validation and there will be little information in the documentation or autocomplete about how to use the function. Create a new function that when given a named list of tags and their attribute names (like below), creates functions with those signatures.

    list(
  a = c("href"),
  img = c("src", "width", "height")
)

    All tags should get class and id attributes.

  • Currently the html doesn't look terribly pretty, and it's hard to see the structure. How could you adapt tag() to do be indenting and formatting?

Latex

The next DSL we're going to tackle will convert R expression into their latex math equivalents. (This is a bit like ?plotmath, but for text instead of plots.) Latex is the lingua franca of mathematicians and statisticians: whenever you want to describe an equation in text (e.g. in an email) you write it as a latex equation. Many reports are produced from R using latex, so it might be useful to facilitate the automate conversion from mathematical expressions from one language to the other.

This math expression DSL will be more complicated than the HTML DSL, because not only do we need to convert functions, but we also need to convert symbols. We'll also create a "default" conversion, so that functions we don't know how to convert get a standard fallback. Like the HTML dsl, we'll also write functionals to make it easier to generate the translators.

Before we begin, let's quickly cover how formulas are expressed in latex.

Latex mathematics

Latex mathematics are complex, and well documented. They have a fairly simple structure:

  • Most simple mathematical equations are represented in the way you'd type them into R: x * y, z ^ 5. Subscripts are written using _, e.g. x_1.

  • Special characters start with a \: \pi = π, \pm = ±, and so on. There are a huge number of symbols available in latex. Googling for latex math symbols finds many lists, and there's even a service where you can sketch a symbol in the browser and it will look it up for you.

  • More complicated functions look like \name{arg1}{arg2}. For example to represent a fraction you use \frac{a}{b}, and a sqrt looks like \sqrt{a}.

  • To group elements together use {}: i.e. x ^ a + b vs. x ^ {a + b}.

  • In good math typesetting, a distinction is made between variables and functions, but without extra information, latex doesn't know whether f(a * b) represents calling the function f with argument a * b, or is shorthand for f * a * b. If f is a function, you can tell latex to typeset it using an upright font with \textrm{f}(a * b)

Goal

Our goal is to use these rules to automatically convert from an R expression to a latex representation of that expression. We will tackle it in four stages:

  • Convert known symbols: pi -> \pi
  • Leave other symbols unchanged: x -> x, y -> y
  • Convert known functions: x * pi -> x * \pi, sqrt(frac(a, b)) -> \sqrt{\frac{a, b}}
  • Wrap unknown functions with \textrm: f(a) -> \textrm{f}(a)

Compared to the HTML dsl, we'll work in the opposite direction: we'll start with the infrastructure and work our way down to generate all the functions we need

to_math

To begin, we need a wrapper function that we'll use to convert R expressions into latex math expressions. This works the same way as to_html: we capture the unevaluated expression and evaluate it in a special environment. However, the special environment is no longer fixed, and will vary depending on the expression. We need this in order to be able to deal with symbols and functions that we don't know about a priori.

to_math <- function(x) {
  expr <- substitute(x)
  eval(expr, latex_env(expr))
}

Known symbols

Our first step is to create an environment that allows us to convert the special latex symbols used for Greek, e.g. pi to \pi. This is the same basic trick used in subset to make it possible to select column ranges by name (subset(mtcars, , cyl:wt)): we just bind a name to a string in a special environment.

First we create than environment by creating a named vector, converting that vector into a list, and then turn that list into an environment.

greek <- c(
  "alpha", "theta", "tau", "beta", "vartheta", "pi", "upsilon", 
  "gamma", "gamma", "varpi", "phi", "delta", "kappa", "rho", 
  "varphi", "epsilon", "lambda", "varrho", "chi", "varepsilon", 
  "mu", "sigma", "psi", "zeta", "nu", "varsigma", "omega", "eta", 
  "xi", "Gamma", "Lambda", "Sigma", "Psi", "Delta", "Xi", "Upsilon", 
  "Omega", "Theta", "Pi", "Phi")
greek_list <- setNames(paste0("\\", greek), greek)
greek_env <- list2env(as.list(greek_list), parent = emptyenv())

We can then check it:

latex_env <- function(expr) {
  greek_env
}

to_math(pi)
# [1] "\\pi"
to_math(beta)
# [1] "\\beta"

Unknown symbols

If a symbol isn't greek, we want to leave it as is. This is trickier because we don't know in advance what symbols will be used, and we can't possibly generate them all. So we'll use a little bit of computing on the language to find out what symbols are present in an expression. The all_names function takes an expression: if it's a name, it converts it to a string; if it's a call, it recurses down through its arguments.

all_names <- function(x) {
  # Base cases
  if (is.name(x)) return(as.character(x))
  if (!is.call(x)) return(NULL)

  # Recursive case
  children <- lapply(x[-1], all_names)
  unique(unlist(children))
}

all_names(quote(x + y + f(a, b, c, 10)))
# [1] "x" "y" "a" "b" "c"
# [1] "x" "y" "a" "b" "c"

We now want to take that list of symbols, and convert it to an environment so that each symbol is mapped to a string representing itself (e.g. so eval(quote(x), env) yields "x"). We again use the pattern of converting a named character vector to a list, then an environment.

latex_env <- function(expr) {
  names <- all_names(expr)
  symbol_list <- setNames(as.list(names), names)
  symbol_env <- list2env(symbol_list)

  symbol_env
}

to_math(x)
# [1] "x"
to_math(longvariablename)
# [1] "longvariablename"
to_math(pi)
# [1] "pi"

This works, but we need to combine it with the enviroment of the Greek symbols. Since we want to prefer Greek to the defaults (e.g. to_math(pi) should give "\\pi", not "pi"), symbol_env needs to be the parent of greek_env, and thus we need to make a copy of greek_env with a new parent. Strangely R doesn't come with a function for cloning environments, but we can easily create one by combining two existing functions:

clone_env <- function(env, parent = parent.env(env)) {
  list2env(as.list(env), parent = parent)
}

This gives us a function that can convert both known (Greek) and unknown symbols.

latex_env <- function(expr) {
  # Unknown symbols
  names <- all_names(expr)
  symbol_list <- setNames(as.list(names), names)
  symbol_env <- list2env(symbol_list)

  # Known symbols
  clone_env(greek_env, symbol_env)
}

to_math(x)
# [1] "x"
to_math(longvariablename)
# [1] "longvariablename"
to_math(pi)
# [1] "\\pi"

Known functions

Next we'll add functions to our DSL. We'll start with a couple of helper closures that make it easy to add new unary and binary operators. These functions are very simple since they only have to assemble strings. (Again we use force to make sure the arguments are evaluated at the right time.)

unary_op <- function(left, right) {
  force(left)
  force(right)
  function(e1) {
    paste0(left, e1, right)
  }
}

binary_op <- function(sep) {
  force(sep)
  function(e1, e2) {
    paste0(e1, sep, e2)
  }
}

Using these helpers, we can map a few illustrative examples from R to latex. Note how the lexical scoping rules of R help us: we can easily provide new meanings for standard functions like +, - and *, and even ( and {.

# Binary operators
f_env <- new.env(parent = emptyenv())
f_env$"+" <- binary_op(" + ")
f_env$"-" <- binary_op(" - ")
f_env$"*" <- binary_op(" * ")
f_env$"/" <- binary_op(" / ")
f_env$"^" <- binary_op("^")
f_env$"[" <- binary_op("_")

# Grouping
f_env$"{" <- unary_op("\\left{ ", " \\right}")
f_env$"(" <- unary_op("\\left( ", " \\right)")
f_env$paste <- paste

# Other math functions
f_env$sqrt <- unary_op("\\sqrt{", "}")
f_env$sin <- unary_op("\\sin(", ")")
f_env$log <- unary_op("\\log(", ")")
f_env$abs <- unary_op("\\left| ", "\\right| ")
f_env$frac <- function(a, b) {
  paste0("\\frac{", a, "}{", b, "}")
}

# Labelling
f_env$hat <- unary_op("\\hat{", "}")
f_env$tilde <- unary_op("\\tilde{", "}")

We again modify latex_env() to include this environment. It should be the last environment in which names are looked for, so that sin(sin) works. (because of R's matching rules wrt functions vs. other objects)

latex_env <- function(expr) {
  # Known functions
  f_env

  # Default symbols
  names <- all_names(expr)
  symbol_list <- setNames(as.list(names), names)
  symbol_env <- list2env(symbol_list, parent = fenv)

  # Known symbols
  greek_env <- clone_env(greek_env, parent = symbol_env)
}

to_math(sin(x + pi))
# [1] "\\sin(x + \\pi)"
to_math(log(x_i ^ 2))
# [1] "\\log(x_i^2)"
to_math(sin(sin))
# [1] "\\sin(sin)"

Unknown functions

Finally, we'll add a default for functions that we don't know about. Like the unknown names, we can't know in advance what these will be, so we again use a little computing on the language to figure them out:

all_calls <- function(x) {
  # Base name
  if (!is.call(x)) return(NULL)

  # Recursive case
  fname <- as.character(x[[1]])
  children <- lapply(x[-1], all_calls)
  unique(c(fname, unlist(children, use.names = FALSE)))
}

all_calls(quote(f(g + b, c, d(a))))
# [1] "f" "+" "d"

And we need a closure that will generate the functions for each unknown call

unknown_op <- function(op) {
  force(op)
  function(...) {
    contents <- paste(..., collapse = ", ")
    paste0("\\mathrm{", op, "}(", contents, ")")
  }
}

And again we update latex_env():

latex_env <- function(expr) {
  calls <- all_calls(expr)
  call_list <- setNames(lapply(calls, unknown_op), calls)
  call_env <- list2env(call_list)

  # Known functions
  f_env <- clone_env(f_env, call_env)

  # Default symbols
  symbols <- all_names(expr)
  symbol_list <- setNames(as.list(symbols), symbols)
  symbol_env <- list2env(symbol_list, parent = f_env)

  # Known symbols
  greek_env <- clone_env(greek_env, parent = symbol_env)
}

to_math(f(a * b))
# [1] "\\mathrm{f}(a * b)"

Exercises

  • Add automatic escaping. Special symbols that should be escaped by adding a backslash in front of them are \, $ and %. Like for sql, you'll need to make sure you don't end up double-escaping, so you'll need to create a small s3 class and then use that in function operators. That will also allow you to embed arbitrary latex if needed.

  • Complete the DSL to support all the functions that plotmath supports

  • There's a repeating pattern in latex_env(): we take a character vector, do something to each piece, then convert it to a list, and then an environment. Write a function to automate this task, and then rewrite latex_env()

Add a custom footer
Add a custom sidebar
Clone this wiki locally