We will consider a very simple domain, that of building nested menus such as the following:

    Questions
  

    Answers
  

    Letters


        To Bill
      

        To Steve
      

It’s simply a nested list in HTML, where each entry is a link to some other page. Clearly, for such a simple example, a DSL may seem redundant. But it does keep the code quite simple, while hopefully keeping enough interest to be of value.

We will implement 3 different versions of the DSL, corresponding to different approaches to DSL development (as per the previous blog entry):

1. Program → Actions: In this case, running the DSL program immediately generates the menu as directly as HTML.
2. Program → Model → Actions: In this case, running the DSL program generates a data structure (the model) representing the menu. A separate operation renders the menu as HTML.
3. Program → Model → Artifacts → Actions: Again, running the DSL produces a model of the menu. In this case, a code generation phase takes the model and produces Java code which can be incorporated into a Swing application to use our menu (albeit, without any functionality).

First, let’s see what the old school approach looks like in the case where an intermediate data structure is constructed.

require 'model'
require 'render'

mymenu = Node.new(:root, 'root')
questions = mymenu.add_node(:q, '/q', 'Questions')
answers   = mymenu.add_node(:a, '/a', 'Answers')
letters   = mymenu.add_node(:letter, '', 'Letters')
bill      = letters.add_node(:bill, '/letter/bill', 'To Bill')
steve     = letters.add_node(:stever, '/letter/steve', 'To Steve')

puts ListRenderer.new(mymenu).render

Firstly, a data structure representing the model is imported, as well as code to render the model into HTML. The menu is created by constructing a root node, to which menu entries and submenus are added by attaching these to the appropriate part of the tree being constructed. Finally, the resulting data structure is passed to an object to render it to HTML.

Incidentally, this code resembles the DOM-style of building XML documents.

The MenuBuilder DSL for the above menu will look like the following.

mymenu = menu do
	item :q, '/q', 'Questions'
	item :a, '/a', 'Answers'
	item :letter, '', 'Letters' do
	  item :bill, '/letter/bill', 'To Bill'
	  item :steve, '/letter/steve', 'To Steve'
	end
end

render mymenu

This code introduces two methods menu and render which construct menus and render them as HTML, respectively. The menu method takes a Ruby block which implements the contents of the menu. The method item which is available only within the block passed to the menu method creates the items of the menu. Each item is given a symbolic label, a target url, and a label, and optionally, as a Ruby block, a submenu.

The difference between the old school code and the DSL code is small but important. The original approach forces the programmer to think in terms of objects and operations upon them which build up a data structure representing the menu. With the DSL, the programmer directly describes the menu. The DSL implementation performs the construction of the underlying data structures. With the DSL, a menu is specified declaratively, rather than constructed imperatively.

In addition, the old school approach also requires a premature commitment to a particular approach, which may need to be changed whenever the requirements change. The DSL program is sufficiently abstract yet contains enough information to specify the menu completely. When the implementation requirements change, the menu code need not (necessarily) change.

1. Direct implementation: Program → Actions

The following code implements the DSL so that running the menu method directly produces the HTML code to standard output. The render command is not required in this example.

class MenuWriter
  def self.menu(&block)
    puts "
"
    instance_eval &block
    puts "

"
  end

  def self.item(sym, url='’, name='’, &block)
    puts "
"
    puts "#{name}"
    if block_given?
      menu &block
    end
    puts "

"
  end
end

def menu(&block)
  MenuWriter.menu &block
end

The code is pretty straightforward. It evaluates the DSL in the order that is is presented on the page, wrapping each item with the appropriate HTML tags.

For a simple DSL with a simple application which is not likely to ever be reused, this is not bad approach.

2. Implementation via Model: Program → Model → Actions

In this approach we introduce a simple class which acts as the model for menus. It is a simple tree data structure.

The following is the code which represents the nesting menu. This is the implementation of the underlying model:

class Node
  @active = false
  attr_accessor :url, :label
  attr_reader :name, :children

  # constuctor
  def initialize(name, url='', label='')
    @children = []
    @name     = name.to_sym
    @url      = url
    @label    = label
  end

  # add a new node to the tree, returning new node
  def add_node(name, url, label)
    node = Node.new(name, url, label)
    @children << node
    ## stores node in a field named name, creating accessor method
    instance_eval "
      @#{name} = node
      def #{name}; @#{name}; end"
    return node
  end
end

This is really the code of a more elaborate and more useful class. Notice the use of meta-programming in the add_node method. The call the instance_eval creates a field with the name of the node, and stores the new node in that field. An accessor method for this field is also created.

The following class renders a menu as an HTML unordered list using the XML Builder library. This is achieved by performing a simple traversal of the Node data structure.

require 'rubygems'
require_gem 'builder', '~> 2.0'

class ListRenderer

  attr_reader :root, :depth, :buffer, :builder

  # pass in the root node of the menu you're trying to render
  def initialize(root)
    @buffer = ''
    @builder = Builder::XmlMarkup.new(:target => @buffer, :indent=>2)
    @root = root
    @depth = 0
  end

  # output HTML -- skips root!
  def render
    render_children @root
    @builder.target!
  end

  private
  # render a single node, recursively render its children
  def render_node(node)
    attr = {}
    @builder.li(attr) { |xml|
      xml.a("#{node.label}", :href=>node.url)
      render_children node
    }
  end

  # render children
  def render_children(node)
    # now descents down the children and do the same
    return if node.children.length == 0
    @builder.ul {
      node.children.each { |child| render_node(child) }
    }
  end
end

Finally, we have the implementation of the DSL constructs, which in this case consists of: a class Creator which holds a reference to the node being created, and itself implements the item operation; the method menu for creating a menu, and the method render for rendering a menu.

class Creator
  def initialize(menu)
    @menu = menu
  end

  def item(sym, url='', name='', &block)
    nextnode = @menu.add_node(sym,url,name)
    if block_given?
      foo = Creater.new(nextnode)
      foo.instance_eval(&block)
    end
  end
end

def menu(sym=:root, name='root', &block)
  nm = Node.new(sym, name)
  creator = Creator.new(nm)
  creator.instance_eval &block
  return nm
end

def render(menu)
  puts ListRenderer.new(menu).render
end

The main reason this approach is better than the first approach is that it enables more reuse. The model can serve as the basis of this reuse. In the next implementation, we will use the same model to generate Java code which calls to Java’s Swing library to generate our menu. Neither the DSL implementation, the DSL code, nor the model code needs to change to accommodate this new feature (apart from changing the call to render).

3. Implementation via Model and Generator: Program → Model → Artifacts → Actions

In this example, we take the existing DSL and its implementation and generate Java code to build a menu using Java’s Swing library. The main point to observe is that neither the DSL implementation, nor the model need change; we simply add another class which traverses the model to spit out Java rather than HTML. We have now entered the world of generators, also called code generators, not to be confused with code generation in a compiler back-end compiler.

Before presenting the generator we need to think about what code it will produce. A good approach to doing this is to first write the desired code for a small example, such as for our example menu. One can then apply techniques which we’ll dub refactorization for code generation to construct the generator.

Here is the Java code we anticipate the generator will output for our example menu:

import javax.swing.*;

public class MenuGen {
  public static void addMenu(JFrame frame) {
    JMenuBar menuBar;
    JMenu menu, submenu;
    JMenuItem menuItem;

    menuBar = new JMenuBar();

    menu = new JMenu("Root");
    menuBar.add(menu);

    menuItem = new JMenuItem("Questions");
    menu.add(menuItem);

    menuItem = new JMenuItem("Answers");
    menu.add(menuItem);

    submenu = new JMenu("Letters");
    menu.add(submenu);

    menuItem = new JMenuItem("To Bill");
    submenu.add(menuItem);

    menuItem = new JMenuItem("To Steve");
    submenu.add(menuItem);

    frame.setJMenuBar(theJMenuBar);
  }
}

Code generators are typically written based on a domain model, although the model need not be formally or explicitly developed; it may be purely a mental model. The model is constructed used what’s called domain analysis. This is a topic for future entries, but the key techniques we wish to point out now is commonality/variability analysis. This analysis helps determine which parts of a model, or, in our case, code, remain constant and which parts vary. Typically, the variation points are features which are represented in DSL programs and the constant parts are present in the framework which underlies the DSL implementation.

The same idea applies to our code generation implementation. We need to work out which parts of the above code are constant and which vary. You could print out the code and use a highlighter to get a rough general idea. We can then attack the code either top-down (from the package/class level inwards) or bottom-up (starting from inside methods, working out to classes), with the aim of writing Ruby code which will generate the Java code.

Starting from the top, we identify a Java package, some import statements, and the start and finish of the method addMenu, the workhorse, as being pretty much fixed. The following Ruby method simply outputs this code as a string, making a call in the middle to a method render, which will generate the code which varies for each menu.

class JavaMenuGenerator
  def generate_class
    puts %(
    import javax.swing.*;

    public class MenuGen {
      public static void addMenu(JFrame frame) {
        JMenuBar menuBar;
        JMenuItem menuItem;
        menuBar = new JMenuBar();
    )
    # renders the model
    render "menuBar", @root
    puts %(
        frame.setJMenuBar(theJMenuBar);
      }
    }
    )
  end
end

A more serious implementation would endeavor to pretty-print the generated code, which is important for both debugging the generator and using the generated code.
Now it’s time to start thinking bottom-up. As we already know, there are two basic elements:
menus and items. Let’s write a method to generate each of these. We also write a generic render method which determines whether a particular node is a menu or an item based on the number of children it has.

class JavaMenuGenerator
  def menu(parent, label, children)
    target = genvar
    puts "JMenu #{target} = new JMenu("#{label}");"
    puts "#{parent}.add(#{target});"
    children.each { |c| render target, c }
  end

  def item(target, label)
    puts %(
      menuItem = new JMenuItem("#{label}");
      #{target}.add(menuItem);
    )
  end

  def render(parent, node)
    if node.children.length == 0 then
      item parent, node.label
    else
      menu parent, node.label, node.children
    end
  end
end

The render method also forms a bridge between this code and the constant code, to loosen the coupling between the two parts of the code generator.

Finally, here’s the remainder of the code which initializes a JavaMenuGenerator object with particular Node:

class JavaMenuGenerator
  attr_reader :root

  # pass in the root node of the menu you're trying to render
  def initialize(root)
    @root = root
    @varid = 0
  end

  # generate new variable names
  def genvar
    @varid += 1
    "var#{@varid}"
  end
end

After creating a menu, the following call generates the Java code:

JavaMenuGenerator.new(mymenu).generate_class

It’s a simple task for the reader to hide this behind a more DSL-like call.

If you run this code, you’ll notice that it doesn’t produce exactly the code that we initially wrote. This is because some simple changes were made along the way to make the code generator easier to write.

Now we have written two generators for the same model. Both generators are not exactly the same, but they have much in common. As pragmatic programmers, we would want to make this code DRY-er to avoid any duplication. The Visitor pattern or some variation on in would certainly come in handy here.

Final Words (Finally)

We’ve covered a lot of ground in this entry, and we’ve employed one very basic technique for writing DSLs in Ruby. Other techniques are possible, but we’ll explore these at another time. (Also look at the comment from Justin for a potential pitfall with the approach we’ve taken.) One lesson to take away is that DSL design and implementation goes well beyond simply writing some methods which have an English like appearance. A lot more code is written to support the DSL. The DSL process works is most effective when you have a clear notion of what your domain model is.

Credits

Jeroen Houben of Lost Boys posed this simple challenge and wrote the initial version of the code.

    Program --> Model --> Artifacts --> Actions

The various components are:

1. a program written in the concrete syntax of the DSL;
2. a model which captures the underlying interpretation of the program, possibly as some form of annotated abstract syntax tree, generally decorated with semantic information gained from analyzing the program;
3. a number of artifacts such as classes, libraries, objects, database entries/tables, binaries, programs, configuration files, documentation, web pages, and so forth which are generated based from the model; and
4. some actions which are the resulting behaviour of the DSL, which could include parsing a particular kind of data stream, playing a sequence of sounds, transforming an XML document, or what have you.

The phases proceed as follows:

Program → Model

Standard compiler technology (lexing, parsing, semantic analysis) converts a program in an external DSL into its abstract model. Internal (or embedded) DSLs are evaluated and part of their execution involves semantic checks and the construction of the underlying model.

Model → Artifacts

Generative programming techniques (meta-programming, templating, “code generation”, program transformation) convert the model into executable code, web pages, documentation, etc. More dynamic approaches convert the model into objects, modify the structure of running classes and objects, instantiate components, and so forth.

Czarnecki and Eisenecker’s book Generative Programming: Methods, Tools and Applications covers these techniques in details, mostly in the context of C++.

Artifacts → Actions

The final step is often the simplest, given that all the code/objects have been set up. These artifacts are incorporated into some program or run standalone, all we need do is run the code.

Sometimes this phase may not be as simple as this, because the generated artifacts need to be integrated into existing code. This is often difficult because the generated code may have complex interfaces which are not apparent from the DSL program. Issues of integration testing arise here, especially when the DSL program is changing.

Variations

All of the steps just described are not necessarily employed. Sometimes different phases are integrated or are totally omitted.

For example, it is often the case that when either XML, JSON or YAML is used, the program and the model coincide. Examples include various XML configuration files (often touted as DSLs), Ant, and so-called Jargons, which incidentally pre-date XML. Sometimes the model is an elaborated version of the program.

The model is often omitted when the language is simple, or provides a one-off effect. Some of the DSL-like features in Ruby on Rail’s ActiveRecord are like this. In this case, evalutating the embedded DSL program directly modifies the class structure using reflection and meta-programming.

Often no artifacts are generated, when a language can be interpreted directly into actions. Jargons adopt this approach explicitly as a part of its methodology by advocating the use of syntax-directed translators which apply to the model.

Models

A number of advantages can be argued for explicitly including a model in a DSL implementation, just as different but related advantages to separating the program syntax and the model exist.

The advantage of using a separate language syntax, as opposed to using a representation of the model directly, is that languages can be designed to more concisely reflect domain knowledge, often in a manner which is familiar to domain experts. This means that the domain knowledge embedded in the program is directly accessible to and modifiable by the domain experts.

The advantage of using a model is the domain knowledge encoded in the model can be more readily reused. For example, from a PADS description, not only is a parser generated, but also a statistics collection engine and other tools. Without a model, these tools would need to derive directly from the language, which typically would clutter the language processor’s implementation. With a separate model, multiple interpreters can traverse it and produce different artifacts without affecting the initial processing phase.

The main advantage of keeping a model separate is that it enables the reuse of domain knowledge. This is important because domain knowledge is an extremely valuable asset.

Please keep commenting. Please keep sending links. Even write a little article if you want. Community support is what will help make this a great little blog.

The project is called Transformations for Abstractions and the topic it addresses lies at the intersection generative programming, wherein programs are produced via automatic transformation, and domain-specific programming, wherein application specific abstractions concepts and notations are directly supported in programming languages. The idea is to incorporate domain-specific notions in general purpose languages, and then compile them away using program transformation techniques, in particular, using the Stratego/XT toolset.

If was not otherwise employed, I’d consider applying for the position myself.

The advantages of DSLs come not without cost: the development time required to design and build a DSL may be prohibitive. The expertise required to develop DSLs is expensive as the skills required are uncommon, often requiring compiler construction skills hand in hand with comprehensive domain understanding.

Programming languages such as Ruby, Python, Haskell and LISP enable the somewhat easier construction of internal DSLs, those which embed domain constructs within a programming language. These languages often use features such as metaprogramming, macros, templates, type-classes, and so forth to implement the embedded DSL features.

DSLs have been around since the dawn of Unix, which was home to “little languages” such as make, pic, lex, yacc, troff etc. Recently, DSLs have broken into the collective consciousness otherwise known as the blogsphere. The interest within the blogsphere of developing internal DSLs in Ruby is particularly vivid (spurred by the popularity of Rails). This is no surprise, as Ruby offers both powerful metaprogramming facilities and palatable syntax, making it suitable for writing DSLs.

Here’s where the “Fistful of Languages” blog comes in. This blog aims to address all aspects of DSL development, focussing particularly on DSLs embedded in Ruby.

Dave & Tobias