1011 lines
No EOL
98 KiB
HTML
1011 lines
No EOL
98 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="en"
|
||
xmlns:og="http://ogp.me/ns#"
|
||
xmlns:fb="https://www.facebook.com/2008/fbml">
|
||
<head>
|
||
<title>Let’s Build A Simple Interpreter. Part 7: Abstract Syntax Trees - Ruslan's Blog</title>
|
||
<!-- Using the latest rendering mode for IE -->
|
||
<meta http-equiv="X-UA-Compatible" content="IE=edge">
|
||
<meta charset="utf-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||
|
||
|
||
|
||
<link rel="canonical" href="index.html">
|
||
|
||
<meta name="author" content="Ruslan Spivak" />
|
||
<meta name="description" content="As I promised you last time, today I will talk about one of the central data structures that we’ll use throughout the rest of the series, so buckle up and let’s go. Up until now, we had our interpreter and parser code mixed together and the interpreter would …" />
|
||
|
||
<meta property="og:site_name" content="Ruslan's Blog" />
|
||
<meta property="og:type" content="article"/>
|
||
<meta property="og:title" content="Let’s Build A Simple Interpreter. Part 7: Abstract Syntax Trees"/>
|
||
<meta property="og:url" content="https://ruslanspivak.com/lsbasi-part7/"/>
|
||
<meta property="og:description" content="As I promised you last time, today I will talk about one of the central data structures that we’ll use throughout the rest of the series, so buckle up and let’s go. Up until now, we had our interpreter and parser code mixed together and the interpreter would …"/>
|
||
<meta property="article:published_time" content="2015-12-15" />
|
||
<meta property="article:section" content="blog" />
|
||
<meta property="article:author" content="Ruslan Spivak" />
|
||
|
||
<meta name="twitter:card" content="summary">
|
||
<meta name="twitter:domain" content="https://ruslanspivak.com">
|
||
|
||
<!-- Bootstrap -->
|
||
<link rel="stylesheet" href="../theme/css/bootstrap.min.css" type="text/css"/>
|
||
<link href="../theme/css/font-awesome.min.css" rel="stylesheet">
|
||
|
||
<link href="../theme/css/pygments/tango.css" rel="stylesheet">
|
||
<link href="../theme/css/typogrify.css" rel="stylesheet">
|
||
<link rel="stylesheet" href="../theme/css/style.css" type="text/css"/>
|
||
<link href="../static/custom.css" rel="stylesheet">
|
||
|
||
<link href="../feeds/all.atom.xml" type="application/atom+xml" rel="alternate"
|
||
title="Ruslan's Blog ATOM Feed"/>
|
||
|
||
</head>
|
||
<body>
|
||
|
||
<div class="navbar navbar-default navbar-fixed-top" role="navigation">
|
||
<div class="container">
|
||
<div class="navbar-header">
|
||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-ex1-collapse">
|
||
<span class="sr-only">Toggle navigation</span>
|
||
<span class="icon-bar"></span>
|
||
<span class="icon-bar"></span>
|
||
<span class="icon-bar"></span>
|
||
</button>
|
||
<a href="../index.html" class="navbar-brand">
|
||
Ruslan's Blog </a>
|
||
</div>
|
||
<div class="collapse navbar-collapse navbar-ex1-collapse">
|
||
<ul class="nav navbar-nav">
|
||
</ul>
|
||
<ul class="nav navbar-nav navbar-right">
|
||
<li><a href="../pages/about.html"><i class="fa fa-question"></i><span class="icon-label">About</span></a></li>
|
||
<li><a href="../archives.html"><i class="fa fa-th-list"></i><span class="icon-label">Archives</span></a></li>
|
||
</ul>
|
||
</div>
|
||
<!-- /.navbar-collapse -->
|
||
</div>
|
||
</div> <!-- /.navbar -->
|
||
<!-- Banner -->
|
||
<!-- End Banner -->
|
||
<div class="container">
|
||
<div class="row">
|
||
<div class="col-sm-9">
|
||
|
||
<section id="content">
|
||
<article>
|
||
<header class="page-header">
|
||
<h1>
|
||
<a href="index.html"
|
||
rel="bookmark"
|
||
title="Permalink to Let’s Build A Simple Interpreter. Part 7: Abstract Syntax Trees">
|
||
Let’s Build A Simple Interpreter. Part 7: Abstract Syntax Trees
|
||
</a>
|
||
</h1>
|
||
</header>
|
||
<div class="entry-content">
|
||
<div class="panel">
|
||
<div class="panel-body">
|
||
<footer class="post-info">
|
||
<span class="label label-default">Date</span>
|
||
<span class="published">
|
||
<i class="fa fa-calendar"></i><time datetime="2015-12-15T07:00:00-05:00"> Tue, December 15, 2015</time>
|
||
</span>
|
||
|
||
|
||
|
||
|
||
</footer><!-- /.post-info --> </div>
|
||
</div>
|
||
<p>As I promised you last time, today I will talk about one of the central
|
||
data structures that we’ll use throughout the rest of the series, so buckle
|
||
up and let’s go.</p>
|
||
<p>Up until now, we had our interpreter and parser code mixed together and
|
||
the interpreter would evaluate an expression as soon as the parser recognized
|
||
a certain language construct like addition, subtraction, multiplication, or division.
|
||
Such interpreters are called <em>syntax-directed interpreters</em>. They usually make a single
|
||
pass over the input and are suitable for basic language applications.
|
||
In order to analyze more complex Pascal programming language constructs, we need to build
|
||
an <em>intermediate representation</em> (<em><span class="caps">IR</span></em>). Our parser will be responsible for building an
|
||
<em><span class="caps">IR</span></em> and our interpreter will use it to interpret the input represented as the <em><span class="caps">IR</span></em>.</p>
|
||
<p>It turns out that a tree is a very suitable data structure for an <span class="caps">IR</span>.</p>
|
||
<p><img alt="" src="lsbasi_part7_realtree.png" width="500"></p>
|
||
<p>Let’s quickly talk about tree terminology.</p>
|
||
<ul>
|
||
<li>A <em>tree</em> is a data structure that consists of one or more nodes organized into a hierarchy.</li>
|
||
<li>The tree has one <em>root</em>, which is the top node.</li>
|
||
<li>All nodes except the root have a unique <em>parent</em>.</li>
|
||
<li>The node labeled <strong>*</strong> in the picture below is a <em>parent</em>. Nodes labeled <strong>2</strong> and <strong>7</strong> are its <em>children</em>; children are ordered from left to right.</li>
|
||
<li>A node with no children is called a <em>leaf</em> node.</li>
|
||
<li>A node that has one or more children and that is not the root is called an <em>interior</em> node.</li>
|
||
<li>The children can also be complete <em>subtrees</em>. In the picture below the left child (labeled <strong>*</strong>) of the <strong>+</strong> node is a complete <em>subtree</em> with its own children.</li>
|
||
<li>In computer science we draw trees upside down starting with the root node at the top and branches growing downward.</li>
|
||
</ul>
|
||
<p>Here is a tree for the expression 2 * 7 + 3 with explanations:</p>
|
||
<p><img alt="" src="lsbasi_part7_tree_terminology.png" width="640"></p>
|
||
<p>The <span class="caps">IR</span> we’ll use throughout the series is called an <em>abstract-syntax tree</em> (<em><span class="caps">AST</span></em>).
|
||
But before we dig deeper into ASTs let’s talk about <em>parse trees</em> briefly.
|
||
Though we’re not going to use parse trees for our interpreter and compiler, they can help
|
||
you understand how your parser interpreted the input by visualizing the execution trace
|
||
of the parser. We’ll also compare them with ASTs to see why ASTs are better suited for
|
||
intermediate representation than parse trees.</p>
|
||
<p>So, what is a parse tree? A <em>parse-tree</em> (sometimes called a <em>concrete syntax tree</em>) is a tree
|
||
that represents the syntactic structure of a language construct according to our grammar
|
||
definition. It basically shows how your parser recognized the language construct or, in other
|
||
words, it shows how the start symbol of your grammar derives a certain string in the
|
||
programming language.</p>
|
||
<p>The call stack of the parser implicitly represents a parse tree and it’s automatically
|
||
built in memory by your parser as it is trying to recognize a certain language construct.</p>
|
||
<p>Let’s take a look at a parse tree for the expression 2 * 7 + 3:</p>
|
||
<p><img alt="" src="lsbasi_part7_parsetree_01.png" width="520"></p>
|
||
<p>In the picture above you can see that:</p>
|
||
<ul>
|
||
<li>The parse tree records a sequence of rules the parser applies to recognize the input.</li>
|
||
<li>The root of the parse tree is labeled with the grammar start symbol.</li>
|
||
<li>Each interior node represents a non-terminal, that is it represents a grammar rule
|
||
application, like <em>expr</em>, <em>term</em>, or <em>factor</em> in our case.</li>
|
||
<li>Each leaf node represents a token.</li>
|
||
</ul>
|
||
<p>As I’ve already mentioned, we’re not going to manually construct parser trees and use them
|
||
for our interpreter but parse trees can help you understand how the parser interpreted
|
||
the input by visualizing the parser call sequence.</p>
|
||
<p>You can see how parse trees look like for different arithmetic expressions by trying out
|
||
a small utility called <a href="https://github.com/rspivak/lsbasi/blob/master/part7/python/genptdot.py">genptdot.py</a> that I quickly wrote to help you visualize them. To use the utility you first need to install <a href="http://graphviz.org">Graphviz</a> package and after you’ve run the following command, you can open the generated image file parsetree.png and see a parse tree for the expression you passed as a command line argument:</p>
|
||
<div class="highlight"><pre><span></span>$ python genptdot.py <span class="s2">"14 + 2 * 3 - 6 / 2"</span> > <span class="se">\</span>
|
||
parsetree.dot <span class="o">&&</span> dot -Tpng -o parsetree.png parsetree.dot
|
||
</pre></div>
|
||
|
||
|
||
<p>Here is the generated image parsetree.png for the expression 14 + 2 * 3 - 6 / 2:</p>
|
||
<p><img alt="" src="lsbasi_part7_genptdot_01.png"></p>
|
||
<p>Play with the utility a bit by passing it different arithmetic expressions and see what a parse tree looks like for a particular expression.</p>
|
||
<p>Now, let’s talk about <em>abstract-syntax trees</em> (<span class="caps">AST</span>). This is the <em>intermediate representation</em> (<span class="caps">IR</span>) that we’ll heavily use throughout the rest of the series. It is one of the central data structures for our interpreter and future compiler projects.</p>
|
||
<p>Let’s start our discussion by taking a look at both the <span class="caps">AST</span> and the parse tree for the expression 2 * 7 + 3:</p>
|
||
<p><img alt="" src="lsbasi_part7_ast_01.png" width="640"></p>
|
||
<p>As you can see from the picture above, the <span class="caps">AST</span> captures the essence of the input while being smaller.</p>
|
||
<p>Here are the main differences between ASTs and Parse trees:</p>
|
||
<ul>
|
||
<li>ASTs uses operators/operations as root and interior nodes and it uses operands as their children.</li>
|
||
<li>ASTs do not use interior nodes to represent a grammar rule, unlike the parse tree does.</li>
|
||
<li>ASTs don’t represent every detail from the real syntax (that’s why they’re called <em>abstract</em>) - no rule nodes and no parentheses, for example.</li>
|
||
<li>ASTs are dense compared to a parse tree for the same language construct.</li>
|
||
</ul>
|
||
<p>So, what is an abstract syntax tree?
|
||
An <em>abstract syntax tree</em> (<em><span class="caps">AST</span></em>) is a tree that represents the abstract syntactic structure of a language construct where each interior node and the root node represents an operator, and the children of the node represent the operands of that operator.</p>
|
||
<p>I’ve already mentioned that ASTs are more compact than parse trees. Let’s take a look at an <span class="caps">AST</span> and a parse tree for the expression 7 + ((2 + 3)). You can see that the following <span class="caps">AST</span> is much smaller than the parse tree, but still captures the essence of the input:</p>
|
||
<p><img alt="" src="lsbasi_part7_ast_02.png"></p>
|
||
<p>So far so good, but how do you encode operator precedence in an <span class="caps">AST</span>? In order to encode the operator precedence in <span class="caps">AST</span>, that is, to represent that “X happens before Y” you just need to put X lower in the tree than Y. And you’ve already seen that in the previous pictures.</p>
|
||
<p>Let’s take a look at some more examples.</p>
|
||
<p>In the picture below, on the left, you can see an <span class="caps">AST</span> for the expression 2 * 7 + 3. Let’s change the precedence by putting 7 + 3 inside the parentheses. You can see, on the right, what an <span class="caps">AST</span> looks like for the modified expression 2 * (7 + 3):</p>
|
||
<p><img alt="" src="lsbasi_part7_astprecedence_01.png" width="640"></p>
|
||
<p>Here is an <span class="caps">AST</span> for the expression 1 + 2 + 3 + 4 + 5:</p>
|
||
<p><img alt="" src="lsbasi_part7_astprecedence_02.png" width="480"></p>
|
||
<p>From the pictures above you can see that operators with higher precedence end up being lower in the tree.</p>
|
||
<p>Okay, let’s write some code to implement different <span class="caps">AST</span> node types and modify our parser to generate an <span class="caps">AST</span> tree composed of those nodes.</p>
|
||
<p>First, we’ll create a base node class called <span class="caps">AST</span> that other classes will inherit from:</p>
|
||
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">AST</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
|
||
<span class="k">pass</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Not much there, actually. Recall that ASTs represent the operator-operand model. So far, we have four operators and integer operands. The operators are addition, subtraction, multiplication, and division. We could have created a separate class to represent each operator like AddNode, SubNode, MulNode, and DivNode, but instead we’re going to have only one <em>BinOp</em> class to represent all four binary operators (a <em>binary operator</em> is an operator that operates on two operands):</p>
|
||
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">BinOp</span><span class="p">(</span><span class="n">AST</span><span class="p">):</span>
|
||
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">left</span><span class="p">,</span> <span class="n">op</span><span class="p">,</span> <span class="n">right</span><span class="p">):</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">left</span> <span class="o">=</span> <span class="n">left</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">token</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">op</span> <span class="o">=</span> <span class="n">op</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">right</span> <span class="o">=</span> <span class="n">right</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>The parameters to the constructor are <em>left</em>, <em>op</em>, and <em>right</em>, where <em>left</em> and <em>right</em> point correspondingly to the node of the left operand and to the node of the right operand. <em>Op</em> holds a token for the operator itself: Token(<span class="caps">PLUS</span>, ‘+’) for the plus operator, Token(<span class="caps">MINUS</span>, ‘-‘) for the minus operator, and so on.</p>
|
||
<p>To represent integers in our <span class="caps">AST</span>, we’ll define a class <em>Num</em> that will hold an <span class="caps">INTEGER</span> token and the token’s value:</p>
|
||
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Num</span><span class="p">(</span><span class="n">AST</span><span class="p">):</span>
|
||
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">token</span><span class="p">):</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">token</span> <span class="o">=</span> <span class="n">token</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">value</span> <span class="o">=</span> <span class="n">token</span><span class="o">.</span><span class="n">value</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>As you’ve noticed, all nodes store the token used to create the node. This is mostly for convenience and it will come in handy in the future.</p>
|
||
<p>Recall the <span class="caps">AST</span> for the expression 2 * 7 + 3. We’re going to manually create it in code for that expression:</p>
|
||
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="kn">from</span> <span class="nn">spi</span> <span class="kn">import</span> <span class="n">Token</span><span class="p">,</span> <span class="n">MUL</span><span class="p">,</span> <span class="n">PLUS</span><span class="p">,</span> <span class="n">INTEGER</span><span class="p">,</span> <span class="n">Num</span><span class="p">,</span> <span class="n">BinOp</span>
|
||
<span class="o">>>></span>
|
||
<span class="o">>>></span> <span class="n">mul_token</span> <span class="o">=</span> <span class="n">Token</span><span class="p">(</span><span class="n">MUL</span><span class="p">,</span> <span class="s1">'*'</span><span class="p">)</span>
|
||
<span class="o">>>></span> <span class="n">plus_token</span> <span class="o">=</span> <span class="n">Token</span><span class="p">(</span><span class="n">PLUS</span><span class="p">,</span> <span class="s1">'+'</span><span class="p">)</span>
|
||
<span class="o">>>></span> <span class="n">mul_node</span> <span class="o">=</span> <span class="n">BinOp</span><span class="p">(</span>
|
||
<span class="o">...</span> <span class="n">left</span><span class="o">=</span><span class="n">Num</span><span class="p">(</span><span class="n">Token</span><span class="p">(</span><span class="n">INTEGER</span><span class="p">,</span> <span class="mi">2</span><span class="p">)),</span>
|
||
<span class="o">...</span> <span class="n">op</span><span class="o">=</span><span class="n">mul_token</span><span class="p">,</span>
|
||
<span class="o">...</span> <span class="n">right</span><span class="o">=</span><span class="n">Num</span><span class="p">(</span><span class="n">Token</span><span class="p">(</span><span class="n">INTEGER</span><span class="p">,</span> <span class="mi">7</span><span class="p">))</span>
|
||
<span class="o">...</span> <span class="p">)</span>
|
||
<span class="o">>>></span> <span class="n">add_node</span> <span class="o">=</span> <span class="n">BinOp</span><span class="p">(</span>
|
||
<span class="o">...</span> <span class="n">left</span><span class="o">=</span><span class="n">mul_node</span><span class="p">,</span>
|
||
<span class="o">...</span> <span class="n">op</span><span class="o">=</span><span class="n">plus_token</span><span class="p">,</span>
|
||
<span class="o">...</span> <span class="n">right</span><span class="o">=</span><span class="n">Num</span><span class="p">(</span><span class="n">Token</span><span class="p">(</span><span class="n">INTEGER</span><span class="p">,</span> <span class="mi">3</span><span class="p">))</span>
|
||
<span class="o">...</span> <span class="p">)</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Here is how an <span class="caps">AST</span> will look with our new node classes defined. The picture below also follows the manual construction process above:</p>
|
||
<p><img alt="" src="lsbasi_part7_astimpl_01.png" width="600"></p>
|
||
<p>Here is our modified parser code that builds and returns an <span class="caps">AST</span> as a result of recognizing the input (an arithmetic expression):</p>
|
||
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">AST</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
|
||
<span class="k">pass</span>
|
||
|
||
|
||
<span class="k">class</span> <span class="nc">BinOp</span><span class="p">(</span><span class="n">AST</span><span class="p">):</span>
|
||
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">left</span><span class="p">,</span> <span class="n">op</span><span class="p">,</span> <span class="n">right</span><span class="p">):</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">left</span> <span class="o">=</span> <span class="n">left</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">token</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">op</span> <span class="o">=</span> <span class="n">op</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">right</span> <span class="o">=</span> <span class="n">right</span>
|
||
|
||
|
||
<span class="k">class</span> <span class="nc">Num</span><span class="p">(</span><span class="n">AST</span><span class="p">):</span>
|
||
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">token</span><span class="p">):</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">token</span> <span class="o">=</span> <span class="n">token</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">value</span> <span class="o">=</span> <span class="n">token</span><span class="o">.</span><span class="n">value</span>
|
||
|
||
|
||
<span class="k">class</span> <span class="nc">Parser</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
|
||
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">lexer</span><span class="p">):</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">lexer</span> <span class="o">=</span> <span class="n">lexer</span>
|
||
<span class="c1"># set current token to the first token taken from the input</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">current_token</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">lexer</span><span class="o">.</span><span class="n">get_next_token</span><span class="p">()</span>
|
||
|
||
<span class="k">def</span> <span class="nf">error</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span><span class="s1">'Invalid syntax'</span><span class="p">)</span>
|
||
|
||
<span class="k">def</span> <span class="nf">eat</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">token_type</span><span class="p">):</span>
|
||
<span class="c1"># compare the current token type with the passed token</span>
|
||
<span class="c1"># type and if they match then "eat" the current token</span>
|
||
<span class="c1"># and assign the next token to the self.current_token,</span>
|
||
<span class="c1"># otherwise raise an exception.</span>
|
||
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_token</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">token_type</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">current_token</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">lexer</span><span class="o">.</span><span class="n">get_next_token</span><span class="p">()</span>
|
||
<span class="k">else</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">error</span><span class="p">()</span>
|
||
|
||
<span class="k">def</span> <span class="nf">factor</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="sd">"""factor : INTEGER | LPAREN expr RPAREN"""</span>
|
||
<span class="n">token</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_token</span>
|
||
<span class="k">if</span> <span class="n">token</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">INTEGER</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">eat</span><span class="p">(</span><span class="n">INTEGER</span><span class="p">)</span>
|
||
<span class="k">return</span> <span class="n">Num</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
|
||
<span class="k">elif</span> <span class="n">token</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">LPAREN</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">eat</span><span class="p">(</span><span class="n">LPAREN</span><span class="p">)</span>
|
||
<span class="n">node</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">expr</span><span class="p">()</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">eat</span><span class="p">(</span><span class="n">RPAREN</span><span class="p">)</span>
|
||
<span class="k">return</span> <span class="n">node</span>
|
||
|
||
<span class="k">def</span> <span class="nf">term</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="sd">"""term : factor ((MUL | DIV) factor)*"""</span>
|
||
<span class="n">node</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">factor</span><span class="p">()</span>
|
||
|
||
<span class="k">while</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_token</span><span class="o">.</span><span class="n">type</span> <span class="ow">in</span> <span class="p">(</span><span class="n">MUL</span><span class="p">,</span> <span class="n">DIV</span><span class="p">):</span>
|
||
<span class="n">token</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_token</span>
|
||
<span class="k">if</span> <span class="n">token</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">MUL</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">eat</span><span class="p">(</span><span class="n">MUL</span><span class="p">)</span>
|
||
<span class="k">elif</span> <span class="n">token</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">DIV</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">eat</span><span class="p">(</span><span class="n">DIV</span><span class="p">)</span>
|
||
|
||
<span class="n">node</span> <span class="o">=</span> <span class="n">BinOp</span><span class="p">(</span><span class="n">left</span><span class="o">=</span><span class="n">node</span><span class="p">,</span> <span class="n">op</span><span class="o">=</span><span class="n">token</span><span class="p">,</span> <span class="n">right</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">factor</span><span class="p">())</span>
|
||
|
||
<span class="k">return</span> <span class="n">node</span>
|
||
|
||
<span class="k">def</span> <span class="nf">expr</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="sd">"""</span>
|
||
<span class="sd"> expr : term ((PLUS | MINUS) term)*</span>
|
||
<span class="sd"> term : factor ((MUL | DIV) factor)*</span>
|
||
<span class="sd"> factor : INTEGER | LPAREN expr RPAREN</span>
|
||
<span class="sd"> """</span>
|
||
<span class="n">node</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">term</span><span class="p">()</span>
|
||
|
||
<span class="k">while</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_token</span><span class="o">.</span><span class="n">type</span> <span class="ow">in</span> <span class="p">(</span><span class="n">PLUS</span><span class="p">,</span> <span class="n">MINUS</span><span class="p">):</span>
|
||
<span class="n">token</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_token</span>
|
||
<span class="k">if</span> <span class="n">token</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">PLUS</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">eat</span><span class="p">(</span><span class="n">PLUS</span><span class="p">)</span>
|
||
<span class="k">elif</span> <span class="n">token</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">MINUS</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">eat</span><span class="p">(</span><span class="n">MINUS</span><span class="p">)</span>
|
||
|
||
<span class="n">node</span> <span class="o">=</span> <span class="n">BinOp</span><span class="p">(</span><span class="n">left</span><span class="o">=</span><span class="n">node</span><span class="p">,</span> <span class="n">op</span><span class="o">=</span><span class="n">token</span><span class="p">,</span> <span class="n">right</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">term</span><span class="p">())</span>
|
||
|
||
<span class="k">return</span> <span class="n">node</span>
|
||
|
||
<span class="k">def</span> <span class="nf">parse</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">expr</span><span class="p">()</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Let’s go over the process of an <span class="caps">AST</span> construction for some arithmetic expressions.</p>
|
||
<p>If you look at the parser code above you can see that the way it builds nodes of an <span class="caps">AST</span> is that each BinOp node adopts the current value of the <em>node</em> variable as its left child and the result of a call to a <em>term</em> or <em>factor</em> as its right child, so it’s effectively pushing down nodes to the left and the tree for the expression 1 +2 + 3 + 4 + 5 below is a good example of that. Here is a visual representation how the parser gradually builds an <span class="caps">AST</span> for the expression 1 + 2 + 3 + 4 + 5:</p>
|
||
<p><img alt="" src="lsbasi_part7_astimpl_02.png" width="780"></p>
|
||
<p>To help you visualize ASTs for different arithmetic expressions, I wrote a small utility that takes an arithmetic expression as its first argument and generates a <span class="caps">DOT</span> file that is then processed by the <em>dot</em> utility to actually draw an <span class="caps">AST</span> for you (<em>dot</em> is part of the <a href="http://graphviz.org">Graphviz</a> package that you need to install to run the <em>dot</em> command). Here is a command and a generated <span class="caps">AST</span> image for the expression 7 + 3 * (10 / (12 / (3 + 1) - 1)):</p>
|
||
<div class="highlight"><pre><span></span>$ python genastdot.py <span class="s2">"7 + 3 * (10 / (12 / (3 + 1) - 1))"</span> > <span class="se">\</span>
|
||
ast.dot <span class="o">&&</span> dot -Tpng -o ast.png ast.dot
|
||
</pre></div>
|
||
|
||
|
||
<p><img alt="" src="lsbasi_part7_genastdot_01.png"></p>
|
||
<p>It’s worth your while to write some arithmetic expressions, manually draw ASTs for the expressions, and then verify them by generating <span class="caps">AST</span> images for the same expressions with the <a href="https://github.com/rspivak/lsbasi/blob/master/part7/python/genastdot.py">genastdot.py</a> tool. That will help you better understand how ASTs are constructed by the parser for different arithmetic expressions.</p>
|
||
<p>Okay, here is an <span class="caps">AST</span> for the expression 2 * 7 + 3:</p>
|
||
<p><img alt="" src="lsbasi_part7_ast_walking_01.png" width="360"></p>
|
||
<p>How do you navigate the tree to properly evaluate the expression represented by that tree? You do that by using a <em>postorder traversal</em> - a special case of <em>depth-first traversal</em> - which starts at the root node and recursively visits the children of each node from left to right. The postorder traversal visits nodes as far away from the root as fast as it can.</p>
|
||
<p>Here is a pseudo code for the postorder traversal where <em><<postorder actions>></em> is a placeholder for actions like addition, subtraction, multiplication, or division for a <em>BinOp</em> node or a simpler action like returning the integer value of a <em>Num</em> node:</p>
|
||
<p><img alt="" src="lsbasi_part7_ast_visit_postorder.png" width="640"></p>
|
||
<p>The reason we’re going to use a postorder traversal for our interpreter is that first, we need to evaluate interior nodes lower in the tree because they represent operators with higher precedence and second, we need to evaluate operands of an operator before applying the operator to those operands. In the picture below, you can see that with postorder traversal we first evaluate the expression 2 * 7 and only after that we evaluate 14 + 3, which gives us the correct result, 17:</p>
|
||
<p><img alt="" src="lsbasi_part7_ast_walking_02.png" width="360"></p>
|
||
<p>For the sake of completeness, I’ll mention that there are three types of depth-first traversal: <em>preorder traversal</em>, <em>inorder traversal</em>, and <em>postorder traversal</em>. The name of the traversal method comes from the place where you put actions in the visitation code:</p>
|
||
<p><img alt="" src="lsbasi_part7_ast_visit_generic.png" width="560"></p>
|
||
<p>Sometimes you might have to execute certain actions at all those points (preorder, inorder, and postorder). You’ll see some examples of that in the source code repository for this article.</p>
|
||
<p>Okay, let’s write some code to visit and interpret the abstract syntax trees built by our parser, shall we?</p>
|
||
<p>Here is the source code that implements the <a href="https://en.wikipedia.org/wiki/Visitor_pattern">Visitor pattern</a>:</p>
|
||
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">NodeVisitor</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
|
||
<span class="k">def</span> <span class="nf">visit</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">node</span><span class="p">):</span>
|
||
<span class="n">method_name</span> <span class="o">=</span> <span class="s1">'visit_'</span> <span class="o">+</span> <span class="nb">type</span><span class="p">(</span><span class="n">node</span><span class="p">)</span><span class="o">.</span><span class="vm">__name__</span>
|
||
<span class="n">visitor</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">method_name</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">generic_visit</span><span class="p">)</span>
|
||
<span class="k">return</span> <span class="n">visitor</span><span class="p">(</span><span class="n">node</span><span class="p">)</span>
|
||
|
||
<span class="k">def</span> <span class="nf">generic_visit</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">node</span><span class="p">):</span>
|
||
<span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span><span class="s1">'No visit_{} method'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">type</span><span class="p">(</span><span class="n">node</span><span class="p">)</span><span class="o">.</span><span class="vm">__name__</span><span class="p">))</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>And here is the source code of our <em>Interpreter</em> class that inherits from the <em>NodeVisitor</em> class and implements different methods that have the form <em>visit_NodeType</em>, where <em>NodeType</em> is replaced with the node’s class name like <em>BinOp</em>, <em>Num</em> and so on:</p>
|
||
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Interpreter</span><span class="p">(</span><span class="n">NodeVisitor</span><span class="p">):</span>
|
||
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">parser</span><span class="p">):</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">parser</span> <span class="o">=</span> <span class="n">parser</span>
|
||
|
||
<span class="k">def</span> <span class="nf">visit_BinOp</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">node</span><span class="p">):</span>
|
||
<span class="k">if</span> <span class="n">node</span><span class="o">.</span><span class="n">op</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">PLUS</span><span class="p">:</span>
|
||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">left</span><span class="p">)</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">right</span><span class="p">)</span>
|
||
<span class="k">elif</span> <span class="n">node</span><span class="o">.</span><span class="n">op</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">MINUS</span><span class="p">:</span>
|
||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">left</span><span class="p">)</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">right</span><span class="p">)</span>
|
||
<span class="k">elif</span> <span class="n">node</span><span class="o">.</span><span class="n">op</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">MUL</span><span class="p">:</span>
|
||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">left</span><span class="p">)</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">right</span><span class="p">)</span>
|
||
<span class="k">elif</span> <span class="n">node</span><span class="o">.</span><span class="n">op</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">DIV</span><span class="p">:</span>
|
||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">left</span><span class="p">)</span> <span class="o">/</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">right</span><span class="p">)</span>
|
||
|
||
<span class="k">def</span> <span class="nf">visit_Num</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">node</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="n">node</span><span class="o">.</span><span class="n">value</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>There are two interesting things about the code that are worth mentioning here:
|
||
First, the visitor code that manipulates <span class="caps">AST</span> nodes is decoupled from the <span class="caps">AST</span> nodes themselves. You can see that none of the <span class="caps">AST</span> node classes (BinOp and Num) provide any code to manipulate the data stored in those nodes. That logic is encapsulated in the <em>Interpreter</em> class that implements the <em>NodeVisitor</em> class.</p>
|
||
<p>Second, instead of a giant <em>if</em> statement in the NodeVisitor’s <em>visit</em> method like this:</p>
|
||
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">visit</span><span class="p">(</span><span class="n">node</span><span class="p">):</span>
|
||
<span class="n">node_type</span> <span class="o">=</span> <span class="nb">type</span><span class="p">(</span><span class="n">node</span><span class="p">)</span><span class="o">.</span><span class="vm">__name__</span>
|
||
<span class="k">if</span> <span class="n">node_type</span> <span class="o">==</span> <span class="s1">'BinOp'</span><span class="p">:</span>
|
||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit_BinOp</span><span class="p">(</span><span class="n">node</span><span class="p">)</span>
|
||
<span class="k">elif</span> <span class="n">node_type</span> <span class="o">==</span> <span class="s1">'Num'</span><span class="p">:</span>
|
||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit_Num</span><span class="p">(</span><span class="n">node</span><span class="p">)</span>
|
||
<span class="k">elif</span> <span class="o">...</span>
|
||
<span class="c1"># ...</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>or like this:</p>
|
||
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">visit</span><span class="p">(</span><span class="n">node</span><span class="p">):</span>
|
||
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">BinOp</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit_BinOp</span><span class="p">(</span><span class="n">node</span><span class="p">)</span>
|
||
<span class="k">elif</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">node</span><span class="p">,</span> <span class="n">Num</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit_Num</span><span class="p">(</span><span class="n">node</span><span class="p">)</span>
|
||
<span class="k">elif</span> <span class="o">...</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>the NodeVisitor’s <em>visit</em> method is very generic and dispatches calls to the appropriate method based on the node type passed to it. As I’ve mentioned before, in order to make use of it, our interpreter inherits from the <em>NodeVisitor</em> class and implements necessary methods. So if the type of a node passed to the <em>visit</em> method is BinOp, then the <em>visit</em> method will dispatch the call to the <em>visit_BinOp</em> method, and if the type of a node is Num, then the <em>visit</em> method will dispatch the call to the <em>visit_Num</em> method, and so on.</p>
|
||
<p>Spend some time studying this approach (standard Python module <a href="https://docs.python.org/2.7/library/ast.html#module-ast">ast</a> uses the same mechanism for node traversal) as we will be extending our interpreter with many new <em>visit_NodeType</em> methods in the future.</p>
|
||
<p>The <em>generic_visit</em> method is a fallback that raises an exception to indicate that it encountered a node that the implementation class has no corresponding <em>visit_NodeType</em> method for.</p>
|
||
<p>Now, let’s manually build an <span class="caps">AST</span> for the expression 2 * 7 + 3 and pass it to our interpreter to see the visit method in action to evaluate the expression. Here is how you can do it from the Python shell:</p>
|
||
<div class="highlight"><pre><span></span><span class="o">>>></span> <span class="kn">from</span> <span class="nn">spi</span> <span class="kn">import</span> <span class="n">Token</span><span class="p">,</span> <span class="n">MUL</span><span class="p">,</span> <span class="n">PLUS</span><span class="p">,</span> <span class="n">INTEGER</span><span class="p">,</span> <span class="n">Num</span><span class="p">,</span> <span class="n">BinOp</span>
|
||
<span class="o">>>></span>
|
||
<span class="o">>>></span> <span class="n">mul_token</span> <span class="o">=</span> <span class="n">Token</span><span class="p">(</span><span class="n">MUL</span><span class="p">,</span> <span class="s1">'*'</span><span class="p">)</span>
|
||
<span class="o">>>></span> <span class="n">plus_token</span> <span class="o">=</span> <span class="n">Token</span><span class="p">(</span><span class="n">PLUS</span><span class="p">,</span> <span class="s1">'+'</span><span class="p">)</span>
|
||
<span class="o">>>></span> <span class="n">mul_node</span> <span class="o">=</span> <span class="n">BinOp</span><span class="p">(</span>
|
||
<span class="o">...</span> <span class="n">left</span><span class="o">=</span><span class="n">Num</span><span class="p">(</span><span class="n">Token</span><span class="p">(</span><span class="n">INTEGER</span><span class="p">,</span> <span class="mi">2</span><span class="p">)),</span>
|
||
<span class="o">...</span> <span class="n">op</span><span class="o">=</span><span class="n">mul_token</span><span class="p">,</span>
|
||
<span class="o">...</span> <span class="n">right</span><span class="o">=</span><span class="n">Num</span><span class="p">(</span><span class="n">Token</span><span class="p">(</span><span class="n">INTEGER</span><span class="p">,</span> <span class="mi">7</span><span class="p">))</span>
|
||
<span class="o">...</span> <span class="p">)</span>
|
||
<span class="o">>>></span> <span class="n">add_node</span> <span class="o">=</span> <span class="n">BinOp</span><span class="p">(</span>
|
||
<span class="o">...</span> <span class="n">left</span><span class="o">=</span><span class="n">mul_node</span><span class="p">,</span>
|
||
<span class="o">...</span> <span class="n">op</span><span class="o">=</span><span class="n">plus_token</span><span class="p">,</span>
|
||
<span class="o">...</span> <span class="n">right</span><span class="o">=</span><span class="n">Num</span><span class="p">(</span><span class="n">Token</span><span class="p">(</span><span class="n">INTEGER</span><span class="p">,</span> <span class="mi">3</span><span class="p">))</span>
|
||
<span class="o">...</span> <span class="p">)</span>
|
||
<span class="o">>>></span> <span class="kn">from</span> <span class="nn">spi</span> <span class="kn">import</span> <span class="n">Interpreter</span>
|
||
<span class="o">>>></span> <span class="n">inter</span> <span class="o">=</span> <span class="n">Interpreter</span><span class="p">(</span><span class="bp">None</span><span class="p">)</span>
|
||
<span class="o">>>></span> <span class="n">inter</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">add_node</span><span class="p">)</span>
|
||
<span class="mi">17</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>As you can see, I passed the root of the expression tree to the <em>visit</em> method and that triggered traversal of the tree by dispatching calls to the correct methods of the <em>Interpreter</em> class(<em>visit_BinOp</em> and <em>visit_Num</em>) and generating the result.</p>
|
||
<p>Okay, here is the complete code of our new interpreter for your convenience:</p>
|
||
<div class="highlight"><pre><span></span><span class="sd">""" SPI - Simple Pascal Interpreter """</span>
|
||
|
||
<span class="c1">###############################################################################</span>
|
||
<span class="c1"># #</span>
|
||
<span class="c1"># LEXER #</span>
|
||
<span class="c1"># #</span>
|
||
<span class="c1">###############################################################################</span>
|
||
|
||
<span class="c1"># Token types</span>
|
||
<span class="c1">#</span>
|
||
<span class="c1"># EOF (end-of-file) token is used to indicate that</span>
|
||
<span class="c1"># there is no more input left for lexical analysis</span>
|
||
<span class="n">INTEGER</span><span class="p">,</span> <span class="n">PLUS</span><span class="p">,</span> <span class="n">MINUS</span><span class="p">,</span> <span class="n">MUL</span><span class="p">,</span> <span class="n">DIV</span><span class="p">,</span> <span class="n">LPAREN</span><span class="p">,</span> <span class="n">RPAREN</span><span class="p">,</span> <span class="n">EOF</span> <span class="o">=</span> <span class="p">(</span>
|
||
<span class="s1">'INTEGER'</span><span class="p">,</span> <span class="s1">'PLUS'</span><span class="p">,</span> <span class="s1">'MINUS'</span><span class="p">,</span> <span class="s1">'MUL'</span><span class="p">,</span> <span class="s1">'DIV'</span><span class="p">,</span> <span class="s1">'('</span><span class="p">,</span> <span class="s1">')'</span><span class="p">,</span> <span class="s1">'EOF'</span>
|
||
<span class="p">)</span>
|
||
|
||
|
||
<span class="k">class</span> <span class="nc">Token</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
|
||
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">type</span><span class="p">,</span> <span class="n">value</span><span class="p">):</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">type</span> <span class="o">=</span> <span class="nb">type</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">value</span> <span class="o">=</span> <span class="n">value</span>
|
||
|
||
<span class="k">def</span> <span class="fm">__str__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="sd">"""String representation of the class instance.</span>
|
||
|
||
<span class="sd"> Examples:</span>
|
||
<span class="sd"> Token(INTEGER, 3)</span>
|
||
<span class="sd"> Token(PLUS, '+')</span>
|
||
<span class="sd"> Token(MUL, '*')</span>
|
||
<span class="sd"> """</span>
|
||
<span class="k">return</span> <span class="s1">'Token({type}, {value})'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span>
|
||
<span class="nb">type</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">type</span><span class="p">,</span>
|
||
<span class="n">value</span><span class="o">=</span><span class="nb">repr</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
|
||
<span class="p">)</span>
|
||
|
||
<span class="k">def</span> <span class="fm">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="fm">__str__</span><span class="p">()</span>
|
||
|
||
|
||
<span class="k">class</span> <span class="nc">Lexer</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
|
||
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">text</span><span class="p">):</span>
|
||
<span class="c1"># client string input, e.g. "4 + 2 * 3 - 6 / 2"</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">text</span> <span class="o">=</span> <span class="n">text</span>
|
||
<span class="c1"># self.pos is an index into self.text</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">pos</span> <span class="o">=</span> <span class="mi">0</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">current_char</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">text</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">pos</span><span class="p">]</span>
|
||
|
||
<span class="k">def</span> <span class="nf">error</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span><span class="s1">'Invalid character'</span><span class="p">)</span>
|
||
|
||
<span class="k">def</span> <span class="nf">advance</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="sd">"""Advance the `pos` pointer and set the `current_char` variable."""</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">pos</span> <span class="o">+=</span> <span class="mi">1</span>
|
||
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">pos</span> <span class="o">></span> <span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">text</span><span class="p">)</span> <span class="o">-</span> <span class="mi">1</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">current_char</span> <span class="o">=</span> <span class="bp">None</span> <span class="c1"># Indicates end of input</span>
|
||
<span class="k">else</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">current_char</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">text</span><span class="p">[</span><span class="bp">self</span><span class="o">.</span><span class="n">pos</span><span class="p">]</span>
|
||
|
||
<span class="k">def</span> <span class="nf">skip_whitespace</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="k">while</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_char</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_char</span><span class="o">.</span><span class="n">isspace</span><span class="p">():</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">advance</span><span class="p">()</span>
|
||
|
||
<span class="k">def</span> <span class="nf">integer</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="sd">"""Return a (multidigit) integer consumed from the input."""</span>
|
||
<span class="n">result</span> <span class="o">=</span> <span class="s1">''</span>
|
||
<span class="k">while</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_char</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_char</span><span class="o">.</span><span class="n">isdigit</span><span class="p">():</span>
|
||
<span class="n">result</span> <span class="o">+=</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_char</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">advance</span><span class="p">()</span>
|
||
<span class="k">return</span> <span class="nb">int</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
|
||
|
||
<span class="k">def</span> <span class="nf">get_next_token</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="sd">"""Lexical analyzer (also known as scanner or tokenizer)</span>
|
||
|
||
<span class="sd"> This method is responsible for breaking a sentence</span>
|
||
<span class="sd"> apart into tokens. One token at a time.</span>
|
||
<span class="sd"> """</span>
|
||
<span class="k">while</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_char</span> <span class="ow">is</span> <span class="ow">not</span> <span class="bp">None</span><span class="p">:</span>
|
||
|
||
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_char</span><span class="o">.</span><span class="n">isspace</span><span class="p">():</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">skip_whitespace</span><span class="p">()</span>
|
||
<span class="k">continue</span>
|
||
|
||
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_char</span><span class="o">.</span><span class="n">isdigit</span><span class="p">():</span>
|
||
<span class="k">return</span> <span class="n">Token</span><span class="p">(</span><span class="n">INTEGER</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">integer</span><span class="p">())</span>
|
||
|
||
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_char</span> <span class="o">==</span> <span class="s1">'+'</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">advance</span><span class="p">()</span>
|
||
<span class="k">return</span> <span class="n">Token</span><span class="p">(</span><span class="n">PLUS</span><span class="p">,</span> <span class="s1">'+'</span><span class="p">)</span>
|
||
|
||
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_char</span> <span class="o">==</span> <span class="s1">'-'</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">advance</span><span class="p">()</span>
|
||
<span class="k">return</span> <span class="n">Token</span><span class="p">(</span><span class="n">MINUS</span><span class="p">,</span> <span class="s1">'-'</span><span class="p">)</span>
|
||
|
||
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_char</span> <span class="o">==</span> <span class="s1">'*'</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">advance</span><span class="p">()</span>
|
||
<span class="k">return</span> <span class="n">Token</span><span class="p">(</span><span class="n">MUL</span><span class="p">,</span> <span class="s1">'*'</span><span class="p">)</span>
|
||
|
||
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_char</span> <span class="o">==</span> <span class="s1">'/'</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">advance</span><span class="p">()</span>
|
||
<span class="k">return</span> <span class="n">Token</span><span class="p">(</span><span class="n">DIV</span><span class="p">,</span> <span class="s1">'/'</span><span class="p">)</span>
|
||
|
||
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_char</span> <span class="o">==</span> <span class="s1">'('</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">advance</span><span class="p">()</span>
|
||
<span class="k">return</span> <span class="n">Token</span><span class="p">(</span><span class="n">LPAREN</span><span class="p">,</span> <span class="s1">'('</span><span class="p">)</span>
|
||
|
||
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_char</span> <span class="o">==</span> <span class="s1">')'</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">advance</span><span class="p">()</span>
|
||
<span class="k">return</span> <span class="n">Token</span><span class="p">(</span><span class="n">RPAREN</span><span class="p">,</span> <span class="s1">')'</span><span class="p">)</span>
|
||
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">error</span><span class="p">()</span>
|
||
|
||
<span class="k">return</span> <span class="n">Token</span><span class="p">(</span><span class="n">EOF</span><span class="p">,</span> <span class="bp">None</span><span class="p">)</span>
|
||
|
||
|
||
<span class="c1">###############################################################################</span>
|
||
<span class="c1"># #</span>
|
||
<span class="c1"># PARSER #</span>
|
||
<span class="c1"># #</span>
|
||
<span class="c1">###############################################################################</span>
|
||
|
||
<span class="k">class</span> <span class="nc">AST</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
|
||
<span class="k">pass</span>
|
||
|
||
|
||
<span class="k">class</span> <span class="nc">BinOp</span><span class="p">(</span><span class="n">AST</span><span class="p">):</span>
|
||
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">left</span><span class="p">,</span> <span class="n">op</span><span class="p">,</span> <span class="n">right</span><span class="p">):</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">left</span> <span class="o">=</span> <span class="n">left</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">token</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">op</span> <span class="o">=</span> <span class="n">op</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">right</span> <span class="o">=</span> <span class="n">right</span>
|
||
|
||
|
||
<span class="k">class</span> <span class="nc">Num</span><span class="p">(</span><span class="n">AST</span><span class="p">):</span>
|
||
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">token</span><span class="p">):</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">token</span> <span class="o">=</span> <span class="n">token</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">value</span> <span class="o">=</span> <span class="n">token</span><span class="o">.</span><span class="n">value</span>
|
||
|
||
|
||
<span class="k">class</span> <span class="nc">Parser</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
|
||
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">lexer</span><span class="p">):</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">lexer</span> <span class="o">=</span> <span class="n">lexer</span>
|
||
<span class="c1"># set current token to the first token taken from the input</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">current_token</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">lexer</span><span class="o">.</span><span class="n">get_next_token</span><span class="p">()</span>
|
||
|
||
<span class="k">def</span> <span class="nf">error</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span><span class="s1">'Invalid syntax'</span><span class="p">)</span>
|
||
|
||
<span class="k">def</span> <span class="nf">eat</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">token_type</span><span class="p">):</span>
|
||
<span class="c1"># compare the current token type with the passed token</span>
|
||
<span class="c1"># type and if they match then "eat" the current token</span>
|
||
<span class="c1"># and assign the next token to the self.current_token,</span>
|
||
<span class="c1"># otherwise raise an exception.</span>
|
||
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_token</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">token_type</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">current_token</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">lexer</span><span class="o">.</span><span class="n">get_next_token</span><span class="p">()</span>
|
||
<span class="k">else</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">error</span><span class="p">()</span>
|
||
|
||
<span class="k">def</span> <span class="nf">factor</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="sd">"""factor : INTEGER | LPAREN expr RPAREN"""</span>
|
||
<span class="n">token</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_token</span>
|
||
<span class="k">if</span> <span class="n">token</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">INTEGER</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">eat</span><span class="p">(</span><span class="n">INTEGER</span><span class="p">)</span>
|
||
<span class="k">return</span> <span class="n">Num</span><span class="p">(</span><span class="n">token</span><span class="p">)</span>
|
||
<span class="k">elif</span> <span class="n">token</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">LPAREN</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">eat</span><span class="p">(</span><span class="n">LPAREN</span><span class="p">)</span>
|
||
<span class="n">node</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">expr</span><span class="p">()</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">eat</span><span class="p">(</span><span class="n">RPAREN</span><span class="p">)</span>
|
||
<span class="k">return</span> <span class="n">node</span>
|
||
|
||
<span class="k">def</span> <span class="nf">term</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="sd">"""term : factor ((MUL | DIV) factor)*"""</span>
|
||
<span class="n">node</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">factor</span><span class="p">()</span>
|
||
|
||
<span class="k">while</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_token</span><span class="o">.</span><span class="n">type</span> <span class="ow">in</span> <span class="p">(</span><span class="n">MUL</span><span class="p">,</span> <span class="n">DIV</span><span class="p">):</span>
|
||
<span class="n">token</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_token</span>
|
||
<span class="k">if</span> <span class="n">token</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">MUL</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">eat</span><span class="p">(</span><span class="n">MUL</span><span class="p">)</span>
|
||
<span class="k">elif</span> <span class="n">token</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">DIV</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">eat</span><span class="p">(</span><span class="n">DIV</span><span class="p">)</span>
|
||
|
||
<span class="n">node</span> <span class="o">=</span> <span class="n">BinOp</span><span class="p">(</span><span class="n">left</span><span class="o">=</span><span class="n">node</span><span class="p">,</span> <span class="n">op</span><span class="o">=</span><span class="n">token</span><span class="p">,</span> <span class="n">right</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">factor</span><span class="p">())</span>
|
||
|
||
<span class="k">return</span> <span class="n">node</span>
|
||
|
||
<span class="k">def</span> <span class="nf">expr</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="sd">"""</span>
|
||
<span class="sd"> expr : term ((PLUS | MINUS) term)*</span>
|
||
<span class="sd"> term : factor ((MUL | DIV) factor)*</span>
|
||
<span class="sd"> factor : INTEGER | LPAREN expr RPAREN</span>
|
||
<span class="sd"> """</span>
|
||
<span class="n">node</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">term</span><span class="p">()</span>
|
||
|
||
<span class="k">while</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_token</span><span class="o">.</span><span class="n">type</span> <span class="ow">in</span> <span class="p">(</span><span class="n">PLUS</span><span class="p">,</span> <span class="n">MINUS</span><span class="p">):</span>
|
||
<span class="n">token</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">current_token</span>
|
||
<span class="k">if</span> <span class="n">token</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">PLUS</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">eat</span><span class="p">(</span><span class="n">PLUS</span><span class="p">)</span>
|
||
<span class="k">elif</span> <span class="n">token</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">MINUS</span><span class="p">:</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">eat</span><span class="p">(</span><span class="n">MINUS</span><span class="p">)</span>
|
||
|
||
<span class="n">node</span> <span class="o">=</span> <span class="n">BinOp</span><span class="p">(</span><span class="n">left</span><span class="o">=</span><span class="n">node</span><span class="p">,</span> <span class="n">op</span><span class="o">=</span><span class="n">token</span><span class="p">,</span> <span class="n">right</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">term</span><span class="p">())</span>
|
||
|
||
<span class="k">return</span> <span class="n">node</span>
|
||
|
||
<span class="k">def</span> <span class="nf">parse</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">expr</span><span class="p">()</span>
|
||
|
||
|
||
<span class="c1">###############################################################################</span>
|
||
<span class="c1"># #</span>
|
||
<span class="c1"># INTERPRETER #</span>
|
||
<span class="c1"># #</span>
|
||
<span class="c1">###############################################################################</span>
|
||
|
||
<span class="k">class</span> <span class="nc">NodeVisitor</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
|
||
<span class="k">def</span> <span class="nf">visit</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">node</span><span class="p">):</span>
|
||
<span class="n">method_name</span> <span class="o">=</span> <span class="s1">'visit_'</span> <span class="o">+</span> <span class="nb">type</span><span class="p">(</span><span class="n">node</span><span class="p">)</span><span class="o">.</span><span class="vm">__name__</span>
|
||
<span class="n">visitor</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">method_name</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">generic_visit</span><span class="p">)</span>
|
||
<span class="k">return</span> <span class="n">visitor</span><span class="p">(</span><span class="n">node</span><span class="p">)</span>
|
||
|
||
<span class="k">def</span> <span class="nf">generic_visit</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">node</span><span class="p">):</span>
|
||
<span class="k">raise</span> <span class="ne">Exception</span><span class="p">(</span><span class="s1">'No visit_{} method'</span><span class="o">.</span><span class="n">format</span><span class="p">(</span><span class="nb">type</span><span class="p">(</span><span class="n">node</span><span class="p">)</span><span class="o">.</span><span class="vm">__name__</span><span class="p">))</span>
|
||
|
||
|
||
<span class="k">class</span> <span class="nc">Interpreter</span><span class="p">(</span><span class="n">NodeVisitor</span><span class="p">):</span>
|
||
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">parser</span><span class="p">):</span>
|
||
<span class="bp">self</span><span class="o">.</span><span class="n">parser</span> <span class="o">=</span> <span class="n">parser</span>
|
||
|
||
<span class="k">def</span> <span class="nf">visit_BinOp</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">node</span><span class="p">):</span>
|
||
<span class="k">if</span> <span class="n">node</span><span class="o">.</span><span class="n">op</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">PLUS</span><span class="p">:</span>
|
||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">left</span><span class="p">)</span> <span class="o">+</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">right</span><span class="p">)</span>
|
||
<span class="k">elif</span> <span class="n">node</span><span class="o">.</span><span class="n">op</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">MINUS</span><span class="p">:</span>
|
||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">left</span><span class="p">)</span> <span class="o">-</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">right</span><span class="p">)</span>
|
||
<span class="k">elif</span> <span class="n">node</span><span class="o">.</span><span class="n">op</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">MUL</span><span class="p">:</span>
|
||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">left</span><span class="p">)</span> <span class="o">*</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">right</span><span class="p">)</span>
|
||
<span class="k">elif</span> <span class="n">node</span><span class="o">.</span><span class="n">op</span><span class="o">.</span><span class="n">type</span> <span class="o">==</span> <span class="n">DIV</span><span class="p">:</span>
|
||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">left</span><span class="p">)</span> <span class="o">/</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">node</span><span class="o">.</span><span class="n">right</span><span class="p">)</span>
|
||
|
||
<span class="k">def</span> <span class="nf">visit_Num</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">node</span><span class="p">):</span>
|
||
<span class="k">return</span> <span class="n">node</span><span class="o">.</span><span class="n">value</span>
|
||
|
||
<span class="k">def</span> <span class="nf">interpret</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
|
||
<span class="n">tree</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">parser</span><span class="o">.</span><span class="n">parse</span><span class="p">()</span>
|
||
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">visit</span><span class="p">(</span><span class="n">tree</span><span class="p">)</span>
|
||
|
||
|
||
<span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
|
||
<span class="k">while</span> <span class="bp">True</span><span class="p">:</span>
|
||
<span class="k">try</span><span class="p">:</span>
|
||
<span class="k">try</span><span class="p">:</span>
|
||
<span class="n">text</span> <span class="o">=</span> <span class="nb">raw_input</span><span class="p">(</span><span class="s1">'spi> '</span><span class="p">)</span>
|
||
<span class="k">except</span> <span class="ne">NameError</span><span class="p">:</span> <span class="c1"># Python3</span>
|
||
<span class="n">text</span> <span class="o">=</span> <span class="nb">input</span><span class="p">(</span><span class="s1">'spi> '</span><span class="p">)</span>
|
||
<span class="k">except</span> <span class="ne">EOFError</span><span class="p">:</span>
|
||
<span class="k">break</span>
|
||
<span class="k">if</span> <span class="ow">not</span> <span class="n">text</span><span class="p">:</span>
|
||
<span class="k">continue</span>
|
||
|
||
<span class="n">lexer</span> <span class="o">=</span> <span class="n">Lexer</span><span class="p">(</span><span class="n">text</span><span class="p">)</span>
|
||
<span class="n">parser</span> <span class="o">=</span> <span class="n">Parser</span><span class="p">(</span><span class="n">lexer</span><span class="p">)</span>
|
||
<span class="n">interpreter</span> <span class="o">=</span> <span class="n">Interpreter</span><span class="p">(</span><span class="n">parser</span><span class="p">)</span>
|
||
<span class="n">result</span> <span class="o">=</span> <span class="n">interpreter</span><span class="o">.</span><span class="n">interpret</span><span class="p">()</span>
|
||
<span class="k">print</span><span class="p">(</span><span class="n">result</span><span class="p">)</span>
|
||
|
||
|
||
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">'__main__'</span><span class="p">:</span>
|
||
<span class="n">main</span><span class="p">()</span>
|
||
</pre></div>
|
||
|
||
|
||
<p>Save the above code into the <em>spi.py</em> file or download it directly from <a href="https://github.com/rspivak/lsbasi/blob/master/part7/python/spi.py">GitHub</a>. Try it out and see for yourself that your new tree-based interpreter properly evaluates arithmetic expressions.</p>
|
||
<p>Here is a sample session:</p>
|
||
<div class="highlight"><pre><span></span>$ python spi.py
|
||
spi> <span class="m">7</span> + <span class="m">3</span> * <span class="o">(</span><span class="m">10</span> / <span class="o">(</span><span class="m">12</span> / <span class="o">(</span><span class="m">3</span> + <span class="m">1</span><span class="o">)</span> - <span class="m">1</span><span class="o">))</span>
|
||
<span class="m">22</span>
|
||
spi> <span class="m">7</span> + <span class="m">3</span> * <span class="o">(</span><span class="m">10</span> / <span class="o">(</span><span class="m">12</span> / <span class="o">(</span><span class="m">3</span> + <span class="m">1</span><span class="o">)</span> - <span class="m">1</span><span class="o">))</span> / <span class="o">(</span><span class="m">2</span> + <span class="m">3</span><span class="o">)</span> - <span class="m">5</span> - <span class="m">3</span> + <span class="o">(</span><span class="m">8</span><span class="o">)</span>
|
||
<span class="m">10</span>
|
||
spi> <span class="m">7</span> + <span class="o">(((</span><span class="m">3</span> + <span class="m">2</span><span class="o">)))</span>
|
||
<span class="m">12</span>
|
||
</pre></div>
|
||
|
||
|
||
<p><br/>
|
||
Today you’ve learned about parse trees, ASTs, how to construct ASTs and how to traverse them to interpret the input represented by those ASTs. You’ve also modified the parser and the interpreter and split them apart. The current interface between the lexer, parser, and the interpreter now looks like this:</p>
|
||
<p><img alt="" src="lsbasi_part7_pipeline.png" width="640"></p>
|
||
<p>You can read that as “The parser gets tokens from the lexer and then returns the generated <span class="caps">AST</span> for the interpreter to traverse and interpret the input.”</p>
|
||
<p>That’s it for today, but before wrapping up I’d like to talk briefly about recursive-descent parsers, namely just give them a definition because I promised last time to talk about them in more detail. So here you go: a <em>recursive-descent parser</em> is a top-down parser that uses a set of recursive procedures to process the input. Top-down reflects the fact that the parser begins by constructing the top node of the parse tree and then gradually constructs lower nodes.</p>
|
||
<p><br/>
|
||
And now it’s time for exercises :)</p>
|
||
<p><img alt="" src="lsbasi_part7_exercise.png" width="280"></p>
|
||
<ul>
|
||
<li>Write a translator (hint: node visitor) that takes as input an arithmetic expression and prints it out in postfix notation, also known as Reverse Polish Notation (<span class="caps">RPN</span>). For example, if the input to the translator is the expression (5 + 3) * 12 / 3 than the output should be 5 3 + 12 * 3 /. See the answer <a href="https://github.com/rspivak/lsbasi/blob/master/part7/python/ex1.py">here</a> but try to solve it first on your own.</li>
|
||
<li>Write a translator (node visitor) that takes as input an arithmetic expression and prints it out in <span class="caps">LISP</span> style notation, that is 2 + 3 would become (+ 2 3) and (2 + 3 * 5) would become (+ 2 (* 3 5)). You can find the answer <a href="https://github.com/rspivak/lsbasi/blob/master/part7/python/ex2.py">here</a> but again try to solve it first before looking at the provided solution.</li>
|
||
</ul>
|
||
<p><br/>
|
||
In the next article, we’ll add assignment and unary operators to our growing Pascal interpreter. Until then, have fun and see you soon.</p>
|
||
<p><br/>
|
||
<span class="caps">P.S.</span> I’ve also provided a Rust implementation of the interpreter that you can find on <a href="https://github.com/rspivak/lsbasi/blob/master/part7/rust/spi/src/main.rs">GitHub</a>. This is a way for me to learn <a href="https://www.rust-lang.org/">Rust</a> so keep in mind that the code might not be “idiomatic” yet. Comments and suggestions as to how to make the code better are always welcome.</p>
|
||
<p><br/>
|
||
Here is a list of books I recommend that will help you in your study of interpreters and compilers:</p>
|
||
<ol>
|
||
<li>
|
||
<p><a href="http://www.amazon.com/gp/product/193435645X/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=193435645X&linkCode=as2&tag=russblo0b-20&linkId=MP4DCXDV6DJMEJBL">Language Implementation Patterns: Create Your Own Domain-Specific and General Programming Languages (Pragmatic Programmers)</a><img src="http://ir-na.amazon-adsystem.com/e/ir?t=russblo0b-20&l=as2&o=1&a=193435645X" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /></p>
|
||
</li>
|
||
<li>
|
||
<p><a href="http://www.amazon.com/gp/product/0470177071/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0470177071&linkCode=as2&tag=russblo0b-20&linkId=UCLGQTPIYSWYKRRM">Writing Compilers and Interpreters: A Software Engineering Approach</a><img src="http://ir-na.amazon-adsystem.com/e/ir?t=russblo0b-20&l=as2&o=1&a=0470177071" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /></p>
|
||
</li>
|
||
<li>
|
||
<p><a href="http://www.amazon.com/gp/product/052182060X/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=052182060X&linkCode=as2&tag=russblo0b-20&linkId=ZSKKZMV7YWR22NMW">Modern Compiler Implementation in Java</a><img src="http://ir-na.amazon-adsystem.com/e/ir?t=russblo0b-20&l=as2&o=1&a=052182060X" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /></p>
|
||
</li>
|
||
<li>
|
||
<p><a href="http://www.amazon.com/gp/product/1461446988/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1461446988&linkCode=as2&tag=russblo0b-20&linkId=PAXWJP5WCPZ7RKRD">Modern Compiler Design</a><img src="http://ir-na.amazon-adsystem.com/e/ir?t=russblo0b-20&l=as2&o=1&a=1461446988" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /></p>
|
||
</li>
|
||
<li>
|
||
<p><a href="http://www.amazon.com/gp/product/0321486811/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=0321486811&linkCode=as2&tag=russblo0b-20&linkId=GOEGDQG4HIHU56FQ">Compilers: Principles, Techniques, and Tools (2nd Edition)</a><img src="http://ir-na.amazon-adsystem.com/e/ir?t=russblo0b-20&l=as2&o=1&a=0321486811" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /></p>
|
||
</li>
|
||
</ol>
|
||
<p><br/>
|
||
<p>If you want to get my newest articles in your inbox, then enter your email address below and click "Get Updates!"</p>
|
||
|
||
<!-- Begin MailChimp Signup Form -->
|
||
<link href="https://cdn-images.mailchimp.com/embedcode/classic-081711.css"
|
||
rel="stylesheet" type="text/css">
|
||
<style type="text/css">
|
||
#mc_embed_signup {
|
||
background: #f5f5f5;
|
||
clear: left;
|
||
font: 18px Helvetica,Arial,sans-serif;
|
||
}
|
||
|
||
#mc_embed_signup form {
|
||
text-align: center;
|
||
padding: 20px 0 10px 3%;
|
||
}
|
||
|
||
#mc_embed_signup .mc-field-group input {
|
||
display: inline;
|
||
width: 40%;
|
||
}
|
||
|
||
#mc_embed_signup div.response {
|
||
width: 100%;
|
||
}
|
||
</style>
|
||
<div id="mc_embed_signup">
|
||
<form
|
||
action="https://ruslanspivak.us4.list-manage.com/subscribe/post?u=7dde30eedc045f4670430c25f&id=6f69f44e03"
|
||
method="post"
|
||
id="mc-embedded-subscribe-form"
|
||
name="mc-embedded-subscribe-form"
|
||
class="validate"
|
||
target="_blank" novalidate>
|
||
<div id="mc_embed_signup_scroll">
|
||
|
||
<div class="mc-field-group">
|
||
<label for="mce-NAME">Enter Your First Name *</label>
|
||
<input type="text" value="" name="NAME" class="required" id="mce-NAME">
|
||
</div>
|
||
<div class="mc-field-group">
|
||
<label for="mce-EMAIL">Enter Your Best Email *</label>
|
||
<input type="email" value="" name="EMAIL" class="required email" id="mce-EMAIL">
|
||
</div>
|
||
<div id="mce-responses" class="clear">
|
||
<div class="response" id="mce-error-response" style="display:none"></div>
|
||
<div class="response" id="mce-success-response" style="display:none"></div>
|
||
</div>
|
||
<!-- real people should not fill this in and expect good things - do not remove this or risk form bot signups-->
|
||
<div style="position: absolute; left: -5000px;"><input type="text" name="b_7dde30eedc045f4670430c25f_6f69f44e03" tabindex="-1" value=""></div>
|
||
<div class="clear"><input type="submit" value="Get Updates!" name="subscribe" id="mc-embedded-subscribe" class="button" style="background-color: rgb(63, 146, 236);"></div>
|
||
</div>
|
||
</form>
|
||
</div>
|
||
<!-- <script type='text/javascript' src='//s3.amazonaws.com/downloads.mailchimp.com/js/mc-validate.js'></script><script type='text/javascript'>(function($) {window.fnames = new Array(); window.ftypes = new Array();fnames[1]='NAME';ftypes[1]='text';fnames[0]='EMAIL';ftypes[0]='email';}(jQuery));var $mcj = jQuery.noConflict(true);</script> -->
|
||
<!--End mc_embed_signup-->
|
||
</p>
|
||
<p><br/>
|
||
<strong>All articles in this series:</strong>
|
||
|
||
<ul>
|
||
<li>
|
||
<a href="../lsbasi-part1/index.html">Let's Build A Simple Interpreter. Part 1.</a>
|
||
</li>
|
||
<li>
|
||
<a href="../lsbasi-part2/index.html">Let's Build A Simple Interpreter. Part 2.</a>
|
||
</li>
|
||
<li>
|
||
<a href="../lsbasi-part3/index.html">Let's Build A Simple Interpreter. Part 3.</a>
|
||
</li>
|
||
<li>
|
||
<a href="../lsbasi-part4/index.html">Let's Build A Simple Interpreter. Part 4.</a>
|
||
</li>
|
||
<li>
|
||
<a href="../lsbasi-part5/index.html">Let's Build A Simple Interpreter. Part 5.</a>
|
||
</li>
|
||
<li>
|
||
<a href="../lsbasi-part6/index.html">Let's Build A Simple Interpreter. Part 6.</a>
|
||
</li>
|
||
<li>
|
||
<a href="index.html">Let's Build A Simple Interpreter. Part 7: Abstract Syntax Trees</a>
|
||
</li>
|
||
<li>
|
||
<a href="../lsbasi-part8/index.html">Let's Build A Simple Interpreter. Part 8.</a>
|
||
</li>
|
||
<li>
|
||
<a href="../lsbasi-part9/index.html">Let's Build A Simple Interpreter. Part 9.</a>
|
||
</li>
|
||
<li>
|
||
<a href="../lsbasi-part10/index.html">Let's Build A Simple Interpreter. Part 10.</a>
|
||
</li>
|
||
<li>
|
||
<a href="../lsbasi-part11/index.html">Let's Build A Simple Interpreter. Part 11.</a>
|
||
</li>
|
||
<li>
|
||
<a href="../lsbasi-part12/index.html">Let's Build A Simple Interpreter. Part 12.</a>
|
||
</li>
|
||
<li>
|
||
<a href="../lsbasi-part13.html">Let's Build A Simple Interpreter. Part 13: Semantic Analysis</a>
|
||
</li>
|
||
<li>
|
||
<a href="../lsbasi-part14/index.html">Let's Build A Simple Interpreter. Part 14: Nested Scopes and a Source-to-Source Compiler</a>
|
||
</li>
|
||
<li>
|
||
<a href="../lsbasi-part15/index.html">Let's Build A Simple Interpreter. Part 15.</a>
|
||
</li>
|
||
<li>
|
||
<a href="../lsbasi-part16/index.html">Let's Build A Simple Interpreter. Part 16: Recognizing Procedure Calls</a>
|
||
</li>
|
||
<li>
|
||
<a href="../lsbasi-part17.html">Let's Build A Simple Interpreter. Part 17: Call Stack and Activation Records</a>
|
||
</li>
|
||
<li>
|
||
<a href="../lsbasi-part18/index.html">Let's Build A Simple Interpreter. Part 18: Executing Procedure Calls</a>
|
||
</li>
|
||
<li>
|
||
<a href="../lsbasi-part19/index.html">Let's Build A Simple Interpreter. Part 19: Nested Procedure Calls</a>
|
||
</li>
|
||
</ul>
|
||
</p>
|
||
</div>
|
||
<!-- /.entry-content -->
|
||
<hr/>
|
||
<section class="comments" id="comments">
|
||
<h2>Comments</h2>
|
||
|
||
<div id="disqus_thread"></div>
|
||
<script type="text/javascript">
|
||
/* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
|
||
var disqus_shortname = 'ruslanspivak'; // required: replace example with your forum shortname
|
||
|
||
var disqus_identifier = 'lets-build-a-simple-interpreter-part-7-abstract-syntax-trees';
|
||
var disqus_url = 'https://ruslanspivak.com/lsbasi-part7/';
|
||
|
||
var disqus_config = function () {
|
||
this.language = "en";
|
||
};
|
||
|
||
/* * * DON'T EDIT BELOW THIS LINE * * */
|
||
(function () {
|
||
var dsq = document.createElement('script');
|
||
dsq.type = 'text/javascript';
|
||
dsq.async = true;
|
||
dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
|
||
(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
|
||
})();
|
||
</script>
|
||
<noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by
|
||
Disqus.</a></noscript>
|
||
<a href="http://disqus.com" class="dsq-brlink">comments powered by <span class="logo-disqus">Disqus</span></a>
|
||
|
||
</section>
|
||
</article>
|
||
</section>
|
||
|
||
</div>
|
||
<div class="col-sm-3" id="sidebar">
|
||
<aside>
|
||
|
||
<section class="well well-sm">
|
||
<ul class="list-group list-group-flush">
|
||
<li class="list-group-item"><h4><i class="fa fa-home fa-lg"></i><span class="icon-label">Social</span></h4>
|
||
<ul class="list-group" id="social">
|
||
<li class="list-group-item"><a href="https://github.com/rspivak/"><i class="fa fa-github-square fa-lg"></i> github</a></li>
|
||
<li class="list-group-item"><a href="https://twitter.com/rspivak"><i class="fa fa-twitter-square fa-lg"></i> twitter</a></li>
|
||
<li class="list-group-item"><a href="https://linkedin.com/in/ruslanspivak/"><i class="fa fa-linkedin-square fa-lg"></i> linkedin</a></li>
|
||
</ul>
|
||
</li>
|
||
|
||
<li class="list-group-item"><h4><i class="fa fa-home fa-lg"></i><span class="icon-label">Popular posts</span></h4>
|
||
<ul class="list-group" id="popularposts">
|
||
<li class="list-group-item"
|
||
style="font-size: 15px; word-break: normal;">
|
||
<a href="../lsbaws-part1/index.html">
|
||
Let's Build A Web Server. Part 1.
|
||
</a>
|
||
</li>
|
||
<li class="list-group-item"
|
||
style="font-size: 15px; word-break: normal;">
|
||
<a href="../lsbasi-part1/index.html">
|
||
Let's Build A Simple Interpreter. Part 1.
|
||
</a>
|
||
</li>
|
||
<li class="list-group-item"
|
||
style="font-size: 15px; word-break: normal;">
|
||
<a href="../lsbaws-part2/index.html">
|
||
Let's Build A Web Server. Part 2.
|
||
</a>
|
||
</li>
|
||
<li class="list-group-item"
|
||
style="font-size: 15px; word-break: normal;">
|
||
<a href="../lsbaws-part3/index.html">
|
||
Let's Build A Web Server. Part 3.
|
||
</a>
|
||
</li>
|
||
<li class="list-group-item"
|
||
style="font-size: 15px; word-break: normal;">
|
||
<a href="../lsbasi-part2/index.html">
|
||
Let's Build A Simple Interpreter. Part 2.
|
||
</a>
|
||
</li>
|
||
</ul>
|
||
</li>
|
||
|
||
<li class="list-group-item">
|
||
<h4>
|
||
<span>Disclaimer</span>
|
||
</h4>
|
||
<p id="disclaimer-text"> Some of the links on this site
|
||
have my Amazon referral id, which provides me with a small
|
||
commission for each sale. Thank you for your support.
|
||
</p>
|
||
</li>
|
||
|
||
|
||
|
||
</ul>
|
||
</section>
|
||
</aside>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<footer>
|
||
<div class="container">
|
||
<hr>
|
||
<div class="row">
|
||
<div class="col-xs-10">© 2020 Ruslan Spivak
|
||
<!-- · Powered by <a href="https://github.com/DandyDev/pelican-bootstrap3" target="_blank">pelican-bootstrap3</a>, -->
|
||
<!-- <a href="http://docs.getpelican.com/" target="_blank">Pelican</a>, -->
|
||
<!-- <a href="http://getbootstrap.com" target="_blank">Bootstrap</a> -->
|
||
<!-- -->
|
||
</div>
|
||
<div class="col-xs-2"><p class="pull-right"><i class="fa fa-arrow-up"></i> <a href="index.html#">Back to top</a></p></div>
|
||
</div>
|
||
</div>
|
||
</footer>
|
||
<script src="../theme/js/jquery.min.js"></script>
|
||
|
||
<!-- Include all compiled plugins (below), or include individual files as needed -->
|
||
<script src="../theme/js/bootstrap.min.js"></script>
|
||
|
||
<!-- Enable responsive features in IE8 with Respond.js (https://github.com/scottjehl/Respond) -->
|
||
<script src="../theme/js/respond.min.js"></script>
|
||
|
||
<!-- Disqus -->
|
||
<script type="text/javascript">
|
||
/* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
|
||
var disqus_shortname = 'ruslanspivak'; // required: replace example with your forum shortname
|
||
|
||
/* * * DON'T EDIT BELOW THIS LINE * * */
|
||
(function () {
|
||
var s = document.createElement('script');
|
||
s.async = true;
|
||
s.type = 'text/javascript';
|
||
s.src = '//' + disqus_shortname + '.disqus.com/count.js';
|
||
(document.getElementsByTagName('HEAD')[0] || document.getElementsByTagName('BODY')[0]).appendChild(s);
|
||
}());
|
||
</script>
|
||
<!-- End Disqus Code -->
|
||
<!-- Google Analytics Universal -->
|
||
<script type="text/javascript">
|
||
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
|
||
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
|
||
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
|
||
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
|
||
|
||
ga('create', 'UA-2572871-3', 'auto');
|
||
ga('send', 'pageview');
|
||
</script>
|
||
<!-- End Google Analytics Universal Code -->
|
||
|
||
</body>
|
||
</html> |