这是一篇被墙的文章,记录了作者如何因为atom的一个拼写错误而费力排错,并因此编写了一个检查代码内atom是否在白名单的功能;循序渐进,是parse_transform难得的入门文章,特全系列搬运在此.
The Atomiser, Part I
Erlang has atoms - one of many language features that I find hard to live without when I have to work with other systems.
But the compiler for Erlang does not check that the atoms you pass to a function are actually used by that function.
While I love programming with Erlang, stuff like this happens to me more often than I care to admit:
1> Greeting = fun
1> (hello) -> "Hooray! You said hello!";
1> (_) -> "You said something else."
1> end.
#Fun
2> Greeting(he11o).
"You said something else."
...
Huh? I said hello!
Didn't I?
The more observant of you will have immediately spotted the number '1's masquerading as 'l's up there, but believe me, depending on the obviousness of the typo and the lateness of the hour this sort of pest can be quite painful to hunt down and eradicate.
It was especially painful when I first started learning the language. I would find some strange behaviour or receive these odd "function_clause" errors, and I had no idea what was wrong.
Wouldn't it be nice if you could optionally give the compiler a list of valid atoms for a source file, and have the compiler warn you if you were using an atom that was not in that list?
I thought so too.
Let's fix it.
The Atomiser, Part II
So now we have some initial requirements to work with:
- Ability to include a list of valid atoms into a source file.
- Have the Erlang compiler tell us when an atom used is not in the valid list.
- Have the Erlang compiler tell us when an atom in the valid list is not used.
To achieve this I am going to write an Erlang parse transformation module. (A parse transformation takes the already-parsed syntax tree of an Erlang program and (usually) applies some modification to it.)
As an aside, programmers are strongly advised not to engage in parse transformations and no support is offered for problems encountered... but where is the fun in that?
In this case we are not actually going to engage in any transformations - we just want to hook into the compiler and scan the source code for atoms at compile-time.
So without further ado, I present the Atomiser, first draft:
-module(atomiser).
-export([parse_transform/2]).
parse_transform(AST, _Options) ->
io:format("The Atomiser is running.~n"),
AST.
All we are doing at this stage is printing a message and returning the unchanged Abstract Syntax Tree.
Once this atomiser.erl file is compiled into atomiser.beam we can put a parse_transform compiler directive in any source file and the compiler will call the Atomiser's parse_transform function automatically.
-compile({parse_transform, atomiser}).
Since we are already engaging in a highly ill-advised activity, we might as well go the whole way and include this directive into the Atomiser itself. (Only do this after the original version has been compiled, or the compiler will hang when it tries to compile this file.)
-module(atomiser).
-export([parse_transform/2]).
-compile({parse_transform, atomiser}).
parse_transform(AST, _Options) ->
io:format("The Atomiser is running.~n"),
AST.
So when we compile this program we should be informed that the Atomiser is actually running:
1> c(atomiser), l(atomiser).
The Atomiser is running.
{module, atomiser}
So far, so good.
The Atomiser, Part III
Right, so we have a skeleton parse transformation module that emits a message and returns the unchanged Abstract Syntax Tree. What do we do now?
There are a few requirements listed in Part II; we might as well start from the top:
* Ability to include a list of valid atoms into a source file.
Implicit in this requirement is that the Atomiser will actually be able to read this list and do something with it. Let's work on that.
To begin with we want to embed our list of valid atoms in our source code. Unfortunately we cannot use just any old syntax to do this - the syntax we choose must be parse-able by the existing Erlang compiler so that it can build us an Abstract Syntax Tree to play with.
Luckily we can use a module attribute to specify our list of valid atoms. For no particular reason I will name this attribute 'atoms':
-atoms([atom1, atom2, atom2, atom4]).
The Atomiser will do a quick pass through the top level of the Abstract Syntax Tree to pull in all the 'atoms' module attributes, storing their contents in a dictionary of valid atoms. The atom names themselves will be the keys of this dictionary; the line number of the attribute that specified the atom will be the value stored against each atom key. (We will use these line numbers for reporting later, if a specified atom is unused.)
Here is a function to scan an AST and print all the atoms attributes it finds:
atoms_find([]) ->
ok;
atoms_find([{attribute,Line,atoms,AtomList}|ASTRest]) ->
io:format("Found atom list on line ~B: ~p~n", [Line, AtomList]),
atoms_find(ASTRest);
atoms_find([_Node|ASTRest]) ->
atoms_find(ASTRest).
And with a slight modification, instead of printing them out we can store the atoms we find in a dictionary:
atoms_from_ast(AST) ->
atoms_from_ast(AST, dict:new()).
atoms_from_ast([], Atoms) ->
Atoms;
atoms_from_ast([{attribute,Line,atoms,AtomList}|ASTRest], Atoms) ->
atoms_from_ast(ASTRest, atoms_from_attribute(Line, AtomList, Atoms));
atoms_from_ast([_|ASTRest], Atoms) ->
atoms_from_ast(ASTRest, Atoms).
Neat, huh?
Well, okay, I haven't yet added the code to extract the atoms from an atoms attribute and store them in the dictionary. It sounds a bit complicated...
atoms_from_attribute(Line, AtomList, Atoms) ->
AddAtom = fun(Atom, Dict) ->
dict:store(Atom, Line, Dict)
end,
lists:foldl(AddAtom, Atoms, AtomList).
...well, maybe that is not too complicated after all.
Oh, wait! If I make the Atomiser report on atoms that have already been specified as valid, then I can totally make this look impressive:
atoms_from_attribute(Line, AtomList, Atoms) ->
AddAtom = fun(Atom, Dict) ->
case dict:find(Atom, Dict) of
{ok, LineAlreadyDefined} ->
io:format(
"Line ~B: Atom ~w already defined on line ~B.~n",
[Line, Atom, LineAlreadyDefined]),
Dict;
error -> dict:store(Atom, Line, Dict)
end
end,
lists:foldl(AddAtom, Atoms, AtomList).
There we go. Now the Atomiser will let us know if we have accidentally specified a valid atom more than once.
We can try this out now. Here is the full listing of the Atomiser so far:
-module(atomiser).
-export([parse_transform/2]).
-compile({parse_transform, atomiser}). % Comment out for initial compile.
-atoms([atom1, atom2, atom2, atom4]).
parse_transform(AST, _Options) ->
Atoms = atoms_from_ast(AST),
io:format("Retrieved these valid atoms: ~p~n", [dict:fetch_keys(Atoms)]),
AST.
atoms_from_ast(AST) ->
atoms_from_ast(AST, dict:new()).
atoms_from_ast([], Atoms) ->
Atoms;
atoms_from_ast([{attribute,Line,atoms,AtomList}|ASTRest], Atoms) ->
atoms_from_ast(ASTRest, atoms_from_attribute(Line, AtomList, Atoms));
atoms_from_ast([_|ASTRest], Atoms) ->
atoms_from_ast(ASTRest, Atoms).
atoms_from_attribute(Line, AtomList, Atoms) ->
AddAtom = fun(Atom, Dict) ->
case dict:find(Atom, Dict) of
{ok, LineAlreadyDefined} ->
io:format(
"Line ~B: Atom ~w already defined on line ~B.~n",
[Line, Atom, LineAlreadyDefined]),
Dict;
error -> dict:store(Atom, Line, Dict)
end
end,
lists:foldl(AddAtom, Atoms, AtomList).
Compiling this atomiser.erl file should give us the list of the three unique atoms specified, and a complaint about atom2 occurring twice:
1> c(atomiser), l(atomiser).
Line 5: Atom atom2 already defined on line 5.
Retrieved these valid atoms: [atom1,atom2,atom4]
{module,atomiser}
2>
This thing had better do some real work soon... it is already past thirty lines of code!
The Atomiser, Part IV
The story so far:
Our intrepid (also whiny, and quite lazy) developer suffers from a serious case of fat fingers and failing eyesight. Mistyped Erlang atoms have caused him countless hours of anguish and hair loss, and the compiler does not even have the decency to emit so much as a warning or apology.
In an effort to fill this massive hole in his development tools, the developer has managed to bolt together the bare bones of a parse transformation module. Soon this unnatural creation will be lumbering around the neighbourhood, causing all sorts of havoc.
The Atomiser currently has just enough intelligence to recognise valid atom lists specified in a given source file, but it is not able to actually do anything with that information. We need to walk the AST and locate the atoms used in the source code.
This walk_ast function will be the work-horse of the Atomiser module:
walk_ast([], Atoms) -> Atoms;
walk_ast([Node|ASTRest], Atoms) ->
io:format("Unknown node:~n~p~n", [Node]),
walk_ast(ASTRest, Atoms).
And we will modify parse_transform to call it:
parse_transform(AST, _Options) ->
Atoms = atoms_from_ast(AST),
_AtomsMarked = walk_ast(AST, Atoms),
AST.
With this simple walk_ast function we will get a huge output list of 'unknown' Abstract Syntax Tree nodes when we compile atomiser.erl[1]. The last unknown node will appear as:
Unknown node: {eof,41}
Obviously we will need to add more function clauses to cater for the different nodes in the Abstract Syntax Tree. Rather than starting from the top of the output list, I will work my way up from the bottom. (It saves scrolling up... see reference to 'lazy' above.)
Keeping the empty-list and the unknown-node function clauses as the first and last entries respectively, we add a clause entry for the eof node:
walk_ast([{eof,_Line}|ASTRest], Atoms) ->
walk_ast(ASTRest, Atoms);
Compiling the atomiser.erl program again[2] shows that, indeed, the {eof,41} node is now missing from the 'unknown node' output.
Continuing on we can enter some more function clauses. Something like this:
walk_ast([], Atoms) -> Atoms;
walk_ast([{call,_Line,Fun,Args}|ASTRest], Atoms) ->
walk_ast(ASTRest, walk_ast(Args, walk_ast([Fun], Atoms)));
walk_ast([{clause,_Line,Args,Guards,Exprs}|ASTRest], Atoms) ->
walk_ast(ASTRest, walk_ast(Exprs, walk_ast(Guards, walk_ast(Args, Atoms))));
walk_ast([{cons,_Line,Head,Tail}|ASTRest], Atoms) ->
walk_ast(ASTRest, walk_ast([Tail], walk_ast([Head], Atoms)));
walk_ast([{eof,_Line}|ASTRest], Atoms) ->
walk_ast(ASTRest, Atoms);
walk_ast([{function,_Line,_Fun,_Arity,Clauses}|ASTRest], Atoms) ->
walk_ast(ASTRest, walk_ast(Clauses, Atoms));
walk_ast([{nil,_Line}|ASTRest], Atoms) ->
walk_ast(ASTRest, Atoms);
walk_ast([{var,_Line,_Name}|ASTRest], Atoms) ->
walk_ast(ASTRest, Atoms);
walk_ast([Node|ASTRest], Atoms) ->
io:format("Unknown node: ~p~n", [Node]),
walk_ast(ASTRest, Atoms).
Hang on, I am typing the same stuff over and over again... this is way too inefficient.
It looks like there is a bit of a pattern going on here. Something along the lines of:
walk_ast([PatternTuple|ASTRest], Atoms) ->
walk_ast(ASTRest, PossiblyNestedCallsToWalkAST(SomeValueInThePattern, Atoms));
Where the possibly-nested calls to walk_ast are based on the elements of the pattern tuple that we are interested in processing recursively.
Now, Erlang does not have a built-in full macro system like Lisp does, but it does at least have basic substitution macros. I think that we can make the code above much nicer to read by using a macro like this:
-define(WALK_AST(Pattern, Expressions),
walk_ast([Pattern|ASTRest], Atoms) ->
walk_ast(ASTRest,
lists:foldl(
fun(AST, AtomsMarked) ->
walk_ast(AST, AtomsMarked)
end,
Atoms,
Expressions))).
Now that hideous function clause
walk_ast([{clause,_Line,Args,Guards,Exprs}|ASTRest], Atoms) ->
walk_ast(ASTRest, walk_ast(Exprs, walk_ast(Guards, walk_ast(Args, Atoms))));
may be written like this:
?WALK_AST({clause,_Line,Args,Guards,Exprs}, [Args, Guards, Exprs]);
Continuing to write a function clause for every unknown node in the AST[3], we get:
walk_ast([], Atoms) -> Atoms;
?WALK_AST({atom,_Line,Atom}, []); % Need to check for valid atom.
?WALK_AST({attribute,_Line,file,_File}, []);
?WALK_AST({attribute,_Line,module,_Module}, []);
?WALK_AST({attribute,_Line,export,_ExportList}, []);
?WALK_AST({attribute,_Line,compile,_CompilerDirective}, []);
?WALK_AST({attribute,_Line,atoms,_AtomList}, []);
?WALK_AST({call,_Line,Fun,Args}, [[Fun], Args]);
?WALK_AST({'case',_Line,Test,Clauses}, [[Test], Clauses]);
?WALK_AST({clause,_Line,Args,Guards,Exprs}, [Args, Guards, Exprs]);
?WALK_AST({cons,_Line,Head,Tail}, [[Head], [Tail]]);
?WALK_AST({eof,_Line}, []);
?WALK_AST({'fun',_Line,{clauses,Clauses}}, [Clauses]);
?WALK_AST({function,_Line,_Fun,_Arity,Clauses}, [Clauses]);
?WALK_AST({match,_Line,Left,Right}, [[Left], [Right]]);
?WALK_AST({nil,_Line}, []);
?WALK_AST({remote,_Line,_Module,_Function}, []);
?WALK_AST({string,_Line,_String}, []);
?WALK_AST({tuple,_Line,Elements}, [Elements]);
?WALK_AST({var,_Line,_Name}, []);
walk_ast([Node|ASTRest], Atoms) ->
io:format("Unknown node: ~p~n", [Node]),
walk_ast(ASTRest, Atoms).
And this is all we need to walk through the Atomiser's current code, with no unknown node messages appearing.
[1] We need to compile atomiser.erl twice to pick up the changes and see them in action. The first time the compilation is performed the new code is compiled and loaded. The second time the compilation is performed the new code is actually run against the source.
[2] Twice!
[3] Rather than consisting of a bunch of pattern matching clauses, the walk_ast function could be made "smarter" by transforming the given node tuple into a list, and applying some rules-based logic to the elements of that list (from the third element onwards). I chose to do things the function-clause way mainly because I am not completely familiar with all of the elements in an Erlang AST. Having newly-encountered nodes come up as unknown elements is a good way to be exposed to the underlying structure of the parse tree, and to ensure that we have caught everything we need to.
The Atomiser, Part V
The Atomiser can now walk its own Abstract Syntax Tree and pluck lists of valid atoms from its own source code.
Next on the list is to
- Have the Erlang compiler tell us when we use an atom that is not in the valid list.
Sounds easy: let's get to it!
We want to turn our placeholder atom clause in the walk_ast function into something a little more useful:
walk_ast([{atom,Line,Atom}|RestAST], Atoms) ->
walk_ast(RestAST, atom_check(Atom, Line, Atoms));
Our new atom_check function will compare the supplied atom against the given atoms dictionary. If the atom exists in the dictionary then a new dictionary will be returned, with that atom's value updated to 'found' (taking the place of the atom's line-of-definition integer). If the atom does not exist in the dictionary then a warning message will be displayed and the original dictionary will be returned:
atom_check(Atom, Line, Atoms) ->
case dict:find(Atom, Atoms) of
{ok, found} ->
Atoms;
{ok, _LineDefinedOn} ->
dict:store(Atom, found, Atoms);
error ->
io:format(
"Line ~B: Atom ~w unexpected.~n",
[Line, Atom]),
Atoms
end.
...and we are almost done.
A bit of a shock - but a good sign - is the number of unknown atoms that appear when you first run the Atomiser on itself. All of the atoms we use in our walk_ast pattern tuples show up as unknown atoms, so let's add some proper valid atom lists to the top of the file. First, the atoms we use in our patterns:
-atoms([atom, attribute, call, 'case', clause, clauses, compile]).
-atoms([cons, eof, export, file, 'fun', function, match, module]).
-atoms([nil, remote, string, tuple, var]).
And then some other atoms we are use elsewhere in the code:
-atoms([atoms, error, found, ok]).
And... all the function names are showing up as unknown atoms, too?
Executive decision time!
Since internal function-calls are already picked up by the compiler when they do not match a function in the module, I think it should be safe to modify the 'call' line to:
?WALK_AST({call,_Line,_Fun,Args}, [Args]);
This will skip validating the called function name.
And now,
1> c(atomiser), l(atomiser).
{module,atomiser}
2>
Beautiful.
The Atomiser, Part VI
We still have one (optional) requirement left for the Atomiser:
* Have the Erlang compiler tell us if an atom in the valid list is not used.
This can be done quite easily: the atom_check function returns an updated dictionary whenever an atom has been found. All we have to do is go through the valid atoms dictionary and pick up all the atom keys that do not have the value 'found' associated with them, and print them out in line-number order.
atoms_unused_print(Atoms) ->
Filter = fun({_Atom, FoundOrDefinedLine}) ->
FoundOrDefinedLine =/= found
end,
AtomsUnused = lists:keysort(2, lists:filter(Filter, dict:to_list(Atoms))),
PrintUnusedAtom = fun({Atom, Line}) ->
io:format("Line ~B: Atom ~w unused.~n", [Line, Atom])
end,
lists:foreach(PrintUnusedAtom, AtomsUnused).
Incidentally, another requirement raised its ugly head during the development of this module. If you ever use the Atomiser when compiling an invalid program, you will receive a horrible 'error' node in the AST. The Atomiser is not interested in these nodes (they will stop the compile process later anyway) and so we just want to ignore them.
A new specialised function clause for walk_ast takes care of this:
?WALK_AST({error,_Details}, []);
Believe it or not, the Atomiser is now pretty much complete.
If you have managed to stay with me so far on this meandering journey, you will probably have noticed that there are a few more minor details we need to clean up before we are finished. The missing pieces are just a few more atoms that need to be identified in the atoms module attributes, and a couple of extra function clauses that need to be added walk_ast. I will leave both of these as an exercise for any impatient readers that cannot wait until the next post, when I will list the entire module in all its naked glory.
The Atomiser, Part VII
As promised, here is the full listing of my current atomiser.erl file:
-module(atomiser).
-author("Philip Robinson").
-export([parse_transform/2]).
%-compile({parse_transform, atomiser}). % Uncomment after initial compile.
-atoms([atom, attribute, bin, bin_element, call, 'case', char]).
-atoms([clause, clauses, cons, eof, 'fun', function, generate]).
-atoms(['if', integer, lc, match, nil, op, 'receive', record]).
-atoms([record_field, remote, string, tuple, var]).
-atoms([atoms, error, found, ok]).
parse_transform(AST, _Options) ->
atoms_unused_print(walk_ast(AST, dict:new())),
AST.
atoms_from_attribute(Line, AtomList, Atoms) ->
AddAtom = fun(Atom, Dict) ->
case dict:find(Atom, Dict) of
{ok, LineAlreadyDefined} ->
io:format("Line ~B: Atom ~w already defined on line ~B.~n",
[Line, Atom, LineAlreadyDefined]),
Dict;
error -> dict:store(Atom, Line, Dict)
end
end,
lists:foldl(AddAtom, Atoms, AtomList).
atom_check(Atom, Line, Atoms) ->
case dict:find(Atom, Atoms) of
{ok, found} -> Atoms;
{ok, _LineDefinedOn} -> dict:store(Atom, found, Atoms);
error ->
io:format("Line ~B: Atom ~w unexpected.~n", [Line, Atom]),
Atoms
end.
atoms_unused_print(Atoms) ->
Filter = fun({_Atom, FoundOrDefinedLine}) ->
FoundOrDefinedLine =/= found
end,
PrintUnusedAtom = fun({Atom, Line}) ->
io:format("Line ~B: Atom ~w unused.~n", [Line, Atom])
end,
lists:foreach(PrintUnusedAtom,
lists:keysort(2, lists:filter(Filter, dict:to_list(Atoms)))).
-define(WALK_AST(Pattern, Expressions),
walk_ast([Pattern|ASTRest], Atoms) ->
Fun = fun(AST, AtomsMarked) ->
walk_ast(AST, AtomsMarked)
end,
walk_ast(ASTRest, lists:foldl(Fun, Atoms, Expressions))).
walk_ast([], Atoms) -> Atoms;
walk_ast([{atom,Line,Atom}|RestAST], Atoms) -> % Check whether atom is valid.
walk_ast(RestAST, atom_check(Atom, Line, Atoms));
walk_ast([{attribute,Line,atoms,AtomList}|RestAST], Atoms) -> % Valid atoms.
walk_ast(RestAST, atoms_from_attribute(Line, AtomList, Atoms));
?WALK_AST({attribute,_Line,_Name,_Value}, []);
?WALK_AST({bin,_Line,Elements}, [Elements]);
?WALK_AST({bin_element,_Line,_Name,_Size,_Type}, []);
?WALK_AST({call,_Line,_Fun,Args}, [Args]);
?WALK_AST({'case',_Line,Test,Clauses}, [[Test], Clauses]);
?WALK_AST({char,_Line,_Char}, []);
?WALK_AST({clause,_Line,Args,Guards,Exprs}, [Args] ++ Guards ++ [Exprs]);
?WALK_AST({cons,_Line,Head,Tail}, [[Head], [Tail]]);
?WALK_AST({eof,_Line}, []);
?WALK_AST({error,_Details}, []); % Ignore compiler errors.
?WALK_AST({'fun',_Line,{clauses,Clauses}}, [Clauses]);
?WALK_AST({function,_Line,_Fun,_Arity,Clauses}, [Clauses]);
?WALK_AST({generate,_Line,A,B}, [[A, B]]);
?WALK_AST({'if',_Line,Clauses}, [Clauses]);
?WALK_AST({integer,_Line,_Integer}, []);
?WALK_AST({lc,_Line,Head,Tail}, [[Head|Tail]]);
?WALK_AST({match,_Line,Left,Right}, [[Left], [Right]]);
?WALK_AST({nil,_Line}, []);
?WALK_AST({op,_Line,_BinaryOperator,Left,Right}, [[Left], [Right]]);
?WALK_AST({op,_Line,_UnaryOperator,_Operand}, []);
?WALK_AST({'receive',_Line,Clauses}, [Clauses]);
?WALK_AST({'receive',_Line,Clauses1,_TimeAfter,Clauses2}, [Clauses1, Clauses2]);
?WALK_AST({record,_Line,_Record,Fields}, [Fields]);
?WALK_AST({record_field,_Line,Field,Contents}, [[Field,Contents]]);
?WALK_AST({record_field,_Line,_Variable,_Record,Field}, [[Field]]);
?WALK_AST({remote,_Line,_Module,_Function}, []);
?WALK_AST({string,_Line,_String}, []);
?WALK_AST({tuple,_Line,Elements}, [Elements]);
?WALK_AST({var,_Line,_Name}, []);
walk_ast([Node|ASTRest], Atoms) ->
io:format("Unknown node: ~p~n", [Node]),
walk_ast(ASTRest, Atoms).
Some final notes, in no particular order:
I am quite pleased with the functionality of the Atomiser, especially considering that it currently weighs in at just under 100 lines of code. I can honestly attribute the relatively small size of this module to the use of the single WALK_AST substitution macro. If this macro had not been used then we would be looking at an increase of 50% in lines of code, at least.
The fact that the Atomiser does not alter the parse tree of the program it is examining made it an ideal project to get used to working with Erlang parse_transform programs. Without support for parse_transform modules I would have had to hack at the source code of the compiler to achieve a similar result... which is not really a viable option if the addition is not accepted into the project.
Obviously in this implementation I have only added walk_ast function clauses for those AST nodes my own programs require; your mileage may vary. If you do run this module over your own code and an unknown node appears then please let me know so I can add an appropriate clause here. Likewise, please feel free to drop me a line with questions, comments, and/or (especially!) suggestions.
Update 10/4/2007:
Yariv Sadan suggested that I take a look at Recless, one of his many ongoing projects (see comments in Part 1). As Yariv did in Recless, I have removed the atoms_from_ast function by rolling the gathering of atoms into the walk_ast function. It saves five lines of code, but more importantly it saves one pass through the top level of the AST. (The down side is that you can no longer expect the Atomiser to validate atoms before the appropriate module attribute is encountered, but I have no problem with that.)
"ayrnieu" posted a link to this series on Reddit, and also mentioned the Dialyzer tool. I have just played with the Dialyzer and have only one thing to say: Use the Dialyzer on your code.
