EUC 2015
Sean Cribbs
Background Macros Techniques for Metaprogramming in Erlang eunit
Parse Tr a n s f o rm s lager Sean Cribbs parse_trans Comcast Cable (T+PD) Syntax Trees erl_syntax @seancribbs Neotoma
mochiglobal
merl
erlydtl
Conclusion Erlang User Conference Stockholm 12 June 2015 About me
EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Tr a n s f o rm s lager Senior Principal Engineer at Comcast Cable parse_trans
Syntax Trees Former Te c h n i c a l L e a d for Riak at Basho
erl_syntax
Neotoma Creator of neotoma,Erlangpackrat-parser toolkit
mochiglobal
merl
erlydtl
Conclusion EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Tr a n s f o rm s
lager
parse_trans
Syntax Trees
erl_syntax Background
Neotoma
mochiglobal
merl
erlydtl
Conclusion Run-time Compile-time
What is Metaprogramming?
EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Tr a n s f o rm s
lager
parse_trans Code writing code
Syntax Trees Programs as data erl_syntax
Neotoma Reflection / reflexivity mochiglobal Homoiconicity merl
erlydtl
Conclusion Compile-time
What is Metaprogramming?
EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Tr a n s f o rm s
lager
parse_trans Code writing code Run-time
Syntax Trees Programs as data erl_syntax
Neotoma Reflection / reflexivity mochiglobal Homoiconicity merl
erlydtl
Conclusion What is Metaprogramming?
EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Tr a n s f o rm s
lager
parse_trans Code writing code Run-time Syntax Trees Programs as data Compile-time erl_syntax
Neotoma Reflection / reflexivity mochiglobal Homoiconicity merl
erlydtl
Conclusion Why Metaprogram?
EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Reduce duplication Tr a n s f o rm s lager Inject optimization parse_trans
Syntax Trees Simplify APIs erl_syntax Neotoma Improve tools mochiglobal merl Implement DSLs erlydtl
Conclusion Metaprogramming Erlang
EUC 2015
Sean Cribbs
Background Source substitute Macros
eunit
Parse tokenize Tr a n s f o rm s
lager parse_trans Tokens rewrite pretty-print Syntax Trees
erl_syntax Neotoma parse mochiglobal
merl erlydtl Abstract form transform Conclusion
compile evaluate
Bytecode load Runtime EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Tr a n s f o rm s
lager parse_trans Te c h n i q u e 1 Syntax Trees
erl_syntax
Neotoma mochiglobal Macros merl
erlydtl
Conclusion Macros
EUC 2015
Sean Cribbs Generates code in preprocessor (epp) Background Operates over Tokens (mostly) Macros
eunit % static term Parse ⌥ ⌅ Tr a n s f o rm s -define(TIMEOUT,5000). lager
parse_trans
Syntax Trees ⌃ ⇧
erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion Macros
EUC 2015
Sean Cribbs Generates code in preprocessor (epp) Background Operates over Tokens (mostly) Macros
eunit % static term Parse ⌥ ⌅ Tr a n s f o rm s -define(TIMEOUT,5000). lager
parse_trans % parameterized
Syntax Trees -define(THUNK(A), fun() -> (A) end). erl_syntax -define(IF(B,T,F), Neotoma mochiglobal begin merl ( (B) true (T); false (F) ) erlydtl case of -> -> end
Conclusion end).
⌃ ⇧ Macros
EUC 2015
Sean Cribbs Generates code in preprocessor (epp) Background Operates over Tokens (mostly) Macros
eunit % static term Parse ⌥ ⌅ Tr a n s f o rm s -define(TIMEOUT,5000). lager
parse_trans % parameterized
Syntax Trees -define(THUNK(A), fun() -> (A) end). erl_syntax -define(IF(B,T,F), Neotoma mochiglobal begin merl ( (B) true (T); false (F) ) erlydtl case of -> -> end
Conclusion end). %% escaped arguments -define(Quote(A), io_lib:format("~s",[??A])).
⌃ ⇧ ⌥Nope =?THUNK(launch(missiles)). ⌅ %% Nope = fun() -> (launch(missiles)) end.
⌃ ⇧ ⌥io:format("The value of ~s is ~p.",[?Quote(Foo), Foo]). ⌅ %% io:format("The value of ~s is ~p.",["Foo", Foo]).
⌃ ⇧
Using Macros
EUC 2015
Sean Cribbs
Background
Macros eunit ⌥gen_server:call(?MODULE, ping,?TIMEOUT). ⌅ Parse %% gen_server:call(mymodule, ping, 5000). Tr a n s f o rm s
lager parse_trans ⌃ ⇧ Syntax Trees
erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion ⌥io:format("The value of ~s is ~p.",[?Quote(Foo), Foo]). ⌅ %% io:format("The value of ~s is ~p.",["Foo", Foo]).
⌃ ⇧
Using Macros
EUC 2015
Sean Cribbs
Background
Macros eunit ⌥gen_server:call(?MODULE, ping,?TIMEOUT). ⌅ Parse %% gen_server:call(mymodule, ping, 5000). Tr a n s f o rm s
lager parse_trans ⌃ ⇧ Nope =?THUNK(launch(missiles)). Syntax Trees ⌥ ⌅
erl_syntax %% Nope = fun() -> (launch(missiles)) end.
Neotoma
mochiglobal merl ⌃ ⇧ erlydtl
Conclusion Using Macros
EUC 2015
Sean Cribbs
Background
Macros eunit ⌥gen_server:call(?MODULE, ping,?TIMEOUT). ⌅ Parse %% gen_server:call(mymodule, ping, 5000). Tr a n s f o rm s
lager parse_trans ⌃ ⇧ Nope =?THUNK(launch(missiles)). Syntax Trees ⌥ ⌅
erl_syntax %% Nope = fun() -> (launch(missiles)) end.
Neotoma
mochiglobal merl ⌃ ⇧ io:format("The value of ~s is ~p.",[?Quote(Foo), Foo]). erlydtl ⌥ ⌅ Conclusion %% io:format("The value of ~s is ~p.",["Foo", Foo]).
⌃ ⇧ Macros - eunit
EUC 2015
Sean Cribbs ⌥-define(assert(BoolExpr), ⌅ Background begin
Macros ((fun () ->
eunit
Parse Tr a n s f o rm s
lager
parse_trans
Syntax Trees
erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion
end end)()) end).
⌃ ⇧ Macros - eunit
EUC 2015
Sean Cribbs ⌥-define(assert(BoolExpr), ⌅ Background begin
Macros ((fun () -> eunit case (BoolExpr) of Parse true -> ok; Tr a n s f o rm s
lager
parse_trans
Syntax Trees
erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion
end end)()) end).
⌃ ⇧ Macros - eunit
EUC 2015
Sean Cribbs ⌥-define(assert(BoolExpr), ⌅ Background begin
Macros ((fun () -> eunit case (BoolExpr) of Parse true -> ok; Tr a n s f o rm s lager __V -> erlang:error({assertion_failed, parse_trans [{module,?MODULE}, Syntax Trees {line,?LINE}, erl_syntax Neotoma {expression,(??BoolExpr)}, mochiglobal
merl {expected, true}, erlydtl {value, case __V of false -> __V; Conclusion _ -> {not_a_boolean,__V} end}]})
end end)()) end).
⌃ ⇧ 1> eunit:test(mymodule).
mymodule: fizzbuzz_test (module ’mymodule’)...*failed*
in function mymodule:’-fizzbuzz_test/0-fun-3-’/0 (mymodule.erl, line 18)
**error:{assertion_failed,[{module,mymodule},
{line,18},
{expression,"10 =:= fizzbuzz ( 10 )"},
{expected,true},
{value,false}]}
Macros - eunit
EUC 2015 fizzbuzz_test() -> Sean Cribbs ⌥ ⌅ ?assert(fizz =:= fizzbuzz(3)), Background ?assert(buzz =:= fizzbuzz(5)), Macros ?assert(fizzbuzz =:= fizzbuzz(15)), eunit ?assert(10 =:= fizzbuzz(10)). Parse Tr a n s f o rm s
lager parse_trans ⌃ ⇧
Syntax Trees
erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion Macros - eunit
EUC 2015 fizzbuzz_test() -> Sean Cribbs ⌥ ⌅ ?assert(fizz =:= fizzbuzz(3)), Background ?assert(buzz =:= fizzbuzz(5)), Macros ?assert(fizzbuzz =:= fizzbuzz(15)), eunit ?assert(10 =:= fizzbuzz(10)). Parse Tr a n s f o rm s
lager parse_trans ⌃ ⇧
Syntax Trees 1> eunit:test(mymodule). erl_syntax mymodule: fizzbuzz_test (module ’mymodule’)...*failed* Neotoma mochiglobal in function mymodule:’-fizzbuzz_test/0-fun-3-’/0 (mymodule.erl, line 18) merl erlydtl **error:{assertion_failed,[{module,mymodule},
Conclusion {line,18},
{expression,"10 =:= fizzbuzz ( 10 )"},
{expected,true},
{value,false}]} Cons: Limited expressivity Appearance
Good for: Small API wrappers like in eunit or eqc Naming constants Compile-time feature-switching (OTP upgrades) Debugging statements
Macros - Summary
EUC 2015
Sean Cribbs
Background Pros: Macros Easy and familiar eunit
Parse Inline with program Tr a n s f o rm s lager Syntax draws attention parse_trans
Syntax Trees
erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion Good for: Small API wrappers like in eunit or eqc Naming constants Compile-time feature-switching (OTP upgrades) Debugging statements
Macros - Summary
EUC 2015
Sean Cribbs
Background Pros: Cons: Macros Easy and familiar Limited expressivity eunit
Parse Inline with program Appearance Tr a n s f o rm s lager Syntax draws attention parse_trans
Syntax Trees
erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion Macros - Summary
EUC 2015
Sean Cribbs
Background Pros: Cons: Macros Easy and familiar Limited expressivity eunit
Parse Inline with program Appearance Tr a n s f o rm s lager Syntax draws attention parse_trans
Syntax Trees
erl_syntax Neotoma Good for: mochiglobal merl Small API wrappers like in eunit or eqc erlydtl
Conclusion Naming constants Compile-time feature-switching (OTP upgrades) Debugging statements EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Tr a n s f o rm s
lager parse_trans Te c h n i q u e 2 Syntax Trees
erl_syntax
Neotoma mochiglobal Parse Transforms merl
erlydtl
Conclusion $erlc-Pmymodule.erl
$catmymodule.P
Parse Transforms
EUC 2015
Sean Cribbs Generates or transforms code after parsing Background Operates over Abstract Form (AST) Macros
eunit %% In your module: Parse ⌥ ⌅ Tr a n s f o rm s -compile([{parse_transform, the_transform_module}]). lager
parse_trans
Syntax Trees %% In the parse transform module: erl_syntax parse_transform(Forms, _Options) -> Neotoma mochiglobal %% ’Forms’ is the AST. ’Options’ are the compiler options. merl %% Traverse/modify ’Forms’ and return it erlydtl Forms. Conclusion
⌃ ⇧ Parse Transforms
EUC 2015
Sean Cribbs Generates or transforms code after parsing Background Operates over Abstract Form (AST) Macros
eunit %% In your module: Parse ⌥ ⌅ Tr a n s f o rm s -compile([{parse_transform, the_transform_module}]). lager
parse_trans
Syntax Trees %% In the parse transform module: erl_syntax parse_transform(Forms, _Options) -> Neotoma mochiglobal %% ’Forms’ is the AST. ’Options’ are the compiler options. merl %% Traverse/modify ’Forms’ and return it erlydtl Forms. Conclusion
⌃ ⇧ $erlc-Pmymodule.erl
$catmymodule.P Parse Transforms - lager github.com/basho/lager
EUC 2015
Sean Cribbs Rewrites calls to lager:SYSLOG_SEVERITY_LEVEL Background Injects producer-side filtering and call-site metadata Macros
eunit lager:warning("Resource threshold exceeded ~p:~p",[Used, Available]). Parse ⌥ ⌅ Tr a n s f o rm s
lager parse_trans ⌃ ⇧
Syntax Trees
erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion Parse Transforms - lager github.com/basho/lager
EUC 2015
Sean Cribbs Rewrites calls to lager:SYSLOG_SEVERITY_LEVEL Background Injects producer-side filtering and call-site metadata Macros
eunit lager:warning("Resource threshold exceeded ~p:~p",[Used, Available]). Parse ⌥ ⌅ Tr a n s f o rm s %% Becomes equivalent of: lager
parse_trans case {whereis(lager_event), lager_config:get(loglevel,{0,[]})}of
Syntax Trees {undefined, _} -> {error, lager_not_running}; erl_syntax {Pid,{Level, Traces}} when (Level band 16) /= 0 orelse Traces /= [] -> Neotoma mochiglobal lager:do_log(warning,[{module, mymodule}, {function, myfunc}, merl {line,5},{pid, pid_to_list(self())}, erlydtl
Conclusion {node, node()} | lager:md()], "Resource threshold exceeded ~p:~p",
[Used, Available], Level, Traces, Pid); _ -> ok
end.
⌃ ⇧ ⌥[{’case’,1, ⌅ {tuple,1,
[{call,1,{atom,1,whereis},[{atom,1,lager_event}]},
{call,1,
{remote,1,{atom,1,lager_config},{atom,1,get}},
[{atom,1,loglevel},{tuple,1,[{integer,1,0},{nil,1}]}]}]},
[{clause,2,
[{tuple,2,[{atom,2,undefined},{var,2,’_’}]}],
[],
[{tuple,2,[{atom,2,error},{atom,2,lager_not_running}]}]},
{clause,3,
[{tuple,3,
[{var,3,’Pid’},
{tuple,3,[{var,3,’Level’},{var,3,’Traces’}]}]}],
[[{op,3,’orelse’,
{op,3,’/=’,
{op,3,’band’,{var,3,’Level’},{integer,3,16}},
{integer,3,0}},
{op,3,’/=’,{var,3,’Traces’},{nil,3}}}]],
[{call,4,
{remote,4,{atom,4,lager},{atom,4,do_log}},
[{atom,4,warning},
{cons,4,
{tuple,4,[{atom,4,module},{atom,4,...}]},
{cons,4,{tuple,4,[...]},{cons,5,...}}},
{string,7,"Resource threshold exceeded ~p:~p"},
{cons,8,{var,8,’Used’},{cons,8,{var,...},{...}}},
{var,8,’Level’},
{var,8,’Traces’},
{var,8,’Pid’}]}]},
{clause,9,[{var,9,’_’}],[],[{atom,9,ok}]}]}]
⌃ ⇧
Parse Transforms - lager Understanding the AST
EUC 2015 {ok, Bin}=file:read_file("lager_snippet.erl"), Sean Cribbs ⌥ ⌅ {ok, Tokens, _}=erl_scan:string(unicode:characters_to_list(Bin)), Background {ok, AST}=erl_parse:parse_exprs(Tokens), Macros AST. eunit
Parse Tr a n s f o rm s ⌃ ⇧ lager
parse_trans
Syntax Trees
erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion Parse Transforms - lager Understanding the AST
EUC 2015 {ok, Bin}=file:read_file("lager_snippet.erl"), Sean Cribbs ⌥ ⌅ {ok, Tokens, _}=erl_scan:string(unicode:characters_to_list(Bin)), Background {ok, AST}=erl_parse:parse_exprs(Tokens), Macros AST. eunit
Parse Tr a n s f o rm s ⌃[{’case’,1, ⇧ lager ⌥ ⌅ parse_trans {tuple,1,
Syntax Trees [{call,1,{atom,1,whereis},[{atom,1,lager_event}]}, erl_syntax Neotoma {call,1, mochiglobal
merl {remote,1,{atom,1,lager_config},{atom,1,get}},
erlydtl [{atom,1,loglevel},{tuple,1,[{integer,1,0},{nil,1}]}]}]}, Conclusion [{clause,2,
[{tuple,2,[{atom,2,undefined},{var,2,’_’}]}],
[],
[{tuple,2,[{atom,2,error},{atom,2,lager_not_running}]}]},
{clause,3,
[{tuple,3,
[{var,3,’Pid’},
{tuple,3,[{var,3,’Level’},{var,3,’Traces’}]}]}],
[[{op,3,’orelse’,
{op,3,’/=’,
{op,3,’band’,{var,3,’Level’},{integer,3,16}},
{integer,3,0}},
{op,3,’/=’,{var,3,’Traces’},{nil,3}}}]],
[{call,4,
{remote,4,{atom,4,lager},{atom,4,do_log}},
[{atom,4,warning},
{cons,4,
{tuple,4,[{atom,4,module},{atom,4,...}]},
{cons,4,{tuple,4,[...]},{cons,5,...}}},
{string,7,"Resource threshold exceeded ~p:~p"},
{cons,8,{var,8,’Used’},{cons,8,{var,...},{...}}},
{var,8,’Level’},
{var,8,’Traces’},
{var,8,’Pid’}]}]},
{clause,9,[{var,9,’_’}],[],[{atom,9,ok}]}]}]
⌃ ⇧ Parse Transforms - lager lager_transform module
EUC 2015
Sean Cribbs
Background parse_transform(AST, Options) -> Macros ⌥ ⌅ eunit TruncSize = proplists:get_value(lager_truncation_size, Options, Parse ?DEFAULT_TRUNCATION), Tr a n s f o rm s
lager Enable = proplists:get_value(lager_print_records_flag, Options, true), parse_trans put(print_records_flag, Enable), Syntax Trees put(truncation_size, TruncSize), erl_syntax
Neotoma erlang:put(records,[]),
mochiglobal %% .app file should either be in the outdir, or the same dir merl erlydtl %% as the source file
Conclusion guess_application(proplists:get_value(outdir, Options), hd(AST)), walk_ast([], AST).
⌃ ⇧ ⌥walk_clauses(Acc,[])-> ⌅ lists:reverse(Acc);
walk_clauses(Acc,[{clause, Line, Arguments, Guards, Body}|T]) -> walk_clauses([{clause, Line, Arguments, Guards, walk_body([], Body)}|Acc], T).
⌃ ⇧ ⌥walk_body(Acc,[])-> ⌅ lists:reverse(Acc);
walk_body(Acc,[H|T]) -> walk_body([transform_statement(H)|Acc], T).
⌃ ⇧
Parse Transforms - lager Recursing through the AST
EUC 2015
Sean Cribbs ⌥walk_ast(Acc,[{function, Line, Name, Arity, Clauses}|T]) -> ⌅ Background put(function, Name), Macros walk_ast([{function, Line, Name, Arity, eunit walk_clauses([], Clauses)}|Acc], T); Parse Tr a n s f o rm s lager ⌃ ⇧ parse_trans
Syntax Trees
erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion ⌥walk_body(Acc,[])-> ⌅ lists:reverse(Acc);
walk_body(Acc,[H|T]) -> walk_body([transform_statement(H)|Acc], T).
⌃ ⇧
Parse Transforms - lager Recursing through the AST
EUC 2015
Sean Cribbs ⌥walk_ast(Acc,[{function, Line, Name, Arity, Clauses}|T]) -> ⌅ Background put(function, Name), Macros walk_ast([{function, Line, Name, Arity, eunit walk_clauses([], Clauses)}|Acc], T); Parse Tr a n s f o rm s lager ⌃ ⇧ parse_trans ⌥walk_clauses(Acc,[])-> ⌅ Syntax Trees lists:reverse(Acc); erl_syntax
Neotoma walk_clauses(Acc,[{clause, Line, Arguments, Guards, Body}|T]) -> mochiglobal walk_clauses([{clause, Line, Arguments, Guards, walk_body([], Body)}|Acc], T). merl
erlydtl
Conclusion ⌃ ⇧ Parse Transforms - lager Recursing through the AST
EUC 2015
Sean Cribbs ⌥walk_ast(Acc,[{function, Line, Name, Arity, Clauses}|T]) -> ⌅ Background put(function, Name), Macros walk_ast([{function, Line, Name, Arity, eunit walk_clauses([], Clauses)}|Acc], T); Parse Tr a n s f o rm s lager ⌃ ⇧ parse_trans ⌥walk_clauses(Acc,[])-> ⌅ Syntax Trees lists:reverse(Acc); erl_syntax
Neotoma walk_clauses(Acc,[{clause, Line, Arguments, Guards, Body}|T]) -> mochiglobal walk_clauses([{clause, Line, Arguments, Guards, walk_body([], Body)}|Acc], T). merl
erlydtl
Conclusion ⌃ ⇧ ⌥walk_body(Acc,[])-> ⌅ lists:reverse(Acc);
walk_body(Acc,[H|T]) -> walk_body([transform_statement(H)|Acc], T).
⌃ ⇧ Parse Transforms - lager Transforming matching calls
EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse transform_statement({call, Line,{remote, _Line1,{atom, _Line2, lager}, Tr a n s f o rm s ⌥ ⌅
lager {atom, _Line3, Severity}}, Arguments0}=Stmt) -> parse_trans case lists:member(Severity,?LEVELS) of Syntax Trees false -> Stmt; %% NB: Don’t modify if it isn’t a severity level! erl_syntax
Neotoma true -> %%...
mochiglobal end; merl
erlydtl
Conclusion ⌃ ⇧ Parse Transforms - lager Checking the logging conditions
EUC 2015
Sean Cribbs ⌥LevelVar = make_varname("__Level", Line), ⌅ Background TracesVar = make_varname("__Traces", Line),
Macros PidVar = make_varname("__Pid", Line), eunit %% case {whereis(lager_event), Parse Tr a n s f o rm s %% lager_config:get(loglevel,{?LOG_NONE, []})} of lager {’case’, Line, parse_trans {tuple, Line, Syntax Trees
erl_syntax [{call, Line,{atom, Line, whereis}, [{atom, Line, lager_event}]},
Neotoma {call, Line,{remote, Line,{atom, Line, lager_config}, {atom, mochiglobal merl Line, get}}, erlydtl [{atom, Line, loglevel}, {tuple, Line,[{integer, Line,0}, Conclusion {nil, Line}]}]}]}, [ %% case clauses ...
]}
⌃ ⇧ Parse Transforms - lager The log dispatch clause
EUC 2015
Sean Cribbs ⌥{clause, Line, ⌅ Background %% Match [{tuple, Line,[{var, Line, PidVar}, {tuple, Line,[{var, Line, LevelVar}, Macros eunit {var, Line,
Parse TracesVar}]}]}], Tr a n s f o rm s
lager % ... parse_trans % Syntax Trees % erl_syntax
Neotoma %
mochiglobal % merl erlydtl % Conclusion % % % % %
%
⌃ ⇧ Parse Transforms - lager The log dispatch clause
EUC 2015
Sean Cribbs ⌥{clause, Line, ⌅ Background %% Match [{tuple, Line,[{var, Line, PidVar}, {tuple, Line,[{var, Line, LevelVar}, Macros eunit {var, Line,
Parse TracesVar}]}]}], Tr a n s f o rm s
lager %% Guards parse_trans [[{op, Line, ’orelse’, Syntax Trees {op, Line, ’/=’,{op, Line, ’band’,{var, Line, LevelVar}, erl_syntax
Neotoma {integer, Line,
mochiglobal SeverityAsInt}}, merl erlydtl {integer, Line,0}}, Conclusion {op, Line, ’/=’,{var, Line, TracesVar}, {nil, Line}}}]], % ... % % %
%
⌃ ⇧ Parse Transforms - lager The log dispatch clause
EUC 2015
Sean Cribbs ⌥{clause, Line, ⌅ Background %% Match [{tuple, Line,[{var, Line, PidVar}, {tuple, Line,[{var, Line, LevelVar}, Macros eunit {var, Line,
Parse TracesVar}]}]}], Tr a n s f o rm s
lager %% Guards parse_trans [[{op, Line, ’orelse’, Syntax Trees {op, Line, ’/=’,{op, Line, ’band’,{var, Line, LevelVar}, erl_syntax
Neotoma {integer, Line,
mochiglobal SeverityAsInt}}, merl erlydtl {integer, Line,0}}, Conclusion {op, Line, ’/=’,{var, Line, TracesVar}, {nil, Line}}}]], %% Statements [ %% do the call to lager:dispatch_log {call, Line,{remote, Line,{atom, Line, lager}, {atom, Line, do_log}},
[ % ...
⌃ ⇧ Parse Transforms - parse_trans github.com/uwiger/parse_trans
EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Tr a n s f o rm s
lager
parse_trans
Syntax Trees Don’t worry. . . erl_syntax Ulf has your back! Neotoma
mochiglobal
merl
erlydtl
Conclusion ⌥do_transform({call, _Line,{remote, _Line1,{atom, _Line2, lager}, ⌅ {atom, _Line3, _Severity}}, _Arguments0}=Stmt) -> %% Do what we did before... transform_statement(Stmt);
do_transform(_) -> continue.
⌃ ⇧
Parse Transforms - parse_trans Rewriting lager’s transform with parse_trans
EUC 2015
Sean Cribbs Write transformations as “visitors” instead of manual
Background recursion
Macros Return NewForm to replace the current form eunit
Parse Return continue to recurse into subexpressions Tr a n s f o rm s
lager parse_trans ⌥parse_transform(AST, Options) -> ⌅ Syntax Trees %% Previously: walk_ast([], AST)
erl_syntax parse_trans:plain_transform(fun do_transform/1, AST). Neotoma
mochiglobal
merl erlydtl ⌃ ⇧
Conclusion Parse Transforms - parse_trans Rewriting lager’s transform with parse_trans
EUC 2015
Sean Cribbs Write transformations as “visitors” instead of manual
Background recursion
Macros Return NewForm to replace the current form eunit
Parse Return continue to recurse into subexpressions Tr a n s f o rm s
lager parse_trans ⌥parse_transform(AST, Options) -> ⌅ Syntax Trees %% Previously: walk_ast([], AST)
erl_syntax parse_trans:plain_transform(fun do_transform/1, AST). Neotoma
mochiglobal
merl erlydtl ⌃ ⇧ do_transform({call, _Line,{remote, _Line1,{atom, _Line2, lager}, Conclusion ⌥ ⌅ {atom, _Line3, _Severity}}, _Arguments0}=Stmt) -> %% Do what we did before... transform_statement(Stmt);
do_transform(_) -> continue.
⌃ ⇧ Parse Transforms - parse_trans Other cool stu !
EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Tr a n s f o rm s
lager
parse_trans ct_expand -compile-timeevaluation
Syntax Trees exprecs -generatesrecord-accessorfunctions erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion Cons: Hides “magic” Di cult to write/debug Only modifies current module
Good for: Injecting optimizations or new semantics Embedded DSLs Generating code in same module
Parse Transforms - Summary
EUC 2015
Sean Cribbs
Background Pros: Macros Powerful eunit
Parse Erlang syntax Tr a n s f o rm s lager Compile-time parse_trans computation Syntax Trees
erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion Good for: Injecting optimizations or new semantics Embedded DSLs Generating code in same module
Parse Transforms - Summary
EUC 2015
Sean Cribbs
Background Pros: Cons: Macros Powerful Hides “magic” eunit
Parse Erlang syntax Di cult to write/debug Tr a n s f o rm s lager Compile-time Only modifies current parse_trans computation module Syntax Trees
erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion Parse Transforms - Summary
EUC 2015
Sean Cribbs
Background Pros: Cons: Macros Powerful Hides “magic” eunit
Parse Erlang syntax Di cult to write/debug Tr a n s f o rm s lager Compile-time Only modifies current parse_trans computation module Syntax Trees
erl_syntax
Neotoma
mochiglobal merl Good for: erlydtl Injecting optimizations or new semantics Conclusion Embedded DSLs Generating code in same module EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Tr a n s f o rm s
lager parse_trans Te c h n i q u e 3 Syntax Trees
erl_syntax
Neotoma mochiglobal Syntax Trees merl
erlydtl
Conclusion Syntax Trees
EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Tr a n s f o rm s
lager
parse_trans Generates code by constructing syntax trees Syntax Trees Operates over Abstract Forms erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion Syntax Trees - erl_syntax
EUC 2015
Sean Cribbs Datatype for Abstract forms
Background Functions for every construct
Macros
eunit
Parse Tr a n s f o rm s
lager
parse_trans
Syntax Trees
erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion Inspecting nodes: type/1 float_value/1 attribute_name/1
Converting: abstract/1 revert/1 revert_forms/1
Tr a v e r s i n g : subtrees/1
Syntax Trees - erl_syntax
EUC 2015
Sean Cribbs
Background Creating nodes: Macros integer/1 float/1 atom/1 variable/1 eunit list/2 cons/2 tuple/1 Parse Tr a n s f o rm s block_expr/1 clause/2,3 fun_expr/1 lager parse_trans conjunction/1 disjunction/1
Syntax Trees function/2 attribute/2 form_list/1 erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion Converting: abstract/1 revert/1 revert_forms/1
Tr a v e r s i n g : subtrees/1
Syntax Trees - erl_syntax
EUC 2015
Sean Cribbs
Background Creating nodes: Macros integer/1 float/1 atom/1 variable/1 eunit list/2 cons/2 tuple/1 Parse Tr a n s f o rm s block_expr/1 clause/2,3 fun_expr/1 lager parse_trans conjunction/1 disjunction/1
Syntax Trees function/2 attribute/2 form_list/1 erl_syntax
Neotoma
mochiglobal Inspecting nodes: merl type/1 float_value/1 attribute_name/1 erlydtl
Conclusion Tr a v e r s i n g : subtrees/1
Syntax Trees - erl_syntax
EUC 2015
Sean Cribbs
Background Creating nodes: Macros integer/1 float/1 atom/1 variable/1 eunit list/2 cons/2 tuple/1 Parse Tr a n s f o rm s block_expr/1 clause/2,3 fun_expr/1 lager parse_trans conjunction/1 disjunction/1
Syntax Trees function/2 attribute/2 form_list/1 erl_syntax
Neotoma
mochiglobal Inspecting nodes: merl type/1 float_value/1 attribute_name/1 erlydtl
Conclusion Converting: abstract/1 revert/1 revert_forms/1 Syntax Trees - erl_syntax
EUC 2015
Sean Cribbs
Background Creating nodes: Macros integer/1 float/1 atom/1 variable/1 eunit list/2 cons/2 tuple/1 Parse Tr a n s f o rm s block_expr/1 clause/2,3 fun_expr/1 lager parse_trans conjunction/1 disjunction/1
Syntax Trees function/2 attribute/2 form_list/1 erl_syntax
Neotoma
mochiglobal Inspecting nodes: merl type/1 float_value/1 attribute_name/1 erlydtl
Conclusion Converting: abstract/1 revert/1 revert_forms/1
Tr a v e r s i n g : subtrees/1 Syntax Trees - Neotoma v1 Getting out my shinebox
EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Tr a n s f o rm s
lager
parse_trans
Syntax Trees
erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion ⌥generate_module_attrs(ModName, Combinators) -> ⌅ ["-module(", atom_to_list(ModName),").\n", "-export([parse/1,file/1]).\n", [ generate_combinator_macro(C)||Combinators /= undefined,
C <- Combinators ], "\n" ].
⌃ ⇧
Syntax Trees - Neotoma v1 github.com/seancribbs/neotoma
EUC 2015
Sean Cribbs ⌥quoted_string <- single_quoted_string / double_quoted_string ⌅ Background %{
Macros used_combinator(p_string), eunit lists:flatten(["p_string(<<\"", Parse escape_string(unicode:characters_to_list(proplists:get_value(string, Node))), Tr a n s f o rm s lager "\">>)"]) parse_trans %}; Syntax Trees
erl_syntax Neotoma ⌃ ⇧ mochiglobal
merl
erlydtl
Conclusion Syntax Trees - Neotoma v1 github.com/seancribbs/neotoma
EUC 2015
Sean Cribbs ⌥quoted_string <- single_quoted_string / double_quoted_string ⌅ Background %{
Macros used_combinator(p_string), eunit lists:flatten(["p_string(<<\"", Parse escape_string(unicode:characters_to_list(proplists:get_value(string, Node))), Tr a n s f o rm s lager "\">>)"]) parse_trans %}; Syntax Trees
erl_syntax Neotoma ⌃ ⇧ mochiglobal ⌥generate_module_attrs(ModName, Combinators) -> ⌅ merl ["-module(", atom_to_list(ModName),").\n", erlydtl "-export([parse/1,file/1]).\n", Conclusion [ generate_combinator_macro(C)||Combinators /= undefined,
C <- Combinators ], "\n" ].
⌃ ⇧ Syntax Trees - Neotoma v2
EUC 2015
Sean Cribbs ⌥quoted_string <- single_quoted_string / double_quoted_string ⌅ Background %{
Macros #string{string = unicode:characters_to_binary(proplists:get_value(string, eunit Node)), Parse index = Idx} Tr a n s f o rm s lager %}; parse_trans
Syntax Trees
erl_syntax ⌃ ⇧
Neotoma
mochiglobal
merl
erlydtl
Conclusion Syntax Trees - Neotoma v2
EUC 2015
Sean Cribbs ⌥case Input of ⌅ Background <<"a string"/binary, Input1/binary>> -> % ...do the success path;
Macros _ -> % ...do the failure path eunit end Parse Tr a n s f o rm s lager ⌃ ⇧ parse_trans
Syntax Trees
erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion Syntax Trees - Neotoma v2
EUC 2015
Sean Cribbs ⌥case Input of ⌅ Background <<"a string"/binary, Input1/binary>> -> % ...do the success path;
Macros _ -> % ...do the failure path eunit end Parse Tr a n s f o rm s lager ⌃ ⇧ generate(#string{string=S}, InputName, Success, Fail) -> parse_trans ⌥ ⌅ Syntax Trees erl_syntax ⌃ ⇧ Neotoma
mochiglobal
merl
erlydtl
Conclusion Syntax Trees - Neotoma v2
EUC 2015
Sean Cribbs ⌥case Input of ⌅ Background <<"a string"/binary, Input1/binary>> -> % ...do the success path;
Macros _ -> % ...do the failure path eunit end Parse Tr a n s f o rm s lager ⌃ ⇧ generate(#string{string=S}, InputName, Success, Fail) -> parse_trans ⌥ ⌅ Syntax Trees Literal = abstract(S), % convert term to syntaxTree() erl_syntax RestName = variable(new_name("Input")), Neotoma
mochiglobal
merl
erlydtl ⌃ ⇧
Conclusion Syntax Trees - Neotoma v2
EUC 2015
Sean Cribbs ⌥case Input of ⌅ Background <<"a string"/binary, Input1/binary>> -> % ...do the success path;
Macros _ -> % ...do the failure path eunit end Parse Tr a n s f o rm s lager ⌃ ⇧ generate(#string{string=S}, InputName, Success, Fail) -> parse_trans ⌥ ⌅ Syntax Trees Literal = abstract(S), erl_syntax RestName = variable(new_name("Input")), Neotoma
mochiglobal case_expr(InputName, % case ... of ... end
merl
erlydtl
Conclusion ⌃ ⇧ Syntax Trees - Neotoma v2
EUC 2015
Sean Cribbs ⌥case Input of ⌅ Background <<"a string"/binary, Input1/binary>> -> % ...do the success path;
Macros _ -> % ...do the failure path eunit end Parse Tr a n s f o rm s lager ⌃ ⇧ generate(#string{string=S}, InputName, Success, Fail) -> parse_trans ⌥ ⌅ Syntax Trees Literal = abstract(S), erl_syntax RestName = variable(new_name("Input")), Neotoma
mochiglobal case_expr(InputName,
merl [clause([binary([binary_field(Literal,[atom("binary")]), erlydtl binary_field(RestName,[atom("binary")])])], Conclusion none,
Success(Literal, RestName)), % success path!
⌃ ⇧ Syntax Trees - Neotoma v2
EUC 2015
Sean Cribbs ⌥case Input of ⌅ Background <<"a string"/binary, Input1/binary>> -> % ...do the success path;
Macros _ -> % ...do the failure path eunit end Parse Tr a n s f o rm s lager ⌃ ⇧ generate(#string{string=S}, InputName, Success, Fail) -> parse_trans ⌥ ⌅ Syntax Trees Literal = abstract(S), erl_syntax RestName = variable(new_name("Input")), Neotoma
mochiglobal case_expr(InputName,
merl [clause([binary([binary_field(Literal,[atom("binary")]), erlydtl binary_field(RestName,[atom("binary")])])], Conclusion none, Success(Literal, RestName)), clause([underscore()], none,
Fail(InputName, error_reason({string, S})))]); % fail path!
⌃ ⇧ Syntax Trees - mochiglobal github.com/mochi/mochiweb
EUC 2015
Sean Cribbs Memoizes frequently used values in code Background Good for high-read, low-write scenarios Macros
eunit %% @doc Store term V at K, replaces an existing term if present. Parse ⌥ ⌅ Tr a n s f o rm s put(K, V) -> lager
parse_trans put(K, V, key_to_module(K)).
Syntax Trees erl_syntax ⌃ ⇧ Neotoma
mochiglobal
merl
erlydtl
Conclusion Syntax Trees - mochiglobal github.com/mochi/mochiweb
EUC 2015
Sean Cribbs Memoizes frequently used values in code Background Good for high-read, low-write scenarios Macros
eunit %% @doc Store term V at K, replaces an existing term if present. Parse ⌥ ⌅ Tr a n s f o rm s put(K, V) -> lager
parse_trans put(K, V, key_to_module(K)).
Syntax Trees erl_syntax put(_K, V, Mod) -> Neotoma mochiglobal Bin = compile(Mod, V), merl code:purge(Mod), erlydtl
Conclusion {module, Mod}=code:load_binary(Mod, atom_to_list(Mod)++".erl", Bin), ok.
⌃ ⇧ ⌥forms(Module, T) -> ⌅ [erl_syntax:revert(X)||X <- term_to_abstract(Module, term, T)].
⌃ ⇧
Syntax Trees - mochiglobal
EUC 2015
Sean Cribbs
Background
Macros eunit ⌥-spec compile(atom(), any()) -> binary(). ⌅ Parse compile(Module, T) -> Tr a n s f o rm s
lager {ok, Module, Bin}=compile:forms(forms(Module, T), parse_trans [verbose, report_errors]), Syntax Trees Bin. erl_syntax
Neotoma
mochiglobal
merl ⌃ ⇧
erlydtl
Conclusion Syntax Trees - mochiglobal
EUC 2015
Sean Cribbs
Background
Macros eunit ⌥-spec compile(atom(), any()) -> binary(). ⌅ Parse compile(Module, T) -> Tr a n s f o rm s
lager {ok, Module, Bin}=compile:forms(forms(Module, T), parse_trans [verbose, report_errors]), Syntax Trees Bin. erl_syntax
Neotoma mochiglobal ⌃ ⇧ merl forms(Module, T) -> erlydtl ⌥ ⌅ [erl_syntax:revert(X)||X <- term_to_abstract(Module, term, T)]. Conclusion
⌃ ⇧ Syntax Trees - mochiglobal
EUC 2015
Sean Cribbs ⌥term_to_abstract(Module, Getter, T) -> ⌅ Background [%% -module(Module).
Macros erl_syntax:attribute( eunit erl_syntax:atom(module), Parse [erl_syntax:atom(Module)]), Tr a n s f o rm s
lager parse_trans ⌃ ⇧ Syntax Trees
erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion Syntax Trees - mochiglobal
EUC 2015
Sean Cribbs ⌥term_to_abstract(Module, Getter, T) -> ⌅ Background [%% -module(Module).
Macros erl_syntax:attribute( eunit erl_syntax:atom(module), Parse [erl_syntax:atom(Module)]), Tr a n s f o rm s lager %% -export([Getter/0]). parse_trans erl_syntax:attribute( Syntax Trees erl_syntax:atom(export), erl_syntax Neotoma [erl_syntax:list( mochiglobal
merl [erl_syntax:arity_qualifier( erlydtl erl_syntax:atom(Getter), Conclusion erl_syntax:integer(0))])]),
⌃ ⇧ Syntax Trees - mochiglobal
EUC 2015
Sean Cribbs ⌥term_to_abstract(Module, Getter, T) -> ⌅ Background [%% -module(Module).
Macros erl_syntax:attribute( eunit erl_syntax:atom(module), Parse [erl_syntax:atom(Module)]), Tr a n s f o rm s lager %% -export([Getter/0]). parse_trans erl_syntax:attribute( Syntax Trees erl_syntax:atom(export), erl_syntax Neotoma [erl_syntax:list( mochiglobal
merl [erl_syntax:arity_qualifier( erlydtl erl_syntax:atom(Getter), Conclusion erl_syntax:integer(0))])]),
%% Getter() -> T. erl_syntax:function( erl_syntax:atom(Getter), [erl_syntax:clause([], none,[erl_syntax:abstract(T)])])].
⌃ ⇧ Combines strategies of:
Macros - ?Q(Text), ?Q(Text,Env)
Parse Transforms Syntax Tree Generation
Included in OTP 18!!!!
Syntax Trees - merl github.com/richcarl/merl
EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Tr a n s f o rm s
lager
parse_trans
Syntax Trees
erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion Don’t worry. . . Richard has your back! Included in OTP 18!!!!
Syntax Trees - merl github.com/richcarl/merl
EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Combines strategies of: Tr a n s f o rm s lager Macros - ?Q(Text), parse_trans ?Q(Text,Env) Syntax Trees erl_syntax Parse Transforms Neotoma
mochiglobal
merl Syntax Tree Generation
erlydtl
Conclusion Don’t worry. . . Richard has your back! Syntax Trees - merl github.com/richcarl/merl
EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Combines strategies of: Tr a n s f o rm s lager Macros - ?Q(Text), parse_trans ?Q(Text,Env) Syntax Trees erl_syntax Parse Transforms Neotoma
mochiglobal
merl Syntax Tree Generation
erlydtl
Conclusion Don’t worry. . . Included in OTP 18!!!! Richard has your back! ⌥Function1 =?Q("_@FunctionName@(_Variables) -> _@FunctionName@(_Variables, [])"), ⌅ ⌃ ⇧
Syntax Trees - erlydtl github.com/erlydtl/erlydtl
EUC 2015
Sean Cribbs Implements Django-style templates
Background Moved from using erl_syntax to merl last year Macros
eunit Function1 = erl_syntax:function( Parse ⌥ ⌅ Tr a n s f o rm s erl_syntax:atom(FunctionName),
lager [erl_syntax:clause( parse_trans [erl_syntax:variable("_Variables")], Syntax Trees erl_syntax none, Neotoma
mochiglobal [erl_syntax:application( merl none, erl_syntax:atom(FunctionName), erlydtl [erl_syntax:variable("_Variables"), erl_syntax:list([])]) Conclusion ]) ]),
⌃ ⇧ Syntax Trees - erlydtl github.com/erlydtl/erlydtl
EUC 2015
Sean Cribbs Implements Django-style templates
Background Moved from using erl_syntax to merl last year Macros
eunit Function1 = erl_syntax:function( Parse ⌥ ⌅ Tr a n s f o rm s erl_syntax:atom(FunctionName),
lager [erl_syntax:clause( parse_trans [erl_syntax:variable("_Variables")], Syntax Trees erl_syntax none, Neotoma
mochiglobal [erl_syntax:application( merl none, erl_syntax:atom(FunctionName), erlydtl [erl_syntax:variable("_Variables"), erl_syntax:list([])]) Conclusion ]) ]),
⌃ ⇧ ⌥Function1 =?Q("_@FunctionName@(_Variables) -> _@FunctionName@(_Variables, [])"), ⌅ ⌃ ⇧ Cons: Verbose Many manual steps AST understanding may be needed
Good for: Implementing new languages & External DSLs “Run-time” code generation
Syntax Trees - Summary
EUC 2015
Sean Cribbs
Background Pros: Macros eunit Most versatile Parse Tr a n s f o rm s Powerful tools lager parse_trans Multiple output Syntax Trees destinations erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion Good for: Implementing new languages & External DSLs “Run-time” code generation
Syntax Trees - Summary
EUC 2015
Sean Cribbs
Background Pros: Cons: Macros eunit Most versatile Verbose Parse Tr a n s f o rm s Powerful tools Many manual steps lager parse_trans Multiple output AST understanding may Syntax Trees destinations be needed erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion Syntax Trees - Summary
EUC 2015
Sean Cribbs
Background Pros: Cons: Macros eunit Most versatile Verbose Parse Tr a n s f o rm s Powerful tools Many manual steps lager parse_trans Multiple output AST understanding may Syntax Trees destinations be needed erl_syntax
Neotoma
mochiglobal
merl
erlydtl Good for: Conclusion Implementing new languages & External DSLs “Run-time” code generation Use erl_syntax, parse_trans,andmerl!
Build cool tools!
Conclusion
EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Metaprogramming Erlang is great! Tr a n s f o rm s
lager
parse_trans
Syntax Trees
erl_syntax
Neotoma
mochiglobal
merl
erlydtl
Conclusion Build cool tools!
Conclusion
EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Metaprogramming Erlang is great! Tr a n s f o rm s
lager
parse_trans
Syntax Trees erl_syntax Use erl_syntax, parse_trans,andmerl! Neotoma
mochiglobal
merl
erlydtl
Conclusion Conclusion
EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Metaprogramming Erlang is great! Tr a n s f o rm s
lager
parse_trans
Syntax Trees erl_syntax Use erl_syntax, parse_trans,andmerl! Neotoma
mochiglobal
merl
erlydtl
Conclusion Build cool tools! EUC 2015
Sean Cribbs
Background
Macros
eunit
Parse Tr a n s f o rm s lager Thanks! parse_trans
Syntax Trees
erl_syntax
Neotoma
mochiglobal
merl erlydtl Tw i t t e r / G i t h u b : seancribbs Conclusion