You can run the program by calling the start function, but first you must generate a cookie and start two named Erlang nodes using the following commands:

1. echo -n “dh32d8yhd8″ > erlang.cookie
2. erl -name Node1 -cookie
3. erl -name Node2 -cookie

Collective Intelligence is a shared intelligence that emerges from the collaboration or competition amongst many individuals, it is used by online retailers such as Amazon.com and Netflix to recommend books and movies.  Today we’ll use these algorithms to compare user preference, and in subsequent posts apply see how these algorithms are used in recommendation systems and clustering algorithms.

Before we continue, there are two unfamiliar concepts which we need to review: list Comprehensions and dictionaries.

List comprehensions are expressions (which are very common to functional programming languages) that allow you to create lists without having to use funs, maps, or filters. This helps you to write programs which are shorter and easier to debug. In Erlang, the syntax is:

[ F(X) || X <- L]

which means, create a list of F(X) where F is a function and X is an element taken from the list L. Additionally, in Erlang we can use qualifiers to filter out elements of the list L.

Dictionaries (Hashes/Maps) are not directly supported by Erlang and to use them in your code you must use the dict module. The module includes the function dict:from_list which converts a list of two element tuples into a dictionary; it also includes the function dict:fetch which takes the Key and the dictionary as an argument and returns the value which matches the key.

Similarity

Below is a list of movie ratings by user

which we can store in Erlang as a dictionary:

  dict:from_list([
	{
	"Lisa Rose",
	dict:from_list( [{"Lady in the Water", 2.5},{"Snakes on a Plane", 3.5},
				 {"Just My Luck", 3.0},{"Superman Returns", 3.5},
				 {"You, Me and Dupree", 2.5},{"The Night Listener", 3.0}])
	},
	{"Mick LaSalle",
	dict:from_list( [{"Lady in the Water", 3.0},{"Snakes on a Plane", 4.0},
				 {"Just My Luck", 2.0},{"Superman Returns", 3.0},
				 {"You, Me and Dupree", 2.0},{"The Night Listener", 3.0}])
	},
	{"Jack Matthews",
	dict:from_list( [{"Lady in the Water", 3.0},{"Snakes on a Plane", 4.0},
				 {"Superman Returns", 5.0},{"You, Me and Dupree", 3.5},
				 {"The Night Listener", 3.0}])
	},
	{"Shey",
	dict:from_list( [{"Snakes on a Plane", 2.5},{"Superman Returns", 3.0},
				 {"You, Me and Dupree", 3.0}])}
	]).

To determine how similar two users preferences are we need to compute a similarity score. An easy way to calculate similarity is to use the scaled Euclidean Distance formula which we learned back in high school geometry. The function returns a value between 0 and 1, where a value of 1 means that two people have identical tastes.

sim_distance(List1, List2) when length(List1) /= 0, length(List1) == length(List1)  ->
  Sum_of_squares = lists:sum([ math:pow(X-Y,2) || {X,Y} <- lists:zip(List1,List2) ]),
  1.0 / (1.0 + math:sqrt(Sum_of_squares)).

The function accepts two lists, each lists contains the user’s rating of a movie, where the ith element of each lists represents the user’s rating for the same movie. Note the use of the lists:zip function which takes two lists of equal length and returns one list of tuples with two elements each, where the first element of each tuple is taken from the first list and the second element is taken from corresponding element in the second list. Finally, the function also has two guard expressions that prevent the user from calling the function with empty lists or lists of unequal length. Another simple distance metric is the Manhattan distance, it represents the absolute differences between coordinates of a pair of objects:

sim_manhattan(X,Y) when length(X) /= 0,  length(X) == length(Y)  ->
  Sum = lists:sum([ abs(Xn-Yn) || {Xn,Yn} <- lists:zip(X,Y)]),
  1.0/ (1.0 + Sum).

In this scenario, a slightly better way to determine the similarity between two objects is to use a Pearson correlation coefficient. The correlation coefficient is a measure of how well two sets of data fit on a straight line and it gives better results in when the data isn’t well normalized– for example, if critics’ movie rankings are routinely harsher than average. You can read about other other distance metrics for comparing items on Wikipedia.

sim_pearson(X, Y) when length(X) == length(Y)  ->
  Len = length(X),

  %% sum
  SumX = lists:sum(X),
  SumY = lists:sum(Y),

  %% sum of squares
  SumXSq=lists:sum([math:pow(N,2) || N <- X]),
  SumYSq=lists:sum([math:pow(N,2) || N <- Y]),

  %% sum of products
  Sum_products = lists:sum( [ A*B || {A,B} <- lists:zip(X,Y)]),

  %% Calculate Pearson Score
  Numer = Sum_products - (SumX*SumY/Len),

  case math:sqrt( (SumXSq - math:pow(SumX,2) /Len) * (SumYSq - math:pow(SumY,2)/Len)) of
    0.0 ->
      0.0;
    Denom ->
      Numer/Denom %% Don't put a comma or semi-colon here
  end.

A complete listing of the code is available online; you can experiment with the different distance metrics using the main function, but due to some limitations of the command line interface, you’ll have to assign the function to a variable before passing it to the main function:

1> c(pci).
2> SimPearson = fun(X,Y) -> pci:sim_pearson(X,Y) end.
3> pci:main("Mick LaSalle", "Lisa Rose", SimPearson).

While this code is (mostly) functional, we haven’t taken advantage of Erlang’s distributed processes, soon we’ll explore the K-Means clustering algorithm which is very well suited to parallel implementations.

Your comments and Feedback are appreciated.

-module(star).
-export([main/3, vertex_node/0, center_node/4]).

%% Write a function which starts N processes in a star,
%% and sends a message to each of them M times.
%% After the messages have been sent the processes
%% should terminate gracefully.

vertex_node() ->
  receive
    {stop, Center_PID} ->
      io:format("recieved ~w, from ~w, exiting~n", [stop, Center_PID]),
	  exit(ok);
	{Something, Center_PID} ->
		io:format("recieved message: ~s, from ~w~n", [Something, Center_PID]),
		Center_PID ! {self(), ok},
		vertex_node()
  end.

center_node(_, _, 0, Original_Nodes ) ->
	lists:map(fun(Vertex_Node) -> Vertex_Node ! {stop, self()} end, Original_Nodes);

center_node(Message, [], Count, Origingal_Nodes) ->
	center_node(Message, Origingal_Nodes, Count -1, Origingal_Nodes);

center_node(Message, Nodes, Count, Original_Node_List) ->
	[Vertex_Node | Remaining_Nodes] = Nodes,
	Vertex_Node ! {Message, self()},
	receive
		{Pid, Msg} ->
			io:format("center received ack: ~w, from ~w~n", [Msg, Pid])
	end,
	center_node(Message, Remaining_Nodes, Count, Original_Node_List).

main(Processes, Messages, Message) ->
	Nodes = lists:map(fun(X) -> spawn(star, vertex_node, []) end, lists:seq(1, Processes)),
	spawn(star, center_node, [Message, Nodes, Messages, Nodes]).

A nicely formatted version is available here.

P.S. If anyone knows how to do add syntax highlighting for Erlang to Windows Live Writer, please let me know.

We need three Erlang primitives to work with processes: spawn,  ! and receive.  The built-in spawn function creates a new process executing a function and returns the new process’ process id. The ! operator sends a message to a process and processes use the receive...end function to match messages in the mailbox to patterns and execute the appropriate functions.

To send a message “BUY MSFT” to the Seller process, we use:

   Seller ! {{buy, msft} , Buyer}

we include the variable Buyer which contains the process id of the buyer, this lets the seller know which process to send the acknowledgment to.  Now we need to create the Seller process, we do this by calling the spawn/3 function– the first argument to spawn is the module name, followed by the function name and ending with a list of arguments:

   Seller_pid = spawn(exchange, seller, [])

Both the Buyer and Seller processes can handle the message using pattern matching in the receive..end clause, Here the Seller process receives the {buy, Symbol} message, processes it, and then acknowledges the transaction by sending an “ok” to the buyer:

seller() ->
     receive
        {finished, _} ->
            io:format("Received finished. no one is buying~n", []);
        {{buy, Symbol}, Buyer_pid} ->
            io:format("sold ~s to buyer~n", [Symbol]),
            Buyer_pid ! ok,
            seller();
    end.

Now, lets put these pieces together in a program in which the Buyer process buys Stocks from the Seller and in return, the Seller process acknowledges the sale; the program terminates when the Buyer has run out of Stocks to buy.

-module(exchange).
-export([start/0, buyer/2, seller/0]).

buyer([], Seller_pid) ->
    Seller_pid ! {finished, self()},
    %% note there's a ";" instead of a "." at the end of the next clause
    io:format("buyer finished~n", []);

buyer(List, Seller_pid) ->
	[Head|Tail] = List,
    Seller_pid ! {Head, self()},
    receive
        ok ->
            io:format("buyer received seller confirmation~n", [])
    end,
    buyer(Tail, Seller_pid).

seller() ->
     receive
        {finished, _} ->
            io:format("Received finished. no one is buying~n", []);
        {{buy, Symbol}, Buyer_pid} ->
            io:format("sold ~s to buyer~n", [Symbol]),
            Buyer_pid ! ok,
            seller()
            %% note there's, no ";" or "," before the end
    end.

start() ->
  Things_to_buy = [{buy, sina}, {buy, msft}, {buy, intc}],
  Seller_pid = spawn(exchange, seller, []),
  spawn(exchange, buyer, [Things_to_buy, Seller_pid]).

The program illustrates the most basic ways to use Erlang processes, I’ve completely skipped over guards, error handling, timeouts and registered processes, but several detailed documents on these topics and  Erlang design principles are available on there ftp site.

Because I love abstractions, I’ll use the function pmap– pmap works like map, but when the function is called it creates one parallel process to evaluate each argument in the list. You can find a copy of the code on the book’s website. Generally, it’s not advisable to use pmap when the the list is very small or very large. Ideally, the implementation of pmap should be modified to spawn n processes, but that is something that I’m not comfortable doing yet.

Full Listing

   1:  -module(quote).

   2:  -import(json_parser).

   3:   

   4:  -export([get_stock_quote/1, get_data_in_parallel/0]).

   5:   

   6:  -define(BASE_URL, "http://www.google.com/finance/info?client=ig&q=".

   7:   

   8:  symbols() ->

   9:      [ "MSFT", "RHIE", "INTC", "DPTR", 

  10:        "RVSB", "BBGI", "SRDD", "DEAR",

  11:        "ALKS", "GOOG", "QQQQ", "AAPL",

  12:        "RIMM", "GEOY", "CBST", "ANGO"

  13:      ].

  14:      

  15:      

  16:  get_google_url(Symbol) ->

  17:      ?BASE_URL ++ Symbol.

  18:     

  19:  get_stock_quote(Symbol) ->

  20:      %% moved inets:start() outside this function

  21:      URL =  get_google_url(Symbol),

  22:      { ok, {_Status, _Headers, Body }} = http:request(URL),

  23:      PureData = lists:subtract(lists:subtract(Body, "// [ "), "] "),

  24:      {_,{_,RealData},_} = json_parser:dvm_parser(list_to_binary(PureData)),

  25:      RealData.

  26:      

  27:   

  28:  get_data_in_parallel() ->

  29:      inets:start(),

  30:      lib_misc:pmap(fun get_stock_quote/1, symbols()).

In Erlang, the -module directive defines an Erlang module, which is how code is organized in Erlang, the -export directive tells Erlang which functions in this module to expose. My Erlang module is called quote, and I’m storing it in the file quote.erl. I’m also exposing a single function get_stock_quote which accepts a single parameter.

-module(quote).
-export([get_stock_quote/1]).

Next I’ll define an Erlang macro called BASE_URL which contains the base URL of the Google Finance API. The function get_google_url builds the full URL by appending the symbol to the base URL.

-define(BASE_URL, "http://www.google.com/finance/info?client=ig&q=".
get_google_url(Symbol) ->
  ?BASE_URL ++ Symbol.

After retrieving a stock quote for MSFT in our browser I noticed that the data  returned from Google Finance is a JSON(ish) object surrounded by some extraneous text which has to be removed before we can do anything else.  Later I’ll use an external library to convert the string into a JSON object and extract the price and date.

get_stock_quote(Symbol) ->
  %% Don't know why I need the following line
  %% mentioned in inets documentation
  inets:start(),
  URL = get_google_url(Symbol),
  { ok, {_Status, _Headers, Body }} = http:request(URL),
  PureData = lists:subtract(lists:subtract(Body, "// [ "), "] ").

The code to make http request is simple, http:request() returns the HTTP status, headers and body but the call was throwing an exception on my machine; after reading the http documentation included with Erlang I learned that a call inets:start() has to be made before making a http request.  The variable PureData contains the string which will transform into our JSON object, I had to use the lists:subtract function to remove extra characters from the beginning and end of the string.

After some searching (and cursing) I found a third party library, json_parser, on Process One’s Comprehensive Erlang Archive Network. I downloaded a copy of the development version, renamed the source file to json_parser.erl, compiled json_praser and referenced it from quotes.erl with the -import(json_parser) import directive.

The documentation for json_parser is fairly sparse—the dvm_parser function in the library returns a tuple with more data then I need and I’m only interested in the actual JSON data which I’ parsed into the RealData variable and passed onto the parse_json_tuple function which extracts the fields I’m interested in.

{_,{_,RealData},_} = json_parser:dvm_parser(list_to_binary(PureData)),
parse_json_tuple(list_to_tuple(RealData)).

Finally, I parse the data in the RealData tuple and returned the CurrentPrice and Quote time:

parse_json_tuple(RealData) ->
  %% this is ugly and needs to be refactored
  {_,_,_,_,{_,CurrentPrice},_,{_,CurrentTime},_,_,_} = RealData,
  {CurrentPrice, CurrentTime}.

To use the code, you have to compile the quote.erl file by calling the  c(quote) function in the Erlang Shell, then call the get_stock_quote function with the a stock symbol: quote:get_stock_quote(“MSFT”) from the Erlang shell.

This isn’t the best use of Erlang, but I wanted to ease into Erlang before exploring the more complicated recursive and distributed functionality, later I’ll refactor the code to be more Erlang-y and modify the program to retrieve quotes in parallel using Erlang’s  concurrency magic.

*Edit*I Just discovered that Google Finance returns a slightly different dataset when the market is closed, I’ll update the parse_json_tuple function with the changes soon.

Full Listing

   1:  -module(quote).

   2:  -export([get_stock_quote/1]).

   3:  -import(json_parser).

   4:  

   5:  -define(BASE_URL, "http://www.google.com/finance/info?client=ig&q=".

   6:  

   7:  get_google_url(Symbol) ->

   8:      ?BASE_URL ++ Symbol.

   9:  

  10:  parse_json_tuple(RealData) ->

  11:      %% this is ugly and needs to be refactored

  12:     {_,_,_,_,{_,CurrentPrice},_,{_,CurrentTime},_,_,_} = RealData,

  13:      {CurrentPrice, CurrentTime}.

  14:  

  15:  get_stock_quote(Symbol) ->

  16:      %% Don't know why I need the following line

  17:      %% mentioned in inets documentation

  18:      inets:start(),

  19:      URL =  get_google_url(Symbol),

  20:      { ok, {_Status, _Headers, Body }} = http:request(URL),

  21:      PureData = lists:subtract(lists:subtract(Body, "// [ "), "] "),

  22:      {_,{_,RealData},_} = json_parser:dvm_parser(list_to_binary(PureData)),

  23:      parse_json_tuple(list_to_tuple(RealData)).