Dependency Injection is an important pattern to use when you wan to create classes that are easier to reuse and unit test. Since the dependencies are external, they can be configured and maintained outside class or function and there’s no need to change the code in order to reuse it. Also, when dependencies are injected into a class or a function it is possible to substitute a mock implementation of the dependency. In unit tests, mock objects are used as replacement for the real implementation, this helps isolate the functionality being tested.

A simple example

The following function has it’s dependencies, a list of numbers, passed to it, and since calculate_average only talks to a list of numbers it is easy to reuse.

def calculate_average(list_of_numbers):
  x = sum(list_of_numbers)
  return x/len(list_of_numbers)

Imagine a similar function which uses a database connection, retrieves a list of numbers then calculates the average.

def calculate_average():
  sql_query = "select numbers from table"
  list_of_numbers = db.query(sql_query)
  x = sum(list_of_numbers)
  return x/len(list_of_numbers)

This function is difficult to reuse because it depends on the database, future users of this function will have to either modify the function or create and maintain a database just to use it. More importantly it’s difficult to test this function. A test would require a database connection and a fixture to populate the database with data.

You should be thinking, but Shey, I would never do this with a simple function like this. You’re right, you wouldn’t do this with a simple function, but as soon as calculate_average turns into a more complicated calculation such as calculate_value_at_risk we start instantiating or dependencies.

A slightly less contrived example

The BillingService class is responsible for charging your customers a fee.

class BillingService
  def initialize(credit_card_processor)
    self.processor = processor

  def charge(account, amount):
    if account.has_balance:
      result = processor.charge(account.cc_number, amount)
    if result == "success":
      account.deduct(amount)

Lets assume that Authorize.net is offering better rates than PayPal. If the class accepts an implementation of a credit card processor as a dependency it is possible to switch to the authorize.net implementation of the credit card processor class without changing the BillingService class. Similarly during unit testing, a mock implementation of a credit card processor can be used to alwasy return “success” and isolate the charge functionality of the BillingService.

class mock_credit_card_processor:
  def charge(cc_number, amount):
    return "success"

    def test_charge():
      bs = BillingService(mock_credit_card_processor)
      a = Account(cc_number="411111111111111", balance=300.00)
      bs.charge(a, 300.00)
      assert(a.balance == 0)

Unfortunately Moneytracking has an annoying limitation which prevents you from tracking tags across accounts, so if you’re buying junk food with cash and credit tagging loses it appeal.  Luckily, they provide an API which will allow us to easily get around the tagging limitation programmatically. 

I can imagine some pretty cool analytics and infographics coming out of 6 months of data so I wrote Moneytrackin.rb, a Ruby interface to the MoneyTrackin API.  It exposes easy-to-use methods to interact with your Moneytrackin account. While it only implements the methods exposed by their API but I intend to add more functionality like tag searches across all accounts soon, imagine piping an amounte-weighted stream of tags to wordle!

With two gems, HTTParty and Mash, I was easily able to create the client interface.  I included the HTTParty module within connection class and it gave it a "get" method that retrieves data over HTTP.   Usage is simple:

response = self.class.get(self.class.base_uri + '/listProjects/')

Then I passed the response to Mash which converts the XML response into a hash that acts more like a an object., it also also recursively descends down the response converting hashes into Mashes making it easy to get the transactions and the tags associated with them.

1. download sqlitedll-3_6_18.zip,
2. copy the two files that are part of the archive to your Ruby bin directory;
3. run “gem install –version 1.2.3 sqlite3-ruby” from the command prompt.

Enjoy!

The few discussions I read online implied that Python doesn’t need an IoC framework because it is a dynamically typed language.  Dynamically typed languages eliminate the need to use interfaces, they do not do away with the need for inversion of control.

IoC is used to decouple components of an application, remove direct dependencies so that replacing a component will not have a side effects on the rest of the system.  As Dave Thomas explains

a DI application is written as a set of loosely coupled components. These components contain no knitting code: nothing in the application code itself is responsible, for example, for making sure that the necessary objects somehow get an instance of the database connection. Instead, the components all run in a container. This container is given a description of the knitting to be done (typically using an XML file). The container then instantiates objects and sets them into components that need them

I’m new to the language and there maybe a more “pythonic” way to handle inversion of control than using a container. Here is small and contrived example of inversion of control using Pinsor, a IoC container in Python. Those coming from a .NET or Java background will find Pinsor easy to use, but slightly lacking in features.

You can run the program by calling the start function, but first you must generate a cookie and start two named Erlang nodes using the following commands:

1. echo -n “dh32d8yhd8″ > erlang.cookie
2. erl -name Node1 -cookie
3. erl -name Node2 -cookie

Benchmark
The benchmarking script sets up 10 queues, 10 producers and 10 consumers, each producer writes to a single queue and each consumer reads from that queue ie: p1 writes to q1 and c1 reads from q1 as fast at it can.  The time measured is the time it takes a message to travel from p1 to the queue and to be read by c1 and it was calculated by appending the timestamp to the message itself.

I chose to run a series of tests with message size varying from 35 bytes to 400 bytes and the message count varying from 50 messages to 100 messages in an attempt to visualize the broker’s behavior under varying conditions.

The benchmark was performed using a server running 2.8GHz Pentium D with 903464 kB RAM running on Kernel 2.6.18-128.1.10.el5 using RabbitMQ 1.5.5 and a machine with similar specifications as the client—both the producer and consumer run on the same machine.  The network is a typical Fast Ethernet with two switches separating the server and client machines.

Conclusion
Inconclusive! The numbers are very surprising I wasn’t expecting the message transfer times to be greater than .5 seconds under this load.  It’s possible that the consumer may not be able to keep up with flood of messages, if that’s the case than a consumer written in C# is needed.  What is obvious is that more tests  with a constant message size and a varying number of messages  are needed.

The standard deviation is also greater than I expected, so I plan on creating longer running tests and see if the broker falls into a steady state.  Finally, I also want to test with 50 messages per producer using 20 queues/producers instead of 10  and see how RabbitMQ responds.

I’m wondering what sort of performance others are experiencing—any and all feedback is greatly appreciated.

Postscript
I intended on using C# to benchmark RabbitMQ but for want of time  I used a modified version of Colin Surprenant’s bunnypunisher ruby script.  There’s a slightly outdated (and dirty) version of the script available on github.  I will upload the latest version along with the “launcher” tomorrow morning.

Kyle Burton at Asymmetrical View has a fairly comprehensive comparison of RabbitMQ and Qpid.

The order management system I work on uses IRC as the messaging middle ware– a message is sent to a channel and another process reads and processes the message.  This gives us great flexibility, letting us write the user interface in .NET and the back-end server code in whichever language we choose and it also allows us to scale the system by moving processes to different machines.

But this setup has problems, if a message is missed by a process, there’s no easy to way to retrieve it and recent benchmarking showed that the latency of the messaging system spikes, unpredictably to 300 ms or more.  Initially IRC made sense but the requirements have grown and it’s clear a more robust solution which provides greater reliability, performance and scalability is needed.

AMQP
The Advanced Message Queuing Protocol (AMQP) is an open standard, vendor neutral binary protocol for messaging middleware. AMQP features: messages, queuing and routing and it was designed by the financial industry to replace existing proprietary message queues.  It seems to be the Promised Land—the perfect replacement for IRC.

After a quick evaluation of messaging brokers, the best candidates appear to be RabbitMQ and Apache Qpid. Both support AMQP, seem to have low latency and have vendor support—Qpid in the form of RedHat MRG.  While both seem to be faster than IRC, I needed to be sure.

AMQP is more complicated than IRC and replacing the current messaging system won’t be easy. I wanted to create a benchmark that simulates the load we see in a production environment, but before I could create the test, I need to better understand what AMQP provides and the terminology it uses.

AMQP Features

Messages are glorified strings which are published to an exchange. They consist of a header and a content body, while headers have several properties; you only need to know one to get started: the routing-key, a field used by exchanges to determine which queue the message will be routed to.

Queues are where your messages wait until they are consumed.  Queues can be configured to let messages die if a consumer is not available, queues along with exchanges can be made durable.  Message can persists if the queue and the exchange are marked as durable and the message’s delivery mode is set to persistent.

Exchanges the help the broker decide which queue to route your message to.  While they’re slated for removal in AMQP  1.0,  but they’re still important and difficult to ignore– there are four types of exchanges:  Fanout, Direct, Topic and Header.

• Direct exchange: Use routing keys, messages sent to a direct exchange are routed to exchanges where the routing key matches exactly.
• Topic exchange: A hierarchical exchange which also uses routing keys, routing keys are matched against a pattern, the hierarchy is established by separating keywords with the dot symbol.
• Fanout Exchange: A 1:N exchange, any message sent to an exchange is sent to all queues bound to that exchange.

A Channel is to AMQP what session is to HTTP, I’m not sure if that’s very accurate– all communication over a channel is stateful, and each instance of your program will have at least one channel.

Conclusion
There’s a lot to digest, I spent the better part of the day reading RabbitMQ’s .NET API Guide.  In part two of this series I’ll walkthrough C# client and benchmark RabbitMQ.  As always, your feedback and comments are appreciated.

Django, a Python framework for creating web applications which is loosely based on MVC architecture.  In the two-three-odd hours I spent with Instant Django I realized that I’ve never had a compelling reason to develop web applications in ASP.NET and  “I know C#” is not a not a reason to learn ASP.NET MVC.

Almost Immediately I noticed I wasn’t fighting the language, interpreted languages are much more compatible with the web development story, web application development with static languages is time consuming and tiresome.  Within the first hour I was in love with the documentation, not that I needed it *too* much, the different components fit well together and that’s fairly significant friction point for me. To do web development with ASP.NET MVC properly you need some sort of ORM, usually NHibernate, a dependency injection framework and 3rd party validators—I’m left wondering, isn’t this getting too complicated, where is my productivity? 

By the second hour I’m sold, my gut tells me that Django is good and my brain is agreeing with my gut on this one.

Collective Intelligence is a shared intelligence that emerges from the collaboration or competition amongst many individuals, it is used by online retailers such as Amazon.com and Netflix to recommend books and movies.  Today we’ll use these algorithms to compare user preference, and in subsequent posts apply see how these algorithms are used in recommendation systems and clustering algorithms.

Before we continue, there are two unfamiliar concepts which we need to review: list Comprehensions and dictionaries.

List comprehensions are expressions (which are very common to functional programming languages) that allow you to create lists without having to use funs, maps, or filters. This helps you to write programs which are shorter and easier to debug. In Erlang, the syntax is:

[ F(X) || X <- L]

which means, create a list of F(X) where F is a function and X is an element taken from the list L. Additionally, in Erlang we can use qualifiers to filter out elements of the list L.

Dictionaries (Hashes/Maps) are not directly supported by Erlang and to use them in your code you must use the dict module. The module includes the function dict:from_list which converts a list of two element tuples into a dictionary; it also includes the function dict:fetch which takes the Key and the dictionary as an argument and returns the value which matches the key.

Similarity

Below is a list of movie ratings by user

which we can store in Erlang as a dictionary:

  dict:from_list([
	{
	"Lisa Rose",
	dict:from_list( [{"Lady in the Water", 2.5},{"Snakes on a Plane", 3.5},
				 {"Just My Luck", 3.0},{"Superman Returns", 3.5},
				 {"You, Me and Dupree", 2.5},{"The Night Listener", 3.0}])
	},
	{"Mick LaSalle",
	dict:from_list( [{"Lady in the Water", 3.0},{"Snakes on a Plane", 4.0},
				 {"Just My Luck", 2.0},{"Superman Returns", 3.0},
				 {"You, Me and Dupree", 2.0},{"The Night Listener", 3.0}])
	},
	{"Jack Matthews",
	dict:from_list( [{"Lady in the Water", 3.0},{"Snakes on a Plane", 4.0},
				 {"Superman Returns", 5.0},{"You, Me and Dupree", 3.5},
				 {"The Night Listener", 3.0}])
	},
	{"Shey",
	dict:from_list( [{"Snakes on a Plane", 2.5},{"Superman Returns", 3.0},
				 {"You, Me and Dupree", 3.0}])}
	]).

To determine how similar two users preferences are we need to compute a similarity score. An easy way to calculate similarity is to use the scaled Euclidean Distance formula which we learned back in high school geometry. The function returns a value between 0 and 1, where a value of 1 means that two people have identical tastes.

sim_distance(List1, List2) when length(List1) /= 0, length(List1) == length(List1)  ->
  Sum_of_squares = lists:sum([ math:pow(X-Y,2) || {X,Y} <- lists:zip(List1,List2) ]),
  1.0 / (1.0 + math:sqrt(Sum_of_squares)).

The function accepts two lists, each lists contains the user’s rating of a movie, where the ith element of each lists represents the user’s rating for the same movie. Note the use of the lists:zip function which takes two lists of equal length and returns one list of tuples with two elements each, where the first element of each tuple is taken from the first list and the second element is taken from corresponding element in the second list. Finally, the function also has two guard expressions that prevent the user from calling the function with empty lists or lists of unequal length. Another simple distance metric is the Manhattan distance, it represents the absolute differences between coordinates of a pair of objects:

sim_manhattan(X,Y) when length(X) /= 0,  length(X) == length(Y)  ->
  Sum = lists:sum([ abs(Xn-Yn) || {Xn,Yn} <- lists:zip(X,Y)]),
  1.0/ (1.0 + Sum).

In this scenario, a slightly better way to determine the similarity between two objects is to use a Pearson correlation coefficient. The correlation coefficient is a measure of how well two sets of data fit on a straight line and it gives better results in when the data isn’t well normalized– for example, if critics’ movie rankings are routinely harsher than average. You can read about other other distance metrics for comparing items on Wikipedia.

sim_pearson(X, Y) when length(X) == length(Y)  ->
  Len = length(X),

  %% sum
  SumX = lists:sum(X),
  SumY = lists:sum(Y),

  %% sum of squares
  SumXSq=lists:sum([math:pow(N,2) || N <- X]),
  SumYSq=lists:sum([math:pow(N,2) || N <- Y]),

  %% sum of products
  Sum_products = lists:sum( [ A*B || {A,B} <- lists:zip(X,Y)]),

  %% Calculate Pearson Score
  Numer = Sum_products - (SumX*SumY/Len),

  case math:sqrt( (SumXSq - math:pow(SumX,2) /Len) * (SumYSq - math:pow(SumY,2)/Len)) of
    0.0 ->
      0.0;
    Denom ->
      Numer/Denom %% Don't put a comma or semi-colon here
  end.

A complete listing of the code is available online; you can experiment with the different distance metrics using the main function, but due to some limitations of the command line interface, you’ll have to assign the function to a variable before passing it to the main function:

1> c(pci).
2> SimPearson = fun(X,Y) -> pci:sim_pearson(X,Y) end.
3> pci:main("Mick LaSalle", "Lisa Rose", SimPearson).

While this code is (mostly) functional, we haven’t taken advantage of Erlang’s distributed processes, soon we’ll explore the K-Means clustering algorithm which is very well suited to parallel implementations.

Your comments and Feedback are appreciated.