We rely on polymorphism and design patterns to help reduce complexity; often those solutions are well suited for static programming languages like Java, but feel ceremonious in languages like Python and Javascript. If you’re programming in a dynamic programming language that supports dictionaries out of the box, then you may find the Dispatch Table “pattern” a useful way to simply your code.

The pattern uses a dictionary where the keys represent possible actions, and the value of each key contains a callable that does the actual work. Compared to if-else statements, dispatch tables require fewer lines of code, and we know fewer lines of code equals fewer bugs.

Here’s a mostly real world example that creates handlers for webhooks. You can imagine how long the if-else statement, that is used to decide which handler to instantiate, can become, especially when the consuming code has to respond to multiple keys.

Neat.

Dependency Injection is an important pattern to use when you wan to create classes that are easier to reuse and unit test. Since the dependencies are external, they can be configured and maintained outside class or function and there’s no need to change the code in order to reuse it. Also, when dependencies are injected into a class or a function it is possible to substitute a mock implementation of the dependency. In unit tests, mock objects are used as replacement for the real implementation, this helps isolate the functionality being tested.

A simple example

The following function has it’s dependencies, a list of numbers, passed to it, and since calculate_average only talks to a list of numbers it is easy to reuse.

def calculate_average(list_of_numbers):
  x = sum(list_of_numbers)
  return x/len(list_of_numbers)

Imagine a similar function which uses a database connection, retrieves a list of numbers then calculates the average.

def calculate_average():
  sql_query = "select numbers from table"
  list_of_numbers = db.query(sql_query)
  x = sum(list_of_numbers)
  return x/len(list_of_numbers)

This function is difficult to reuse because it depends on the database, future users of this function will have to either modify the function or create and maintain a database just to use it. More importantly it’s difficult to test this function. A test would require a database connection and a fixture to populate the database with data.

You should be thinking, but Shey, I would never do this with a simple function like this. You’re right, you wouldn’t do this with a simple function, but as soon as calculate_average turns into a more complicated calculation such as calculate_value_at_risk we start instantiating or dependencies.

A slightly less contrived example

The BillingService class is responsible for charging your customers a fee.

class BillingService
  def initialize(credit_card_processor)
    self.processor = processor

  def charge(account, amount):
    if account.has_balance:
      result = processor.charge(account.cc_number, amount)
    if result == "success":
      account.deduct(amount)

Lets assume that Authorize.net is offering better rates than PayPal. If the class accepts an implementation of a credit card processor as a dependency it is possible to switch to the authorize.net implementation of the credit card processor class without changing the BillingService class. Similarly during unit testing, a mock implementation of a credit card processor can be used to alwasy return “success” and isolate the charge functionality of the BillingService.

class mock_credit_card_processor:
  def charge(cc_number, amount):
    return "success"

    def test_charge():
      bs = BillingService(mock_credit_card_processor)
      a = Account(cc_number="411111111111111", balance=300.00)
      bs.charge(a, 300.00)
      assert(a.balance == 0)

The few discussions I read online implied that Python doesn’t need an IoC framework because it is a dynamically typed language.  Dynamically typed languages eliminate the need to use interfaces, they do not do away with the need for inversion of control.

IoC is used to decouple components of an application, remove direct dependencies so that replacing a component will not have a side effects on the rest of the system.  As Dave Thomas explains

a DI application is written as a set of loosely coupled components. These components contain no knitting code: nothing in the application code itself is responsible, for example, for making sure that the necessary objects somehow get an instance of the database connection. Instead, the components all run in a container. This container is given a description of the knitting to be done (typically using an XML file). The container then instantiates objects and sets them into components that need them

I’m new to the language and there maybe a more “pythonic” way to handle inversion of control than using a container. Here is small and contrived example of inversion of control using Pinsor, a IoC container in Python. Those coming from a .NET or Java background will find Pinsor easy to use, but slightly lacking in features.

Benchmark
The benchmarking script sets up 10 queues, 10 producers and 10 consumers, each producer writes to a single queue and each consumer reads from that queue ie: p1 writes to q1 and c1 reads from q1 as fast at it can.  The time measured is the time it takes a message to travel from p1 to the queue and to be read by c1 and it was calculated by appending the timestamp to the message itself.

I chose to run a series of tests with message size varying from 35 bytes to 400 bytes and the message count varying from 50 messages to 100 messages in an attempt to visualize the broker’s behavior under varying conditions.

The benchmark was performed using a server running 2.8GHz Pentium D with 903464 kB RAM running on Kernel 2.6.18-128.1.10.el5 using RabbitMQ 1.5.5 and a machine with similar specifications as the client—both the producer and consumer run on the same machine.  The network is a typical Fast Ethernet with two switches separating the server and client machines.

Conclusion
Inconclusive! The numbers are very surprising I wasn’t expecting the message transfer times to be greater than .5 seconds under this load.  It’s possible that the consumer may not be able to keep up with flood of messages, if that’s the case than a consumer written in C# is needed.  What is obvious is that more tests  with a constant message size and a varying number of messages  are needed.

The standard deviation is also greater than I expected, so I plan on creating longer running tests and see if the broker falls into a steady state.  Finally, I also want to test with 50 messages per producer using 20 queues/producers instead of 10  and see how RabbitMQ responds.

I’m wondering what sort of performance others are experiencing—any and all feedback is greatly appreciated.

Postscript
I intended on using C# to benchmark RabbitMQ but for want of time  I used a modified version of Colin Surprenant‘s bunnypunisher ruby script.  There’s a slightly outdated (and dirty) version of the script available on github.  I will upload the latest version along with the “launcher” tomorrow morning.

Kyle Burton at Asymmetrical View has a fairly comprehensive comparison of RabbitMQ and Qpid.

The order management system I work on uses IRC as the messaging middle ware– a message is sent to a channel and another process reads and processes the message.  This gives us great flexibility, letting us write the user interface in .NET and the back-end server code in whichever language we choose and it also allows us to scale the system by moving processes to different machines.

But this setup has problems, if a message is missed by a process, there’s no easy to way to retrieve it and recent benchmarking showed that the latency of the messaging system spikes, unpredictably to 300 ms or more.  Initially IRC made sense but the requirements have grown and it’s clear a more robust solution which provides greater reliability, performance and scalability is needed.

AMQP
The Advanced Message Queuing Protocol (AMQP) is an open standard, vendor neutral binary protocol for messaging middleware. AMQP features: messages, queuing and routing and it was designed by the financial industry to replace existing proprietary message queues.  It seems to be the Promised Land—the perfect replacement for IRC.

After a quick evaluation of messaging brokers, the best candidates appear to be RabbitMQ and Apache Qpid. Both support AMQP, seem to have low latency and have vendor support—Qpid in the form of RedHat MRG.  While both seem to be faster than IRC, I needed to be sure.

AMQP is more complicated than IRC and replacing the current messaging system won’t be easy. I wanted to create a benchmark that simulates the load we see in a production environment, but before I could create the test, I need to better understand what AMQP provides and the terminology it uses.

AMQP Features

Messages are glorified strings which are published to an exchange. They consist of a header and a content body, while headers have several properties; you only need to know one to get started: the routing-key, a field used by exchanges to determine which queue the message will be routed to.

Queues are where your messages wait until they are consumed.  Queues can be configured to let messages die if a consumer is not available, queues along with exchanges can be made durable.  Message can persists if the queue and the exchange are marked as durable and the message’s delivery mode is set to persistent.

Exchanges the help the broker decide which queue to route your message to.  While they’re slated for removal in AMQP  1.0,  but they’re still important and difficult to ignore– there are four types of exchanges:  Fanout, Direct, Topic and Header.

• Direct exchange: Use routing keys, messages sent to a direct exchange are routed to exchanges where the routing key matches exactly.
• Topic exchange: A hierarchical exchange which also uses routing keys, routing keys are matched against a pattern, the hierarchy is established by separating keywords with the dot symbol.
• Fanout Exchange: A 1:N exchange, any message sent to an exchange is sent to all queues bound to that exchange.

A Channel is to AMQP what session is to HTTP, I’m not sure if that’s very accurate– all communication over a channel is stateful, and each instance of your program will have at least one channel.

Conclusion
There’s a lot to digest, I spent the better part of the day reading RabbitMQ’s .NET API Guide.  In part two of this series I’ll walkthrough C# client and benchmark RabbitMQ.  As always, your feedback and comments are appreciated.

Django, a Python framework for creating web applications which is loosely based on MVC architecture.  In the two-three-odd hours I spent with Instant Django I realized that I’ve never had a compelling reason to develop web applications in ASP.NET and  “I know C#” is not a not a reason to learn ASP.NET MVC.

Almost Immediately I noticed I wasn’t fighting the language, interpreted languages are much more compatible with the web development story, web application development with static languages is time consuming and tiresome.  Within the first hour I was in love with the documentation, not that I needed it *too* much, the different components fit well together and that’s fairly significant friction point for me. To do web development with ASP.NET MVC properly you need some sort of ORM, usually NHibernate, a dependency injection framework and 3rd party validators—I’m left wondering, isn’t this getting too complicated, where is my productivity? 

By the second hour I’m sold, my gut tells me that Django is good and my brain is agreeing with my gut on this one.

Source control is the bedrock of your development effort, it contains the history of your entire project and it’s too important to trust it to VSS.  Visual Source Safe is a horrible source control system and i keeps you from being productive.

Teams using VSS love to use exclusive locks on source files, this only creates contention for files and kills the team’s productivity.  Even worse, VSS’s doesn’t support branching which makes refactoring your code doubly difficult, all your changes have to be made offline and manually merged into the “trunk,” and only is VSS insanely slow it has no concept of transactions, if a bug fix required you to change and check-in multiple files at once, there’s no way to figure out which files were checked-in together using VSS’s history feature.

SVN is a much better, free alternative to VSS, which supports many features that you expect from a modern source control system: branching  and merging and transactions and atomic commits.

If you think your project’s history is stuck in VSS, it’s not, there’s a great migration tool called vss2svn which lets you migrate your entire VSS repository along with the project history to SVN.

You’ll need to download and setup SVN server and create a repository and users for your projects first, I recommend using VisualSVN Server if you’re on Windows.  What follows is a painfully long migration, but it’s well worth it in the end.

1. Download and extract vss2svn to a hard drive which has plenty of space, as a general rule, you should have enough space to hold three copies of your VSS repository;
2. After making sure that your repository does not have any files checked out, take your VSS repository offline by disabling all access to it, then create a backup of your VSS directory; at this time, you should also delete/destroy any files/projects you do not wish to migrate to svn.
3. Run the analyze command on your VSS repository, you’ll have to run analyze several times if the analyze utility encounter errors in your repository– this is very time consuming and but an important step.
4. Now you’re ready to run the migration utility from the command prompt.   You’ll need to use the following command:  vss2svn.exe –vssdir <dir> where <dir> is the directory which contains the scrsafe.ini file.  vss2svn.exe will generate a svn dump file which you’ll later import into your svn repository;
5. finally, import the dump file generated by vss2svn by using the following command on the machien where your svn repository resides: svnadmin load /path/to/repository < dumpfile.txt

As aspiring Software Craftsmen we are raising the bar of professional software development by practicing it and helping others learn the craft. Through this work we have come to value:

Not only working software, but also well-crafted software

Not only responding to change, but also steadily adding value

Not only individuals and interactions, but also a community of professionals

Not only customer collaboration, but also productive partnerships

That is, in pursuit of the items on the left we have found the items on the right to be indispensable.

© 2009, the undersigned.
this statement may be freely copied in any form,
but only in its entirety through this notice.

As of right now, I am the only Pakistani signee of the Manifesto for Software Craftsmanship, why? 

If this sounds like you, then read on:  I’m trying to form study group of programmers who are interested in studying software development.  The group would meet once or twice a month to present new ideas, discuss technology related articles and read and refactor code.   

The main focus of the group would be Object oriented programming and design; initially I would like to cover topics such as ORM, Domain Driven Design and Inversion of Control.

Obviously this is for Karachi only, if you’re still interested drop a note in the comments.

Project managers will stand behind us, nodding in agreement, because no project manager is brave enough to openly say that design isn’t a worthwhile activity, instead they’ll rephrase their disagreement: “We have a deadline approaching and we need to get this functionality out of the door fast!” A dangerous phrase which means they want you to churn out code as fast as you can without regard to future costs.

Software developers use the term “Technical Debt” to describe the obligation an organization incurs when it chooses a design or approach that’s quicker in the short term but increases complexity of the software and is more costly in the long term. 

Imagine, skipping the dishes after dinner, instead of washing them you let them pile up in your sink, you’ll save time now, but eventually, you’ll have a huge mess in your kitchen and you won’t have any dishes to eat on. Even worse, imagine that your in-laws drop in on a surprise visit and you don’t have any dishes to serve them food on—these imaginary in-laws are clients asking you for new functionality.

Technical debt prevents you from meeting your customer’s needs quickly, modifications to the code base take exponentially longer and as your technical debt accumulates you expose yourself to more risk.

Sometimes, just like financial debt, it is beneficial to take on technical debt to exploit an opportunity, but like credit card debt, the interest on technical debt can catch up on you, all too often we don’t manage our debt, it spirals out of control and we spend most of the our money paying of the interest, not the principle.

Software development is a series of compromises, but we can’t fool ourselves into believing that we can trade good design for greater development speed.

The problem with sacrificing design is that the code becomes unstable, adding a feature will break the software in other places; the code becomes to modify, a simple change requires modifying other parts of the system and finally, the code becomes harder to read, developers spend days trying to figure out which files need to change.  Eventually, your developers will become frustrated and they’ll start proposing the big rewrite of the system.

Development in the absence of design yields systems which have a high level of coupling, where functionality is spread across the entire application and often duplicated in several locations across the code base.

As the application evolves, these duplicate functions become unsynchronized, making bug fixes more difficult and time consuming. Maintaining a code base with high coupling is difficult and time consuming, your developers waste time looking for the code to modify searching for the duplicate code in ten different files, instead of developing new functionality; they waste time re-understanding the code. All of these problems compound to make maintenance and further development extremely expensive.

Programmers are lazy and if good design actually increased the effort and time required we’d be against it.  Software design is an effortful activity, but one which leads to a net increase in productivity and efficiency.

For software to remain relevant and profitable, it must change often, add value by growing in features, if your code is brittle and fragile, then adding new features to your application is difficult, eventually you’ll have to turn away customers because you can’t meet their requirements in time. Extrapolate this over five years and imagine the profit you had to forgo because your code is unstable.

Good design ensures functionality is confined to a single location, it encourages code reuse, makes maintenance easier and it makes it easier to add new features, in short good design saves you money.