Last Two Weeks
This last mini vacation provided me with an opportunity to spend quality time learning data mining/machine learning; just as importantly it reaffirmed my belief in the application of science and technology over “common sense” to solving complicated problems

Programming Collective Intelligence has beautiful examples of how science can be applied to real world problems, but it only offers an introduction to a very complicated very diverse field of machine learning. 

Another Piece of the Puzzle
Data mining, software engineering, Python, Erlang and financial risk management are just pieces of a very complicated puzzle: algorithmic trading. On Monday, at my new job, another piece of puzzle will fall into place: the FIX Protocol. The next few weeks I’ll be busy learning FIX  but I know that working with FIX and getting re-exposed to the industry will help me make better decisions regarding which technology to use and which to avoid.

Going Forward
I still plan on studying and experimenting in my free time.  My roadmap includes: Introduction to Data Mining which I want to supplement with Orange; Mnesia, a distributed database which is used in conjunction with Erlang or possibly MonetDb which should be more suitable for data mining and tick databases and finally some tinkering with Markcetera and market simulators.

Parting Thoughts
This is a roadmap without a timeline, one thing I should have learned by now is that learning takes time and I have to mentally prepare myself for the long haul.  I don’t know if this combination of technology will yield the results I expect, but no knowing is exciting—the experimenting is half the fun. 

As a software developer, I have unintentionally surrounded myself with other software developers who share the same ideals as myself. If you asked any them if good design is worth the effort, they will answer with an unequivocal yes!  They may ask “How can you question design?  It is the heart of software development.”

Project managers will stand behind us, nodding in agreement, because no project manager is brave enough to openly say that design isn’t a worthwhile activity, instead they’ll rephrase their disagreement: “We have a deadline approaching and we need to get this functionality out of the door fast!” A dangerous phrase which means they want you to churn out code as fast as you can without regard to future costs.

Software developers use the term “Technical Debt” to describe the obligation an organization incurs when it chooses a design or approach that’s quicker in the short term but increases complexity of the software and is more costly in the long term. 

Imagine, skipping the dishes after dinner, instead of washing them you let them pile up in your sink, you’ll save time now, but eventually, you’ll have a huge mess in your kitchen and you won’t have any dishes to eat on. Even worse, imagine that your in-laws drop in on a surprise visit and you don’t have any dishes to serve them food on—these imaginary in-laws are clients asking you for new functionality.

Technical debt prevents you from meeting your customer’s needs quickly, modifications to the code base take exponentially longer and as your technical debt accumulates you expose yourself to more risk.

Sometimes, just like financial debt, it is beneficial to take on technical debt to exploit an opportunity, but like credit card debt, the interest on technical debt can catch up on you, all too often we don’t manage our debt, it spirals out of control and we spend most of the our money paying of the interest, not the principle.

Software development is a series of compromises, but we can’t fool ourselves into believing that we can trade good design for greater development speed.

The problem with sacrificing design is that the code becomes unstable, adding a feature will break the software in other places; the code becomes to modify, a simple change requires modifying other parts of the system and finally, the code becomes harder to read, developers spend days trying to figure out which files need to change.  Eventually, your developers will become frustrated and they’ll start proposing the big rewrite of the system.

Development in the absence of design yields systems which have a high level of coupling, where functionality is spread across the entire application and often duplicated in several locations across the code base.

As the application evolves, these duplicate functions become unsynchronized, making bug fixes more difficult and time consuming. Maintaining a code base with high coupling is difficult and time consuming, your developers waste time looking for the code to modify searching for the duplicate code in ten different files, instead of developing new functionality; they waste time re-understanding the code. All of these problems compound to make maintenance and further development extremely expensive.

Programmers are lazy and if good design actually increased the effort and time required we’d be against it.  Software design is an effortful activity, but one which leads to a net increase in productivity and efficiency.

For software to remain relevant and profitable, it must change often, add value by growing in features, if your code is brittle and fragile, then adding new features to your application is difficult, eventually you’ll have to turn away customers because you can’t meet their requirements in time. Extrapolate this over five years and imagine the profit you had to forgo because your code is unstable.

Good design ensures functionality is confined to a single location, it encourages code reuse, makes maintenance easier and it makes it easier to add new features, in short good design saves you money.

I’ve been incredibly busy at work again and instead of writing something coherent I’ll just dump my brain in this post. I’ve also developed a very severe case of Nerd ADD.

Earlier this week I hit a snag in my Project Server/Navision integration, lets just say that portions of Project Server’s web service isn’t documented very well. I wasted a lot of time trying to figure out researching exactly which functionality the Time Sheet web service exposes; it wasn’t until I found an example on Christophe Fiessinger’s Blog that I finally figured out what I wanted.

Working with Project Server has gotten me excited about programming languages again and I want to learn C# 3.0 and Ruby– C# has gone through some major changes, they’ve added extensions which are similar to Ruby mixins, lambda expressions and anonymous variables, it’s a completely different language now, much more flexible, much less like Java and much more like Ruby.

Tenerife Skunkworks, a Erlang blog has peaked my interest in Erlang and day trading in general. In my former life I was a quant in training and a software developer working on an algorithmic trading platform. Algorithmic trading is resource intensive and our system distributed ‘jobs’ over several computers to manage the processor load, looking back I wish I programmed the core of the system in Erlang instead of C++. That’s hindsight for you. I’d like to get back to my ‘roots’ and work with financial systems again, maybe the FIX protocol or a trading simulator or platform, I should spend some time searching for open source projects that I could get involved in.

I finally got around to installing an evaluation copy of Resharper– I’m disappointed, maybe my expectations were too high, or I haven’t had enough time to use it– while it’s an improvement over the default VS 2005’s interface it still falls behind Eclipse’s refactoring code generation. Does anyone have any Resharper tips for me?

Definitely a severe case of Nerd ADD.