“It’ll all have to come down.” –

Builders, programmers, and the costs of rewriting software systems.


I love a good builder analogy when it comes to software development.

Often when talking to business stakeholders it is useful to use terms that they are familiar with and concepts they can easily grasp and are analogous to those we use in developing software. As software developers, we are skilled tradesmen. We have years of training to develop the skills needed to build projects up from a greenfield site to something of value. We build with bits and bytes rather than bricks but often our concepts can be very similar.

A typical analogy I use is to explain why it takes longer to rewrite a badly constructed, poorly performing software project again from scratch than it did to write the software first time round.

Take it you had a garage built onto the side of your house. The builders you had in to do the job didn’t build the foundations properly. It looked a bit of a shoddy job, but you had invested a lot of money in it so you put up with it. A few years later, there are severe cracks showing in the walls, you know that it won’t last much longer, and so you bring in another builder to help you with it.

He tells you that it all needs to come down because the foundations are really poor, and we need to build the whole thing again. However, this time it will take a lot more time and cost a lot more money. Why?

One reason is that over the last couple of years you have integrated the utilities of the main building into it, running electrics, water, and waste through the extension, and all these need to be resolved. You’ve also added a further bedroom extension on top of the garage and so you not only have to rebuild the garage, but rebuild that too. In addition, all of these things are constantly in use – you need the garage for your vehicle, and the bedroom is your daughter’s room.

Software projects are often very similar. You have a legacy system with a fundamentally bad design which is causing massive problems within your business, and the only recourse, due to that initial poor design, is to do it again. However, since the software was first written you now have various integration points (the utilities) and other subsystems which look like they may not need rebuilding but rely directly upon the code in the badly performing system (the daughter’s bedroom). In addition everybody in your business needs to keep using these systems whilst they are changed over and will need training upon the new system, plus any data migrations need to take place to enable the move to happen easily.

Frequently, the subsystem also has to be rewritten with very little input from the end users, who may have been heavily involved in the original development, but now don’t have the time or the inclination to be involved in a technical rewrite. Original requirements documents, if they exist, are rarely useful, as during the process of development the requirements change and documents are rarely kept up to date. Imagine rebuilding the garage and the bedroom above it with little to no involvement from the homeowner to ensure they will be happy with the finished product!

Due to budget limitations, sometimes the only feasible way forward is to attempt remedial measures. Put in some props, dig some trenches around the foundations and pour in some concrete. This leaves you with an ugly mess, but at least the house won’t fall down!

Business stakeholders can often be bamboozled by technical talk, but the principles of constructing software are similar to any project that takes a concept from design through to completion and use.

Don’t talk to me about backups…

.. talk to me about high availability and data recovery

Blocked toilet

A few recent events have thrown into focus the ability for even large well established businesses to handle adverse IT events.

Firstly, the recent British Airways outage had many experienced IT professionals scratching their heads. The outage was blamed on a power issue which was, according to BA, “not an IT failure and had nothing to do with outsourcing of IT, it was an electrical power supply which was interrupted”. We were scratching our heads because the inability of an IT system to handle a power outage is very much an IT failure!

Secondly, for a client of ours, there was a recent issue with one of their databases where somehow some of the data within the database had been deleted. They had a high availability strategy in place ( the database was being replicated to another instance in case the primary went down ) but as this was data being deleted from the database, the deletion action had also been replicated to the other database instance. The data was gone. ( Luckily Roar IT had integrated another system into this database and this second system had a copy of a lot of the data that had been deleted, albeit in a different format. This enabled us to work with the client to reconstruct the majority of the lost data. )

The first event showed a lack of high availability planning, the second a lack of data recovery planning.

The language here is important, as too often I hear management staff referring to ‘backup’s. “Do we have a backup?” Can we put everyone on the backup system?” “It’s OK because we have a backup.”

In the BA incident, they may well have had a ‘backup’ strategy in that they were storing a historical record of their data, but they did not have a high availability strategy meaning that they could move their business activity to another set of systems if there was an outage.

In the second incident, the management were convinced they had a ‘backup’ strategy because they had another set of systems to switch to, but they weren’t keeping a historical record of their data, and so had no data recovery strategy.

For businesses that rely on their IT systems, then they need to consider both their high availability, and their data recovery. Let’s keep the talk of ‘backup’s to the fields of parking and drainage!

Walking into a doctor’s surgery and asking for penicillin

One of the problems I have found when adopting an agile approach to developing systems (and I mean truly agile, according to the agilemanifesto.org rules, rather than a particular project management methodology), is that the high level of end user interaction in the development process can sometimes lead to the users, in effect, designing the systems.  They often come with an idea for a technical solution to their problem and ask us developers to build that, rather than allowing analysts to design the solution.

I often use the analogy of a visit to the doctor’s surgery to illustrate this.

In recent times the old role of systems analyst or analyst/developer seems to have evolved into two separate roles – the business analyst (often with little or no programming skills or experience) and the software developer.  In our doctor’s surgery analogy, the roles of user, business analyst and software developer are fulfilled by the patient, doctor and pharmacist.   

It is the doctor’s role to investigate the symptoms, diagnose the problem and come up with a treatment plan, which they agree with the patient. It is then the role of the pharmacist to provide that treatment.

In software development projects, it is frequently the case that the patient (user) walks into the surgery with their own treatment plan in mind. They enter the room and ask for penicillin.  It is too often, in my experience, that the doctor (analyst) then simply writes out a prescription for penicillin and hands that over to the pharmacist (developer).

Imagine the case, where the pharmacist spots a problem with the treatment plan (say the patient has a virus), doesn’t think it is wise, and says to the doctor, “I don’t think this is right”, only for the doctor to respond with: “That’s what the patient asked for, give it to them”.

I’ve seen this story play out in many a troubled software project!

Even worse, I’ve often seen the equivalent of the doctor simply leaving blank prescriptions at reception for patients to fill in themselves and hand to the pharmacist!

The business analyst role is not simply to liaise between users and developers (I’ve known business analysts whose time was mostly taken up being an email relay, or a chinese whisperer!). They need to be looking at the business needs behind any request and designing a solution that fits with any current systems that are in place, before bringing that to a developer.

In addition, it’s rare that an analyst with no technical or programming experience will have the ability to know the full extent of what a software system can achieve, or design solutions that fit into existing systems well.

Maybe it’s time to bring back the combined role of analyst/developer.