Bad Code: The Invisible Threat

One of the first things said by the non-believers of the software craftsmanship movement was that good and clean code is not enough to guarantee the success of a project. And yes, they are absolutely right. There are innumerable reasons which would make a project to fail ranging from business strategy to competitors, project management, cost, time to market, partnerships, technical limitations, integrations, etc.

Due to the amount of important things a software project may have, organisations tend not to pay to much attention to things that are considered less important, like the quality of the software being developed. It is believed that with a good management team, deep hierarchies, micro management, strict process and a large amount of good documentation, the project will succeed.

In a software project, the most important deliverable is the software itself. Anything else is secondary.

Many organisations see software development as a production line where the workers (developers) are viewed as less skilled than their high-qualified and much more well paid managers. Very rarely companies like that will be able to attract or retain good software developers, leaving their entire business on the hands of mediocre professionals. 


Look after your garden

Rather than construction, programming is more like gardening. - The Pragmatic Programmer

Code is organic, not mechanical. Like a garden, code needs constant maintenance. For a garden to look nice all year round, you need to look after its soil, constantly remove weeds, regularly water it, remove some dead plants, replant new ones, trim or re-arrange existing ones so they can stay healthy and look nice as whole. With basic regular maintenance, the garden will always look great but if you neglect it, even if for a short period, the effort to make it nice again will be much bigger. The longer you neglect it, the harder it will be to make it look nice again and you may even loose some or all of your plants.

Code is no different. If code quality is not constantly looked after, the code starts to deteriorate. Bad design choices, lack of tests and poor use of languages and tools will make parts of the code to rot. Bit by bit other parts of the code will also be contaminated up to the point that the whole code base is so ill that it becomes extremely painful to change it or add new features to it.

The Invisible Threat

When starting a greenfield project, everything is great. With a non-existent code base, developers can quickly start creating new features without the fear of breaking or changing any existing code. Testers are happy because everything they need to test is new, meaning that they don't need to worry about regression tests. Managers can see a quickly progress in terms of new features added and delivered. What a fantastic first month the team is having. 

However, this is not a team of craftsmen. This is a team of average developers structured and treated like unskilled production line workers. 

As time goes by, things are getting messier, bugs start to appear (some with no apparent explanation) and features start taking longer and longer to be developed and tested. Very slowly, the time to deliver anything starts to stretch out. But this is a slow process. Slow enough that takes months, sometimes, over a year or two to be noticed by the management.

It's very common to see projects where, at the beginning of a project, a feature of size X takes N number of days to be implemented. Over the time, as more bad code is added to the application, the same feature X (or a feature of the same size) takes much longer to be implemented than it used to take at the beginning of the project. As the quality of the code decreases, the amount of time to implement a new feature, fix a bug or make a change increases. The lower the quality, the higher the number of bugs, the harder is to test and less robust and reliable the application becomes. 

Some people say that they just don't have time to do it properly but, in general, a lot more time and money is spent later on on tests and bug fixing.  

Hostage of your own software

When the code base gets into the situation where changes or additional features take too long to be implemented or worse, developers and managers are scared to touch existing code, an action must be taken immediately. This is a very dangerous situation to be since business progress is being impeded or delayed by the software instead of being helped by it.

To keep business progress, schedule and budget under control, high quality code needs to be maintained at all costs.

Organisations may need to cancel the implementation of some features or postpone changes just because of the amount of time and money that they may cost to be built. Having poor quality of code responsible for it is totally unacceptable. 

The biggest problem here is that bad code is invisible to everyone besides developers. Other members of the team will just realise that something is wrong when it is too late. This means that it is the developers responsibility to look after the quality of the code. Some times, developers expose the problem to project managers but the request for having some time to "re-factor" the code is often ignored for various reasons, including a lack of understand of the impacts of bad code and the inability of developers to explain it. On the other hand, when developers come to a point where they need to ask for some formal time to do refactoring, this means that for one reason or another, they neglected the code at some point in the past.

Hire craftsmen not average developers
 
With the amount of literature, tools, technologies, methodologies and the infinite source of information available on the web, it is just unacceptable to have a team of developers that let the code to rot.

Craftsmen are gardeners and are constantly looking after the code base, quickly refactoring it without fear since they are strongly backed by a good battery of tests that can test the entire application in just a few minutes. Time constraints or change in requirements will never be used as excuses for bad code or lack of tests due to the good design principles and techniques constantly used throughout the application.

Having an empowered team of craftsmen can be the difference between success and failure of any software project.

Quality of code may not guarantee the success of a project but it can definitely be the main invisible cause of its failure. 

Beyond the manifesto: The Software Craftsmanship Attitude

Being an aspiring software craftsman goes way beyond than just saying it. I'll quote my own definition of software craftsmanship from my previous post.

Software craftsmanship is a long journey to mastery. It's a lifestyle where developers choose to be responsible for their own careers and for improving their craft, constantly learning new tools and techniques. Software Craftsmanship is all about putting responsibility, professionalism, pragmatism and pride back into software development.

Software craftsmanship is all about attitude. The attitude of raising the bar of professional software development starting with our own skills and professionalism.

The responsibility shift

Not long ago, I was speaking to a developer and he was complaining about his company, saying that they didn't have a good career plan, that he did not have any training, that he was not given the opportunity to learn new technologies and, of course, that he was not paid enough. Apparently, from his perspective, his employer was responsible for his career.

Imagine that we need a doctor. Would we pay a doctor to learn while he cut us open or give us a diagnosis? Would we pay an engineer to learn while he draws the plan for our new house? Would we go to a concert and pay the musician to learn how to play the guitar during the show? What about a chef in a restaurant?

So why is the employer's obligation to pay us training courses and pay us to learn new technologies and tools while we are working on a project? Should the employers be responsible for what we learn and what we don't learn.

Software development is not a 9 to 5 profession. To be a professional software developer, we need to take our own time and money to keep learning and improving. As professionals, we should be paid for what we know, our ability to learn fast and for the quality of the work we do. We own our careers and are responsible for them. Working for a customer/employer that helps us with our career in terms of training, books, conferences, seminars, etc, is great but should be considered a bonus.


... but how can we learn and keep ourselves up-to-date?

Different developers have different preferences but here is a list of ways I find useful.

Literature 

Books, many books. Having your own library is essential. Books give you a good overview of a specific technology or subject quickly. No, it does not mean you will be proficient and next day you will be a specialist. What a book will give you is an understanding of what a technology or subject is about. It will be up to you then to decide if you want to practice what you've learned and become proficient. If you don't have the habit, try reading 3 to 4 technical books per year. Once you get the habit, try one per month. The most common excuse is that you don't have time. The cool thing about books is that you can read them during periods of  "dead" time, like on the tube, bus, in your dentist's waiting room, on the bed before going to sleep, toilet, etc.  

Blogs are now one of my favourite types of reading. They tend to fit more the software craftsmanship model since they are much more personal and in general, related to personal findings, opinions, successes and failures. Reading blogs from more experienced professionals and subject matter experts is a good, quick and free way for apprentices and journeymen to learn from multiple master craftsmen at the same time. But don't think that just experienced professionals should write blogs. Every software developer should write his or her own blogs, sharing their experiences and findings, helping to create a great community of professionals.  

Technical websites are also good in order to keep yourself up-to-date with what's going in the market. New technologies, techniques, etc.

Practice, practice, practice

Pet project(s) is for me, by far, the best way to learn and study. A pet project is a real project but without the boring bits. There are no deadlines, does not need to make money, you control the requirements and most importantly, you use the technologies and methodologies you want, whenever you want, wherever you want. You are the boss. Pet projects give something for you to focus and help you to understand why and how you can use certain technologies. It gives you the experience you need to apply what you've learned into real projects. Pet projects are meant to be fun. 

Contributing to open source projects can also be a great thing. There are thousands of them out there. Find a project that is related to what you want to learn or know more about and download the source code. Start running and reading the tests, if any. Inspect and debug the code. If you want to contribute, start small. Add some documentation and write some tests. Then, check the list of features to be implemented and pick a simple one and give it a go. You can also propose and implement a new and small one to start with.

Pair-programming can be a great experience. Many developers are afraid to try or think they will feel uncomfortable. That's what I thought as well before I tried. Today I really enjoy it. Pair-programming gives you an opportunity to discuss your ideas, to learn new tricks and techniques from your pair and the resulting code is much better. Pairing with someone from your team is good but pairing with someone that you barely know can be a very interesting experience.

If learning a new language or new technique like TDD / BDD or trying different approaches to OOP or functional programming, try a code kata. Code kata is a small exercise that in general can be solved in a few minutes or in a few hours. Code katas were created to help developers to focus in a problem while improving their skills. You can find a good source of code katas at codingkata.org and codekata.pragprog.com

Community and Events

Be part of your local community. Software craftsmanship is all about a community of professionals, learning and sharing with each other and elevating the level of professionalism and maturity of our industry. Join your nearest user groups and participate in their events. User groups are the best way for making contacts, sharing ideas and learning things. Many user groups promote free talks, coding activities and social events. A great aspect of being part of a community if the feeling that you are not alone. There are many people out there having the same problems you've got and many others that will happily share their solution to it.

Raising the bar

The main changes proposed by the software craftsmanship movement are related to the developers attitude. In summary, it's about being proud about your work, the code you produce and the software you create. It's about constantly trying to improve your skills and learn new ones.

An aspiring software craftsman is intolerant of bad code will constantly apply the Boy Scout Rule. 

If you find that the good code you wrote one year ago is still good enough today, it means you didn't learn anything during this period and this is unacceptable.

Productive partnerships

The attitude of a software craftsman goes beyond good code. The relationship between a software craftsman and his or her customer (or employer) is of a productive partnership and not employer / employee. As a productive partnership, it's our role to constantly question the requirements and propose improvements and new features. It's our job to warn our customers about the problems we see. Due to the knowledge of the code and technology, developers are well positioned to help their customers improve their business.

However, sometimes some customers or employers don't want or don't see the advantages of this partnership and will treat developers as production line workers that should just do what they are told. Coding monkeys. In cases like that, any aspiring software craftsman should move on and find another job. Staying in a place where your skills are not appreciated is a career suicide.  

Raising the bar of our industry is on our own interest. The better we become in writing code and delivering valuable software, the better our lives will be, professionally and financially. 

PS: If  you are in London or nearby, join the London Software Craftsmanship Community.

Software Craftsmanship

So what is software craftsmanship? 

A better metaphor: In a very simplistic way, we can say that software craftsmanship is a better metaphor for software development than software engineering, as I wrote in a previous post. Software craftsmanship sees software as a craft and compares software developers to the medieval blacksmiths. Apprentices would work with more experienced blacksmiths, travelling from place to place, working with and for different masters, learning different tools and techniques, improving their craft until the point they were good enough to become master themselves. (There is more to it, but let's keep it simple for now).

Wikipedia Definition:  is an approach to software development that emphasizes the coding skills of the software developers themselves. It is a response by software developers to the perceived ills of the mainstream software industry, including the prioritization of financial concerns over developer accountability. (read in full)

I personally don't like too much the Wikipedia's definition. It's very dry and I don't think it captures the essence of what being a software craftsman means to a software developer.

A more personal definition: Software craftsmanship is a long journey to mastery. It's a lifestyle where developers choose to be responsible for their own careers and for improving their craft, constantly learning new tools and techniques. Software Craftsmanship is all about putting responsibility, professionalism, pragmatism and pride back into software development

A software craftsman cares and is proud of his or her work and is extremely professional and pragmatic when it comes to its implementation.

The Software Craftsmanship Movement

The software craftsmanship movement is basically an evolution of ideas that started probably in the late 90ies, early 2000 with the publication of The Pragmatic Programmer by Andy Hunt and Dave Thomas (1999) and Software Craftsmanship: The New Imperative by Pete McBreen (2001). In 2008, Uncle Bob proposed "Craftsmanship over Execution" (originally Craftsmanship over Crap) as the fifth value for the Agile Manifesto. In 2009, the Manifesto for Software Craftsmanship was created, defining the values of the movement and international Software Craftsmanship conferences emerged in the US and UK.

In the manifesto, the essence of the software craftsmanship is capture in its subtitle: Raising the bar. The manifesto was idealised by very experienced developers that had enough of project failures mainly caused by poor management, ill processes and, of course, badly-written code.

Developers are taking the matter into their own hands and are trying to change how the industry sees software development not just proposing new and revolutionary processes but showing customers that they care about what they do and that they want to work together with their customers in order to produce great and long-lived software.  

The values of the Software Craftsmanship Movement

Not only working software, but also well-crafted software

Working code is not good enough. Think on a 5 year old application (it could be 2 or 10) where we are scared to change some of its parts (fix bugs, add new features, etc.) because we don't understand how it works and have no confidence that we will not break anything else. This application is working software but is it good enough? Well crafted software means that regardless how old the application is, developers can understand it easily, side effects are well known and controlled, high test coverage, clear design, business language well expressed in the code and adding or changing features does not take longer that it used to take at the beginning of the project, when the code base was small.

The code must be maintainable and predictable. Developers must know what is going to happen when changing the code and must not fear to change it. Changes should be localised and not cause impact in other parts of the application. Tests will guarantee that nothing else was broken.

Not only responding to change, but also steadily adding value

This is not just about adding new features and fixing bugs. This is also about constantly improving the structure and cleanliness of the code. The software must be seen as an asset and the constant maintenance of it will make it more valuable during its lifetime, instead of letting it rot and devalue. 

The Boy Scout Rule (defined by Uncle Bob) states that we should always let the code a bit cleaner than we found it. This is a paraphrase of Boy Scouts rule: leave the camp cleaner than you found it. 

If customers want to keep benefiting from adding and changing features quickly, they will need high quality code to enable them to do it.  

Not only individuals and interactions, but also a community of professionals

This is somehow related to the idea of apprentices, journeymen and masters, where software craftsmanship masters will mentor apprentices and help them in their journey. The software craftsmanship community is responsible for training the next generation of professionals. Knowledge and ideas must be shared and discussed within the community in order to keep moving the software development industry forward. 

Not only customer collaboration, but also productive partnerships

Software craftsmen need successful projects to build their reputation and are proud of their achievements. Successfully delivering high quality software is essential for any software craftsman journey. With this in mind, software craftsmen will do whatever they can for a project to succeed. They don't act like simple employees that just do what they are told to do. They want to actively contribute to the success of the project, questioning requirements, understanding the business, proposing improvements and productively partnering with their customers or employers. This is an interesting shift of perspective, if you like, and the advantages for the customer and for the project success are enormous. A well-motivated team has a much bigger chance to make any project succeed. However, if the customer is not prepared to have this partnership and sees software development as an industrial process and the least important part of the project, this customer will never have real software craftsmen working for him for too long. Not getting involved with the business, not questioning requirements, not proposing improvements and not knowing the customers needs is not a partnership. Real software craftsmen make the customer needs, their needs.

Conclusion

Recently, Uncle Bob Martin said during an interview:

The original torch of the Agile message has changed hands, and is now being carried by the Software Craftsmanship movement. These are the folks who continue to pursue the technical excellence and professionalism that drove the original founders of the Agile movement.

The Software Craftsmanship Movement is another step forward towards better and healthier software projects. Instead of just focusing on processes, it focuses on the quality of the software that is produced and most importantly on the attitude and competence of the people involved.


Software Craftsmanship brings pride and professionalism into software development.

Oh, Sugar

I have good news! With a minimal simplification of my INADEQUATE filter I have been able to rescue the last cross-peak of cholesterol. It had been rejected because too near to the diagonal. I changed the code saying: "if it's on the diagonal, it is bad; if it's just near, let's accept it". So it is possible to have the perfect INADEQUATE of cholesterol, with all the expected cross-peaks IN and everything else OUT.
Yesterday I received another INADEQUATE spectrum, this time of sucrose. The S/N is still high enough to make my filter unnecessary. If I play with the contour plot all the noise disappears while the 12 carbon atoms and their 10 bonds remain. Only a spurious peak remains at the coordinates 103.7;-22.9. I have not received the 1-D external projection, so I created it artificially. The spectral width is the same in both dimension (instead of being doubled for the DQF axis). The consequence is that two cross-peaks fall just on the boundary and are partially folded. This is the spectrum:

I have applied the filter with the same parameters used for the cholesterol (C-C coupling; linewidths in the two dimensions) while the threshold corresponds to the above plot. The result is perfect. All the cross-peaks are resolved and they are all present. Nothing else survives.

This time all the peaks are regular anti-phase doublets. The Js are generally larger than in cholesterol.
Click on the thumbnails to see the full-size pictures. There is an expansion to help counting the correct number of cross-peaks.
Do you want to send another spectrum? I can clean it for free. Remember to enclose the 13-C of the same sample.

Interstellar Space

One of the most ancient 2-D experiments has always been more talked about than practiced. The INADEQUATE was invented 30 years ago (an era in which many chemists were still using CW-NMR) and has always been regarded as a the future thing, like travels to the Moon. Everybody agrees it is useful, but the experimental difficulties are discouraging.
The information that can be extracted by this experiment is a formidable aid to unveil the structure of unknown natural compounds. For a long time, however, the experiment has been nearly impossible. You needed a powerful transmitter, because the nominal 180° pulse must be a true 180° pulse over a large spectral width. You also needed a sensitive probe. A cryo-probe is the best. Today we can have both things.
I am not mentioning here the many attempts to increase the actual sensitivity with experimental tricks, because this is a blog about software. 19 years ago the S/N limit was overcome by a pure numerical method, see this paper, commercialized as CCBond (also sold under the trademark FRED). I understand that the same program is now part of the larger NMRanalyst (™). No software that I know has ever changed the world. Having never personally used this particular product, I take for granted what the paper says while observing, at the same time, that its popularity is limited.
From the cited paper I read that the program failed to detect a few bonds in the INADEQUATE spectrum of cholesterol; the authors also explained why (the presence of second order effects). I think this is a serious issue. You can't use such a tool with confidence. A user has no way to verify if these second order effects are present or not, then he cannot verify if his spectrum can or cannot be handled by this particular program.
A few months ago I received a nice INADEQUATE spectrum. Guess which compound it was? Cholesterol! This is the first time I see this particular spectrum, so I can't tell if it was acquired correctly or not. I can see ALL the bonds with possibly one exception. I see the peaks corresponding to the bond between C20 and C22, yet it's not clear if they fall at the correct frequency or, instead, at the frequencies of C10 and C20. It is amazing, however, to see all the bonds without the aid of any special software. It is like discovering that we can go to the Moon with RyanAir.
A bitter surprise came from the observation that not all the peaks have the same shape. The very few articles I have read on the subject say that every cross peak is an anti-phase doublet (one peak goes up, the other peak goes down). Here, instead, I can see several different deviations from this simple model. This means that all the numerical methods previously described cannot be applied to this example. Should I try to improve this spectrum, I am going to lose one or more peaks, which would be a pity since I can already see all the peaks in the untreated spectrum. Maybe all the other INADEQUATE spectra ever acquired only have doublets and mine is the worst INADEQUATE ever. Maybe my spectrum is OK and the model is too simplistic. I hope some reader can solve this fundament doubt.
For the moment being I assume that my spectrum is perfect, simply because it's the only spectrum I have got. I have devised a new numerical method to improve it. It works like a filter and can clean the spectrum. I start from the known frequencies of the carbon atoms. They are easy to get from the standard 13-C spectrum reported at the top in my pictures. Everybody knows that a genuine peak can only fall at the frequency of an existing carbon. The cross-peaks also come in horizontal pairs. The vertical position (DQ frequency) of a pair is given by the sum of the two chemical shifts. Anything that does not fall at these predictable frequencies cannot be a genuine signal. My method filters it out. The signal/noise remains the same, but the plot is much easier to read.
My method, in theory, requires 4 parameter, provided by the user:
- A threshold level. This is the lowest contour level.
- The approximate width of a generic multiplet in the X dimension.
- The width in the Y dimension.
- The C-C coupling constant.
The method is not dumb, though. If nothing is found using the user's parameters, it will automatically try with different starting values. The last 3 parameters are easy to set, either by observation or by mining the literature. The most critical parameter is the threshold. Fortunately, it's not very critical.
I am going to show how tolerant my method can be. The previous picture shows the optimal threshold. [Click on the thumbnails to see the full-size pictures]. When I start from this value of threshold (1620), no cross-peak is lost and no false positive is created (spectrum not shown).
In the next picture the threshold has been reduced to less than the half, quite far from the optimal value. The threshold is now 620.

My method yields this:

The only difference between this and the spectrum processed with the optimal threshold, is the additional presence of the two cross-peaks labeled 22->20. This is good to know, because it means that the threshold is not critical. You can enter a wrong value and the result will almost be the same. This bond C20-C22 actually exists, but the cross-peaks on the left falls at the wrong frequency. It makes you think that C20 is bonded with C10 instead. As you can see, the column of C-10 now contains 5 cross-peaks (5 bonds??).
C-22 falls at 36.2 ppm. C-10 falls at 36.5. C-20 falls at 35.8. It is difficult to demonstrate if two atoms are bonded when their chemical shifts are so near.
I am very glad to see the cross peaks 5-10, 5-4, 3-4, 3-2 because they deviate from the simple doublet model. A simplistic model would never recognize these peaks, but my filter assumes a more generic model.
If I raise the threshold to 2620, the following cross-peaks disappear: 5-10, 1-10, 14-13 and 25-27. They are not so many, yet the basic lesson is: it is better to underestimate the threshold than to overestimate it.
I need more examples to test this method against. I feel it is promising indeed. The algorithm is simple and fast, which are always good qualities. It is amazing how far you can go with very little math.