Scrum & Testing

In fact, a Scrum team need not have a manager at all. In Scrum, the team is ‘self-organizing’. To as large a degree as possible, they manage themselves.

How many ways can you code

Not surprising our leaning towards manufacturing, given the budget production line obsession of business towards development.

The recent story on the cheapest meal, put me in mind of some dole/student classics. Poor mans pizza was rice and ketchup. Anything new can taste interesting, as with the toast sandwich. But you will find after 2/3 experiences, that culinary surprise will dissipate into reality of cheap vinegary tomato flavoured starchy rice. Adding tuna will allay that feeling for a few more meals but ultimately you have to spend more and more to get it to taste nicer. By which point, you may as well have gone for making a decent budget casserole or lasagne.

Basically, you can go too cheap. And many saw Agile as a way to do it on the cheap – why? Because the traditional structure of a project was changed, with generally less roles which immediately followed was less people.  First error – assume downsizing was part of Agile solutions!  They don;t need repeating, but the 4 core statement in the Agile manifesto were:-

1. Individuals and interactions over processes and tools
2. Working software over comprehensive documentation
3. Customer collaboration over contract negotiation
4. Responding to change over following a plan

Of the four core Agile Manifesto directive, the overall experience for myself was that: (1) was done the wrong way round, (2) highlighted development wasn’t that agile, (3) companies couldn’t separate the two and (4) led weak/poor project management.  My favourite of the principles regards development was: Continuous attention to technical excellence and good design enhances agility.   For myself, Agile was about doing things better, not doing things cheaper.

With Lean, there isn’t so much room for subterfuge or being dogmatic.  A startup illustrates to benefits of Lean (and Kanban), as it is more given to dynamic decision-making and calculated risks. Whereas Agile collated principles from existing, though disparate, methodologies; Lean has far more focused sense of continium and improvement – not principles for development but any kind of engineering, where the process to produce a product is just as important as the product.  To manage such a demanding development flow, Kanban provides ideal based to manage a project.  The “Work in Progress” principle behind Kanban, illustrates acknowledgement how rapidly things can change, and the balance work evenly and maintain a realistic map of progress.

Lean and Kanban are deceptively simple, because of course as we all know, management is by far the riskier element on modern web projects.  It takes skill and ability to think on your feet.  Some of us actually enjoy that!

R & D

Research and development – less of a specialised area, more normality.  Of course there is a lot of “keeping up with the Jones’s” marketing around a lot of newer technologies and techniques emerging. You can end up like a pinball, bouncing between different things, that are actually the same. If you alter the way to see coding, it can help.  A lot of the practices advocated in Agile and others, is Iterative and Incremental, i.e. the practice of shorter release cycles, and incremental releases within those cycles.  The ultimate aim is to produce a continuum of development, integration tested at source.  At at the end of each short release, the software should be demonstrable.

To achieve this not only takes applied effort to make the process work, but flexibility and adaptability. These are not two fashion words, they are important in Iterative and Incremental development, enforced with a Thor hammer with Lean development.  The Incremental part demands effective Continuous Integration, i.e.  the regular (usually daily) check-in of all code work and a set of automated tests run (usually overnight).  It leads to a better ongoing development and testing process, that can be sustained.  And avoid pointless stress. Burning out project team members is still ridiculously common, and it is not wonder developers and testers lean more to contracting.

In a truly communicative team, it is surprisingly how many changes can be made mid-stream.  In fact, if this voicing-up is encouraged, you can prevent things slipping further down a bad path.  If that isn’t encouraged  or an atmosphere of blame-culture exists, you are stuffed because problems can slip far down the line, to give you a step-down type bite on the behind.  There was a reason step-down approach to development is costly on modern IT projects.  There was an area of risk that never could be accounted for, until point of testing.  And even then, a more obscure issue could pop up on live system.

This is really to hammer home the reduction of risk, when adopting an R&D approach to development. It takes effective management and someone who has acute gift for time and resource management.  R&D doesn’t have to be limited to development, testers have long been using the open source world, or even their own programming skills, to assist the QA effort.  Sometimes a requirement only becomes apparent once a project is in movement. A project can learn from others of course, but will also have it’s own demands.

The best way to spread knowledge across multiple projects is not weekly meetings in the canteen or Starbucks. It is sharing resources around effectively (this is where Kanban can help). You don’t have to latch onto a methodology or set your store in clunky all-in-one “Agile” software. But taking ones more specific to your needs is a sensible approach.  You don’t find the methodology, it finds you – so to speak. And commit to the damn thing – I have seen so much weak Scrum-mastering over the last decade, it almost makes me weep for the fine methodology it is.  But even that approach seems slow now – the march of time!

Whilst it’s important to keep up with the IT world, its also important not to become a slave to it.  Development is evolving in great directions – more programming language agnostic directions, to help bridge gaps between requirements and code.  Testers are required to program these days, whether Selenium scripting or being able to debug code.  There is no escaping developers and testers have to adapt faster now, and project management even more so. Its a good time to dump the labels, and get to the core of a project which is the management of development. You can argue that R&D is a fashion label, and you are probably right.  Maybe it’s is better to say it’s right, if you have justification to use it – not just using the word!

Throwing out more proverbial babies

I knew it wouldn’t be long before Scrum came under scrutiny , and hence, in the firing line.  Scrum is far from perfect, but provided some kind of transition form palatable to both business and development.

True Agile is happy productive team

Something that’s plain post-Agile is it wasn’t granular enough for public consumption.  Using the principles and devising your own way of working was far more practical. Of course people were already doing this prior to Agile becoming formalised manifesto.  Test driven development (TDD), Extreme Programming (XP), Scrum, Kanban all existed prior to the manifesto in 2001.  In fact Agile seems to be an aggregation of the the collective evolution of advances in quality process within engineering, production, and team management.  As with any latest fad, Agile became a start point, instead of a renaissance; it was a merge of all those advances, in media-friendly soundbites.

All of a sudden everything pre-Agile became wrong, by default.  Previously disinterested business management, suddenly found a new tool to advance their careers, which they could promote without really understanding.  Agile was introduced as a term first, a cry to be “more Agile”.  They may not have understood the term, but they did know it sounded positive and new.

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

The classic example of the principle of having all important information “above the fold” is demonstrated in the Twelve Principles of Agile Software. It is very plain the principles that were adopted stopped around “Build projects around motivated individuals ….”. Yes, but what kind of motivation – advancement in career, more money, more power? Corporations need to tread very carefully around these principles, and need to be especially clear on definitions.

The disclaimer underneath is essential approach. It does not suggest one is right, and the other is wrong. It got to ridiculous stage when you suggested tools or processes, people would wrinkle their noses in Agile-disgust. And then carried on with their efficient bug-tracking system using email. Programming is all about processes – without them it is structureless and chaotic. Maybe there is a future programming language that will work like that, but it isn’t here yet. Running a project in an ad-hoc fashion, having too many meetings and unrealistic schedules creates more stress than on projects running using older methodologies. The part people generally avoid is Agile and psychology. Have a happy productive cross-functional team, and you have an Agile team. And it doesn’t happen by magic.

A lack of trust can be the biggest barrier for any team. Building trust and shared values on a team is critical. Professional Team Management Tips For Creative Folks

Business management still relies on subterfuge as part of their career defence, and in fact the whole of business still plays an old fashioned game. So whatever changes are made to be more Agile on projects, it can be scuppered at management level by too much self-interest, and disengagement form the team. This can lead to a more lacklustre application of Agile at development level, as the project management becomes unrealistic. Quality of code will go downhill and people will not be so forthcoming is they are having problems with workloads. Management of the project team and their workload is paramount in Agile, so use more effective processes to deal with the specific roles. It is also worth promoting what you do to others, who can maybe learn from the successes/failures of Everything is up for review/change principle in Agile, including processes – and people!

BDD

“I am struggling to do this”, or “I need some help” or “I’ve finished, give me something else” are not common phrases to hear on projects. It’s the old ways again, now Agile has lost it’s grip as people more freely slate it. Agile was not a hard enough kick, but Lean development certainly is.  Lean development is not disconnected with Agile, as Agile takes some principle from lean (manufacturing).  Scrum was first applied in manufacturing to streamline the product quality process. Apply them to the more people-sensitive world of IT, and you have a lot of psychology to learn quickly to overcome the most common point of failure for any project – people. Or rather the understanding and communication.

It doesn’t take much to make someone feel appreciated in their efforts. It should quickly be established that the important part is the journey, not the destination. Struggles overcome, or mistakes learnt from as advantages for the project, and knowledge for future projects. It is all very well applying process, but without the team selling up to same ideal and following it, it’s a fruitless exercise where the only surprise will be how the project will fail. It make still work, it may still even make money, but in the shallow world of IT and Sales, website looks that kill, can easily mask a hideous mess underneath.

TDD is a very clean approach to programming – a little too simple for business requirements though, but BDD covers other sides in more detail, aimed at entire project rather than just development. Asides fighting a more established acronym (Body Dysmorphic Disorder), BDD works on sound basis that tests should directly correlate to requirements. And that being the case, then a further link in the chain could be added. The User Story.

What BDD and TDD share in common though is a very structured approach to development, with adherently more administration and maintenance. This is why many developers do not like this approach – it forces them to develop in a structured (and auditable) fashion. BDD can feel like a roller-coaster, but run well extremely efficient. Apply Kanban to the project management (Scrum would struggle to keep up). but requirements and code will merge into one in the future. For now BDD can forge a path, given it’s ideal suited for DSM/DSL approach to development. I go into depth with these in other posts, but just imagine the next steps in programming where what we consider as programs now are rapidly becoming components – the whole nature of the evolution of programming.

The BDD tools available use DSL approach to generates test script framework from entered User Stories/Scenarios. These vary a little from traditional format as they can also include such things as tables of data, and more detailed pre-requisites information. It is not for the faint-hearted, as even those with a lot of experience with requirements will have to work hard to make sure these User Stories hang together. Indeed, the process can quickly highlight requirements that may contradict each other, or cause technical issues. The method is within the area of DSL, the concept of generating code from directly from requirements written in real language.

The tools (such as Cucumber and Specflow) got part way down this path, generating a good test script from the entered User story. Cucumber and Specflow use the Gherkin language for User stories. Any amendments will update the test automatically, but it won’t update the code! This is where developers have to be particularly aware.

Another good side of adopting this approach is it quickly identifies project “wallflowers”. It is uncomfortable truth that companies do carry dead-weight. I am not talking about lack of skills – these can be gained in right environment with team members who can mentor and guide. I am talking about attitude. Modern development approaches are supposed to inspire, as well as help deliver projects well. The “wallflowers” are ones who simply go through the motions in the workplace, and have little desire to change their workload. There are places for these people, but not in modern development.

Kanban/BDD Case Study

The first lesson I had with Behaviour Driven Development (BDD),  is that Kanban is ideal project management match. And particularly suitable for re-factoring projects, though prepare yourselves for more admin and management.

The remit was to implement a clearly defined set of new features, into an unknown arena. No developer likes picking up code in such advanced state as there are many possibilities as to what can go wrong. There were no registered open bugs of any note, as but development progressed, issues regularly surfaced at integration point. Here we hit what we realised was accidental Kanban. Because of the unpredictability of re-factoring projects, the development timeline became more fluid and subject to change. Although not generally done as default, I created new feature files in the Specflow BDD tool to cover the bugfix or where possible added or edited a scenario to an existing feature file (a feature file comprises of a user story and one or many scenarios). This meant that the developer fixing the bug could still follow the BDD process.

Exploratory Testing (ET) is much used (and abused) term, but is very valid description of testing done outside of BDD boundaries. However there is no reason that you can’t still keep to the feature file process when devising ET scope. No application is perfect, so the record of no outstanding issues looked likely to be a simple project closure exercise. So I used the ramp of issues closed near project end date as start point and sure enough, found many existing issues. Due to resources these issues were reviewed with harsh eye, only prioritising ones that could not be postponed. For example, one of the first issues found was that weekly newsletter emails were not being sent out, and had accumulated on mailbox server.

Security testing was not considered in the original project work, but with any website I believe is performing standard checks for security to highlight the more common XSS-scripting and SQL Injection vulnerabilities. It is surprisingly how many developers leave basic holes such a password field autocomplete set active. Security testing can go to real depths, but you can catch most hackers out with basic tests using specific lightweight tools. Remember, most hackers are amateurs.

Kanban provided a dynamic way to manage Work in Progress, which is at Kanban core. The idea is the keep your Kanban board well weighted, with work spread out to resources evenly and identifying the cross-functional elements of the team. Due to my overall experience I was able to offer consultancy on improvements from Kanban perspective. Scrum is iterative in approach, but Kanban is Iterative AND Incremental – managing the parts and the whole. More challenging from management perspective, but extremely effective at gelling a team.

 

BDD – Behaviour Driven Development is a set of practices which help software development teams to have conversations about the behaviour of their system and how it delivers value to the project stakeholders. BDD has changed from its early roots as a replacement to TDD and now works as a mini-methodology across the whole software lifecycle. Over the last few years the adoption of BDD has grown globally, with dozens of tools created, used by hundreds of projects around the world. In this talk we look at the original reasons behind the creation of BDD, bringing the focus away from the tools and back to its purpose and the philosophies which support it, including Real Options, Deliberate Discovery and Feature Injection.

Comedy by Kanban

I am big comedy fan, and in more nerdy moments, looked at sites about how sitcoms are made in the US. And it struck me how Agile they were as a default. They have many story writers, and each has their own specialist topics (so can be cross-functional). They employ storyboarding and continual review/improve process to scripts. Script reviews are with the whole team – when problems are ironed out. Episodes broken down into smaller storylines (sometimes more than one writers). Each writer has their own character that they write best for, so they will certainly be very active in cross-functional capacity. Some episodes require work outside of the core team (special effects, location finders, etc.). The audience (stakeholders) are sounded out, and work still goes on to hone the script.

Below is a general process that the Seinfeld team employed. It touches on Agile, Scrum & Kanban ethos, in the process of getting an idea on our TV screens. But if I had to name one to associate, it would be Kanban approach.

1. Idea for show (Project Initiation) – The main thing that makes for a good episode is the initial idea. The idea should be simple yet worthy of exploration, be easily expressible in about two or three sentences.

2. “Yeah let’s do it” (Project Inception) – After the initial idea/project initiation, it still needs to be pitched to someone else to get the go ahead to be made (and here comes the Client …)

3. Episode Outlines (User Stories)

4. Script (Scenarios) – This is the actual script/scenarios process (from those initial outlines/stories). This is the stage where peer review, and good management is necessary. Also, if the writing/coding starts to go bad here, the muscles atrophy and the script/application dies immediately. Some ideas/scenarios look good, even in an outline/user story, but just are not writable/programmable right then.

5. Read-through/Rewrite/Rehearsal/Rewrite (Tasks) – This time gets the now created “body” off the table and walking on its own. This is the development cycle at work (Enter Kanban).

6. Shoot night (Acceptance testing) – You have to think on your feet, you have to deal with people seeing your work and watching your brain work as you field questions, and that it’s a weird feeling to get used to. This stage shows, too, that there is never a time when editing/review isn’t done.  Editing/reviewing is always going on.

7.Final Editing (Deployment) – Just before it goes on air/deployed, you go back through and do a final edit for cosmetics and the like.

8. Celebration (Review/Improve) – After each show/delivery, there should be a celebration ritual. It is important to celebrate before starting on another project. Celebrate the fact that you all did it and it’s done. And then, get ready to do the whole thing again!

Real-life Kanban

Have you ever found yourself in accidental Kanban on software development project? When the developer is efficiently managing his own WIP (Work in progress), and driving forward changes without clearing with the business? Not ideal, as a manager should be dealing with the WIP, and changes to plan should be by consesnsus. Developers like this don’t grow on trees, so there has to be a way of designing new processes (process fast becoming a dirty word, but as yet no effective substitute) to steer projects on a Kanban course.

Stop Starting Start Finishing (http://www.software-kanban.de)

Kanban introduces improvements in momentum and sustaining that momentum. It is a methodology for managing CHANGE – it is not a project or development methodology. As with any methodology, the simplicity of the fundamentals belies the underlying complexities. Managing WIP cannot be slapdash or applied weakly. Software development has careered forward in wake of Agile, and took hits on quality. With stakeholders and developers (supposedly) more actively testing, testers became an almost endangered species at some points. With shorter, smaller scale web projects, sometimes testing was victim of budget or time constraints. Quality is not omitted as a default, but degrees of success can be achieved without dedicated test resource, even on badly-run projects. In manufacturing this would not be possible for long-term (even short-term) survival.

Quality Assurance has long been in manufacturing industry, and carries weight at every stage of the process. In the thinking of end-to-end path to product from requirements, QA transcends industry – but for some reason, despite it’s lucrative nature, software development remain in a delivery-panic mode. The holy grail became how to make products as quickly and as marketable as possible. In the efforts to shorten the journey there have been compromises, mainly on quality. Though “We have had to cut the testing budget” sounds a lot less alarmist than “We have had to increase risk compromising quality”. I used a similar style recently, when defending case for proper validated HTML and CSS code (traditional back-burner task). In response to pushback I said “OK, so we are all happy with not fixing the invalid code, that may also be causing performance issues?”. OK, melodramatic, but I was just making a point.

With Lean IT, you can streamline development to be relevant and efficient – and Kanban is very effective way to manage the development cycle. Kanban ensures that rules are in place to streamline flow of requirements to code to production. It is oriented around concepts of ultizing your resource and skills, without overusing them. The review and improve in continuous flow is essential for this to work, and therein lies the biggest challenge for IT with Lean methodologies.

What struck me most about Kanban was in the implementation challenge, within different country and work cultures. The UK, along with others, had problems with the Agile ethos, and that was due to many cultural factors. For instance, there is a general lack of openness and transparency in British business. So concepts of learning through mistakes looks logical, but implementing it in a culture where failure is to be avoided, and In the US, they tend to have a more team-oriented way of thinking. Communication is a big deal in Lean and Agile thinking, and it’s importance and definition varies from country to country and industry to industry. The trouble is we can still get away with a lot in software development, and I speak from experience in development. To produce code of quality and robustness is difficult to demonstrate to the customer – it has no instant gratification for someone who will judge based primarily on what they see. But the customer can have visibility of the processes that transfer what that ask for, into end product.

What Agile suggested with a nudge, Lean/Kanban drives home with a sledgehammer – transparency, communication, adaptability.

Kanban Polyglot

…. the essence of software engineering consists of working out the specification, design, and verification of a highly precise and richly detailed set of interlocking concepts. What makes software development difficult is its essential complexity, conformity, changeability, and invisibility.
“No Silver Bullets—Essence and Accident in Software Engineering” (Fred Brooks, April 1987).

The Polyglot Man - Orange (Joan Miro - 1969)

The title sounds like a character from Hitchikers Guide To The Galaxy, but the two are very relevant to each other. Kanban has been rolling around my thoughts and online rants for a while now. I am still learning – I would never be arrogant enough to assume my learning has stopped. And that I can get trapped down a rabbit hole as easily as anyone else when trying to work out the best process. The wave that carried Agile was rough, but it was also an essential learning curve. But in widespread application as purely as a user guide, rather than using it as guidelines, there were errors of judgement. The most common being that the processes devised (or more likely copied) were not being followed through. This was down to general misunderstanding of what Agile was to start with, but also the resistance to change – a risk with any change within a company, not just software development.

Whilst business world nodded to Agile, liked the ethos, liked the idea of faster and better quality delivery, it singularly failed to go Agile itself. Or modernize at all. For all the advancement, I still see a gulf between business and development – in fact, even more so with Agile/Lean, because of their promotion of transparency, experimentation, embracing mistakes as learning opportunities. These aren’t natural bedfellows for business. But it is too easy to waste too much space whining about that problem. Perhaps a better strategy is to implement processes to development cycle by stealth. Largely business don’t actually care about the process, they care about the product.

Software development has become too machine like – the idea that anyone can be a programmers is a tempting chant. It’s accurate but let’s face it, just actual coding (especially with all the helpful IDE’s these days) is a small part of the project story.

I decided to learn a new language, or as it’s C++, relearn something I haven’t touched for many years.  Programming languages generally evolve over time, I mean the fundamental ones like C++ which really brought object-oriented programming to the fore.  When I was choosing, I decided to go for an established and still popular language.  It’s faster to run than many modern languages, and more robust and scalable.  The impatient fervour around Ruby on Rails was too early, as people essentially lactched to its flashy timesaving features, rather than being concerned with the substance.

Creating a programming language is one thing, but it has to go through years of implementations and learning to discover it pros and cons (and all languages have them).  Polygot programming illustrates ideas of using using languages for different features – one language is rarely appropriate for an entire system.  Choosing a fundamental language to learn seems a better approach, as when learning a fundamental languages, other languages can be learnt a lot more quickly.  The jump from C++ to Ruby is a lot less painful than vice versa.  Rather than waste time on more narrow fluff languages, I prefer to learn one that inspired many other languages.

http://polyglotprogramming.com/
http://www.jedi.be/blog/2010/02/12/what-is-this-devops-thing-anyway/