The modern tester

I choose to healthily ignore the various various debates around the place of the tester, as it hasn’t changed that much. With every wave of new developers, comes along the same misplaced arrogance that they can do no wrong.  The chips have yet to leave the shoulders.  

Scrum Case Study 1

In the most perfect of methodology application, there is always room for improvement. To assume otherwise is misguided over-confidence. Take Scrum, the most used (and abused) of the modern methodologies. The principle, as always is simple – better project management and more control over requirements. It requires more future planning, consitent review pattern, and following through to the development process. It is no good having good project management, if the development teams are not following through with code.

The project was a middleware service providing Always-On email system. Though common now, this was in 2005 so the concept of a true mobile LAN email service was relatively new. The first action was devising a test strategy, which defines the high-level testing scope and testing cycles. Though a middleware service, it still has to be verified for the input and output, i.e. email sent, delivery and verification of content. The company had adopted a system of using real phones and emulators to test email content and user experience. The focus of our testing was on the core application and database, which would pontentially servic a high level of traffic.

The project was working with an efficient Scrum management and Continuous integration process. They were very competent developers and the architectural team also very skilled. But what comes with these talents is overlooking chances for improvements. The end-user experience was paramount in gauging success of the middleware. End to end testing was the most efficient and useful method of testing. I am not there to pointlessly increase scope of testing for financial gain. Consultancy should have morals. What I needed to add to ensure a controlled QA gate for releases, was a test environment, and test framework to provide the external and better overview that QA can provide. The problem I identified to begin with is that they were working in solely Incremental fashion, when Iterative AND Incremental is a better approach to deliver to requirement. People may now at the mention of “milestones”, but when else is a client going to get what they want?

For the essential load testing I chose STAF, an open source load testing framework, driven by xml-based user profiles. This was perfect, as user profiling was very appropriate for email users. The advantage of using open source tools is they act as a kcik-start to what you are actually looking for. Rather than letting the tool dictate what you test and how your test. It some work to get it up to the standard you want, but ultimately more useful and can be included in development cycle. Developers will find them useful also, and more likely to gravitate to open source, over bespoke commercial products.

RIP Test Manager (Role)

The Test Manager’s role, as traditionally defined is becoming less and less relevant. As more testers are expected to plan and scripts, test manager’s role has devolved into coordination and more than likely, actual testing (if a 1-man band). This is not a negative thing, as I believe this role is being superseded by that of QA Manager (much more concerned with overall processes of delivering a product). As such, a much more valuable commodity.

This has happened more by accident than design – Agile’s focus on more rapid cycles, and lighter documentation has seen the Test Manager’s role evolve into more QA. In such a position you can better influence project processes, based on sound QA principle. Milestones are traditional business measure of progress, and although Sprint demos for stakeholder partly fulfil this, there really is no milestone until the end. This led to Waterscrum, where Sprints became lengthier in a more traditional Waterfall way. For Test Managers to survive they must become more flexible in scope of their roles.

Testing is widely misconstrued as a task rather than a role. Therefore the “task” is assigned to anyone half-relevant to a project. Anyone can test – that is true – but testing itself is the activity, but there are other tasks around it just as essential, which is ensuring the test process is planned executed and recorded in line with the overall project process. Also, testing is not just about finding defects, sticking rigidly to scope – issues can be raised against requirements, improvements, etc. With this wholesale delusion about what a tester is, suddenly the idea of a Test Manager seemed a little unnecessary.

Of course this is wrong, but the Test Manager role has changed – not through any formal change, it happened out there in the field – professional defining their role within a project, and adopting the same principles as development, i.e. being part of cross-functional team. The new Test Manager will be defining Test Strategy, contributing to Sprint planning sessions, coordinating with Product Owner on stakeholder demonstrations, putting forward plan for process improvements. The role has got bigger and become a QA Manager.

Behaviour Driven Development

Following recent experience with Agile Methodology BDD – I was impressed by how this appraoch provided tighter bond between user story and resulting code.  And with maintenance in mind – every time a new story added or amended, these changes propogate down through to source code level.  BDD also provides the clear scope for testing, and by using scenrarios as basis for a complete regression test set.  A lot in Agile is relaint on the quality of User Stories (as in fact any methodology is reliant on quality of requirements).  The BDD approach does ensure that user Stories are firmly in forefront of the SDLC.

Once a user story is defined, should we assume that is final and that if issues occur outside of its borders, they should be ignored? Test planning based on user stories is a constantly evolving documentation – only through continued development and testing is some issues arrive that weren’t considered in original scope. That is normal. Exploratory testing aims to catch these type of issues before they cause endemic problems in the system. What the stakeholder wants is important – but they are not going to be impressed by pedantic attitude of “well, you asked for it, so you got it”. To test firmly within boundaries is a risky business, becasue you are making dangerous assumption on the quality of user stories/requirements. I was always taught that you test requirements too. And you raise bugs against them if necessary, which in the days when business anaylsts were commonplace, was standard practice.

BDD (Behavior Driven Development) tools such as http://www.specflow.org, and Cucumber can help with exploratory testing, as with these tools, the person designing user stories will be more conscious of the requirement to code flow. BDD approach helps avoid the need for out-of-scope exploratory testing, as outside case scenarios can be addressed and included in earlier stages. The creator will be more aware of cases outside of their scenarios, as code generated can be reviewed as user stories as worked on. These tools encourage earlier reivew between analysis and development, but generating pseudo code based on user story input (which of course means requirement to code is traceable and more likely to be accurate to begin with. There is an overhead in planning/design stages, but less issues will be result. These tools developed from lessons learnt in Agile – the path from user story to code must be smooth and efficient, and if the user stories are properly integrated into the SDLC, then less basic mistakes will happen.

In specflow, this is how a user story is defined and broken down into scenario (which is turn can have 1 or more scenario outlines). This is nothing new fundamentally, but because the BDD workflow focus is heavily on the user story -> coding path. A major area of failure.

User story for a user dashboard, using Specflow’s BDD approach (the tables are decision tables – identifying test data prerequistes, and also examples of input/output).

Feature: Status Lights
In order to quickly view Section status
As Client User
I want a traffic lights type system to represent Section statuses

Background:
Given I am logged in as valid user
And I am on Homepage
And there are Sections and Sub-sections following the criteria in table below

| Type | Status | Worfklow |
| Sub-section | grey | Sub-section should be marked as no longer applicable |
| Sub-section | grey | Section marked as no longer applicable |
| Sub-section | tick | Final step in workflow has been signed off |
| Sub-section | green | Final step not signed off, and due date > 30 days away |
| Sub-section | Amber | Final step not signed off, and due date 7 days away |
| Sub-section | Red | Final step not signed off and due date 30 days away |
| Section | Amber | Step not signed off and due date 7 days away |
| Section | Red | Steps not signed off and due date <= 7 days away |
| Section | Red | Steps not signed off and due date already passed |

@StatusLights
Scenario: Status Lights
Given has Section(s) in various statuses
And I am logged in as preparer or reviewer
When I view Company Status
Then Company Status displays according the Section(s) status

@StatusLightsChangeCompany
Scenario Outline: Company Status Lights Change
Given A Section contains "”
And Section has “” at either Sub-section or Section workflow status
When I view Company status
Then Company Status displays the lowest “” of “” of any Section

Examples:
| Status | Rank |
| Red | 0 |
| Grey | 1 |
| Amber | 2 |
| Green | 3 |
| Tick | 4 |

@StatusLightsChangeSection
Scenario Outline: Section Status Lights Change
Given A Sub-section contains “”
And Sub-section has “”
When I view Section status
Then Section Status displays the lowest “” of “” of any Sub-section

Examples:
| Status | Rank |
| Red | 0 |
| Grey | 1 |
| Amber | 2 |
| Green | 3 |
| Tick | 4 |

Test Manager … anyone?

With the ascent of Agile/SCRUM, and its development/client focussed approach, QA made way for simply testers – testers who remit was solely driven by development. A double-edged sword. As project management and development has methodology, so does QA. The weak links in Agile pontetially are the SCRUMMASTER and Product Owner. Certain assumptions are made in Agile, and a key one oft forgotten is to place people with the right skills in the right role. This rarely happens.

The Art of User Stories

User story approach is sound, but as with all things simple, it can end up complicated.  The main reason for this is not observing the fundamentals of a user story.  Also, it is worth keeping in mind a user story AND the scenrios breakdowns should be clear and understandable to ANYONE, whether they are a user, stakeholder, developer or tester.

To address what a user story is, lets start by saying what it isnt:

It isnt a “mini” Use Case
It isnt a complete specification
It isnt a contract (User Stories can be amended at any point – Agile welcomes change)
It isnt intended to be interpreted without a Product Owner (this is simply not Agile)

A user story template can be illustrated as follows:

As a <User or role> I want <Business Functionality> so that <Business Justification>”

Example:

As an Editor, I want to be able to enable my authors to publish articles immediately to live, in order to avoid delays where delays are unaceptable.

This user story can be broken down into senarios, i.e. the different conditions where an author would need publish to live, without going through an Editor.

Scenario 1: When an article needs editing because of libellious content
Scenario 2: When no Editor is available to vet articles.
Scenario 3: Articles delivered from trusted authors external to company.

…… etc.

These scenarios become the breakdown of the main user story, and indication of work required.

Depending on scope of the scenarios, will determine if the User Story can be deifned as an “Epic”. An “Epic” are  compound Stories, that can be broken down into several smaller, more focused stories, and may encompass enough work for several Sprints (iterations).

Some guidelines to writing good User Stories

• Tangible acceptance tests can be written against User Story
• The scope of the User Story is manage-able enough for the team to provide an Estimate
• Is independent and do not rely on other Stories
• Sized appropriately (i.e. have a level of effort which the team can comfortably achieve in the duration of a single iteration).

World Usability Day

Human Factors International are running an essay competition, with a hard topic – How can the User Experience Community support the future of sustainability? My definition of User Experience may vary from others, as there are a lot of differing opinions as to aims and scope.  This highlights the importance of the user experience, and as everyone sees the world a little differently to each other.  But I believe in keeping things simple – User Experience incoroporates the design and functionality heuristics of a website.  Pandering to users can be a disaster, so it is a fine line between Good/Bad User Experience.   The “customer is always right” can be disasterous in software development.  Just because someone knows what they want, it doesnt mean it is what they need, or indeed if they actually want it!  How it can affect sustainability is a very (deliberate, I’m sure) open question.

World Usability Day

Where Open Source meets Usability

A common problem in open source development (though forgivable, given nature of open source), is completeness.  http://www.openusability.org is an effort to redress this from a usability angle at least.  Still early days, but well worth supporting.

Getting to a state where it is usable for ‘the common computer users’ is one of the greatest challenges for Free/Libre and Open Source Software (FLOSS). During the last years, the awareness for usability has significantly increased among FLOSS developers. Still, little knowledge exists about the scope, the methodology and the application of a user-oriented design process.

Changing this is one the major goal of the OpenUsability project. It provides a platform to bring Open Source developers and usability contributors together. The idea is to establish a long-term relationship between usability and FLOSS development, and to foster the establishment of user-oriented design processes.

OpenUsability – Where Open Source meets Usability » Projects

Test Consultancy Chancers

There is a bewildering array of “test consultancies”, quote marked as many are just agencies in disguise. If your need is pure resourcing, identified within your company, then I am sure there is little difference between them. But give then access to your project, and you will pay the price. Consultancies have a vested interest in making you feel bad about yourself. For myself, a consultant should point out weaknesses but not before coming up with possible solutions. Sadly, upper management are generally easily conned by the QA-sell, and no slur of them – upper management generally make decisions based on what is said, not what is evaluated. And QA is awash with great terminology. An effective Test Consultant can easily pull a project to pieces, and most certainly will not recommend that any of your existing test process remain. Think about it – what consultancy is ever going to tell you that you are doing a good job, nto when their scope of work is based on how bad you think you are doing. But that is not what they should be doing.

There are more honest consultancies out there, and we like to think we are one of them, as behind our analytical approach, is understanding that a project comprises of human beings, and it is very easy to demoralise staff under stress. Be wary of consultancies promising answers to all your ills – they are lying – improvements within constraints of your resources and budget are far more valuable. From our experience in aggressive start-ups, we have learnt anything in possible in any timeframe. And contingency planning is key here. For example, you have to have a go-live with minimal testing due to timeframe? Fine, go for it – but back it up with structured live user support, and ongoing testing on live system. Go-live does not mean end of testing – a very common misconception.

Consultancy or Agency?

I have slated ISEB numerous times, and the robotic generation of QA it has resulted in.  Our specialism is in taking sound QA  principles, and adapting to environments.    We have done this for many years, and doubt there are many consultants out there with as much modern test experience in our chosen areas of web/media.   We choose Agile as overall approach, but also take parts from other methdologies as seems appropriate.   SCRUM, for example, is more given to projects where all project members are dedicated and in same location.