Test coverage

My perception:
Test coverage is the quality metric suggesting the quality of test cases and hence of test adequacy criteria.

In short .... we are trying to quantify test adequacy.

A test case/suite is meant to cover or execute test objects like statements, conditions, paths etc as per its test adequacy criterion. The test adequacy criteria might at times be part of agreement between the client / customer and the service provider.

The more number of test objects the test adequacy criteria covers, more will be its coverage. And the more test coverage given by a test case/suite, better will be its quality.

General definition says ...
Test coverage = (no. of test objects covered or executed at least once) / (total number of test objects)

The concept can be illustrated as follows:
Suppose one test case/test suite covers 80 stmts out of 100, 8 branches out of 10, and 20 paths out of 25 then it is giving 80% stmts coverage, 80% branch coverage and 80% path coverage. And it will be a better test case/suite than that which provides lesser coverage than 80%.As an ideal case test cases/test suite should give 100% test coverage. But in practice it does not happen.

Actually test coverage analysis is a type of the code coverage analysis. The academic world more often uses the term "test coverage" while practitioners more often use "code coverage". Test coverage provides a quantifiable measure of how well the test suite actually tests the product. The most basic form of test coverage is to measure what procedures were and were not executed during the test suite.

There are many other test coverage measures.
1.Statement Coverage
2.Decision Coverage
3.Condition Coverage
4.Multiple Condition Coverage
5.Condition/Decision Coverage
6.Path Coverage
7.Function Coverage
8.Call Coverage
9.Data Flow Coverage
10.Object Code Branch Coverage
11.Loop Coverage
12.Relational Operator Coverage and many more.


Test coverage specifies the extend to which test suite ( at times test criteria) fulfils its defined objectives in the system from its inception to retirement like analysis,design,...... for practitioners (BRD, FDD, TDD, ......)
In each phase of SDLC, nature of test coverage varies but purpose remains same.

Test suites which encompasses test cases and results of each one of them, provides big time support during implementation.
In my opinion, test coverage techniques need to mature a little more so that there are no consequences where one phase (or may be more but rarely all) has higher test coverage but others strive to obtain average.

Hence result of that either others are not performed well or suppose to be eliminated.

Interesting Debate

Me and Cem have been having a very interesting debate over complete testing.

According to Cem the correct answer to the question should be choice ‘b’
Qs ) Which is the best definition of complete testing:
Choose one answer.
a. You have completed every test in the test plan.
b. You have discovered every bug in the program.
c. You have reached the scheduled ship date.
d. You have tested every statement, branch and combination of branches in the program.

And I say answer should be choice ‘d’. Cem reasons, How can testing be complete if there might still be bugs remaining?

And I go back and say that I feel ‘b’ and ‘d’ complement each other.
‘d’ is probably one way to achieve ‘b’.

I reason that all the business units across the world ( I am leaving the small software’s out of scope here, as they can be tested completely most of the times) plan their product releases. Before a release, it is certified that the testing has been complete. The products are released, but still we find 'n' number of customer bugs reported and which directly or indirectly lead to more regressions introduced in the software. These regressions are sometimes identified by the QA teams and sometimes are reported back as new bugs by the customers.
I have till date not seen any white paper from any of the big business houses where they have not had a single bug reported ( either internal or external) against a release.
Now, coming back to my interpretation….. What do you actually try to do in complete testing scenario ?? you try and have such test cases that will check all possible statements, branches and combination of branches in the program. Basically, I try and ensure that all my complexity node points are well covered during testing.

Yes, when I say ‘complete testing’, my idea is that ideally what ever I can think of as a test engineer ( with all my white box and black box test cases) I have covered all the possible scenarios I can think of. But factually, there is always a chance that there are few uncovered, un-trodden pathways lurking around the numerous code lines. More the components, more the dependencies, more the chances of failures. This is the simple testing ‘mantra’ that fits into my thoughts……..

It will be nice if we can have your views on my thoughts ……..

Before I end ….Can anyone recall embarrassment of Bill Gates as he stood to release Windows ’98 and then publicly had to call off the release coz of a failure at the time of release and all this was going on live ……….broadcasting all over the world ……

and last but not the least ....... I am not debating against the definition, but the thought that goes behind the definition ......

what is happenning

Lots of things .......

BBST-IC .......its becoming interesting day by day.............. learning other aspects of things that I had probably not laid much importance to while conducting classes .........

Moving towards Dreams

I always revered people like Cem Kaner, James Bach, Michael Bolton and the likes ....... My interest in the field of testing started because of them ........ there was a time I'd spend unlimited hours in the library and in my lab at the department reading their papers in various journals and magazines ....... My topic for PhD thesis and work was also formalized around software testing .............. all this started way back in 1997 when I was doing my MTech ........... it was a dream that someday I could meet and talk to these icons .........and here in 2007, last week of Jan the dream came true .......

Today, I have joined a course that is being taught by all my icons and I feel so happy and proud to be their student .......I still have a long way to go ........ this is just a start of another dream come true......... but at least I have touched the tip of the iceberg......... hoping that I can measure the whole length, breadth and depth of it in near future ........

I also take the opportunity to thank God for all that he's given me or rather should say, blessed me with !!

Testing - Context Driven

I very strongly believe in the context-driven school of testing.
Learn more at: http://www.context-driven-testing.com/

I always reason.......how can testing happen without a context ?
When ever you start testing, you already have a goal in mind. The goal can be the expected output of the program. It can be expectations from you as a tester, in sync with the Requirement document, Business requirement document, test design document or functional design document and so on ........ (I plan to keep adding to the list :) )

The crux ....... there will always be a start point and an end point to anything under test in this practical world ...... and as soon as you define these ........everything with this population has to be contextual !!

So whatever you subject to testing will be related to some or the other context.

Why 'Testing' is more important than 'Automation' as a career

Today I wrote to David Gilbert asking him why he is wanting to move away from automation and his reply was worth every cents ............ He wrote ........

Meeta -- it is not that I want to get away from automation, but that I want to shift the focus of my career.
I enjoy doing automation work, I find it mentally challenging and rewarding, in the right environments. Unfortunately, it is often done badly, and then I just find it frustrating to be a test code monkey doing things that I believe will not add value long term.
But the larger point is, as a technical automation implementation expert, I am very limited as to where my career can go. Tool jockies are a dime a dozen, and no one really cares about their opinion. If I want to continue to grow my career, influence, and recognition in the industry...if I want to try to be influential and do more than just make a living...then I have to break out of the mold of just being another tool jockey. I have to become a methodologist, and a teacher and promoter of ideas. It is fortunate that this also provides an outlet for my more creative and inquisitive nature, and that it allows a venue for my opinions about what good testing is and is not.
David

The Start

Well ......here I go ....... not an expert at blogging, but had been contemplating for some time ..... Cannot give any big excuse, but probably was just procrastinating the idea to leave a lot of personal info online for people to see .........

Then last month Cem Kaner asked me if I have a blog that I maintain ...... and then I decided that I just need to kick start somewhere .......sometime .......... and here I am ......... this is the first post ....... nothing technical to write in, but yes, since the new year resolution (not planned) has started, be sure to check in to some stuff !!