When a Test Fails and How to Deal with it?

When testing software for unwanted errors, quality assurance engineers can paradoxically be a source of many different failures themselves. It’s widely known that the best protection against making mistakes is to learn from them and, even more significantly, be aware of their existence. Here’s a list of the possible test fails and an explanation of how to deal with them and gain more expertise with every project.

Focusing on a test, and not on a product

One of the reasons why a test might fail, common especially among people who have just started writing scripts, is not getting to know the software thoroughly before launching automation. The sad trend of novice testers is to write tests with the idea that they should pass and work and that there should be as many of them as possible. More assertions, after all, mean more tests and customer or product owner satisfaction because everything comes out beautifully on the reports.

At some point, there is a risk that by making sure the automation covers as many modules as possible, we forget the creative approach in our testing. We focus more on making the tests pass than testing our project.

What can a tester do?

• Awareness of this risk and proper test planning helps solve this problem.

• It's also a good idea to combine automated testing with manual testing, where we know that automation only covers, for example, smoke tests.

Not knowing a product well enough

It is pretty common that the project documentation, which forms the basis for testing, is not of the best quality. Functionality is not described in sufficient detail, it doesn’t cover all the possible process flows, and the specification doesn’t include data descriptions or exceptions. How to plan and implement testing in such a situation?

Poor analytical documentation doesn’t help in understanding how the application works. A tester must often guess and interpret the functionality according to their understanding. Sometimes it also happens that a tester cannot empathize with a system’s user and can't imagine the purpose of particular functionality.

What can a tester do?

• When planning and preparing test cases and scenarios, rely on the documented specification (if any) or functional requirements.

• Describe the error with a specific requirement or section of the specification.

• You can’t expect a developer to decide how to solve a problem related to an error in the specification. If there is any doubt about the expected functionality, consult the analyst and clarify the issue before reporting the problem to the programmer.

• If you don't know how a function should work - ask. The analyst's job is to create the system's documentation and provide all necessary explanations of the purposes and functioning of individual modules.

Misunderstandings in a team

As controversial as it sounds - a software tester cannot trust anyone completely. Even if you hear that a bug has been fixed and you can send a product to a customer or a developer says everything works just fine, you should still test the software once again. Sometimes, a programmer hasn’t corrected a mistake, although he said otherwise. On the other hand, however, it should be remembered that if you don't know the project sufficiently, developers may be right in saying the famous phrase: "it's not a bug, it's a feature."

What can a tester do?

• Remember that a tester is not someone who "points out bugs." Testing ensures the software's quality by verifying that it conforms to the project. Of course, the world would be more beautiful if a tester would only prove that everything works 100% well, with no rollbacks, but that’s simply impossible.

• Bugs happen even in projects written by the most experienced seniors. Finding a defect and reporting it is not tantamount to attacking a programmer.

• Proper communication with a team is very important when working with experienced programmers and juniors. Bugs are no one's fault, and unnecessary emotions and negative words should be avoided in our reporting and conversation.

Not enough error description

The most laconic description of the error one can encounter is: this button doesn't work. The problem may be evident for a person who reports it since they saw this button on the screen in front of their eyes and could tell the malfunction. A person receiving this notification is no longer so sure about the issue, missing the description of the module, the page in the application with the button’s location, and an explanation of how the button currently functions and how it should be according to the specification.

The case described is one of the most drastic, but it perfectly explains why reporting bugs with a sufficiently detailed description is so important. Lack of sufficient information when reporting bugs is one of the most important problems in the testing process. Not only does it drastically reduce the efficiency of testing and prolong the bug-fixing process, but it also has a clear impact on the deterioration of the atmosphere between the teams of testers and developers.

When reporting a bug, a tester should always provide:

• A brief description, enabling a developer to locate the problem and its scale quickly.

• Initial conditions, e.g., operator data or the system's state necessary to reproduce the error.

• The restoration procedure, i.e., actions performed step by step to obtain the error. It is advisable also give test data because some defects occur only when using specific information.

• The expected result of what the system should do after implementing the above procedure.

• The actual result of the system's performance and why it is incorrect (a reference to documentation, excerpt from the specification, etc.).

• A screenshot allows a programmer to quickly see where the problem is and what it consists of. It is advisable to add screenshots for each procedure step in more complex cases to reproduce the problem.

• Priority determined according to established rules and criteria so that it gives a reliable picture of the scale of the problem.

• The system version allows a developer to reproduce and fix the bug in the corresponding version of the application.

• Data of the reporting person allows to contact the tester in case of doubt

• The reporting person should always check their error after the programmer has fixed it.

Inaccurate superficial tests

A common mistake is to perform tests superficially and inaccurately. This is particularly evident in the case of tests based on functional specifications or other documentation requirements. Sometimes testers assume that they will only test what is documented without digging into the details and looking for gaps or inconsistencies in the description. If documentation doesn’t specify all possible paths of the process, they will check only those described. For example, if the specification does not include input validation but only addresses optimistic scenarios, the tester will check only the positive ones. They won’t test data input of the wrong format, erroneous from a business perspective (e.g., performing transfers of money for an amount greater than available in the test account), or the dependencies between fields.

A good tester:

• Works methodically and systematically, with great caution and skepticism, especially when making assumptions of any kind.

• Acts diplomatically but firmly when necessary to prove a point.

• Has a unique sense of observation and noticing details.**

• Is characterized by excellent verbal and communication skills to inform other project team members about the problems found and unambiguously explain their essence. Even the most thorough and perceptive tester with in-depth knowledge of the application under test will not accomplish much if they cannot communicate his insights and observations.

• Senses situations or features of the application that others may not understand (i.e., programmers) and anticipates problems based on their experience.

• Is curious and tries various combinations of actions and scenarios to see what happens.

Summing up

Testing is not as simple an activity as it may seem at first glance. One of the main goals of QA work is to find bugs and irregularities, striving to improve the system and analytical documentation. However, testing quality decreases significantly when the process is carried out incorrectly or by people with insufficient qualifications and aptitude. This article has outlined a few basic mistakes in the planning and test implementation phase. Analyzing these mistakes and studying the guidelines can help eliminate some fundamental failures in work management and organization, making the testing process more efficient.

Read more

1. Manual vs. Automation testing: Pros and Cons
2. What are flaky tests in Cypress and how to prevent them?
3. How to build a test process in an ongoing project?