Unit Testing and System Testing? Are these the best methods we have?

 

In this post we will discuss some of the tests related concepts and guidelines. Some of the ideas here are contradicting with school based guidelines, but base on my experience these are usually the best method to ensure quality. I will specify in the post below whenever the guidelines contradicts with the common known testing methods.

Why do we need testing?

Delivering a product is a complicated process!

We need to go through the process of the following steps:

design, implementation, testing,  delivery

If we use Agile development methods, we'll go through these steps over and over again in each iteration.

Each of these steps could have mistakes and bugs. 

If we have no proper testing, the customer will suffer from our mistakes after the delivery.

We want to find the problems in the prior steps rather than in the later steps, since the cost of a problem in a prior step would affect all the following steps, and the fixing price would be much higher.

The testing step can include automatic testing and manual testing.

Both of these can be either manual or automatic.

Guess which one is better? ...

What is the difference between unit testing and system testing?

By school methodology the unit testing would test a single unit of code. It woill have zero dependencies with any entities external to the unit. These external entities include anything from network access, database access, files access, and up to other code unit access. The unit testing should mock any of these external items access with stubs, enabling the independent testing of the unit.

The system testing includes testing the system as a whole, even as a black box. These tests should simulate full scenario (of a user story) from it beginning to the final result. We will not use any stubs here, so we will need a full running system, including the database, persistence entities, network, and compute resources.

What are the pros and cons for unit testing vs. system testing?

Unit testing is created by the developer when implementing the specific unit of code. It requires development resources not only for implementing the code unit, but also to implement the unit test code, and to make the code unit adjustable behaviour by using stubs for any external APIs. This stub tailor work also complicates the code unit itself, as it should perprared to work with APIs (interfaces) for any external access. However, the benefits are a clear independent testing of a code unit. These tests are usually very fast due to the zero dependencies.

A major disadvantage in unit testing, is that we lake the ability to check a real integration of the code unit with any external entity, hence we can only write stubs that we think that simulate the real input/output to the code unit, and the expected flow of of code unit invocations.

System testing is mostly handled by dedicated automation team. It also requires development resources, but the costs of these resources is sometime lower. The system testing does not assume anything about the APIs between the code units, hence we get a really important advantage. However, the are problems with the system testing. It is developed by another team, so we need more knowledge transfer for any new feature. Each test must be run on a full system, hence it requires compute resources dedicated for each test, and we usually do not run tests in parallel on the system. We also suffer from the time limits. Running the tests on a real system is very very slow compared to unit testing. Not just that, think how can we check something that simulates analyzing data of last month? Do we run the tests for a full month?

What we can do? Project Testing!

Ok, this is where I diverge from the school methodology. It is up to you to think about it and decide if I am right. I can only testify from my years of experience that this had proven to be the best method I have ever used. It had some disadnatages, but the gain is high.

We want to have the benefits of the system testing AND the benefits of the unit testing. These are our goals:

• Testing should be fast
• Testing should not consume the full system compute resources
• Testing should not complicate the code
• Testing should check the real integration between code units
• Testing should not require additional team effort

How do we create project testing?

Project testing uses the same framework as the unit testing, but the difference is in its scope. We treat the entire project code as our single code unit. We still create mocks for external APIs: databases, files, network access, but any integration between the variuos code units is run as it run in the real system.

For that we will need to have all the code accessible to the project testing code. Either we use a mono repo, or we clone the git repositories to sub folders of the project testing code. 

The project testing code can check both internal processes (similar to the unit testing) that not a direct result of a user story scenario, and external processes (similar to the system testing) that simulate end-to-end full user story actions.

The goodies in this method are clear:

Do you want to move the time a month ahead? No problem, use a stub for the current time API.

Do you want to run tests in parallel? No problem, every test can run in its own process, and that all the compute resource it will use. No external entities such as databases are required.

Do we complicate the code? Much less than unit testing. Only external APIs are mocked, whose amount is usually much less than external APIs.

And we've done real code units integration tests, and no additional team effort was required.

Final Note

The project testing does not mean we need to stop using unit testing and system testing. Both of these are important, but we can invest more testing time resources in the project testing, and use less of the other methods.