Skip to content

Unit Testing – The Big Picture

This is not a crash course of what characters one needs to type in his code editor to produce unit tests. This is fuel the brain requires before attempting such actions.

The subject of Unit Testing is not as simple as one might think. Many of us, developers, go into unit testing based on pressure from clients, peers, colleagues, our heroes and so on. We quickly learn the value of it, and, once the tech setup is done, there is a tendency to forget the big picture, if it was ever learnt. This article will provide a short insight into what is and isn’t unit testing in PHP and in general, and unit test place in the quality assurance realm.

What is testing?

Before diving into unit tests, you should have a look at some deeper theory of what testing actually is, so you don’t make mistakes like one of the most popular PHP frameworks on their website showing integration tests and calling them unit tests. They’re not, Laravel, they’re really really not. I still love you though.

Software testing is defined as “investigation conducted to provide stakeholders with information about the quality of the product” as oppose to “Software testing is developers wasting project money doing nothing important and then asking for extra time and budget because “nothing” can be really expensive”. It also isn’t anything new. Have a look at my compiled brief history of testing:

  • 1822 – Difference engine (Charles Babbage)
  • 1843 – Analytical Engine (Ada Lovelace)
  • 1878 – Edison coins the term “bug”
  • 1957 – Program testing vs debugging (Charles Baker)
  • 1958 – First software test team (Gerald Weinberg)
  • 1968 – Software crisis (Friedrich Bauer)
  • 1970’s – Waterfall model, Relational model, Decomposition, Walkthroughs, Design and code inspections, Quality and metrics, Design patterns idea
  • 1980’s – CRUD analysis, System architecture, AutoTester, V-model, Reliability, Cost of quality, Use cases, OOP design patterns
  • 1990’s – Scrum, usability testing, MoSCoW, heuristic testing, software and test automation

If you are a millennial like me, you could be shocked that there were software test teams way, waaay before you were born. Take a moment. Relax. Deep breaths.

What the history tells us is the type of testing software required to be “good enough” for the stakeholders over time. Here are approximate testing orientation phases:

  • … – 1956 debugging
  • 1957 – 1978 demonstration
  • 1979 – 1982 destruction
  • 1983 – 1987 evaluation
  • 1988 – … prevention

Ergo, the unit testing is required to prevent discrepancies between the design and implementation.

What really is testing?

There are multiple ways to classify software testing, I will mention the most generally accepted ones to better understand the place of unit tests.

Tests can be categorized as: Static VS Dynamic, Boxed (White box, Black Box, Grey Box), and Levels and Types. Each of these 3 approaches classify tests by different criteria.

Static & Dynamic testing

Static testing is what one does without actually executing the code, such as proofreading, verification, code reviews (over the shoulder / pair programming), walkthroughs, inspections etc. Dynamic testing is when the actually code needs to be executed to receive a valid result, for instance, unit tests, integration tests, system tests, acceptance tests etc; It is testing with dynamic data, testing various inputs and outputs.

The Box Approach

According to the box approach, all software tests can be divided into 3 boxes:

White Box tests verify and validate internal structures, units, ignores end-user expected functionality. This could be API testing, fault injection, unit testing, integration testing. The contrasting Black Box testing is more interested in what the software does, not how it does is. This means that tester must have no knowledge on subject, no understanding on how things work under the hood. It’s all about that end user, his experience interacting with the visible surface of the product. Black box testing types include model based testing, use case testing, state transition tables, specification testing and so on. The third type falls sort of in between. Grey Box tests are designed with knowledge of software algorithms and data structures (white box), but executed at user level for experience (black box), such as regression testing or pattern testing.

Now, to screw with you, let me mention that unit testing can also be Black Box testing because you might have knowledge about the single unit you are testing, but not about rest of the system. I still think of it as white-box-only though, and wish you do the same.

Test Levels

Test levels vary, there are usually 4-6 tiers and all of the ones I’ve seen makes sense. The names also change, depending on companies culture you might know Integration tests as Functional tests and System tests as Automated tests and so on. So to keep things simple I will describe testing levels in these 5 tiers:

  1. Unit testing
  2. Integration testing
  3. Component interface testing
  4. System testing
  5. Operational Acceptance testing

Unit testing verifies functionality of a specific section of code, mostly one function at a time. Integration testing verifies interfaces between components, verifies that units combined together form a working system as designed. This is an important point, because large portion of unit tests out there are actually integration tests, and developers assume they are unit. If it involves multiple units – it is the integration between them that is being tested, not the unit itself. Component interface testing verifies data passed between various units. For example, get data from unit one – verify – pass to unit 2 – verify outcome. System testing is end to end testing to verify all requirements are met. Operational Acceptance testing is conducted to verify operational readiness. It’s non-functional, it tests if services are healthy, if any sub-system does not damage environment and other services.

Testing types

Every type of testing can also have a type regardless of its level. There are more than 20 accepted software test types. The most widely spread are:

  • Regression testing
  • Acceptance testing
  • Smoke testing
  • UAT
  • Destructive testing
  • Performance testing
  • Continuous testing
  • Usability testing
  • Security testing

Types are relative to the purpose the test is executed for. The ones in bold are PHP unit tests. Ones in italic – sort of / sometimes / get off by back. If one really wanted to, he could put unit testing under each and every of those terms. Really, though, main type of unit tests are regression tests – to verify if all units of the system still execute validly after code changes have been introduced.

So now you know that unit tests are dynamic tests, white-box tests, unit level tests, they are regression tests, but can have multiple test types associated with them. But what are unit tests, really?

What is Unit Testing?

V-model graphically represents previously mentioned levels, types, and their purpose in software development life-cycle.

v-model

When detailed product requirements are verified and set, and code writing has started, the first line of defense against any discrepancies are unit tests. This is why companies who understand what they are doing are pushing developers for mandatory unit tests, or even TDD, because it will cost so much less to fix bugs here than in later stages.

And it makes sense. There are plenty of benefits of having unit tests. They:

  • Isolate each part of the program and validate its correctness
  • Help to find problems early
  • Force developers to think in inputs, outputs, and error conditions
  • Shape code in testable form, facilitate future refactoring
  • Simplify integration of working(!) units
  • Partially substitute technical documentation
  • Enforce interface separation from implementation
  • Prove that the code of the unit works as expected (at least mathematically)
  • Can be used as low level regression test suite
  • Demonstrate progress of unfinished system integration
  • Reduces cost of fixing bugs (even more with TDD)
  • Creates better application designs by defining unit responsibilities
  • If you can test it – you can join it to the rest of your system
  • Unit testing is FUN!

There are, however, a few limitations, that you probably imagined reading the list of wonders above. Unit testing:

  • Does not catch integration errors
  • Every Boolean statement requires at least 2 tests, and it grows quickly
  • Unit tests are as buggy as the code they test
  • Binding tests to couple of specific frameworks or libraries can limit the testing workflow
  • Most tests are written after development is finished… which is sad. Go TDD!
  • Possible that after small refactoring system is working as before, but tests – failing
  • Increases development costs
  • Human error – commenting out broken tests
  • Human error – adding workarounds in code just for unit tests to pass

The last one pisses me off more than anything. In (almost) every project I pick up there are lines right inside the source code of production application like “If this is unit test, load stubbed SQLite db, otherwise load other db” or “if this is unit test, don’t send email, otherwise do” etc. If the application design is bad, don’t pretend you can fix crappy software with good tests, it won’t make it better.

There have been plenty of discussions with my colleagues and client developers on what good unit test is. The bottom line – Good Unit test is:

  • Fast
  • Automated
  • In full control of all of it’s dependencies
  • Reliable – runnable in any order, independent from other tests
  • Runnable in memory only (no real database interactions, filesystem r/w)
  • Always returns the same result
  • Readable and maintainable
  • Doesn’t test SUT (system under test) configuration
  • Clearly defined SINGLE GOAL
  • Well named (and clearly enough to avoid debugging to know what failed)

To the ones smirking about “automated” – I don’t mean PHPUnit or JUnit integration in CI pipelines. I mean – if you change your code, save it, and do not know whether or not modified units still confirm to their tests – they’re not automated, and they should be. File watchers FTW.

What to unit-test?

In a healthy system unit tests should be written for:

  • Units – smallest isolated parts of system that perform single task (a function, a method, a class)
  • Public methods
  • Protected methods, but only in rare cases, and when nobody’s looking
  • Bugs & Bug fixes

The definition of what is a unit comes from developer, who wrote the code. In my case in PHP it is almost always a class method or a function, because that is the smallest part of the software that has meaning by itself. Few times I have also seen developers treating array of mini one-method classes as one unit, and it also makes sense if the minimum viable meaning requires multiple objects to exist.

So define whatever is a unit for you. Or test methods one by one and make life easier for the next guy that picks up the code.

If you don’t do unit testing, I suggest getting into it after your next bug comes along. Verify which method it comes from, write a failing test with proper args and result, fix the bug, re-run test. If unit test passes, you are sure that was the last time you needed to fix that bug (with your defined input scenarios).

This way the unit testing process becomes easier to grasp. Look at each method separately, it has an input and an output. Data Providers can help you define inputs and outputs for all scenarios that come in your mind, making sure whatever happens – you know what to expect.

What NOT to test

Defining what not to test is slightly trickier. Here I have tried to compile a list of items that should not be unit tested:

  • Functionality outside units scope(!)
  • Units integration with other units(!)
  • Non-isolated behavior (unmockable dependencies, real DB, network)
  • Private, Protected methods
  • Static methods
  • External libraries
  • Your framework

I am sure that you should not unit test any of the above, apart from static methods. I like to use the argument that being static basically means procedural, in many cases global. If static method calls another static method there is no way to override that dependency. Which means you are no longer testing in isolation. Which means this isn’t a unit test anymore. On the other hand, it is a piece of code able to live by itself, it has a meaning, and it should be tested to make sure whatever part of that sorry-ass system is calling it doesn’t break. So I think it is fine to test static methods if you can make sure that no other test can change the output of this test, and that language or framework allows doing that natively.

How to write a Unit test?

  • Write unit testable code, then test it
  • Write unit testable code, then test it
  • Write unit testable code, then test it

In case “then test it” is not enough, laracasts.com has very good videos on the topic for PHP, worth the time. Plenty of other sites for JS/Java/YouNameIt. There is no point for me to explain how I unit test code, because the tools for this methodology change quite rapidly, and, by the time you read this, I might have switched from PHPUnit to Kahlan. Or not. Who knows.

But to the first one – How to write unit testable code? – the answer is a bit easier and does not seem to change over time much:

  • SOLID
  • DRY
  • No new keyword in constructor
  • No loops in constructor (and switches, if clauses)
  • No static methods, parameters, classes
  • No setup() methods – Object must be fully initialized after construct
  • No Singletons (Global state) and other untestable anti-patterns
  • No God objects
  • No mixed concern classes
  • No hidden dependencies

Now – knowing what unit tests are, what they aren’t, what to test, and what not, and where is unit test place in software development life-cycle should make it easier to implement them. What’s left is to find a framework or library you like and stick with it. When in doubt – go with de-facto standard for framework/language.

In conclusion – unit tests are very important to both developers and businesses, they should be written, and there are best practices that help cover your units with tests easily, mostly by preparing units themselves. But none of that practical application makes sense without the testing theory I have provided above – one needs to differentiate unit tests from all other types of tests, and, once the difference is clear, it makes it so much more easier to write them.

I hope you enjoyed reading this as much as enjoyed writing it, and I hope that now you have better understanding of what exactly unit tests are, and why they are the most important type of software testing out there. Apart from integration tests.

Thank you for reading! 🙂

7 Comments

  1. When you say:

    “Non-isolated behavior (unmockable dependencies, real DB, network)”

    You want want would say that don’t should to test them thinking in a unit testing or not test them of no way ..

    Because the real DB .. could be testable with mocks..

    • Gundars Gundars

      When writing test for a single unit – never test the real Database, mocking is fine as long as no real connections are made, and the purpose is to assess that unit interacts with DB layer correctly.

  2. Wow, what a great write up! Earlier this year, I also published a post specific to unit testing:
    https://tfrommen.de/an-introduction-to-unit-testing-for-wordpress/

    I spent a little more time on the core concepts and included a few code as well as test examples, but didn’t go into that much detail regarding other test levels, or individual test types and techniques. I therefore think the two posts are good additions to each other, and I will definitely mention yours in the References & Further Reading section of my posts and talks/slides.

    Cheers,
    Thorsten

    • Gundars Gundars

      Hi Thorsten, looks good. Some of the things I just mention you explain pretty well in detail, good read.

  3. Thank you so much for this article (and folks from Jetbrains’ PHP Annotated Monthly put this read on it).

    It is really tough to find any good resources about software testing, and most texts, demos and so on we find out are way more damaging to productivity and reliability than good.

    • Gundars Gundars

      Thank you for reading, I hope you found it not too boring 🙂 But agree, some articles focus a bit too much on the code part, leaving out the important decisions, I tried not to.

Leave a Reply

Your email address will not be published. Required fields are marked *