Unit Testing

Testing is a very important part of coding. We all test our code in one way or another – otherwise, how would we even know that code works?

When you first start coding, you simply run the code over and over until you get the expected results. This seems to work… until you get points deducted on a lab assignment because there was a small bug that you didn’t find! :( And that’s unfortunate.

Surely, there must be a better solution to this problem. How do Facebook and Google test their code? Do you think they simply have people clicking random things to see if the website still works?

Of course not.

They have Quality Assurance engineers whose entire job is simply to test code. But how do they test code?

There are many different kinds of tests, and the most basic and essential is called unit testing.

Unit testing is a type of testing where you test a single function with various different inputs to make sure it behaves as expected.

Folder Structure

In the first bonus section, I showed you how to run your code outside of Eclipse, on the command line. Here, we’re going to discuss how to set up the directories in a formal manner.

A very common way to set up project directories is in the following manner:

ProjectName/
| -- README
| -- setup.py
| -- bin/
|    | -- project
| -- project/
|    | -- test/
|    |     | -- __init__.py
|    |     | -- test_main.py
|    | -- __init__.py
|    | -- main.py

Wow, there’s a lot going on here, and most of it you won’t need in your projects. So let’s break it down.

So let’s take an example Lab1 directory. Here, we’ll say our ProjectName is Pong.

Pong/
| -- README
| -- pong/
|    | -- test/
|    |    | -- __init__.py
|    |    | -- test_pong.py
|    | -- __init__.py
|    | -- pong.py
|    | -- cs1lib.py

Great. Of course, in reality, you may need more than one file to complete the pong lab, but in this example, pretend that everythign is contained within pong.py. Similarly, you probabily will need multiple test files, but for now, imagine all the tests are within the test_pong.py file.

From now on, you should be setting up your folders like this.

There are some mysteries, however. For instance, what on earth is the __init__.py file? Here’s what the official Python documentation says.

The init.py files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur later on the module search path. In the simplest case, init.py can just be an empty file, but it can also execute initialization code for the package or set the all variable, described later.

So what does this mean? It means that if we want to import something from a different directory, then that directory must contain a __init__.py file. That is two underscores followed by “init”, followed by two underscores. They can be empty files in your project directories.

Nose

So what’s a unit test? A unit test is meant to test a single function. We give this function all sorts of inputs, and for each of the inputs we give it, the function must give us the correct answer.

Imagine we have a single function in our pong.py file – a function that tells us whether the paddle is not out of bounds.

PADDLE_X = 40
PADDLE_Y = 80

WINDOW_X = 300
WINDOW_Y = 300

def in_bounds(paddle_x, paddle_y):
    return paddle_x >= 0 and paddle_x <= WINDOW_X - PADDLE_X \
           and paddle_y >= 0 and paddle_y <= WINDOW_Y - PADDLE_Y

So we want to test this out_bounds function to make sure it really will do as it claims. How do we do this? The Python unit testing framework we will be working with is called Nose, and it’s widely used.

First, let’s edit the test_pong.py file.

from pong.pong import in_bounds

def test_out_bounds():
    assert in_bounds(0, 0)

Okay, so this is a fairly simple test. If we have a paddle where the lower left corner is at the origin, it should be in bounds. Couple things to notice though.

  1. Notice the import statement. The first pong refers to the folder name – our folder is called pong. The second pong refers to the file name pong.py. Thus, we are importing the function in_bounds from the folder and then from the file.
  2. Notice the name of the function – test_out_bounds. All test functions must begin with the word test_. Usually, what follows is the name of the function you are testing, but this does not have to be the case.
  3. Notice the name of the file. Test files have to begin with test_, and they usually end with the name of the file you are testing, but this does not have to be the case.
  4. An assert is used in every test. If what follows the assert statement evaluates to True, then the test is passed; otherwise, the test has been failed.

So how do we run this? First, you have to install Nose. On a Mac, this is done using pip install nose in your terminal. See the Nose documentation page (linked above) on how to install if you run into any problems.

Now, go to the pong folder, and type nosetests. That’s it! Nose is smart enough to find out where your tests are and run the tests! In fact, it even compiles the results together. Here is what you should get when you run the command.

.
----------------------------------------------------------------------
Ran 1 test in 0.005s

OK

So what does all of this mean? Let’s look at the first line. The first line has one character per test. In this case, we of course only had one test, so there’s just a single character. A . means the test was passed. If you had failed the test, a different character would show up.

Since there were no failures, it didn’t output very much. Then comes the summary – how many test were run, how long it took. Ultimately, if you passed all the tests, you should see the OK at the very bottom.

Now let’s see what happens if we fail a test. Edit your test_pong.py to the following.

from pong.pong import in_bounds

def test_out_bounds():
    assert in_bounds(0, 0)
    assert in_bounds(400, 400)

Now the second assert is clearly False, so we will fail. After all, a paddle should not be 100 pixels outside of the screen. Is this a problem with our code? Of course not. We are just putting a false test to see what happens when we fail.

Let’s run nosetests again and see what output we get.

F
======================================================================
FAIL: pong.test.test_pong.test_out_bounds
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/Users/chanchan/Desktop/Pong/pong/test/test_pong.py", line 6, in test_out_bounds
    assert in_bounds(400, 400)
AssertionError

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)

Now, notice that instead of the . we saw last time, we get an F. An F means the test was failed. It tells you what line you failed in and why. Great! Lastly, the summary has also been updated with the number of failures.

You may be thinking why there is still only one test even though we have two assert statements. That is because teh number of tests is equal to the number of functions within the test_pong.py, and not the number of assert statements.

Good Principles

Now you know the basics of unit testing with Python. Unit testing is a great way to make sure your code works, but even more than that, it’s used to make sure the code keeps working.

For example, let’s say you write some code, make sure it works, and then later on you have to make some changes. Wouldn’t it stink to have to remember every way in which the code can break, and have to manually re-test all of them? Of course it would.

With unit testing, however, you have a standard array of tests you can just throw at your code, and whatever changes you made, so long as it passes these tests, you’re good to go!

The other reason to use unit testing is because it actually forces you to have better design of your code. How you ultimately end up testing your code determines to a large degree what kinds of functions you’ll write and how you break down the problem. This will force you to think about design patterns, because a badly designed code base is nearly impossible to test!

The last thing to keep in mind is all the ways in which the code could go wrong. Let’s take the in_bounds function for example. What are all the types of cases we can think of?

There are four cases, at least, that we can think of right off the bat! A good unit tester will test against all kinds of possible inputs to be very sure. One tell-tale sign of bad unit testing is when you have bugs despite passing the unit tests.

Often times it means your testing wasn’t rigorous enough.

Of course, should you choose to submit your unit tests along with your labs, you will of course receive extra credit.

Good luck, and happy testing!