Testing is a very important part of coding. We all test our code in one way or another – otherwise, how would we even know that code works?
When you first start coding, you simply run the code over and over until you get the expected results. This seems to work… until you get points deducted on a lab assignment because there was a small bug that you didn’t find!
:( And that’s unfortunate.
Surely, there must be a better solution to this problem. How do Facebook and Google test their code? Do you think they simply have people clicking random things to see if the website still works?
Of course not.
They have Quality Assurance engineers whose entire job is simply to test code. But how do they test code?
There are many different kinds of tests, and the most basic and essential is called unit testing.
Unit testing is a type of testing where you test a single function with various different inputs to make sure it behaves as expected.
In the first bonus section, I showed you how to run your code outside of Eclipse, on the command line. Here, we’re going to discuss how to set up the directories in a formal manner.
A very common way to set up project directories is in the following manner:
ProjectName/ | -- README | -- setup.py | -- bin/ | | -- project | -- project/ | | -- test/ | | | -- __init__.py | | | -- test_main.py | | -- __init__.py | | -- main.py
Wow, there’s a lot going on here, and most of it you won’t need in your projects. So let’s break it down.
README is a very common practice in coding projects. It tells the readers of the code anything they should know – set up instructions, software requirements (does it not work on Windows? What version of Python should they use?), and any common problems and their solutions.
The next is
setup.py, which is very common for scripting with Python. The projects in CS1 will not need to use this, so feel free to ignore this section.
bin/ folder contains executables that your project might need - already compiled and pre-made scripts ready to be run. Again, your projects in this class won’t use them, so feel free to ignore this folder.
project folder is where all of your code will reside. This is where all of your source code will be.
project folder is another folder called
test. Here is where we will place all of our unit testing code.
So let’s take an example Lab1 directory. Here, we’ll say our
Pong/ | -- README | -- pong/ | | -- test/ | | | -- __init__.py | | | -- test_pong.py | | -- __init__.py | | -- pong.py | | -- cs1lib.py
Great. Of course, in reality, you may need more than one file to complete the pong lab, but in this example, pretend that everythign is contained within
pong.py. Similarly, you probabily will need multiple test files, but for now, imagine all the tests are within the
From now on, you should be setting up your folders like this.
There are some mysteries, however. For instance, what on earth is the
__init__.py file? Here’s what the official Python documentation says.
The init.py files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur later on the module search path. In the simplest case, init.py can just be an empty file, but it can also execute initialization code for the package or set the all variable, described later.
So what does this mean? It means that if we want to import something from a different directory, then that directory must contain a
__init__.py file. That is two underscores followed by “init”, followed by two underscores. They can be empty files in your project directories.
So what’s a unit test? A unit test is meant to test a single function. We give this function all sorts of inputs, and for each of the inputs we give it, the function must give us the correct answer.
Imagine we have a single function in our
pong.py file – a function that tells us whether the paddle is not out of bounds.
PADDLE_X = 40 PADDLE_Y = 80 WINDOW_X = 300 WINDOW_Y = 300 def in_bounds(paddle_x, paddle_y): return paddle_x >= 0 and paddle_x <= WINDOW_X - PADDLE_X \ and paddle_y >= 0 and paddle_y <= WINDOW_Y - PADDLE_Y
So we want to test this
out_bounds function to make sure it really will do as it claims. How do we do this? The Python unit testing framework we will be working with is called Nose, and it’s widely used.
First, let’s edit the
from pong.pong import in_bounds def test_out_bounds(): assert in_bounds(0, 0)
Okay, so this is a fairly simple test. If we have a paddle where the lower left corner is at the origin, it should be in bounds. Couple things to notice though.
pongrefers to the folder name – our folder is called
pong. The second
pongrefers to the file name
pong.py. Thus, we are importing the function
in_boundsfrom the folder and then from the file.
test_out_bounds. All test functions must begin with the word
test_. Usually, what follows is the name of the function you are testing, but this does not have to be the case.
test_, and they usually end with the name of the file you are testing, but this does not have to be the case.
assertis used in every test. If what follows the
assertstatement evaluates to
True, then the test is passed; otherwise, the test has been failed.
So how do we run this? First, you have to install Nose. On a Mac, this is done using
pip install nose in your terminal. See the Nose documentation page (linked above) on how to install if you run into any problems.
Now, go to the
pong folder, and type
nosetests. That’s it! Nose is smart enough to find out where your tests are and run the tests! In fact, it even compiles the results together. Here is what you should get when you run the command.
. ---------------------------------------------------------------------- Ran 1 test in 0.005s OK
So what does all of this mean? Let’s look at the first line. The first line has one character per test. In this case, we of course only had one test, so there’s just a single character. A
. means the test was passed. If you had failed the test, a different character would show up.
Since there were no failures, it didn’t output very much. Then comes the summary – how many test were run, how long it took. Ultimately, if you passed all the tests, you should see the
OK at the very bottom.
Now let’s see what happens if we fail a test. Edit your
test_pong.py to the following.
from pong.pong import in_bounds def test_out_bounds(): assert in_bounds(0, 0) assert in_bounds(400, 400)
Now the second
assert is clearly
False, so we will fail. After all, a paddle should not be
100 pixels outside of the screen. Is this a problem with our code? Of course not. We are just putting a false test to see what happens when we fail.
nosetests again and see what output we get.
F ====================================================================== FAIL: pong.test.test_pong.test_out_bounds ---------------------------------------------------------------------- Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/Users/chanchan/Desktop/Pong/pong/test/test_pong.py", line 6, in test_out_bounds assert in_bounds(400, 400) AssertionError ---------------------------------------------------------------------- Ran 1 test in 0.001s FAILED (failures=1)
Now, notice that instead of the
. we saw last time, we get an
F means the test was failed. It tells you what line you failed in and why. Great! Lastly, the summary has also been updated with the number of failures.
You may be thinking why there is still only one test even though we have two assert statements. That is because teh number of tests is equal to the number of functions within the
test_pong.py, and not the number of assert statements.
Now you know the basics of unit testing with Python. Unit testing is a great way to make sure your code works, but even more than that, it’s used to make sure the code keeps working.
For example, let’s say you write some code, make sure it works, and then later on you have to make some changes. Wouldn’t it stink to have to remember every way in which the code can break, and have to manually re-test all of them? Of course it would.
With unit testing, however, you have a standard array of tests you can just throw at your code, and whatever changes you made, so long as it passes these tests, you’re good to go!
The other reason to use unit testing is because it actually forces you to have better design of your code. How you ultimately end up testing your code determines to a large degree what kinds of functions you’ll write and how you break down the problem. This will force you to think about design patterns, because a badly designed code base is nearly impossible to test!
The last thing to keep in mind is all the ways in which the code could go wrong. Let’s take the
in_bounds function for example. What are all the types of cases we can think of?
There are four cases, at least, that we can think of right off the bat! A good unit tester will test against all kinds of possible inputs to be very sure. One tell-tale sign of bad unit testing is when you have bugs despite passing the unit tests.
Often times it means your testing wasn’t rigorous enough.
Of course, should you choose to submit your unit tests along with your labs, you will of course receive extra credit.
Good luck, and happy testing!