Acceptance Test Plan

Overview

Once you have completed your project, you need to demonstrate that it satisfies its requirements. There are several ways to do this. One is to develop an automatic test suite that covers all requirements. Once the test suite passes, the software can be shipped. This approach can be combined with an defect tracking system that ranks the severity of observed defects. In this case, the software can be shipped once all defects above a specific severity level have been fixed. However, creation of automatic test suites is a time-consuming process. Additionally, the test suite requires validation to ensure that is has sufficient coverage of project requirements, and the tests themselves are correct. Unfortunately, there isn't enough time in one quarter to develop your project and create a significant automatic test suite (though modestly scopes test suites are feasible, and could be quite useful).

Another approach is to manually perform an acceptance test. An acceptance test plan document details a series of tests performed to validate the software meets its requirements. An acceptance test is usually held during an official "sell-off" of the software to the customer. Representatives from the customer verify the test executes as specified in the test plan document. Deviations are noted, and result in either a correction to the software, or to the test plan, or both, depending on where the fault lies. After correction, the test is re-run until it passes completely. At this point, the customer accepts delivery of the software (and pays the remaining amount of the development contract costs).

Acceptance Test

During the last week of class, you will perform a sell-off of your project to the TA and the Professor (acting as the customer), using an acceptance test. During this sell-off, your group will execute the test procedure outlined in your acceptance test plan.

The test plan document describes a series of operations to perform on the software. This is a very detailed document. No key gets pressed, no menu item is selected, unless the acceptance test plan says so. Whenever there is a visible change in the state of the software (new window, change in the display, etc.) the test plan document asks the testers to observe the changes, and verify that they are as expected, with the expected behavior described precisely in the test plan. As the test is executed, testers check-off each operation performed, and write down observations made as the software executes.

For this class, the acceptance test is limited to an hour in length. It can be shorter than this, but no longer. Since the goal of the acceptance test is to demonstrate that the software meets its requirements, the hour limitation may mean there isn't enough time to cover all requirements. In this case, it is necessary to prioritize your test coverage, so the most important requirements are covered first.

Since the acceptance test is used by the customer to determine whether to accept delivery of the software, the customer needs to approve the acceptance test plan prior to its execution. So, in addition to receiving a grade on the initial submission of your acceptance test plan, you must revise and resubmit your test plan document until the TA formally approves it. Similarly, if you do not initially pass your acceptance test, your group must revise the software or the test (depending on the source of the observed problem) and then re-run the acceptance test. Failure to pass the acceptance test after two tries will have significant adverse affect on the "acceptance test plan performance" component of your class grade.

Other Verification and Validation Activities

The acceptance test, along with the Software Inspection, comprise the only graded verification and validation (V&V) activities for your project. However, they should not be the only V&V actions you take. A successful acceptance test is usually preceded by significant unit testing and integration testing of the software, as well as a dry-run through the acceptance test itself. This implies that coding is completed far in advance of the acceptance test, so time and attention can be focused on testing. Your group should develop its own V&V strategy to ensure the software meets its requirements, and easily passes the acceptance test plan.

For details about how the acceptance test will be used, see the acceptance test plan execution description.

Acceptance Test Plan Template

Your acceptance test plan document should have the following parts:

• Title page
  
  
• Change log -
  note that once your acceptance test plan has been approved by the TA, any subsequent changes must also be approved by the TA.
  
  
• Table of contents -
  • every test listed, with page numbers
  
  
• Overview -
  • brief description of the purpose of the document (to validate that the project meets its requirements),
  • what order the tests must be run in,
  • reasons for choosing the requirements to test (if you don't have enough time to test everything).
  
  
• Tests -
  A series of tests, with each step described in detail.
  
  Each test must include the following information:
  • Test Number
    
    
  • Brief description of what capabilities are being tested ("this test will verify that the save game functionality works as described in requirements XX, YY, ZZ").
    
    
  • Which requirements are being verified by this test. Your test plan will not be approved without this traceability information.
    
    
  • Test preconditions - addresses concerns surrounding the state of the software prior to executing this test, such as:
    • Does this test assume that previous tests have already been run?
    • Does this test need the software to be in a certain state (e.g., to test game winning behavior, you need to win the game)?
    • Does the software need to be in a special test mode?
    • Does the software need to be running, or do the test instructions involve starting the software?
    • Does any additional software also need to be running (such as a server, or specialized test equipment)? How can we determine that this additional software is up and running?
    
    
  • Test steps. The specific instructions to follow while performing the test. Each test step must:
    • Be individually numbered (i.e., with a document section number)
    • Have an explicitly labeled location where a check mark can be made ("[__]") when the test step has been completed.
    • Give either a detailed instruction ("Go to the File menu, and select Open") or a detailed observation ("Observe that the File... Open dialog box appears, and shows the contents of the user's home directory").
    
    
  • Test postconditions - what operations must be performed to leave the software in the state expected by following tests.
  
  
• Meeting Notes (as usual)
• Time Logs (like always)

Note that if your project involves significant amounts of randomness, you may wish to create an explicit testing mode, where random values are pre-assigned to ensure the behavior of the system is predictable in advance.