~stolowski/unity-2d/wallpaper-kludge-removal

=======================
User Acceptance Testing
=======================

Outline
-------
Repeatedly and reliably reproducing the inputs a user and comparing the outputs 
to known good values is an excellent way to maintain the existing functionality 
of a piece of software. However manually performing these tests is tiresome and 
prone to error.

Automation is the key! As we use Qt, we are spoiled with a tool called 
"Testability" which is designed to automate this testing process. We make use of
this tool for testing Unity 2D.


Testability Introduction
------------------------
Testability is a UX testing framework, whose core feature for us is that it 
allows inspection of the tree of QObjects in a Qt application to read properties
(write to some) and call signals, as well as faking mouse/keyboard/gesture 
inputs, grabbing visual outputs, and measuring both CPU and graphical 
performance.

Testability is comprised of the following parts:

On the System Under Test (SUT)
- A plugin that the Qt-based software to be tested loads - usually with the 
  “-testability” switch.
- A server app that allows inspection and interaction with the SUT [qttasserver]

On the Host machine (can be same as the SUT)
- Ruby libraries to act as client for qttasserver, so that testing is scriptable 
  [testability-driver & testability-driver-qt-sut-plugin]
- GUI to allow easy inspection of SUT and writing tests [tdriver-visualizer],  
  but not needed for running tests.

For all tests, the SUT is assumed to *not* be the Host machine.


Testability installation instructions
-------------------------------------
Add this repo to your apt sources:

$ sudo add-apt-repository ppa:gerboland/testability
$ sudo apt-get update

Testability is comprised of the following parts:
- testability-qttas
    The server binary and plugin library for the SUT.
- ruby-testability-driver
    Basic client interface for qttasserver
- ruby-testability-driver-qt-sut-plugin
    [Depends on ruby-testability-driver]
    Plugin for testability-driver to add QObject inspection support
- testability-visualizer
    User interface - not needed for testing


To get started, install all these and additional dependencies with:

$ sudo apt-get install rubygems testability-qttas \
  ruby-testability-driver-qt-sut-plugin testability-visualizer \
  librmagick-ruby1.8 xdotool xsel x11-utils gir1.2-unity-5.0 python-gi

Note: gir1.2-unity-5.0 and python-gi are required by fake_progress.py.

You need to configure Testability for your usage. Assuming you’ll be testing on 
your host machine, this involves editing /etc/tdriver/tdriver_parameters.xml to 
just contain:

<parameters>
   <sut id="sut_qt" template="qt">
       <!-- use default values -->
       <parameter name="qttas_server_ip" value="127.0.0.1" />
   </sut>

   <!-- overload default behaviours parameter (see generic.xml in defaults folder) -->
   <parameter name="behaviours" value="behaviours.xml" />
</parameters>

Also to get log output from these processes, you should create the directory 
/logs/testability [FIXME]


Running Tests
-------------
Before any test are run, you must start the qttasserver on the SUT:
$ qttasserver&

To run tests on this source tree, make sure you run cmake & compile. Otherwise 
tests will be run on the installed applications.

You run the entire Unity 2D test suite by executing the run-tests.rb script:
$ cd tests
$ ruby run-tests.rb

This script enters specified subdirectories and runs the tests contained in any 
ruby script with particular format.

You can run individual test suites (one per file) by entering the directory and 
calling:
$ cd launcher
$ ruby autohide_show_tests.rb

Single test cases ("Position with Empty Desktop") can be run with
$ ruby autohide_shot_tests.rb --name "test_Position_with_Empty_Desktop"

Automatically generated reports on Test successes/fails/errors can be generated by
running any ruby script with the "-report" switch. These reports are in Junit XML
and HTML format.


Test Suite Syntax
-----------------
Please take a look at this example:



require '../run-tests.rb' unless $INIT_COMPLETED #include necessary libs, etc

################################# Test Suite ###################################
context "Test Suite summary" do
  # Run once at the beginning of this test suite
  startup do
  end
  
  # Run once at the end of this test suite
  shutdown do
  end

  # Run before each test case begins
  setup do
    # Execute the application 
    @app = $SUT.run( :name => "/absolute/path/to/application", 
    		         :arguments => "-testability", 
    		         :sleeptime => 2 )
  end

  # Run after each test case completes
  teardown do
    #@app.close        
    #Need to kill Launcher as it does not shutdown when politely asked
    $SUT.execute_shell_command "pkill -nf unity-2d-launcher"
  end

  ##############################################################################
  # Test cases

  test "Short description of first test case" do
    verify_equal( 66, TIMEOUT, 'These two values are not equal'){
      @app.Unity2dPanel()['width']).to_i 
    }

    verify_equal( 'true', TIMEOUT, 'This quantity is not true'){
      @app.Unity2dPanel()['x_absolute']
    }
  end

  test "Another test case description" do
  end
end


Using Testability Visualizer
----------------------------
First off, ensure you're running the "qttasserver"

Run
$ tdriver_visualizer

In the menu bar, open “Applications” and select “Start New Application” and 
enter the absolute path to the Qt binary you want to inspect. The application 
will appear, but also will the Visualizer, with a preview of the application in 
the left pane. In the center is a tree of QObjects in this application, and on 
the right you can inspect the properties, methods and signals of each QObject as
you drill into the tree.

Note you should go to “View” -> “Docks and Toolbars” -> “Code Editor” to get a 
text editor. See that you can right-click QObjects and you get the option to 
paste a QObject path from the root - saves time!

Hit F9 to execute the test.