GNOME desktop testing automation and how to use Mago
Ara Pulido reviews FOSS desktop application testing using a new API suite called Mago. Read on for an introduction to get you started in applying it to your open source desktop application.
Testing the GNOME Desktop
Growing complexity of software and, therefore, the amount of tests required before launching an application makes it necessary to automate some of those tests. Unit tests are more and more common in FOSS code but, sometimes, it is also useful to automate tests that interrogate the human interface.
Desktop testing automation refers to those tests that automatically test the GUI of an application, mimicking the mouse and keyboard movements that a human user normally performs. If you look for software testing automation in your favorite search engine, you will find a lot of search results about testing automation of web applications, lots of proprietary software to test proprietary desktop software, lots of FOSS to test web applications and some blogs about testing proprietary desktop software. What about testing desktop FOSS? What are the available solutions?
GNOME Accessibility & Desktop Testing Automation
In GNOME there are several frameworks, but they are all based in the accessibility information. The ATK layer in GNOME was originally written to be able to use GNOME with assisting technologies such as screen readers or Braille terminals. ATK describes a set of interfaces that applications need to implement in order to be accessible. It turns out, however, that the information exposed by this layer is really useful for automated desktop testing. Using this information, GNOME desktop testing frameworks can know, at anytime, which applications are running and which tasks can be performed.
Accerciser is one of the AT browsers to check accessible applications.
Any of these frameworks get the accessibility information from the AT-SPI layer, a communication layer based on CORBA. This communication layer is now being ported to D-Bus to make it work in the new GNOME 3.0.
Mago, A Desktop Testing Initiative
When I started with GNOME desktop testing automation there were some things I missed in the landscape of testing automation:
- A common repository: It is better to have a common repository of tests that everybody can benefit from.
- A defined process and better maintenance: One of the key issues of desktop testing automation is that tests are written, but then they are not maintained. A defined process will make maintenance easier for the original writer, or for someone else.
- Consistent run and reporting: Different tests written by different people should be run the same way and should get the same report layout.
Dogtail, Strongwind and LDTP are the three main available frameworks for desktop testing automation in GNOME. These frameworks expose a very complete API to control the applications: buttons, labels, menu, etc. Dogtail and Strongwind are both written in Python using the pyatspi library. LDTP, on the contrary, has its core written in C using the cspi layer which is now deprecated. LDTP2, however, is completely rewritten in Python, is an ongoing project, and a stable release is expected later this year.
So, is Mago one of these frameworks? The answer is no, Mago is not one of these frameworks. In fact, I like to call Mago an initiative, rather than a framework. Mago tries to solve the above issues by sitting on top of one of these frameworks, specifically LDTP.
The main reason why Mago is based on LDTP and not other frameworks is that LDTP is better maintained than the other two frameworks. The main problem of LDTP is that it is based on a library that is now deprecated. Fortunately, the LDTP2 API will be compatible with its predecessor.
How does Mago try to solve these problems?
The first thing that you will see when branching Mago is a folder structure to store tests. Those folders are named by application and you can easily browse the available tests for each of them.
Also, Mago recommends a way to write tests. Although Mago runs Python code and accepts any LDTP command, the recommendation is that each application needs an application wrapper and that each test suite needs to be a subclass of TestSuite, implementing the setup, teardown and cleanup methods:
- setup() is called once when the test suite starts running. It includes the operations needed in the application to leave it in a state where the test cases work.
- cleanup() is called between test cases in the same test suite. It includes the operations needed to leave the application in the same state as when the test suite started.
- teardown() is called once when the test suite ends. In includes the operations to leave the system in the same state as before it ran the suite (killing processes, closing applications, etc).
In most of the cases, you need to create test suites for a single application with little or no interaction with other applications. For those cases, in the setup method you might desire to open the application, while in the teardown method you might desire to close it. Mago already includes a TestSuite child called SingleApplicationTestSuite for these cases.
An instance of SingleApplicationTestSuite is always associated with an application
Lastly, Mago includes its own test harness and reporting system, that allows tests to be run consistently and get reports in both XML and HTML.
Let’s see how Mago can help you write your desktop tests using a simple example for gedit.
As you may guess, in the gedit folder, there are gedit test cases. Each test suite is defined in a XML file. In this file, you will define three things: the python class that contains the TestSuite child class, test cases and their corresponding python methods along with some arguments to pass to the test case methods.
<?xml version="1.0"?> <suite name="gedit chains"> <class>gedit_chains.gEditChain</class> <description> Tests which verify gedit's save file functionality. </description> <case name="Unicode Tests"> <method>testChain</method> <description>Test Unicode text saving.</description> <args> <oracle>data/utf8.txt</oracle> <chain>This is a japanese string: 広告掲載 - ビジネス</chain> </args> </case> <case name="ASCII Test"> <method>testChain</method> <description>Test ASCII text saving.</description> <args> <oracle>data/ascii.txt</oracle> <chain>This is a very basic string!</chain> </args> </case> </suite>
The actual code is where the magic happens. As we already have the gedit application in the Mago API and as we already have separated the data from code (with the XML/Python combination), the script is kept simple and easy to reuse and maintain:
import os from time import time, gmtime, strftime
from mago.test_suite.gnome import GEditTestSuite from mago.check import FileComparison, FAIL
def testChain(self, oracle=None, chain=None): test_file = strftime( "/tmp/" + "%Y%m%d_%H%M%S" + ".txt", gmtime((time()))) self.application.write_text(chain) self.application.save(test_file) # oracle file path is assumed to be relative # to test case code oracle = os.path.join(os.path.dirname(<i>file</i>), oracle) testcheck = FileComparison(oracle, test_file) if testcheck.perform_test() == FAIL: raise AssertionError, "Files differ"
If you run that test suite with mago, you will see this sequence:
- gedit is opened (setup)
- The first test case is run
- gedit closes the document and starts a new one (cleanup)
- gedit is opened (setup)
- The second test case is run
- gdit is closed (teardown)
The opening, clean up and closing of the application is done for you on the TestSuite level and you will only need to care about what you want to test in your test case.
What if you want to test an application that it is not yet in the Mago API? How do I tweak Mago to be used in my build system? How do I turn accessibility on? How do I run tests using Mago? There is a lot of information about Mago that I couldn’t cover in this article. The Mago website includes documentation and examples and the GNOME desktop testing mailing list is read by many Mago developers, contributors and users. These are great resources to further learn to utilize Mago for your custom application testing.
About the Author
Ara Pulido is a Spanish software tester working for Canonical as part of the Ubuntu QA Team. She believes that testing is a very much needed activity in FOSS and she will try to convince you of it.
When not at work you will find her cooking, traveling or lying on the beach.