When a company produces a wide array of products, it can be hard to find a consistent way to test them all. As an Automation Engineer at IAS, one of my first projects was to find a way to do just that. Learning how a new product works is hard enough without having to learn a new testing methodology. The goal was to create a system through which anyone in QA could understand how to write and run tests for any product without having to learn an entirely new set of tools. Historically, most of the tests were scripts written in Python. For this reason, I created an object-oriented testing framework that used Python with Pytest and Selenium – we called it Charmer.
Creating a BaseTest
To make each test uniform, a BaseTest class was created which inherits from the unittest.TestCase class. This, in turn, runs the typical steps used for all steps in Setup() and Teardown() methods.
This made it easy for QA to run tests against Saucelabs, Browserstack or our in-house Selenium Grid in a uniform way. It also allows us to execute tests locally, in mobile environments or against xvfb.
Creating an Entry Point (test-runner)
In order to dynamically pass different testing scenarios to our tests, I created an entry point script called test-runner. Test-runner’s job is to set up the environment before calling Pytest. Things like the access keys and hosts machines that we are testing against are set as environment variables and are then available during testing. This allows us great flexibility in which browsers/os/mobile combinations we use for each test instead of hardcoding it in the framework itself.
In order to have one test run on many different browsers at the same time, I created a decorator called on_platforms. This allows a user to supply the desired capabilities for the Selenium nodes directly from the command line (or from a YAML config file which will be discussed later). Also, the instantiated test would be named using the browser/os/mobile args along with the test name itself. This makes it easier to track down the test when searching for its results.
A Simple Example
An example of a simple test that we would write can be seen below:
As you can see in the example above, we instantiated a page called LoginPage. We use Selenium’s Page Object Model to create our pages and locator elements.
We added a lot of functionality to Charmer. One example is the implementation of whitelists and blacklists. Examples of this are below.
Notice that we have shortcuts (or defaults) for certain browsers so that the user does not have to type out the entire browser argument. We can also support anything provided by Saucelabs, Browserstack or our own Selenium Grid by using the –browser_args option. An example of using this option from the command line is shown below:
This allows us the option to test against specific browser/OS combinations that fail when all others seem to pass and narrow down where the bugs are being generated.
Creating a flexible testing tool while trying to maintain simplicity is a very difficult feat. Just keeping up with new testing technologies and maintaining test coverage on new and old software is a big challenge. With Charmer, we’ve leveraged Pytest and Selenium to write robust tests quickly while maintaining test integrity.