Skip to content

TestRail#

TestRail is a web-based test case management tool. It is used by testers, developers and team leads to manage, track, and organize software testing efforts. TestRail allows team members to enter test cases, organize test suites, execute test runs, and track their results, all from a modern and easy to use web interface.

Integration with Zebrunner allows users to:

  1. Link automated tests to test cases in TestRail
  2. Preview linked test cases in Zebrunner UI
  3. Push automated test execution results to corresponding test cases in TestRail

Note: integration is supported on a per-project basis, meaning that different projects in Zebrunner can be connected to different TestRail instances.

Preconditions#

  • Your Zebrunner account should have Administrator role within a project, where integration needs to be enabled.
  • Your TestRail account should have administrative permissions

Configuration#

The following steps needs to be completed to configure TestRail integration:

  1. Log in to your TestRail workspace
  2. Go to the Administration view and select Site Settings section. Tick Enable API checkbox and press Save Settings button. Site Settings configuration view NOTE: It is possible to skip steps 3-5 and use password instead of API key (token). However, this is not a recommended approach.
  3. Click My Settings button My Settings configuration view
  4. Go to API KEYS view and press Add Key button API KEYS view Add API Key view
  5. Store key in a secure location, press Add Key and Save Setting buttons
  6. Log in to your Zebrunner account and go to Integrations section of a project, where integration needs to be configured
  7. In the grid of available integrations find TestRail tile and click on it
  8. Enter your TestRail hostname, username (or email) and API key to the corresponding fields and hit Save TestRail configuration view

Integration is now configured. You can check if TestRail is configured properly and reachable by clicking on Test button. Additionally, it can be enabled or disabled if needed by clicking power icon in the upper right corner.

Note: Zebrunner will perform a requests to TestRail API on behalf of the user, whose username and API key where used for configuration. Make sure that user has sufficient read (to fetch test case previews) and write (to push execution results) permissions within TestRail workspace for integration to work correctly.

Adding references to test cases in your test code#

Zebrunner provides an ability to upload test results to external TCMs on test run finish. One of the systems is TestRail.

For successful upload of test run results in TestRail, two steps must be performed: 1) Integration with TestRail is configured and enabled for Zebrunner project; 2) Configuration is performed on the tests side.

Configuration#

Zebrunner agent has a special TestRail class with a bunch of methods to control results upload: - #setSuiteId(String) - mandatory. The method sets TestRail suite id for current test run. This method must be invoked before all tests. Thus, it should be invoked from @BeforeSuite method. If your suite is composed of multiple suites, you should invoke this method only for the first sub-suite; - #setCaseId(String) or @TestRailCaseId(array of Strings) - mandatory. Using these mechanisms you can set TestRail's case associated with specific automated test. It is highly recommended using the @TestRailCaseId annotation instead of static method invocation. Use the static method only for special cases; - #disableSync() - optional. Disables result upload. Same as #setSuiteId(String), this method must be invoked before all tests; - #includeAllTestCasesInNewRun() - optional. Includes all cases from suite into newly created run in TestRail. Same as #setSuiteId(String), this method must be invoked before all tests; - #enableRealTimeSync() - optional. Enables real-time results upload. In this mode, result of test execution will be uploaded immediately after test finish. This method also automatically invokes #includeAllTestCasesInNewRun(). Same as #setSuiteId(String), this method must be invoked before all tests; - #setRunId(String) - optional. Adds result into existing TestRail run. If not provided, test run is treated as new. Same as #setSuiteId(String), this method must be invoked before all tests; - #setRunName(String) - optional. Sets custom name for new TestRail run. By default, Zebrunner test run name is used. Same as #setSuiteId(String), this method must be invoked before all tests; - #setMilestone(String) - optional. Adds result in TestRail milestone with the given name. Same as #setSuiteId(String), this method must be invoked before all tests; - #setAssignee(String) - optional. Sets TestRail run assignee. Same as #setSuiteId(String), this method must be invoked before all tests.

By default, a new run containing only cases assigned to the tests will be created in TestRail on test run finish.

Example#

In the example below, a new run with name "Best run ever" will be created in TestRail on test run finish. Suite id is 321 and assignee is "Deve Loper". Results of the awesomeTest1 will be uploaded as result of cases with id 10000, 10001, 10002. Results of the awesomeTest2 will be uploaded as result of case with id 20000.

import com.zebrunner.agent.core.annotation.TestRailCaseId;
import com.zebrunner.agent.core.registrar.TestRail;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.Test;

public class AwesomeTests {

    @BeforeSuite
    public void setUp() {
        TestRail.setSuiteId("321");
        TestRail.setRunName("Best run ever");
        TestRail.setAssignee("Deve Loper");
    }

    @Test
    @TestRailCaseId("10000")
    @TestRailCaseId({"10001", "10002"})
    public void awesomeTest1() {
        // some code here
    }

    @Test
    public void awesomeTest2() {
        // some code here
        TestRail.setCaseId("20000");
        // meaningful assertions
    }

}

Once you'll execute test run with tests having references set, it would be possible to preview test cases in Zebrunner UI and push test execution results to TestRail. Test case preview

NOTE: To re-sync test results with the test case management system configured for your Zebrunner project (ex. results have changed after run finish: tests are marked as passed/failed, or have new issues attached), open 3-dot menu in test run view and click Sync with TCM button.

Sync with TCM button

Notifying about test case changes#

In some organizations test cases, once created, are not further managed by the people who automate them. If one day a test case gets updated, test automation engineer may not be notified about the change and the automated test will continue executing outdated steps. At best, the test will fail the next time it is executed. In the worst case, the test will continue executing the outdated scenario with no visible effect.

In order to mitigate the risk of executing outdated test scenarios and communicate changes to automation engineer, Zebrunner provides an ability to configure a notification mechanism for test case updates.

There are 2 types of notifications that can be configured: * Update notifications, informing that the entire test case or some of its steps has been updated * Deprecation notifications, informing that the test case is no longer valid and is retired

The notification may have an optional message that can provide extra change context for automation engineer. That, and notification type itself will allow test automation engineer to perform required changes on his end. If no message was provided, a default message will be shown.

Configuration#

Zebrunner requires some extra configuration on TestRail side to receive notifications. 1. Add a custom case field for notification types
This field can have any Label, Template and list of Projects. But it must have System Name zebrunner_notification_type (zebrunner prefix is used to avoid conflicts with existing fields) and Dropdown type. The dropdown values may have any name, but must have strict ids associated with the particular notification type: * 1 for update notifications * 2 for deprecation notifications

Example of such a configuration Notification Type field

  1. Add a custom case field for notification message
    Same as field for notification types, this field can have any Label, Template and list of Projects. But it must have System Name zebrunner_notification_message and be of the String type.

Example of the configuration Notification Message field

Usage flow#

  1. Test case owner updates or deprecates a test case and sets notification type field to reflect type of the change. If desired, he or she can specify a message to be included as a part of the notification
  2. The next time the suite is executed, the corresponding test case labels will be displayed in a special way: updates will have an orange dot next to the case id; deprecations will have a grey dot and the case id will be strikethrough

Note: notifications will be received and displayed in Zebrunner right after the test execution finish, even if the suite is still running 4. The automation engineer makes the necessary updates 5. The automation engineer edits the test case in TestRail: sets the notification type option to None and removes the message 6. The next time the suite is executed, the corresponding test case labels will not have any notifications (if a new notification was not added during this time, of course)

Examples#

Example 1 - Updated test case#

The test case shown on the following figure has an Update notification along with message saying that the automation engineer should take a look at the step 2, since the API response was change. Test case with Update notification After the next test execution, it will have the following notification Test case with Update notification

Example 2 - Deprecated test case#

The test case shown on the following figure has an Deprecated notification without a message saying the reason. Test case with Deprecation notification After the next test execution, it will have the following notification Test case with Update notification