Skip to content

Test Execution Duration Analysis#

Test Case Execution Duration Analysis report gives insights into the test execution results both for all the test cases and for each test case separately. This report provides advanced information on test case durations by calculating key metrics such as average, median, and P95(95th percentile) values. By emphasizing duration metrics, this report efficiently identifies slow-performing test cases that may be suitable candidates for automation.

Building the report#

In order to build the Test Case Execution Duration Analysis report, you should go to the Reports section and select the corresponding report type from the library.

You need to select the Dimension that will be used to narrow down test executions for the report. The two dimensions that are currently supported are Period and Milestone. The Period dimension will include in the report all test executions that happened within the period boundaries (inclusive; a date range picking control will be rendered once Period is selected). Meanwhile, the Milestone dimension would require you to define a set of milestones, which will be used to only include test executions belonging to them.

Once you have selected the dimension and configured it, click Generate to generate your report.

Creating a report

Report data#

Sample report

Once the report is generated, the following data will become available to the user in the report's header:

  • Execution results donut chart: interactive chart allowing you to see test execution results distribution
  • Number of executions: total number of executions that were used to generate the report
  • Total average duration: percentage of test cases that have automated executions
  • Total median duration: percentage of test cases that have automated executions
  • Total P95 duration: percentage of test cases that have automated executions

A table with execution results for each test case will follow the header. This table will contain key test case attributes (its key, title, test suite, priority and automation state) along with some aggregates on execution data (its average, median, p95 execution time, and execution results summary bar).

Duration chart#

The key feature of the Test Execution Duration Analysis report is its graphical representation of test case durations in various formats (average, median, p95). This report utilizes histograms to display test case durations segmented into intervals, highlighting key duration metrics.

By emphasizing duration metrics through graphical visualization, this report efficiently identifies slow-performing test cases that may benefit from automation.

Note

The duration chart is interactive, allowing users to engage with the data more dynamically. By clicking on a specific interval within the histogram, users can instantly filter and view the test cases associated with that selected duration range.

Table configuration

Report controls#

Within the Test Case Execution Duration Analysis report, several controls are available to manipulate and customize the displayed data.

Filtering#

Test Case Execution Duration Analysis report offers a range of filters to refine and focus the displayed data based on specific criteria:

  • Date(this filter is mandatory if the Milestone filter is not used. It allows selection of a specific time period, including all test executions carried out within that period);
  • Milestone(this filter is mandatory if the Date filter is not used. It involves selecting a set of milestones, which will restrict the report to only include test executions associated with these milestones);
  • Priority(allows filtering based on the priority level assigned to test cases (e.g., High, Medium, Low));
  • Automation state(allows filtering based on their automation states (e.g., Automated, Manual));
  • Result(allows filtering based on the result of test case executions (e.g., Passed, Failed));
  • Execution type(allows filtering based on the type of test execution (Manual, Automated));

Note

When filters are applied, they are automatically saved to the URL.

Table configuration

Grouping#

Test Case Execution Duration Analysis report provides the ability to group test cases using various criteria:

  • Suite (test cases are grouped based on their assigned test suites. This grouping allows for a holistic view of test case execution performance within specific functional areas or modules);
  • Priority(Test cases are grouped according to their priority levels (e.g., High, Medium, Low). This grouping helps identify and analyze the execution duration trends based on the criticality of test cases);
  • Automation state(test cases are grouped based on their automation status (e.g., Automated, Manual). This grouping allows users to evaluate the impact of automation on test case execution duration and overall testing efficiency.)

Note

When a group is applied, it is automatically saved in the URL.

Table configuration

Configuring and dragging table columns#

Columns of the table with test cases can be configured: it is possible to show or hide columns and rearrange their order. In order to do so, you can use the control in the top right corner of the table:

Note

When the configuration changes, it is automatically saved in the URL with the column order preserved.

Table configuration