AI-infused DevOps: Key takeaways from our latest research
Tricentis experts discuss findings of our AI in DevOps report and...
We’ve recently fielded a lot of questions about using Jira for test case management. From how to use Jira for testing to which solutions best complement a Jira process, both Jira and testing seem to be the topics of the hour these days.
This popularity is not necessarily surprising given a recent Tricentis poll in which 66% of respondents reported having used Jira for the past several months or years.
So what exactly do you need to know when it comes to using Jira for test case management? This guide will walk you through everything from soup to nuts, including:
For those who don’t know, Jira is an issue and project tracking software from Atlassian. Currently, it’s the number one software development tool used by Agile teams, and as Agile adoption continues to grow, more and more organizations are introducing Jira. The recent IPO as well as the massive wave of companies leaving legacy ALM providers have only fueled the widespread adoption of Jira.
Although JIRA is designed for issue and project tracking, many teams are using it for test case management so that development and testing can stay in one system. But how exactly is JIRA set up for testing? Let’s take a look.
Going back to the question of how JIRA is set up for testing, the short answer is that JIRA isn’t. The reality is a lot more complicated, especially given how many organizations are using it for testing today.
When you begin looking at how you can build a testing process in Jira, the most common place to start is with an issue type. You’ll see that Jira has several different types of issues (Bug, Epic, Improvement), but nothing specifically for tests. This is because Jira is not a test case management solution. It’s really meant for issue tracking.
You can customize Jira in certain ways to use it for some test case management processes, but these are makeshift customizations and they have limitations as a result. For example, every workflow built inside Jira has an endpoint of “done.” But in testing, even if you complete a test, that doesn’t necessarily mean that you’re actually done with that test.
This setup means that if you customize Jira or certain issues for test case management, you’ll always find yourself moving to a done status when you might not be “done” once a test executes. What you really want is to be able to use that test case for multiple types of testing, whether it be for regression testing or running the same test case under various parameters and configurations to validate a new feature. However, since Jira issues are always meant to be pushed to “done,” you are limited in the flexibility to reuse the issue for testing.
Atlassian’s own documentation on Jira says that it can work for manual testing, acceptance testing and so on, but it clearly warns that there is no testing-specific functionality built naively into Jira. Also, if you are using any type of automation, you need to use another solution.
Even with manual and acceptance testing, there are some limitations. Let’s say that Jira does a really good job of running a manual test case. It might work one time or for a brand new feature, but then you move it to the done status and what happens next? What if you want to do regression or session-based testing? How are you going to reuse those objects across projects, assign multiple ticket items and show that coverage? Due to the way Jira is set up, you can’t.
Despite the limitations posed, many teams are using JIRA for test case management, and they’re typically doing so by customizing issues in one of two ways: adding a “test case” issue type, or using the “user story.” Let’s take a closer look at how each of these options works:
Deep dive: Adding a “Test Case” issue type
To customize Jira for test case management by adding a “Test Case” issue type, you need to take the following steps:
While this approach works in theory, in reality it presents several challenges:
Deep dive: Using the “User Story”
Alternatively, you can customize Jira for test case management by tweaking the “User Story”. Here’s how this approach works:
Once again, this approach poses challenges because the subtask acts as the test case and the test run. Specifically, it poses challenges around:
As the above examples illustrate, while Jira is not designed for testing you can customize it to fulfill certain test case management needs. But, should you do this? Let’s consider the pros and cons of using strictly Jira for test case management:
Pros of using Jira for test case management:
Cons of using Jira for test case management:
All of this means that things you would normally do with a test case and environment execution tool, such as grouping testing cycles and test executions, running test configurations and handling version control, are not supported. This makes you think: Is Jira really built for testing or are you forcing the issue (no pun intended, since everything in JIRA is an issue)? In most cases, you’re forcing the issue and there is likely a better way.
Because Jira is not designed for testing, it poses many limitations when it’s used for testing purposes. Arguably the biggest limitation is the inability to reuse and centralize testing efforts.
So if you’ve made the transition to Jira and your developers love it, what can you do? Is there any way for your test team to get value out of Jira? Integrations can help.
Because Jira is not designed for testing, it poses many limitations when it’s used for testing purposes. Arguably the biggest limitation is the inability to reuse and centralize testing efforts.
There are two types of integrations you can use to add testing functionality to Jira:
Let’s take a closer look at what each of these integration options entails:
What about Jira apps or add-ons?
Add-ons are available via the Atlassian Marketplace. A few commonly used add-ons for testing include:
These types of add-ons extend the functionality of Jira to make it better equipped for test case management. For example, Xray automatically adds a “Test” issue type to the list of issues and allows you to add test case steps to the Test issue type. This setup alleviates the problem of creating a custom issue type that requires you to add in a bulk section that includes completed steps and expected outcomes (which in turn leaves you with huge chunks of testing that you have to do all at once and the inability to put in line by line where the steps failed, where it was executed and what your actual execution was).
Meanwhile, if you’re trying to do BDD testing, it’s hard to follow the Gherkin “given, when, then” syntax in JIRA alone, but an add-on like Tricentis qTest Scenario makes it possible. Tricentis qTest Scenario allows you to import your dot feature files along with your scenarios as issue types so that you no longer have to create standard issue types. Instead, you can add in the Gherkin style test scripting language in the scenario piece, where you can then create a dot feature file. You can also create scenario files that attach to those dot feature files inside of JIRA, which allows you to add them to a User Story.
Pros of using add-ons
In general, the testing add-ons try to bring together the groups of testing occurring in Jira so that you can pull all of the test cases you have into cycles or suites. They also allow you to see the latest execution results of the test cases for each grouping, which creates visibility into the grouping of test executions.
Other pros of using add-ons to improve the testing functionality of Jira include:
Cons of using add-ons
While the add-ons do help a lot, you’ll still be a bit strapped in terms of what you can and can’t do with your text execution because the add-on functionality is limited based on what the Jira infrastructure allows.
For example, the add-ons create a bit more of a manual process than you would have with a dedicated test case management solution. Additionally, everything you do inside the add-on is limited to the particular project at hand, meaning you can’t put the same issue in multiple projects.
The add-ons also present some version limitation when you’re creating a test case. Typically, if you create an issue and go to run the issue, the run is simply a duplication of the test case. Let’s say you have five steps in a test case that’s in a cycle to be run and you add another step. Jira will automatically push that data over into every test case run, meaning you can’t keep different versions. With Tricentis qTest and some other tools, you can actually keep track of different versions of test cases and run past versions when you execute, but you need to have a one-to-one version history of that test case execution in order to do that.
Finally, when you’re using an add-on, a test case equals a Jira issue. It may not appear that way, but that’s how it is with all the add-ons, and that setup poses limitations.
Other cons of using add-ons to improve the testing functionality of Jira include:
Deep dive: Dedicated test case management solution
Most testing teams struggle to be very efficient. Often it’s because they’re simply not using the right tool. A dedicated test case management tool can help improve efficiency, and there are many that now offer some level of external integration with JIRA in order to bring more testing-specific functionality to your testing process while still allowing you to use JIRA.
Most testing teams struggle to be efficient. Often it’s because they’re simply not using the right tool.
What do test management tools do?
Test management tools:
Pros of using a dedicated test case management tool
By offering the functionality described above, integrating a dedicated test case management tool can go a long way toward improving testing within Jira. Pros of this approach include:
Cons of using a dedicated test case management tool
Despite all the value integrating a dedicated test case management tool with JIRA can provide, it still has its limitations. Cons of this approach include:
With these downsides in mind, there are a couple points to consider to help determine whether or not the pros of an external integration with a dedicated test case management tool outweigh the cons:
While a dedicated test case management tool offers the best experience for testing inside of Jira, it typically still poses some downsides. However, it is possible to get the best of both worlds between an enterprise test case management tool and Jira.
The key to achieving this balance is understanding that not all integrations are created equal and identifying the type of solution that offers the optimal integration experience for your needs. For example, most test solutions use a scheduled sync, but Tricentis qTest Manager has a real-time integration that uses webhooks.
With a scheduled sync, you have to map your entire test case management tool schema to your JIRA schema, which means you constantly have to pull data and use resources to keep the system current. A scheduled sync is also prone to delays, poor performance due to APIs constantly trying to figure out what’s been updated and data corruption if any schedule collisions occur. All of this requires a lot of resources to manage the integration.Meanwhile, with a real-time sync, you get a straight mapping between a project (and its issues, whether they’re User Stories, Epics, subtasks, etc.) and JIRA, and you can then show test case coverage against those issues. This real-time sync offers a direct submission into JIRA for defects, browser automation and embedded reporting, plus it ensures data accuracy since JIRA is always the source record for user stories and defects.The real-time sync option keeps both developers and testers happy because it allows everyone to keep doing all of their updates inside of Jira and only requires you to pull the issues that you care about into the test case management tool, where you can then create test cases and test runs and map those to the requirements from Jira. Once you’ve executed those runs, you can simply push the defect back out to Jira to interact with it and start working on a resolution. This allows everyone to keep doing all of their updates inside of JIRA and only requires you to pull the issues that you care about.With this type of integration in place, your developers can stay inside of Jira (rather than going to another tool to see everything) and your testers gain access to the enterprise features that are missing from a lot of the add-ons that are available — all without a complicated, time-consuming integration to manage. In other words, you get the best of both worlds.
Ultimately, which mix of solutions — Jira only, Jira plus an add-on or Jira plus an external integration — is right for your team depends a lot on the size of your team and the types of testing your doing.
To help you determine the solution mix that best fits your needs, we’ve mapped out our recommendations in the following chart:
Tricentis experts discuss findings of our AI in DevOps report and...
Tricentis qTest integrates with Jira to enhance two-way feedback...
In this webinar, our experts will show how Tricentis LiveCompare...
For most public sector agencies, transitioning to an Agile test...
Explore the vital role of data integrity in Oracle environments and...