Zephyr vs Xray vs TestRail



  • We are thinking of implementing a test management tool for our test cases which are currently in Excel sheets. So far we have zeroed in on these three tools. Some of the features we are looking for are:

    • integration with JIRA
    • ability to import existing test cases from Excel sheets
    • cost effective
    • ability to integrate automated tests
    • quality of the JIRA plugin
    • in case of Zephyr and Testrail, is it advisable to use the JIRA plugin only without using their enterprise service?

    It would be good if someone could provide a Matrix Comparison for these tools based on these features.



  • We are using the Jira addin with locally running TestRail without any issues. What I like about Test Rail is the ease of use. We went from install to all QA team using it in a few weeks. Also Gurock respond very quickly to development requests.

    Mark Smith



  • @zymir l'm not sure if you're adopting Jira cloud or Jira datacenter, as the options you have will depend on that.
    Anyway, let me give you some food for thought 🙂
    I work at Xray, but I'll do my best to provide some useful information (non-sales related).
    I will answer inline, below.

    integration with JIRA

    Both Xray and Zephyr (squad and scale) work inside Jira, so therefore their integration is native => the most complete you can have.
    However, some teams may need just to have a way to upload test results to Jira and see their impacts on the related stories. If that's the case, then TestRail is a viable option.
    Here, you'll have to think if you want people (devs/testers) to use one tool, or several tools.

    ability to import existing test cases from Excel sheets

    In the case of Xray, it provides a built-in Test Case Importer that you can use to import test cases in bulk.

    cost effective

    I won't answer this one on purpose; that's something you have to evaluate.

    ability to integrate automated tests

    Xray supports a bunch of test automation tools/frameworks/libraries, including Junit, TestNG, xUnit, Nunit, Cucumber, Cypress, Playwright, Postman, etc. You may find around dozens of test automation tutorials and integrations with CI/CD tools in Xray docs.

    quality of the JIRA plugin

    I would recommend you to look at:

    • reviews on Atlassian Marketplace, for example for Xray cloud
    • frequency of releases, and how frequent are bugfixes and new releases. You can see it also on Atlassian Marketplace, on the Versions tab>pick datacenter>then see all versions under Resources section, which shows info both for the product on Jira cloud and on Jira server/datacenter; example for Xray

    in case of Zephyr and Testrail, is it advisable to use the JIRA plugin only without using their enterprise service?

    Please check what each vendor provides exactly with their enterprise offering and see if it matches your needs; also check the SLAs.
    Some organizations demand priority support, customer representative, 24x7 support, etc.. check exactly your needs and what is offered by the the enterprise offering. I would also consider the frequency of releases and the policy of bugfixes as something to consider.. because you can have a "enterprise kinda of service" but bugfixes could take a while to be released.

    It would be good if someone could provide a Matrix Comparison for these tools based on these features.

    Xray has this comparison page, which is not complete.
    I'll give you some few things that you can use to help you pick tools in this space. Please don't consider this an exhaustive list 🙂

    • what testing approaches (manual test cases, test automation, exploratory testing) are we following? does it support all of those?
    • are our teams working in siloed environments? Or are they part of the scrum / agile team, or even embed as skills within the team? Siloed teams may prefer to have separate tools (even though I don't think it promotes collaboration, transparency, agility)
    • do we need to track coverage? and full, bi-directional traceability between epics<=>stories<=>tests<=>runs<=>defects?
    • do we need to track the status of our product on multiple versions/devices/environments at the same time?
    • is the product flexible to enought so we can adapt processes and tooling, including test automation frameworks and CI/CD tools, at any time?

    Hope this helps, anything please let me know 🙂




Suggested Topics

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2