@zymir l'm not sure if you're adopting Jira cloud or Jira datacenter, as the options you have will depend on that.
Anyway, let me give you some food for thought
I work at Xray, but I'll do my best to provide some useful information (non-sales related).
I will answer inline, below.

integration with JIRA

Both Xray and Zephyr (squad and scale) work inside Jira, so therefore their integration is native => the most complete you can have.
However, some teams may need just to have a way to upload test results to Jira and see their impacts on the related stories. If that's the case, then TestRail is a viable option.
Here, you'll have to think if you want people (devs/testers) to use one tool, or several tools.

ability to import existing test cases from Excel sheets

In the case of Xray, it provides a built-in Test Case Importer that you can use to import test cases in bulk.

cost effective

I won't answer this one on purpose; that's something you have to evaluate.

ability to integrate automated tests

Xray supports a bunch of test automation tools/frameworks/libraries, including Junit, TestNG, xUnit, Nunit, Cucumber, Cypress, Playwright, Postman, etc. You may find around dozens of test automation tutorials and integrations with CI/CD tools in Xray docs.

quality of the JIRA plugin

I would recommend you to look at:

• reviews on Atlassian Marketplace, for example for Xray cloud
• frequency of releases, and how frequent are bugfixes and new releases. You can see it also on Atlassian Marketplace, on the Versions tab>pick datacenter>then see all versions under Resources section, which shows info both for the product on Jira cloud and on Jira server/datacenter; example for Xray

in case of Zephyr and Testrail, is it advisable to use the JIRA plugin only without using their enterprise service?

Please check what each vendor provides exactly with their enterprise offering and see if it matches your needs; also check the SLAs.
Some organizations demand priority support, customer representative, 24x7 support, etc.. check exactly your needs and what is offered by the the enterprise offering. I would also consider the frequency of releases and the policy of bugfixes as something to consider.. because you can have a "enterprise kinda of service" but bugfixes could take a while to be released.

It would be good if someone could provide a Matrix Comparison for these tools based on these features.

Xray has this comparison page, which is not complete.
I'll give you some few things that you can use to help you pick tools in this space. Please don't consider this an exhaustive list

• what testing approaches (manual test cases, test automation, exploratory testing) are we following? does it support all of those?
• are our teams working in siloed environments? Or are they part of the scrum / agile team, or even embed as skills within the team? Siloed teams may prefer to have separate tools (even though I don't think it promotes collaboration, transparency, agility)
• do we need to track coverage? and full, bi-directional traceability between epics<=>stories<=>tests<=>runs<=>defects?
• do we need to track the status of our product on multiple versions/devices/environments at the same time?
• is the product flexible to enought so we can adapt processes and tooling, including test automation frameworks and CI/CD tools, at any time?

Hope this helps, anything please let me know

As @Kadir mentioned, from a technical standpoint, you can link issues in Jira to Confluence pages but you may not need that.

@rosemadder I've prepared this article some time ago precisely on the topic of compliance in regulated environments.
In Xray you have full traceability, from epic > stories > tests > runs > defects.
But this traceability is not static; actually, it's multidimensional. In other words, in Xray the traceability is calculated depending on the version/environment/scope you aim to analyze. For example, traceability and related coverage for v1 of the product can be completly different from the one obtained for v2 of the product (e.g., due to different code, different bugs, different test results)

Your code commits can be linked to any jira issue, including epics, stories, and even Tests (which are Jira issues), in case you're implementing the related test automation code.
You can also generate a report, manually or automatically, with your testing specification, planning, results (please see the article for more info).
Hope that helps

@zymir l'm not sure if you're adopting Jira cloud or Jira datacenter, as the options you have will depend on that.
Anyway, let me give you some food for thought
I work at Xray, but I'll do my best to provide some useful information (non-sales related).
I will answer inline, below.

integration with JIRA

Both Xray and Zephyr (squad and scale) work inside Jira, so therefore their integration is native => the most complete you can have.
However, some teams may need just to have a way to upload test results to Jira and see their impacts on the related stories. If that's the case, then TestRail is a viable option.
Here, you'll have to think if you want people (devs/testers) to use one tool, or several tools.

ability to import existing test cases from Excel sheets

In the case of Xray, it provides a built-in Test Case Importer that you can use to import test cases in bulk.

cost effective

I won't answer this one on purpose; that's something you have to evaluate.

ability to integrate automated tests

Xray supports a bunch of test automation tools/frameworks/libraries, including Junit, TestNG, xUnit, Nunit, Cucumber, Cypress, Playwright, Postman, etc. You may find around dozens of test automation tutorials and integrations with CI/CD tools in Xray docs.

quality of the JIRA plugin

I would recommend you to look at:

• reviews on Atlassian Marketplace, for example for Xray cloud
• frequency of releases, and how frequent are bugfixes and new releases. You can see it also on Atlassian Marketplace, on the Versions tab>pick datacenter>then see all versions under Resources section, which shows info both for the product on Jira cloud and on Jira server/datacenter; example for Xray

in case of Zephyr and Testrail, is it advisable to use the JIRA plugin only without using their enterprise service?

Please check what each vendor provides exactly with their enterprise offering and see if it matches your needs; also check the SLAs.
Some organizations demand priority support, customer representative, 24x7 support, etc.. check exactly your needs and what is offered by the the enterprise offering. I would also consider the frequency of releases and the policy of bugfixes as something to consider.. because you can have a "enterprise kinda of service" but bugfixes could take a while to be released.

It would be good if someone could provide a Matrix Comparison for these tools based on these features.

Xray has this comparison page, which is not complete.
I'll give you some few things that you can use to help you pick tools in this space. Please don't consider this an exhaustive list

• what testing approaches (manual test cases, test automation, exploratory testing) are we following? does it support all of those?
• are our teams working in siloed environments? Or are they part of the scrum / agile team, or even embed as skills within the team? Siloed teams may prefer to have separate tools (even though I don't think it promotes collaboration, transparency, agility)
• do we need to track coverage? and full, bi-directional traceability between epics<=>stories<=>tests<=>runs<=>defects?
• do we need to track the status of our product on multiple versions/devices/environments at the same time?
• is the product flexible to enought so we can adapt processes and tooling, including test automation frameworks and CI/CD tools, at any time?

Hope this helps, anything please let me know

Hi @Alberto ,
first of all, as an initial disclaimer, I work for Xray team, which is a competitor of Zephyr (on the differences between the two)
There are some free tools that I'm aware (e.g., testlink) but I advise you to try them out.. it all depends on your needs. Some tips for thought:

• what kind of approaches do you need to support? manual scripted test cases, test automation, exploratory testing?
• what CI tools/testing frameworks do you need to integrate with?
• do you need to track coverage?
• is your dev/testing team the same? How will they navigate information from one side to the other side? Do you want to promote this separation?

As @terrea mentioned, there are also "free integration plugins" on Jira/Atlassian marketplace that usually require a paid subscription elsewhere.
I would recommend to dig a bit more on what people mean with "Plugins that integrate with Jira". Do they mean just sending the test results to Jira? What exactly? And is that enough for you?
Of course I'm biased, but it also depends on the level of collaboration and visibility you want to have around Jira.

As @guilherme answered, that's a sign you're missing a configuration step.
The quick setup instructions detail the typical setup operation you need to follow to setup a project. Xray is quite flexible, so it means you can decide several things during that process.

Cucumber tests can be executed manually or through automation.
No matter what, I would recommend to use Xray as the single source-of-truth.
To distinguish what tests are to be executed manually or through automation, you can define your own rule for that, like having a specific label on the test. Then, you can build a Jira filter to filter out those tests.. and in your CI tool (e.g., Jenkins) you can use that filter to only pick the relevant tests to be run by the automation.

@kalyn said in Is it possible to configure Jira/X-Ray so that the X-Ray tests can be used as sub-tasks?:

What Xray provides is the ability to create Sub Test Executions as sub-tasks of a Story, for example. More info here: https://confluence.xpand-it.com/display/XRAY/Sub-Test+Execution

Since a Test is somehow like a test case template reusable, even for later version, maybe it makes more sense to have the "execution" related task as a sub-task instead. With Sub Test Executions you can track them directly in the Agile boards, as something right below the related Story, as depicted here: https://confluence.xpand-it.com/display/XRAY/Agile+Enhancements#AgileEnhancements-QuickviewofExecutionsforRequirementsfromAgileBoard

Precisely!

Hi @amarantha,
there are two directories/files:

/Users/<your user>/Library/ApplicationSupport/Xray\ Exploratory\ App/logs/

and

~/Library/Logs/Xray Exploratory App/main.log

Hi @magpie ,
@fbio has already answered; in sum, you can generate Junit XML report, using available reporters, and them submit it to Xray using a free Jenkins plugin available for that purpose. You could also generate a Xray JSON file but you would need to find/build a newman reporter for that,
I wanted also to share this Postman tutorial which may be useful for you; the previous tutorial is for Xray on Jira cloud. If you're using Xray for Jira datacenter/server, the proper tutorial is this one instead.

Hi @Raziyah00 ,
Xray supports not only Cucumber but also other Gherkin based frameworks.
The first thing to decide is where will be your master the information: either Jira/Xray or git.
You can see here the possible workflows.
Xray (Jira cloud) documentation provides a tutorial for Java+Cucumber, exercising the two workflows.
In sum, a Scenario/Scenario Outline will be translated to a Test issue in Xray, of gherkin/cucumber type; that's it.
If you want to test multiple examples/data row combinations, for the time being this has to be handled inside the Test issue itself, where you compose the full scenario outline; data-driven testing for cucumber scenarios is not yet possible.
There is a online course focusing on test automation, where cucumber flows are also shown.

Hi @Amritpal , if you have any specific questions I can try to answer.
Disclaimer: I'm solution architect at Xray team.