Automated tests are failing when ran together but pass when are ran individually



  • I'm using Python+pytest and Helium to create my tests. Each time I run a test individually, it works 100% of times. However, if I try to run more than 5-6 tests together, some are failing, some are passing. The errors are random and never the same(from "LookUp" errors to "NoSuchWindowException" and other). I already checked the answers from similar questions asked:

    • no preconditions are altered
    • no dependencies between my tests
    • no alteration of data that is used in another test
    • i'm not using xdist, I am only trying to run them sequentially
    • I created different structures and arranged the tests in multiple ways trying to narrow down if a common element exists - nothing showed up. No matter what I include or exclude, just after I try to run 5+ tests, this behaviour is reproduced. I'm always starting the tests with start_chrome(URL) and end them with kill_browser(). My best guess is that somehow, in the background, the instances are overlapping or something similar to this, but I have no idea where to start the debug process.

    I'm pretty new to this, any help is welcomed. I'm running my tests using python3 -m pytest. Here is my pip list output if it helps:

    attrs        21.4.0
    chromedriver 2.24.1
    distlib      0.3.4
    filelock     3.6.0
    helium       3.0.8
    iniconfig    1.1.1
    packaging    21.3
    pip          21.2.4
    platformdirs 2.5.1
    pluggy       1.0.0
    py           1.11.0
    pyparsing    3.0.7
    pytest       7.1.1
    selenium     3.141.0
    setuptools   58.1.0
    six          1.16.0
    tomli        2.0.1
    urllib3      1.26.9
    virtualenv   20.13.4
    

    Edit per comment: here are some of the errors:

    - AttributeError: module 'locators' has no attribute 'checkbox_terms_and_conditions'
    - AttributeError: module 'locators' has no attribute ‘set_date’
    - action = . at 0x7f80dcba6ee0>
    - 
    -     def perform(self, action):
    -       from helium import Config
    -       end_time = time() + Config.implicit_wait_secs
    -       # Try to perform `action` at least once:
    -       result = self._perform_no_wait(action)
    -       while result is None and time() < end_time:
    -           result = self._perform_no_wait(action)
    -       if result is not None:
    -           return result
    - >     raise LookupError()
    - E    LookupError
    

    Please note that each 2 runs have different errors, mostly LookUpError, NoSuchWindow,or AttributeError for locators file.

    in my tests, I use an import statement for the file that contains my web elements locators, like ids, css class, xpath etc. each folder of tests has its own “locators” file. I’m not sure if this is the best method, any help is welcomed; thank you.



  • Since you have shared that there are no lookup errors or no such window exception, I think it might be an issue related to waiting time. This is why when you are running an individual test, it works fine. However, when you must be running multiple tests together, no wait time between the tests restricts the elements in the script to load.

    Let me know if you have added the wait time in-between the tests. And if adding wait time resolves your issues, I would like to hear it too.

    Also, here I am sharing a link for a similar query that I think might help you solve the problem https://www.typemock.com/answers/9694/running-all-tests-fails-some-tests-that-pass-individually




Suggested Topics

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2