How to build a Testing Virtual lab using VMs,Vagrant,Puppet,Chef etc
My team of testers and me have a big collection of functional tests (written in python and run with pytest). The thing is that all the tests are run on a single testing server, an ubuntu VPS with selenium,webdriver,firefox,chrome,xvfb,pytest etc. What I would like us to do is create many different VMs (combinations of OSs and browser versions) that would be launched/run based on a configuration file, and would run the automated scripts that we have on our github repo. Example: I want to run these X scripts on a Windows 7 VM with IE8. I am currently familiarizing myself with Vagrant as a possible solution. Would it be possible to create some base boxes with specific configurations and then "somehow" trigger them from an interface? I am referring to Puppet and Chef in the title because I have read many articles that combine them with Vagrant. I have no experience with them. It would be helpful if I could understand the architecture that could be implemented... and maybe the limitations. Essentially i would like to make a small and local implementation of SauceLabs/Browserstack.
My team has created some Gradle tasks that spin up VM's using vagrant and then deploy stuff to the VM's using Chef (similar to puppet). It would be possible to do something like this for your test infrastructure, but it is a lot of work not only to build the initial infrastructure, but also to maintain those VM's over time, especially with the number of configurations you're talking about. I would suggest an alternative. Sauce Labs is a service that already exists to do what you want to do and is not too expensive. It allows you to execute your WebDriver tests on their grid and their site has a lot of great documentation, making it relatively easy to get set up. In addition, you can watch your tests executing in real time, view videos of the tests after the fact, and get screenshots, all easily configurable. Their logging/reporting infrastructure is also pretty mature, you can easily use it out of the box to cover most of your reporting needs. If you haven't investigated it, I would highly recommend doing so.