I would like to test software installation(on windows) after each release. Since installer doesn't support silent mode i wonder if there is one good aproach that can minimize list of used tools and automate task as much as possible. I can think of the following steps:
- Create new clean machine in cloud using packer or terraform
- Download latest version of the app
- Download some tools for UI automation and script that will run installer
- Try to run app.
- Somehow verify if everything is correct. Maybe by taking a screenshot
So far, I'd say that you're on the same track that I would take myself, although I haven't tested desktop applications since long before these automation tools existed.
1) Packer - I've never heard of it before and just took a look at it. I'm going to have to look deeper, but, if it could include something like Winium in the golden image. If you are not necessarily testing just installation of your application, it might be good to have an image that contains your app.
2) For the update, if it just needs to download the latest copy, I would ideally suggest seeing if there's a way to include it in the image that you use. If not, I would probably write a powershell script that will tie in with the next step that would begin by downloading the application to be installed.
3) Powershell script to call either Winium, AutoIT, or another suitable scripting tool to kick off and execute the installation. If the installation is always going to be the same, it probably doesn't matter which one. If you're able to use Winium though and a proper test runner, I would move to the next sentence and then number 4, the more that you can test and change easily, the better. If you have tests already in NUnit or similar, it could pull them over to be run. This could go for the execution and/or the tests you run on the AUT.
4) Powershell script does hand over to test runner, which executes your tests. This would normally be a command line command with arguments. Ensure that your image has a UNC defined where it can write logs/results to (less manual work for you to do). Feel free to have it save screenshots here as well. If you're not expecting much different, you could even do an image comparison against the previous if you wanted to (I've had mixed results doing this). When it's finished. or ran into an error, have it notify you.
5) Look at the results and action as necessary
Hope this helps, although I'd love to see some answers from people who do this on a regular basis.