Automation in Testing

[dialex]

Links

• Def https://responsibleautomation.wordpress.com/2016/12/02/automation-first-a-definition/
• Example http://www.developsense.com/blog/2018/02/efficiency/
• http://www.lucassaldanha.com/unit-tests-vs-acceptance-tests/
• what to automate https://www.developsense.com/blog/2021/11/what-tests-should-i-automate
• https://sites.google.com/site/unclebobconsultingllc/tdd-with-acceptance-tests-and-unit-tests
• What its not http://www.developsense.com/blog/2017/04/deeper-testing-2-automating-the-testing/
• https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html
• http://automation-beyond.com/2016/10/10/how-do-you-choose-which-test-cases-to-automate/
• https://blog.testproject.io/2021/07/21/the-10-10-10-rule-of-test-automation/
• https://www.eviltester.com/blog/eviltester/testautomation/automate-for-yourself-first

.

• tools https://geteasyqa.com/blog/best-automation-testing-tools/
• PageObject http://www.thefriendlytester.co.uk/2014/05/pageobject-pattern-why-how-and-more.html
• https://medium.com/@TuckerConnelly/good-practices-for-testing-react-apps-3a64154fa3b1#.vpr2elq2k
• https://medium.com/@bertold/dangers-of-generalization-in-software-testing-c4eeaf4e605e
• Cross Browser https://dojo.ministryoftesting.com/lessons/getting-started-with-cross-browser-testing
• Cross browser https://club.ministryoftesting.com/t/cross-browser-testing-getting-started/1218

.

• When to automate http://blog.eviltester.com/2016/10/q-when-do-we-prefer-manual-testing-over.html
• https://kentcdodds.com/blog/how-to-know-what-to-test
• where to test GUI vs API http://blog.eviltester.com/2017/02/should-i-test-at-gui-level-or-api-level.html
• http://thetesteye.com/blog/2017/03/implication-of-emphasis-on-automation-in-ci/
• http://blog.eviltester.com/2017/04/that-moment-where-you-should-have.html
• flaky https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html
• https://katrinatester.blogspot.pt/2017/05/three-styles-of-automation.html
• performance https://dojo.ministryoftesting.com/lessons/30-days-of-performance-testing
• https://katrinatester.blogspot.pt/2017/07/test-automation-canvas.html
• https://www.ontestautomation.com/on-supporting-continuous-testing-with-fitr-test-automation/
• https://www.mariedrake.com/post/api-testing-using-jest-and-supertest
• challenges https://dev-tester.com/top-challenges-of-automated-end-to-end-testing

.

• devops infrastructure testing https://www.infoq.com/presentations/opencredo-ruby
• devops https://skillsmatter.com/skillscasts/10732-testing-programmable-infrastructure
• http://blog.eviltester.com/2017/09/rethinking-test-process-automation-modelling.html

.

• regression https://club.ministryoftesting.com/t/defining-regression-in-testing/1411
• regression https://dojo.ministryoftesting.com/lessons/how-to-do-good-regression-testing
• regression http://blog.eviltester.com/2017/10/what-is-regression-testing-software.html

.

• https://alisterbscott.com/2021/12/08/10-tips-for-successful-e2e-web-app-test-automation/
• https://katrinatester.blogspot.pt/2018/01/three-types-of-coding-dojo-for-test.html
• spec by example https://www.thoughtworks.com/insights/blog/specification-example
• https://responsibleautomation.wordpress.com/2018/01/02/my-automations-not-finding-bugs-but-thats-ok/
• https://katrinatester.blogspot.pt/2018/02/how-do-you-decide-what-to-automate.html
• http://www.developsense.com/blog/2018/06/which-test-cases-should-i-automate/
• https://dojo.ministryoftesting.com/dojo/lessons/rethink-your-automation-setup
• https://thelifeofoneman.com/test-automation
• https://thelifeofoneman.com/beware-of-automation
• http://www.developsense.com/blog/2018/11/what-should-i-automate/
• Smoke/Sanity http://www.developsense.com/blog/2011/11/smoke-testing-vs-sanity-testing/
• http://www.developsense.com/blog/2019/01/breaking-the-test-case-addiction-part-1/
• https://www.linkedin.com/posts/john-ferguson-smart_agiletesting-testautomation-bdd-activity-6711181127962378240-3LOx/
• Against https://building.nubank.com.br/why-we-killed-our-end-to-end-test-suite/

Personalities

• Alister Scott
• Joe Colantonio

[dialex]

P.S: We are currently experimenting with this strategy. It may be tweaked as we go. When we have a definitive strategy, we will formalise the diagrams above. Our strategy was greatly influenced by this talk. Here is a brief summary:

15:29 - what to test

Linting? -> ShellCheck
Deployment scripts unit tests? -> Unit InSpec
Are services running? -> Acceptance InSpec

18:44 - tooling

(before provisioning)

• Unit testing: bash scripts, terraform scripts
  • Linting: quick sanity check, run in CI before committing
  • Low value on testing configuration?

(after provisioning)

• Integration testing: packages installed, services running, ports listening
  • Serverspec/Inspec: readable, quick run time, can SSH into instances
• Acceptance testing: SSHing into machines, using apps deployed on the machine
  • Cucumber: readable for devs and business, reporting, executable specification
• Smoke tests: run before everything else, really quick, catches obvious errors, not complex tasks

[dialex]

https://twitter.com/theBConnolly/status/915614905016795142

Also http://blog.getcorrello.com/2015/11/20/how-much-automated-testing-is-enough-automated-testing/#

“How much testing is enough?” is “It depends!”
(see “The Complete Guide to Software Testing”).

It depends on risk: the probablity of something going wrong and the impact of it. We should use risk to determine where to place the emphasis when testing by prioritizing our test cases. here's also the risk of over-testing (doing ineffective testing) and leaving the real needed testing behind.

[dialex]

Attachments on https://trello.com/c/cb96u4Xm/232-%E2%9C%85-automation-tester

[dialex]

Test case: Specific, explicit, documented, and largely confirmatory test ideas — like a recipe.

Note: A test case is not a test, any more than a recipe is a meal, or an itinerary is a trip. Open your mind to the fact that heavily scripted test cases do not add the value you think they do. If you are reading acceptance criteria, and writing test cases based on that, you are short-circuiting the real testing process and are going to miss an incredible amount of product risks that may matter to your client. More on the value (or lack thereof) of test cases here: http://www.developsense.com/blog/2017/01/drop-the-crutches/

[dialex]

As an industry, we are obsessed with automation for all the wrong reasons. The view that we can take a complex cognitive activity and distil it into code is a fallacy which results in both bad testing and bad automation. To be successful with automation we need to think deeply about what we do in testing as well as what we can do with automation. This has been my feeling for most of my testing career.

"Automation in Testing" vs "Test Automation"

http://www.mwtestconsultancy.co.uk/automation-in-testing/

---

AUTOMATION THAT SUPPORTS TESTING, and not TESTING AUTOMATED
https://automationintesting.com/

[dialex]

There is a pattern I see with many clients, often enough that I sought out a word to describe it: Manumation, A sort of well-meaning automation that usually requires frequent, extensive and expensive intervention to keep it 'working'.

You have probably seen it, the build server that needs a prod and a restart 'when things get a bit busy'. Or a deployment tool that, 'gets confused' and a 'test suite' that just needs another run or three.

Did it free up time for finding the important bugs? Or are you now finding the real bugs in the test automation, while the software your product owner is paying for is hobbling along slowly and expensively to production?

FROM: http://www.investigatingsoftware.co.uk/2018/02/manumation-worst-best-practice.html

[dialex]

https://twitter.com/FriendlyTester/status/965508649559961601

[dialex]

A test script will check if what was expected and known to be true, still is.

[dialex]

https://club.ministryoftesting.com/t/how-do-you-create-reliable-automated-tests/32016

[dialex]

https://madeintandem.com/blog/five-factor-testing/

Good tests can…

1. Verify the code is working correctly
2. Prevent future regressions
3. Document the code’s behavior
4. Provide design guidance
5. Support refactoring