UI Automation Out of Control

From：http://blogs.msdn.com/imtesty/archive/2009/08/01/automation.aspx

• Not all automated UI tests save time!
  Tests that require constant massaging and tweaking because they constantly throw false positives take up a huge amount of a tester’s time in wasted maintenance.
• Sometimes a human is a more efficient oracle than a computer algorithm!
  Sure, just about anything a computer does can be automated to some degree in some fashion, but there really are clearly some tests where it is more prudent and simpler to rely on a tester.
• Don’t rely on automation to emulate your customers!
  Test automation does not effectively emulate a human user. Sure, we have test methods in some of our internal automation frameworks to slow down simulated keystrokes (the actual keys are not being pressed on the keyboard), or simulate multiple or repeated clicks on a control or the mouse, and other tricks that try to emulate various user behaviors; however, test automation is generally poor at detecting behavioral issues such as usability, ease of use, or other customer value type assessments. Rely on the feedback from internal and external customers who are dog-fooding, self-hosting, and beta-testing your product (and act on it).
• Go under the covers!
  I think many testers rely too heavily on UI automation because they think it emulates user behavior (although most things such as populating a text box are simulated via Windows APIs), or perhaps because they don’t know how to dig into the product below the surface of the UI. Whatever the case, think about the specific purpose of the test. If it is easier to check a return value, or call an API to change a setting then go deep…and stop messing around on the surface. (It only complicates the test, wastes valuable machine cycles, reduces reuse across multiple versions, and often leads to long term maintenance costs. (For an example of this see my previous post.)
• Constantly massaging code contributes to false negatives!
  I have seen many cases where a tester designs a a UI automated test, and then tweaks a bit here and there to get it to run. Often times this tweaking contributes to a tests ineffectiveness in exposing problems, and may even hide other problems. Also, some tweaks are geared around synchronization issues (sync’ing the automated test with the system under test) and involve artificially slowing down the automation (usually by stopping or ‘sleeping’ the automated test process for a specific period of time). Other tweaks might hard-code parameters that then make the test fail on a different resolution or non-portable across different environments.
• STOP trying to automate every damn test!
  As I stated before…just because we can automate something doesn’t mean that we should try to automate everything! We need to make rational decisions about what tests to automate, and what is the best approach to automating that test.