Goals

  • Ensure that the product reaches the expected performance (throughput &
    response times notably)

  • Ensure that performance does not regress as updates(QFEs & service
    packs) are added

  • Understand where the bottlenecks in terms of performance are located in
    order to resolve them when time permits

  • Provide a way for customers and services teams to evaluate HW needs based on
    performance requirements (benchmarks)

Non-goals



  • Test whether performance increases as HW is added, that is part
    of scalability testing

  • Test whether the system stays stable for days, that is part of reliability
    testing

Tasks & ownership


Performance testing is divided between a central performance team (alias:
amlperf) and
individual feature owners


Individual feature owner



  • Ensure feature performance meets requirements set forth in spec

  • Ensure feature performance does not regress

  • Find, understand and investigate bottlenecks in feature

Central performance team



  • Ensure overall integrated product performance meets requirements

  • Ensure overall integrated product performance does not regress

  • Find, understand and investigate bottlenecks in the integrated product

  • Oversee design & development of the product to ensure it will be able to
    meet performance requirements

  • Support individual feature owners as they build their feature performance
    tests by


    • reviewing their test plans

    • reviewing feature performance requirements

    • providing tools to help feature owners create the perf tests

    • providing consulting role to feature owners to help them create the perf
      tests and understand the results

  • Provide externally consumable benchmarks & best-practices document for
    capacity planning

How to test performance of a feature



  • Ensure you have clear performance requirements defined. You need to know
    what to measure, under what circumstances, and where the line between PASS and
    FAIL is.

  • Define what pieces of the distributed system you need


    • Most probably you don't need an entire setup.

    • Of the pieces you need, some might not need to be setup as per the
      recommended topology because they're used only in the beginning or very
      infrequently, so you could get away with putting all those on one VM for
      example.

  • Define the HW you need, not very precisely what that HW is (# of cores, RAM,
    type of storage used, settings used for IIS, the OS, Amalga, ...)


    • You need to ensure that all runs are done against the exact same setup to
      allow comparison in future regression passes, so the exact setup must be
      known

    • If your setup calls a database frequently, you'll need to use a physical
      database server backed up by a SAN for storage. The Performance team can help
      you procure such a system

  • Define the configuration to use for your test (ACF rules, events for
    eventing, etc...) so that it's realistic and representative of a customer use
    (the requirements should be defined based on such a configuration)

  • Use, or write, a tool to exercise your feature. You need to ensure this tool
    is highly efficient and isn't in itself a bottleneck. You don't want to get poor
    perf numbers because that tool sends requests too slowly, takes too long to read
    responses from the server, etc...


    • Ask the performance team about what tools to use, to review the tools you
      will use or the tools you write

  • Measuring the performance should be done through performance counters or
    otherwise PLI. If your feature doesn't expose the values you need through these
    mechanisms, file a bug to have the dev correct this. The performance teams
    provides you with a framework to simplify measuring of the performance of the
    system as well as reporting and analysis of trends, see the Tools section for
    its location and how to use it.

  • Your tool should ideally gradually increase the load on the system instead
    of immediately flooding it. Think of it as gradual steps of at least a minute
    each. In order to control the load you can use a framework (Distribution Engine)
    built by the performance team that will allow you to precisely control the rate
    at which requests/messages/... are sent, see the Tools section for its location
    and how to use it.

ToDo & DontDo list

  • Don't change hardware or software configuration between runs, or you will
    not be able to compare your results (or worse: mistakenly miss a
    regression)

  • Don't run your tests on random VMs from the operations or test services
    teams, those VMs are shared with many other VMs on the same hosts and their
    performance can vary

  • Always use a physical machine for the databases