A bank needs to process transactions in more than two million checking accounts. A large company must calculate payroll for 500,000 employees overnight. A pharmaceutical company is legally obliged to retain data for several years, blowing the database up beyond 10,000 gigabytes. The type and scale of business needs are as diverse as companies themselves. But they all expect their SAP solutions to function reliably and predictably, especially during phases of extremely high load.
To test and prove the scalability of SAP solutions and the hardware they run on, SAP and its hardware and technology partners have collaborated for over 15 years to develop a suite of SAP Standard Application Benchmarks. They use these benchmarks to prove that the particular software and hardware components tested are scalable and can be expanded and reduced – in size, volume or number of concurrent users, for example – and still continue to function properly and reliably.
Business-relevant performance indicators
In the IT industry, benchmarks are standardized load tests that yield key performance figures, which enable a quantitative comparison of the hardware and software being tested. SAP Standard Application Benchmarks specifically measure and report the performance behavior of individual SAP system components, such as database systems or application software, on a particular piece of underlying hardware.
What differentiates SAP Standard Application Benchmarks is that they combine the measurement of system performance with a real-life business application that is productively used in customer implementations. By executing actual business processes, they render application-specific and business-relevant performance indicators, for example the number of users that can work simultaneously in the system, navigation steps or user interactions per hour, or business throughput figures such as fully processed order line items per hour.
Customers benefit from these strong correlations of benchmarks with customer implementations. By analyzing benchmarking results, available at www.sap.com/benchmark, they can anticipate how a particular hardware and software configuration may behave under high load. Since benchmarks are also the basis for predicting hardware resource requirements – a process called hardware sizing – they help customers define a system configuration suitable for their particular business environment.
Standardized benchmarks for reproducible results
Since the early 1990s, SAP had been running in-house load tests in cooperation with their hardware and technology partner competence centers. The goal was to gain insight into the performance behavior across applications, releases, and hardware platforms. Very quickly, it was realized that to ensure reliable and irreproachable results – and to enhance the quality of the comparisons – these load tests had to be standardized.
SAP and its hardware and technology partners developed the concept of SAP Standard Application Benchmarks, along with a defined, standard process for those benchmarks. This process ensured that the benchmarks were portable across multiple platforms and operating systems, and that they generated reproducible and publishable results, the integrity of which is beyond doubt. SAP and its hardware and technology partners prescribed clear definitions and rigorous criteria for permissible configuration and tuning activities. They defined a set of benchmark tools and a standardized benchmark environment. They also established the SAP Benchmark Council, an independent governing body with members from SAP and its hardware and technology partners. This council oversees the certification of existing benchmarks and promotes the development of new benchmarks for emerging technologies.
Chronology of business challenges
The first application benchmark to be formally submitted, certified, and published within this new context was the Sales and Distribution (SD) benchmark in 1995, followed closely by the Financial Accounting (FI) benchmark in the same year. The SD benchmark evolved as one of the most popular, influential, and credible online transaction processing (OLTP) benchmark in the industry. Since then, a host of application benchmarks have joined the ranks to cover the full range of SAP Business Suite, Industry solutions, and SAP NetWeaver platform solutions.
Under the guidance of the SAP Benchmark Council, application benchmarks are defined and developed in response to – or in anticipation of – concrete customer requirements. Or, they’re simply triggered by an industry-wide need to understand and measure emerging high-volume business processes. A glance at the evolution of the benchmarks over the years is like reading a chronology of the driving market forces, business challenges, and hot topics at a particular point in time.
SAP uses the benchmark scripts for quality assurance within its own development community. Because benchmark scripts are stable, they enable comparative evaluation across different platform versions, software releases, or architectures. For example, the SAP Enterprise Portal benchmark, defined in 2006 to answer the growing need for benchmarking results for the SAP NetWeaver Application Server Java, helped SAP understand the behavior of different virtual machines, as well as the effects of Java garbage collection in multi-user environments.
Partners also use the benchmarks in their QA efforts. For example, one major hardware provider runs regression tests with SAP Standard Application Benchmarks for its new servers. The number of benchmarks that partners are certifying each year is steadily on the rise. To date, partners have invested substantial time, effort, and money to submit almost 700 benchmarks, subject them to the scrupulous investigation of the SAP Benchmark Council and its members, and have them certified and published.
Last year partners for the first time ran SD benchmarks on virtualized platforms, thus responding to the interest of customers in employing SAP solutions in virtualized environments. Virtualization helps companies optimize server utilization, cut cost for power, cooling and staffing and reduce overall administration efforts.
In productive systems, the database CPU utilization is the determining factor for throughput and response time; whereas all other performance-relevant components – such as application server CPU time, network bandwidth, main memory, and hard disk – can be configured and extended as needed to meet the performance requirements.
But customers are not interested in the number of parallel users or response times alone. Another important performance metric is throughput per unit of time. SAP Application Benchmarks define throughput numbers in business application terms – such as “fully processed order line items” in the SD benchmark, or “navigation steps per hour” in the BI Data Mart benchmark – and then map them to the most prominent hardware components including CPU, memory, and disk size. The results are presented in SAP Application Performance Standard (SAPS), a hardware-independent unit of measurement.
Balance between continuity and progress
SAP Standard Application Benchmarks have enjoyed a long history of helping IT companies celebrate new performance records and helping customers configure and size SAP business solutions for their productive systems. It’s the foresight, dedication, and cooperation of the members of the SAP Benchmark Council that has kept the suite of SAP Standard Application Benchmarks untainted and reliable. Christian Kowarschick from Fujitsu Siemens Computers, a member of the SAP Benchmark Council since its founding, says, “Straight from the start, the meetings of the Benchmark Council have been fascinating to me because its members – even though they represent companies usually engaged in tough competition – pursue a common goal: to strengthen the importance of the SAP benchmarks as industry-standard benchmarks. Often, opinions on how this goal is to be reached diverge significantly, leading to ardent discussions. Even after 15 years, these discussions have not lost any of their enthusiasm, and I am convinced that it will continue that way in the years to come.”
This year, it is time to honor the members of the SAP Benchmark Council. On March 12, colleagues from more than 15 partner companies followed the invitation of SAP to celebrate the Council’s 15th anniversary. The members of the SAP Benchmark Council have successfully collaborated to make it a steadfast institution in a world of rapid technological change and have been able to strike a pivotal balance between continuity and progress. And if the past is any indication – it will continue to do so going forward.