TPC Benchmark

Introduction

When comparing different database management systems, it’s important to use a consistent and trustworthy method to evaluate their performance. TPC benchmark helps evaluate how well different database systems perform and grow in size. This article will cover the basics of TPC benchmarks. It will explain why they are important and how they can help organizations choose a DBMS wisely.

What is the Transaction Processing Performance Council (TPC)?

The Transaction Processing Performance Council (TPC) is a non-profit organization founded in 1988. Our goal is to create standards for transaction processing and database benchmarks, and provide accurate performance data to the industry. The TPC consists of major computer and database vendors, leading industry experts, and academic representatives who collaborate to develop industry-standard benchmarks.

Understanding Database Management System Benchmarks

A DBMS benchmark is a test to measure how well a database management system performs under certain workload conditions. These benchmarks mimic real-life situations and give measurements like transactions per second, response time, and price/performance ratio. By running these benchmarks, organizations can compare different database systems and make informed decisions based on their specific requirements.

Types of TPC Benchmarks

The TPC offers several benchmarks tailored to different workload scenarios:

1. TPC-C: This benchmark simulates an online transaction processing (OLTP) environment, such as order entry and processing systems. It calculates how many orders are completed in a minute and gives data on transactions per minute and cost efficiency.
2. TPC-H is for decision support systems. It tests how well complex queries perform with big data. It measures the power and throughput of a system, providing metrics like queries per hour (QphH) and price/performance ($/QphH).
3. TPC-DS is a benchmark similar to TPC-H. It models aspects of a decision support system, including queries and data maintenance. It provides metrics such as queries per hour (QphDS) and price/performance ($/QphDS).

When to Use TPC Benchmarks

TPC benchmarks are valuable tools in various scenarios:

1. Assessing new database systems can be done by running TPC benchmarks. These benchmarks help evaluate the performance and scalability of a new DBMS. They also allow for comparison with other options available in the market.
2. Comparing different configurations: TPC benchmarks help assess how hardware and software changes affect database performance. This helps in optimizing the system for specific workloads.
3. Capacity planning includes running TPC benchmarks with varying data sizes and concurrent users. This helps determine the hardware and software resources required for future growth and performance.

Benchmarking NoSQL Databases

Researchers created TPC benchmarks for traditional databases, but they can also be used to assess the speed of NoSQL databases. NoSQL databases are diverse and serve different purposes, so there is no one-size-fits-all benchmark for all systems. Some NoSQL vendors provide their own benchmarks or adapt existing TPC benchmarks to suit their specific database model.

Examples of TPC Benchmark Results

Let’s look at some examples of TPC benchmark results for both non-clustered and clustered databases:

Non-Clustered Database Example

Consider a TPC-C benchmark run on a single-node database server with the following configuration:

• Hardware: 2x Intel Xeon E5-2699 v4 @ 2.20GHz, 512GB RAM
• Database: PostgreSQL 13
• Dataset: 5,000 warehouses

The benchmark results show:

• Transactions per minute (tpmC): 1,500,000
• Price/performance ($/tpmC): $0.50

This shows that the database system can handle 1.5 million transactions every minute for $0.50 each.

Clustered Database Example

Now let’s consider a TPC-H benchmark run on a clustered database with the following setup:

• Hardware: 8x nodes, each with 2x Intel Xeon E5-2699 v4 @ 2.20GHz, 512GB RAM
• Database: Greenplum 6.0
• Dataset: 10 TB

The benchmark results show:

• Power@Size (QphH@10TB): 1,000,000
• Price/performance ($/QphH@10TB): $50

The grouped database system can handle 1 million queries in an hour on a 10 TB dataset. It costs $50 per query per hour in terms of performance.

Conclusion

TPC benchmarks provide a standardized and reliable way to evaluate the performance and scalability of database management systems. Organizations can improve decision-making when selecting a DBMS by understanding various benchmarks and how they meet their requirements. Different types of benchmarks can help organizations evaluate the performance of different DBMS options.

By understanding how benchmarks apply to their specific needs, organizations can make more informed decisions. This knowledge can lead to selecting a DBMS that best fits the organization’s requirements. Whether you’re evaluating a traditional relational database or a NoSQL system, TPC benchmarks offer valuable insights into the performance characteristics of these systems.

To simplify the process of securing and monitoring your databases, consider using DataSunrise. DataSunrise offers user-friendly and flexible tools for database security, audit, and compliance. Our team would be happy to provide you with an online demo to showcase how DataSunrise can help safeguard your database environment.