Optimizing Vault Performance: Insights from Real-World Benchmarking

Introduction to Vault's Performance Dynamics

As organizations expand their use of Vault for managing secrets, certificates, and authentication, it becomes essential to understand how the platform performs under varying loads. Initially, deployments often start modestly, but as systems increasingly depend on Vault, concurrency levels rise, making it vital to comprehend Vault's behavior under these conditions.

In this analysis, we scrutinized Vault's performance through a series of benchmarks simulating real-world scenarios encompassing key areas like the KV, SSH, and PKI secrets engines. This paved the way for insights into planning and tuning Vault deployments effectively.

Benchmarking Methodology

All tests leveraged the HashiCorp Validated Design (HVD) specifically tailored for AWS. Vault Enterprise (v1.17.3+ent) was activated with integrated Raft storage located in the AWS us-west-2 region. Testing conditions were created using the k6 load testing tool, while system performance metrics were channeled to Datadog for insights.

Tests employed a ramp-up approach that systematically increased virtual users (VUs), starting from one user and peaking at 500 for KV workloads. SSH and PKI tests varied from 1 to around 200-300 VUs based on the specific operation being assessed. The simulated workloads mirrored common patterns encountered in production settings.

Performance Insights from the Benchmarks

The following findings encapsulate the significant performance observations made during the benchmarks:

The metrics we gathered refer to key performance thresholds termed as the knee point, saturation point, and failure point. The knee point signifies when latency begins to exceed throughput growth, the saturation point indicates a plateau in throughput despite rising concurrency, and the failure point marks instances when requests start timing out.

1. KV Reads vs. Writes

KV engine benchmarks highlighted a persistent disparity between read and write operations. Throughput for read operations consistently surpassed that of writes, exhibiting enhanced stability as concurrency increased. In fact, reads were approximated to be 2.3 times quicker than writes, demonstrating resilience even at higher levels of load. In contrast, write operations faced stress more rapidly, being the first to encounter failures under significant load.

Payload size accentuated this difference, particularly with large secrets. For instance, KV2 engine write operations began failing with about 50 to 100 VUs for 1 MB secrets, while read operations continued to thrive within higher concurrency levels. Consequently, this behavior illustrates an operational facet of Vault: write-heavy loads exert greater pressure on the Raft storage pathway, limiting scalability.

2. Impact of Payload Size

A clear trend from benchmarks was the considerable impact of payload size on throughput. Smaller secrets showed excellent scalability regardless of the concurrency level, exemplified by 1 KB reads, which maintained high performance at 500 VUs. On the flip side, larger payloads, such as those at 100 KB and 1 MB, faced knee and saturation points much earlier, inhibiting the number of concurrent operations the cluster could support.

This decline in throughput with increased payload size stems from the heightened cost of processing and storing larger data chunks. Bigger secrets amplify disk write activity and escalate the Raft replication demands, straining the storage subsystem. Moreover, the introduction of KV2 involved metadata overhead that could further hinder throughput.

3. Resource Utilization Dynamics

Resource metrics during KV benchmarks painted a vivid picture of performance. Memory usage surged as concurrency increased, hitting 100% utilization during scenarios with 500 VUs, indicating memory saturation as the primary limiting factor. In contrast, CPU utilization remained relatively low, generally between 20% and 40%, suggesting that compute capacity was not the main restriction.

Disk performance contributed significantly, particularly during large payload writes, which correlatively increased tail latency. Since Vault utilizes Raft integrated storage, write operations must be stored on disk to be considered successful, making disk performance critical for overall efficiency.

4. SSH Workload Scalability with ED25519

When examining SSH secrets engine benchmarks, a comparison between RSA-2048 and ED25519 cryptographic algorithms revealed significant performance differences. RSA-2048 operations faced knee points at around 100 VUs, while ED25519 doubled this capacity, sustaining operations at about 200 VUs before latency increases were noted. Even in saturation conditions, ED25519 exhibited resilience, maintaining throughput without hitting full saturation as rapidly as RSA.

Notably, the benchmarks demonstrated that ED25519 workloads remained considerably more resource-efficient, as CPU usage peaked lower in comparison to RSA workloads, indicating it as a more effective choice for high-concurrency SSH signing needs.

5. The Latency Constraints of PKI Workloads

The PKI engine's operations proved to be more computationally demanding, particularly during certificate generation and revocation. The benchmarks revealed that latency began significantly increasing at around 25 concurrent users. Within this sample, throughput reached a plateau while latency escalated sharply.

With about 560 milliseconds being the baseline latency for issuing a certificate with a single virtual user, comparisons across cryptographic algorithms showcased minimal discrepancies. Still, as concurrency surpassed 100 users, several operations began to fail, highlighting that PKI workloads faced stricter latency limits under concurrent demands compared to KV workloads.

Operational Recommendations for Vault Teams

Based on the insights gathered, several operational recommendations emerge for teams managing Vault in high-concurrency environments:

1. Prioritize Small Secret Payloads

Keeping secret sizes below 100 KB proved beneficial to scalability in KV benchmarks. Although these tests were conducted with audit logging disabled, organizations should account for this overhead in production scenarios, as it may amplify the impact of larger payloads.

2. Optimize Storage Solutions

Given that write-heavy workload performance is deeply intertwined with disk efficacy, selecting storage solutions with high write throughput and optimized fsync capabilities—like cutting-edge SSDs or NVMe drives—will help minimize latency.

3. Scale Memory Appropriately

The significant memory utilization observed at high concurrency levels confirms that upgrading memory can enhance stability when dealing with numerous concurrent requests. Anticipating these scaling needs is vital for uptime and user experience.

4. Choose Efficient Cryptographic Algorithms for SSH

Given the demonstrated advantages in concurrency handling, opting for ED25519 over RSA-2048 becomes crucial in environments reliant on SSH certificates to maximize performance.

Final Thoughts

Vault is equipped to service demanding workloads, yet its deployment success is highly contingent upon understanding the interplay between workload types, concurrency, and infrastructure. For platform teams, insights from this analysis emphasize the relevance of considering not just traditional metrics like CPU and memory but also concurrency patterns and payload sizes for effective capacity planning.

Comprehending these dynamics enables teams to architect resilient Vault deployments that can scale appropriately alongside increasing infrastructure demands.

Further Reading

To dive deeper into tuning Vault for optimal performance, including operating system configurations and storage best practices, refer to the official Vault documentation.