How Do I Set Rollback Thresholds for a Shared CPU Test?

Running cost optimization pilots on always-on small services that use shared CPU instances is notoriously tricky. These services often hide cloud waste because their apparent average CPU utilization looks low and benign. Yet beneath the averages, the real performance constraints and risks live in peak CPU usage and occasional spikes that impact your service objectives. When testing shared CPU instances—whether on AWS, Click for more info Azure, or other clouds—you must design rollback criteria based on meaningful peaks, backed by appropriate percentile statistics and spike duration, rather than averages or oversimplified CPU counts.

In this post, I’ll share what actually works for setting rollback thresholds during a shared CPU instance pilot test, referencing tools like AWS Compute Optimizer and Azure Advisor, as well as the quirks of different cloud providers’ shared CPU definitions.

Why Shared CPU Pilots Are Risky Without Proper Rollback Criteria

Developers and infrastructure teams often want to optimize costs by moving always-on small services from dedicated vCPU instances to shared CPU (burstable or t-series style) instances. Unfortunately, these pilots frequently fail or result in degraded service because the pilot was based on average CPU measurements or incorrect assumptions about performance guarantees.

• Always-on small services hide cloud waste: Their average CPU usage can be deceptively low, leading to a false sense of safety.
• Shared CPU definitions differ by cloud provider: What “burstable” means on AWS differs from Azure and Google Cloud, affecting how and when you might see CPU throttling.
• Average CPU is a lie: Relying on average utilization risks missing occasional CPU spikes that violate your service objectives.

Setting rollback thresholds requires engineering rigor: understanding peak utilization percentiles, spike durations, and precise service objectives.

Understanding Shared CPU Models Across Providers

Before diving into metrics and rollback criteria, it is crucial to understand the shared CPU model variations:

Cloud Provider Instance Type Shared/ Burstable CPU Model Performance Scaling Basis AWS T3, T4g, T4i CPU credits accumulated when under baseline, consumed when bursting above Credits & baseline utilization % Azure B-series CPU credits that accumulate and deplete, limits set per instance size CPU credits & burstable vCPU performance Google Cloud E2, N2D Shared-CPU Shared physical cores with limited guaranteed CPU allocation vCPU & physical core sharing https://bizzmarkblog.com/are-bots-and-internal-services-good-on-shared-cpu-if-concurrency-is-low/

Despite similarities, the CPU borrowing and bursting mechanisms differ. For example, AWS’s T-series uses a CPU credit system that allows sustained baseline CPU, plus bursts consuming credits. Azure’s B-series follow a similar credit approach but have different baseline and burst rates. Google’s E2 shared-core VMs do not grant CPU credits but instead share physical cores, resulting in varied contention profiles.

Step 1: Define Your Service Objectives Before the Pilot

Before touching instance types or running a shared CPU pilot, always define critical service objectives for your small always-on service. Your rollback thresholds must be based on real business needs and technical SLIs rather than cost alone.

1. Identify Key SLIs: For example, 99th percentile response latency, error rate, or system throughput.
2. Establish SLOs and Error Budgets: For instance, 99.9% successful requests with latency under 150ms.
3. Gather Historic Performance Data: Measure and analyze the P95 and P99 CPU usage and latency from the current environment.

Do not rely on averages or max CPU alone! Always ask: what do the P95 or P99 CPU and latency metrics look like under typical and peak load?

Step 2: Choose the Right Observation Window for Peak Measurement

When measuring CPU peaks to inform rollback thresholds, the size of your observation window dramatically impacts your view of burstiness and contention.

• Short windows (e.g., 1-minute averages): Capture transient bursts which matter for latency-sensitive services.
• Medium windows (5-15 mins): Show sustained CPU pressure that can deplete burst credits.
• Long windows (30 mins+): Smooth out spikes and risk hiding issues.

Use multiple windows concurrently. For example, AWS CloudWatch’s default 1-minute CPU metrics plus 5-minute derived metrics provide contrasting insights. Your rollback criteria should monitor P95 or P99 CPU usage over both 1 and 5 minute windows to detect spikes and sustained contention.

Example:

If your P99 CPU for 1-minute average is 40% but the 5-minute P99 climbs above 70%, your service may be experiencing sustained CPU bursts that shared models might not support well.

Step 3: Analyze Percentiles and Spike Duration Instead of Averages

Average CPU does not account for burstiness and may underestimate the risk of throttling or contention in shared CPU environments.

Metric What It Reveals Why It Matters for Shared CPU Average CPU Utilization Smoothes out usage over time Can hide short spikes causing throttling Percentiles (P95, P99) Captures heavy utilization tail Reflects true peak demand impacting latency and throttling Spike Duration (seconds/minutes over threshold) Highlights how long CPU stays above a critical level Indicates possible CPU starvation or credit depletion

For rollback criteria, your pilot guardrails should specify limits such as:

• P95 CPU usage less than 80% on 1-minute window
• Number of spikes > 90% CPU lasting longer than 15 seconds does not exceed X times per hour
• Latency P99 below your SLO threshold without significant increase compared to baseline

Step 4: Incorporate Cloud Provider Tools to Inform Your Pilot

Cloud-native tools like AWS Compute Optimizer and Azure Advisor provide valuable insights but must be interpreted carefully.

• AWS Compute Optimizer: Analyzes historical usage to recommend instance types and sizes—pay attention to how it treats burstable instances and whether it flags potential CPU credit depletion risks.
• Azure Advisor: Offers right-sizing recommendations and burstable instance suitability, but review the underlying assumptions, especially CPU credit usage patterns.

Both tools provide baseline utilization metrics; however, do not blindly trust their average CPU recommendations. Combine these insights with your percentile and spike duration analysis and existing SLIs to finalize rollback criteria.

Step 5: Define Clear, Measurable Rollback Criteria Before Your Pilot

Before starting your shared CPU pilot, write down explicit, quantifiable rollback conditions. This guardrails your pilot and avoids guessing whether performance problems are due to the instance type or other unrelated causes.

Sample rollback criteria example:

If during the pilot: - P99 CPU utilization > 85% for more than 5% of 1-minute intervals per hour, OR - Number of CPU spikes > 90% lasting > 30 seconds exceeds 10 per hour, OR - P99 request latency exceeds SLO threshold by more than 20% for > 10 minutes total in any hour, THEN rollback to dedicated vCPU instances.

Setting these guardrails lets your team quickly identify problematic performance and safely abort the pilot before service impact grows.

Summary Checklist for Setting Rollback Thresholds

1. Always define service objectives (SLIs/SLOs) before touching instance types.
2. Understand shared CPU model differences between cloud providers.
3. Measure CPU using appropriate percentiles (P95, P99) rather than averages.
4. Use multiple observation windows to capture transient spikes and sustained bursts.
5. Incorporate spike duration metrics to evaluate CPU contention impact.
6. Use AWS Compute Optimizer and Azure Advisor as inputs, not gospel.
7. Write explicit rollback criteria before pilot start.
digitalocean droplets
8. Continuously monitor your SLIs during the pilot and act fast if thresholds are breached.

Conclusion

Optimizing for cost by using shared CPU instances for always-on small services is tempting but fraught with hidden risks. The key to a successful pilot—and avoiding wasted time and cloud spend—is to define clear, measurable rollback thresholds that focus on P95/P99 CPU utilization and spike duration rather than averages. Leverage cloud tools like AWS Compute Optimizer and Azure Advisor as advisory inputs, but trust your own detailed performance observation over realistic windows to safeguard your service objectives.

Remember: always ask what the peak P95 and P99 CPU usage and latency look like before switching instance types. Doing so sets you up for a smooth pilot, confident cost savings, and reliable application uptime.