predictive analysis graph

Database scaling for seasonal increases

Share this post on:

Timing of seasonal demand depends on the industry, but a cyclical increase in traffic applies to all industries. Maybe your cycle is shorter than a full year. Or maybe it’s related to things like weather patterns or fashion.

You know when your business gets the most traffic. 

  • Retail? Black Friday/Cyber Monday.
  • Health and fitness? New Year’s Day.
  • Pizza restaurant? Halloween and Thanksgiving, surprisingly.

You do know your business well. But do you know how to make sure your database infrastructure can keep up with traffic during these busy periods? 

In this post, I’ll share five things to consider as you prepare your database scaling to support temporary increased demand.

Collect previous database traffic

The more data points you have about typical utilization and previous spikes in traffic, the better you can predict what the next spike will look like. This is one very good reason to install database monitoring tools like Prometheus and to keep at least a year of raw metrics. You’ll want to capture at least CPU, memory, disk i/o, and disk space metrics. 

By the way, many metrics collectors will roll up or downsample data… averaging the max values with regular levels can cause you to lose important data for analysis. A database that is useful on average is not a useful database.

Then you can easily judge how close you are regularly to your thresholds — for example, how much disk space are you currently using? — as well as how much you increased utilization during the last spike — for example, did your disk space grow to 2x last December?

Remember that any software, including database software, cannot use 100% of all resources 100% of the time. For example, in some databases, overhead disk space needs to be retained for compaction, so you would not want your disk utilization to be close to 100%. As another example, some database software is generally CPU-bound, so you’ll rarely expend a large percentage of memory resources.

Consider all the bottlenecks

Your database doesn’t exist in a vacuum. It sits on servers, communicates over a network, and might work alongside various caching or queueing technologies. It doesn’t do any good to have spare capacity on the database if your network is saturated.

Just like you’re gathering metrics and understanding utilization on the database, do the same on these adjacent pieces. 

Understand your workload

Measurement of the system resources is only one part of the equation in database scaling. Another part is knowing how many operations you are sending to the system. This is important because it helps you understand increased traffic due to non-cyclical reasons. 

Knowing details about recent or upcoming releases will give you more insight into changes in base workload. Did you just add a new feature? Did this result in a new set of tables, or a list of new queries? How many operations per second did this add to the database? 

And, just as you know your business well, you know your industry well. What trends are affecting changes in workload? Especially focus on any short-term trends that can ramp up traffic at a steeper rate than you’ve seen in the past.

This all assumes the current application performance is meeting requirements. If it’s not, you have an additional problem to solve.

Give yourself runway to scale

Understand how long it takes you to scale your database. On-prem installations that require hardware requisitioning can take weeks to months. Even cloud installations can require budget approvals and time to plan a maintenance period. Also consider any change freeze periods your company enforces. 

It’s not unreasonable to begin planning for December spikes in mid-August.

Run a forecast

Finally, do some calculations. Eyeballing or drawing an average line through the graph can exclude some important outliers. For the kind of capacity planning you need during scaling, max values are very important. You’ll want to take your regular utilization, add in increased operations from feature releases, with overhead to support known spikes last cycle. 

One open-source tool available to do this kind of data analysis and predictive forecasting is Facebook’s Prophet. See the presentation my colleague and I gave at Datastax Accelerate for more details on using Prophet.

Any forecast works best with more data. In this case, a long collection of metrics (see above) will result in a better forecast.

Preparing your database for seasonal demand requires thinking ahead… setting up monitoring to graph your workload, knowing how long it takes to implement scaling, and doing predictive forecasts. As they say, it’s not hard, it’s just work. 

The payoff is immense, though, saving your company time and money spent on emergency remediation. Perhaps most importantly, it keeps your engineers in their happy place, writing code, instead of in sweaty war rooms. 

Let me know if you’d like help in creating a plan for capacity planning in your environment.

Author: Valerie Parham-Thompson

View all posts by Valerie Parham-Thompson >

Leave a Reply

Your email address will not be published. Required fields are marked *