Best Practices for Monitoring YugabyteDB

Operational visibility is key to understanding your database across dimensions of uptime, performance, and capacity planning

Valerie Parham-Thompson
Database Engineering

At DSS 2021, I provided a comprehensive orientation to monitoring YugabyteDB, focusing on how to interpret and leverage built-in metrics for operational visibility.

Three key dimensions of database monitoring:

  • Uptime (Alerting): Ensuring the system is running and healthy through critical alerts.
  • Performance (Trending): Tracking historical performance to detect changes and optimize queries.
  • Capacity Planning (Forecasting): Using metrics to project future infrastructure needs based on current utilization and expected growth.

Types of monitoring:

  • Alerting (supports uptime): Immediate notification of urgent issues via health checks and email/pager integration.
  • Historical Trending (supports performance): Dashboards for tracking performance over time.
  • Forecasting (supports capacity planning): Using trends to anticipate future resource requirements.

Metrics and dashboards:

  • Platform Health Check: Highlights urgent issues (e.g., node failures, time synchronization, disk utilization).
  • Platform Metrics Dashboard: Provides detailed performance insights, including operation counts and latency for both YSQL and YCQL APIs.
  • Command Line Exports: All metrics can be exported for integration with external monitoring tools like Prometheus.

Critical metrics to monitor:

  • System Level:

    • Remote bootstrap events (unexpected cluster changes)
    • NTP server status and clock skew (time synchronization)
    • Disk utilization (avoid performance degradation)

  • Application Level:

    • Overall and per-operation latency (read/write)
    • Operation counts (detect connectivity or workload issues)

  • Database Internals:

    • Reactor delays (system overload indicator)
    • Write-ahead log (WAL) performance
    • RPC queues (detect bottlenecks and need for tuning)

Best practices:

  • Drill down to per-node metrics for troubleshooting (e.g., identifying which node has clock skew).
  • Use latency and queue metrics as early warning signs of system stress.
  • Monitor CPU and memory usage at both node and process levels, focusing on application-impacting metrics like latency rather than raw resource consumption.
  • Regularly review and tune alert thresholds based on workload and environment specifics[1].

See the video here: https://www.youtube.com/watch?v=7aPG4R3K7is