Embrace the Chaos! Exploring the World of Chaos Engineering

For our case study, we chose to apply chaos engineering principles to an internal product, and we selected Azure Chaos Studio as our tool. With a very limited learning curve, we were able to begin targeting resources, selecting from a list of simulation conditions, and creating chaos.

Because it’s best to start simple and work toward increasingly more complex chaos, we started with a straightforward CPU pressure test on one of the VMs where our QObserver process resides. We didn’t impose any additional abnormal conditions on other areas of the architecture (e.g., DB or QCloud). By gradually ramping up the CPU utilization and monitoring system performance, we were able to determine the threshold at which system latency started occurring. We used this information to adjust our autoscaling policies to obtain the optimal balance between cost efficiency and performance/availability.

Although we did not choose to do so for our case study, we could have also refactored areas of the code or architecture to determine if one-time changes (e.g., implementing caches) might allow us to get more processing headroom without adding infrastructure costs that would be incurred month over month, year after year.

Next, we targeted our client DB by slowly ramping up the number of connections until we started seeing errors related to the maximum DB connections. This allowed us to determine the maximum number of concurrent users that our DB could support with the currently allocated resources.

Knowing this, we can now make efficiency changes, such as identifying long-running queries that can be improved to free up DB connections more quickly; or adding small amounts of memory and/or GPU so that the size of the connection pool can be increased. We can tweak the various settings until we find the best “bang for the buck.” Better understanding these limitations of our product also allows our operations team to set thresholds for monitoring and alerting more accurately.