Robotic Analytical Framework: the gateway to fully automated performance testing

In case you haven’t heard, we are entering a post-digital era where companies are expected to not just have an online presence, but excellent digital performance. Now more than ever, business takes place in the fast-paced virtual world as an increasing number of companies have become partially or completely reliant on their digital channels to help them gain and keep customers.
article-5092-robotic-analytical-framework-performance-testing-header-v1.jpg

What's more, great digital performance not only grows your customer base but also helps to boost your revenue. Our Robotic Analytical Framework (RAF) is built to meet this need.

However, with all this increased online competition in this post-digital era, the bar has been raised on the performance and stability of digital platforms. Customers no longer accept slow loading times, error pages, or anything less than a stellar digital experience.

Customers no longer accept [...] anything less than a stellar digital experience

This demand for high performance has facilitated a shift to an agile way of working, emphasizing continuous improvement and delivery. With increasingly rapid deployments, the ability to fully automate performance testing is the key to secure stability and speed while optimizing your digital channels. 

Making the switch towards an agile workflow 

You may have the most beautifully designed, customer-centered website outfitted with high-quality images and innovative AI driven chatbots, but if it doesn’t perform it won’t deliver and you risk losing customer trust and revenue quickly. 

Today’s customers expect a responsive, high-speed online experience on top of a great platform. The world is moving at breakneck speed and your digital channels need to keep up. There is a continuous focus on performance to constantly improve your digital experience. 

This means that the old waterfall method of delivery just isn’t cutting it anymore. The days where a project would take months or even years to deliver are long gone. Deployment cycles have become shorter and more frequent, demanding a shift towards an agile mindset.

Agile mindset - Robotic Analytical Framework: automated performance testing - Accenture

This constant need for delivery and the efficient breakdown of workflow has given way to a continuous integration and continuous deployment (CI/CD) pipeline. In this way of building, deployment cycles are shortened to a matter of weeks, days, or even hours. With this high velocity of building, automation is the key to keeping a consistent and competitive pace. 

To ensure that what you are deploying will have minimal impact on the current function of your site, testing is important. It’s a crucial way to identify constant areas of improvement as well as prevent outages when you’re expecting a high traffic rate in a short period of time. 

While automated testing is widely implemented, there is still a manual component when it comes to the analysis of results and the decision-making aspect to performance testing. 

RAF—How this unique framework revolutionizes performance testing 

Continuous automated performance testing is nothing new. However, manual analysis of test results—when deciding to PASS or FAIL a build based on the outcome of these tests—has been a roadblock when it comes to keeping pace. 

Among our clients, we've seen that this manual intervention slowed down the automated process, impeding the agile workflow. This is why our engineering team designed a unique, fully automated performance testing solution we refer to as the Robotic Analytical Framework (RAF)

RAF is an innovative framework that fully automates performance testing in a CI/CD pipeline. The framework offers increased speed, accuracy and freedom to further improve your digital channels. 

RAF is an innovative framework that fully automates performance testing in a CI/CD pipeline and offers increased speed, accuracy and freedom

Using a Smart Analytics Engine and machine learning algorithms, RAF provides an accurate, machine generated assessment of the risk that new deployed features pose to the stability and performance of your digital channels. 

This analysis is based on every single individual response time measurement (also referred to as raw data), not on averages or aggregated data. This helps reveal system bottlenecks that aggregate data fails to show.

The framework combines this raw data with throughput, error rate, and average response time to give a comprehensive Test Execution Score (TES). The TES determines the status of a test divided into three basic categories: PASS, FAIL, or UNDER REVIEW. The higher the score, the closer the test measures the benchmark to the baseline.

Test and learn - Robotic Analytical Framework: automated performance testing - Accenture

RAF uses machine learning instead of static manual analysis meaning the baseline is dynamic in nature and adapts to the new information gathered from each test. The more tests are run, the more data is used to determine a baseline and decide which tests pass or fail. 

This means that over time, the framework will perform more and more accurate assessment as the entire library of raw data is utilized to make decisions on subsequent test executions. 

What RAF means for online businesses 

While the concept of RAF may seem good in theory, what measurable results can it actually bring your company? 

Though RAF is designed to work best within an agile framework in combination with any load test tool that exposes the raw date, with some development and adjustments it can provide the following benefits in a number of different contexts: 

  • Fully automate all aspects of performance testing, from execution to analysis and decision -making.
  • Bring performance testing into the CI/CD pipeline so that your team can continuously deliver without risking your application’s performance.
  • Free your performance engineers from repetitive analysis work, so they can focus on more innovative endeavors like risk assessment, test approach, coaching, and designing efficient solutions.
  • Improve the quality and consistency of testing and continually increase accuracy and quality over time.

In the end, the goal of any online business is to make the digital experience for the end user as close to perfect as possible. In order to do this, you need to continuously improve your platform in an agile way. 

To ensure that this continuous improvement doesn’t get in the way of your customers’ experience, automating performance testing to keep pace with your deployment will improve the stability and function of your platform.   

How RAF helped increase our client’s online performance by 500 percent

When a financial client approached us looking to solve their platform performance issues, we saw this as a perfect opportunity to implement RAF to help improve their online function. 

During their seasonal peaks around January and April, their core system experienced outages and poor performance due to overloading. That's why their digital channels were often rendered useless, creating dissatisfied customers and a high volume of calls to the call center. No surprise here that 61 percent of the customers switch companies due to poor service.

Already in the process of moving from the traditional waterfall to an agile delivery method, the conditions were perfect to implement the RAF within the agile switch. The goal at the outset was to focus on optimizing and improving their digital performance to the point that there was no change in performance regardless of load. 

Once the framework was in place, it gave them the ability to automatically compare and optimize their executions. We integrated the RAF with Application Performance Monitoring (APM) tooling to fine-tune and optimize their application performance. 

Essentially, the framework allowed them to implement the right tools in the right context and the result was huge—during the following seasonal peak, their application performance improved 500 percent with no downtime.

During the following seasonal peak, the client's application performance improved 500 percent with no downtime

Stabilize and improve your online performance with RAF

RAF is an innovative new framework with the potential to vastly improve the way your company performs online. In a world that demands ever increasing speed and accuracy when it comes to delivering an exceptional customer experience, this framework allows you to automate performance testing and decision making so you can deploy new business features at breakneck pace without risking your platform performance. 

By implementing a machine learning component, our framework has the unique ability to make more accurate assessments over time leading to the continuous optimization and improvement of your digital channels.

Interested in learning more about RAF and how you can stabilize and improve your online performance? We are here to help!

Author: Stijn Schepers