I: Continuous Integration Testing
Continuous Integration (CI) is a development practice which aims to ensure the correctness of software under development. After each code commit, a suite of tests run automatically to ensure the software is running without any defects. In an event of Test Case failure, immediate feedback is generated apprising about the broken build. Simply put, Continuous Integration is a process of verifying the correctness of a software while it’s still being developed. Some of the Continuous Integration tools are Jenkins, TeamCity, Bamboo, Travis, CircleCi, Bitbucket, and CruiseControl. This article is focused on the Jenkins scenario, but the techniques described carry over to the other CI tools quite directly.
It makes sense that Continuous Testing goes hand-in-hand with Continuous Integration to make sure that bugs are found early and are easy to fix. While many changes are made on a daily basis, there’s an opportunity each time for those changes to disrupt a previously working part of the code.
As more and more teams implement Continuous Integration and Delivery to gain a competitive edge, it’s important to acknowledge that ensuring quality is equally important in the eyes of the end user — no one wants to use a new feature that introduces a bug. By automating the right tests to match the speed of Continuous Integration, rapid delivery can be more effectively achieved and acceptable standards of quality can be met simultaneously.
II: Continuous Deployment Testing
Continuous Integration is the practice of testing each change done to your codebase automatically and as early as possible. Continuous Deployment follows the testing that happens during Continuous Integration and pushes changes to a staging or production system.
Continuous Deployment promotes an overall higher level of product quality than that of traditional QA testing workflows. Continuous Deployment Testing encourages developers to take ownership and stake in the end user experience and the quality of the features they put out. Continuous Deployment lays a framework that makes it easier for company-wide focus and discussion on release quality.
Implementing a robust software testing strategy is the foundation of Continuous Deployment, and automation is the key to a successful continuous delivery pipeline. Some of the Continuous Deployment tools are uDeploy, Octopus Deploy, Chef, TeamCity, Bamboo, Travis.
III: Using Cyara to Automate Testing during CI and CD
In response to either new code committed by a software developer or new code pushed to production, a rapid and deterministic execution of a battery of tests quickly evaluates a situation of uncertainty and provides a binary pass/fail result on which to act. This is an ideal application of the Cyara Platform, in which various Campaigns can be set up and managed for your various CI and CD scenarios.
The diagram below depicts how this can be accomplished
IV: A CI Example: Jenkins-Cyara Integration
Jenkins is the leading CI automation server with broad adoption throughout software development in nearly every industry. Jenkins provides Continuous Integration and Continuous Deployment service for software development. It extends support to automate all sorts of tasks related to building, deploying and automating any project. It is used for automatic test executions and scheduled builds. Using Jenkins, Cyara customers publish results and send email notifications to the appropriate team members.
Together, Jenkins and Cyara can be used to validate systems that deliver customer interactions via multiple channels. Integrating Cyara’s CX assurance platform with Jenkins has many advantages and it permits testers to continuously test and integrate source code advancements in the software development process. An added advantage of Jenkins and Cyara CX assurance platform integration is that it automates the code progression from the development environment to the testing, staging and production environments.
In addition to the traditional practice of automating a small collection of unit and component tests, Jenkins uses Cyara to provide additional levels of assurance to validate an entire end-to-end customer service environment with integration testing driven by real-life customer and agent interactions. Automation of SIT/UAT tests compliment unit tests for components and can find common classes of defects in the complicated interactions between components. Cyara judges the correctness of code and configuration changes from the outside in, looking through the customer’s perspective.
Cyara provides a set of RESTful APIs for performing the testing and validating the CX platform under development. There are two versions of REST API offerings available – a legacy v2.5API and a more recent v3.0 API. This document will mainly refer to the recommended API v3.0 approach, but for those customers running their own, older version of the Cyara Platform, there is an appendix describing the v2.5 API approach to doing CI/CD integrations.
Calling Cyara’s REST API endpoints, you can execute test cases and gather the execution results.
Here is an example of using the Cyara platform for continuous testing during Continuous Integration and Continuous Delivery:
Some assumptions made in this example use of Jenkins and Cyara for CI:
- A Jenkins tool will be used to orchestrate the workflow.
- The Test Cases to validate the check-in and build in the test environment already exist and are included in a Campaign.
- The Cyara Platform API is reachable via HTTP/HTTPS from your instance of Jenkins.
- Jenkins can connect to Cyara API using authorization.
- The User is attached to an Account that owns the Test Cases and CI Campaign and has the Administrator Role.
- The Jenkins execution environment will manage concurrency. Anything that attempts to modify or run the Campaign used by this process, while this process is running, will cause indeterminate behavior. This can include executing this script multiple times in parallel and running / updating the Campaign manually in the Cyara Portal UI while this script is running.
- There is an important distinction between Campaigns and Campaign Runs. Campaign Runs are execution instances of a Campaign. Campaign IDs and Campaign Run IDs are the IDs for their respective entities.
- In our example below, we’re passing two Campaign IDs in to our example script, one to execute the Outbound Campaign that will receive the calls (as a stand in for an IVR), and one to execute the Inbound Campaign that we’ll be reporting on. The script only retrieves results from the Campaign Run for the last Campaign ID passed in; it’s implicit that the last Campaign ID is the Campaign we’re gating our deployment on.
V: Configuring Cyara Campaigns for CI/CD
Here are a few preparatory steps for CI/CD Testing using Cyara APIs:
- Login to Cyara portal
- Create or identify any specific test cases that you may need to execute in CI/CD
- Create the campaigns and associate the identified test cases in the step above
Navigate to Campaign Details page and note down the campaign Id as per the screenshot below:
- Once you are done with the steps above, you are good to call the APIs with campaign ID.
Cyara APIs are easily integrated into your CI/CD toolchain to provide a comprehensive set of tests for Customer Experience validation. Customers using a Continuous Integration and Continuous Deployment pipeline can have their Customer Experience certified by making calls to the Cyara APIs to ensure their code check-ins aren’t causing failure and the latest deployed build is stable. Cyara APIs report failure to the CI/CD toolchain can further propagate the failure to the concerned developer/team. This testing mechanism should not be the same as a full regression test but instead it is a quick test that would turn around the results in less than 20 mins. The CI/CD Testing using Cyara APIs is not limited to any specific toolchain and can be applied to any CI/CD toolchain that can invoke Cyara REST APIs.