Skip to main content

Welcome to the Cyara Knowledge Center

Search

Entering Voice Test Case Steps

Updated: tech writers Edited:

Each Test Case must have at least one Step added to it. Each Step has the following parameters:

  • Step Number
  • Step Description
  • ‘Expect To Hear’ Prompt
  • Voice Quality (Select from the available voice quality algorithms)
  • Reply With
  • Reply Type
  • Min/Max Pause Time
  • Minor/Major Threshold
  • Major Confidence Level for Multi-Pass Speech Recognition (MPSR)/MOS Voice Quality using PESQ (available only if Mean Opinion Score (MOS) is enabled for Account)
  • Minor Confidence Level for Multi-Pass Speech Recognition (MPSR)/MOS Voice Quality using PESQ (available only if Mean Opinion Score (MOS) is enabled for Account)
  • PSST (Post-Speech Silence Timeout)
  1. To create a New Step, click the New Step button to add Step 0. A dialog box as shown below appears to create a new Step:
  2. Click the Add Step button to add Step 0. Step 0 must be present and is a special case that represents the call connection time (ringing). Step 0 is already present in the New Test Case dialog box filled with default values ready for adding to the Test Case. Some fields for Step 0 are not editable and are grayed out.
  3. Click New Step to add Step 1.
    1. Enter the Step No. The Step number can be adjusted to insert a Step above the existing Steps.
    2. Enter a brief description of the Test Step in the Description field.
      Note: A number of the Reports generated by the Cyara Web Portal are able to summarize Campaign results by ‘Step Description’. It is recommended that you formulate a naming convention for similar Steps across Test Cases (for example, Steps that use the same Contact Center application or backend database). For more information, see the Reporting Chapter.
    3. Select the Voice Quality.
      Select from the available Voice Quality algorithms, MPSR, PESQ or (POLQA if enabled).
      Voice Quality: Allows the user to choose between MPSR (Multi Pass Speech Recognition) and PESQ (Perceptual Evaluation of Speech Quality (ITU-T P.862)). When using MPSR, the prompt is entered in plain text, whereas PESQ allows the user to upload a reference audio file to use for PESQ analysis. For more information on PESQ, see the PESQ/MOS Guidelines section.
      When either POLQA or PESQ is selected as the Voice Quality algorithm to be used, both of these algorithms will reference an audio file to perform the analysis.

      Note: to enable the POLQA voice quality algorithm on your Cyara account to test WebRTC voice quality, please contact you account representative.

    4. Select the Reply Type field as DTMF or Speech or Audio File based on the type of IVR application being tested. Responses can either be in DTMF, Speech, or Audio File.
      Responses can often be DTMF tones or Speech, and either numeric digits or text can be entered in the Reply With field to match your selection as necessary. The Audio File option specifies that the Cyara Platform should respond with an audio file. For more information on this feature, see the Audio File Management and Playback in Test Cases section.
      Note: Selecting ‘Speech’ will result in a Step failing if the Test Case is validated from a Campaign with a Plan Type set to Cruncher Lite, because Text to Speech is not available in Cruncher Lite.
    5. Enter the prompt that the IVR is expected to present in the Expect to Hear field. The expected prompt is the plain (phonetic) transcription of the prompt expected to be played by the IVR. For example, Welcome to A B C Company. Please press one for billing. If Step recognition fails, experiment with different spellings for words. Note that the prompt may require fine-tuning for maximum accuracy.
      It is best practice to type numbers greater than nine as words (for example, thirty-two, instead of 32) and you may need to add spaces between individual letters and numbers (for example, ‘w w w dot’ instead of ‘www.’ and ‘zero 3 9 9 9 9’ instead of ‘03999’).
      The Expect to Hear field supports the "Language" tag; this allows you to change the language of the prompt played by the IVR. For more information, see the Tags section.
      Use the Prompt Analyzer to fine-tune the text to match the recording of the expected prompt.
      Note: For creating Velocity Test Case Steps, see the Transcribe "Expect to Hear" section below.
    6. Specify the response the Cyara Platform should reply with in the Reply With field. Like with the Expect To Hear, you should ensure that text is entered phonetically to produce the desired response.

      For example, if the application requires DTMF input and the desired response is DTMF 1 followed by #, enter 1 and # in the field.

      If the application is a speech application, enter the desired speech to respond with. For example, if the prompt is “Would you like Sales or Support?’’ and the desired reply is “Support’’, type in “Support” and the Cyara Platform will generate “Support” in the form of speech. You can also change the default language for a speech response by using the "Voice" tag. For more information, see the Tags section.

      Tip: Place commas between DTMF digits to add pauses between digits. A period (.) equates to a two-second delay and a comma (,) equates to a 0.25 second delay. By using commas and periods, you can mimic DTMF input patterns. For example, younger callers may input digits more quickly than older callers.

    7. Enter the Min/Max Pause Time values.
      Step 0:
      Inbound Test Case: For Step 0 of an inbound call, these parameters are not taken into account. For more information on Campaigns, see the Campaigns chapter.
      Outbound Test Case: This is the amount of time in seconds to wait before answering the call. For more information, see the Creating an Outbound Test Case section.
      Steps other than Step 0:
      For Steps other than Step 0, these values specify the delay after which the response will be sent to the IVR. A random pause time will be used between this minimum and maximum. This provides the ability to simulate different types of users interacting with the IVR. For example, young caller (0.1 sec) versus elderly caller (2 secs+).
    8. Enter the Minor/Major Threshold values.

      The Major Threshold is the maximum allowable response time for each Step before the Step counts as a ‘fail’ result. For example, the IVR's normal response time to a Step is 5 seconds or less and the Minor and Major Thresholds for the Step are configured at 5 and 12 seconds respectively. If the IVR takes more than 12 seconds to respond, this is deemed to be a critical issue and causes the Step and Test Case to be marked as a failure. Note: if Step 0 is marked as a failure, the rest of the Test Case will not continue to run.

      The Minor Threshold is the maximum allowable response time for each Step before the Step counts as a 'satisfactory’ result. Minor Threshold violations are Step response times deemed to be of a non-critical nature. In the above example, the IVR may occasionally take between 5 and 12 seconds to respond and these violations are classified as ‘satisfactory’.

      Response times less than the Minor Threshold are counted as ‘pass’ test results.

    9. Enter the Major Confidence Level /Minor Confidence Level.

      Major Confidence Level: This value specifies the minimum acceptable threshold for a speech recognition match between the ‘Expected Prompt’ and the ‘Actual Prompt’ as determined by the ASR (Automatic Speech Recognition) engine. If the actual confidence level is lower than the ‘Major Confidence Level’ configured, the call is flagged as a failed call. The default value for ASR is configured through Accounts > Edit Account > Default Confidence (at the bottom of the page).

      MOS: This value specifies the minimum acceptable Mean Opinion Score for a recognition match between the audio file and the ‘Actual Prompt’. The value must be between 1.0 and 5.0.

      Minor Confidence Level: This value specifies the Success threshold for a speech recognition match between the ‘Expected Prompt’ and the ‘Actual Prompt’ as determined by the ASR (Automatic Speech Recognition) engine. If the actual confidence level is lower than the ‘Minor Confidence Level’ configured, but higher than the 'Major Confidence Level', the step is flagged as a Satisfactory. This value must be higher or exactly same as Major Confidence Level. if the two values are exactly same, there will be no Satisfactory range. The default value for ASR is configured through Accounts > Edit Account > Default Confidence .

      MOS: This value specifies the Success threshold for Mean Opinion Score for a recognition match between the audio file and the ‘Actual Prompt’. The value must be between 1.0 and 5.0.

    10. Enter the Post-Speech Silence Timeout field. Post-Speech Silence Timeout (PSST) is the continuous silence in seconds that needs to be detected to assume the end of the phrase to be recognized from the Expect To Hear. After the phrase is recognized, the Cyara Platform will play the reply. There is usually a period of silence at the end of each Step (phrase) and the period of silence is usually more than one second. Therefore a safe value for the PSST is one second. The PSST can also be used to barge-in on long prompts by setting a low value (for example, 0.5 seconds). This will force the Cruncher solution to detect a small pause in between phrases in the prompt. Upon detection, the application will assume that the phrase has finished and will play the reply. A more accurate way to achieve barge-in is to use the {BargeIn} tag (for more information, see Voice Tags).
    Calculating Response Time in a Test Case

    For example, consider a Test Case to ensure the following scenario:

    Step Expect To Hear Reply With Minor/Major PSST
    1 For telephone banking, press 1. To speak with a representative, press 2. 2 3/5 1.0
    2 In a few words, please tell me how we can help you today. 3/5 1.0
    3 You can say things like "new mortgage" or "stolen credit card". 2/3 1.0

    The Response Time measured is as shown:

    In the above scenario, the Response Time for Step 2 is measured from when Cyara replies with DTMF 2 in Step 1. If you decide that the Response Time of Step 2 should be less than 3 seconds and must be less than 5, since the Step 2 has no reply, Step 3’s Response Time is measured from when the IVR finishes speaking in Step 2.

    Therefore, Step 3’s Response Time includes Step 2’s PSST.

    Since Step 3’s Response Time is 2.5 seconds, this test would produce a Satisfactory result indicating that Step 3’s Response Time exceeded the Minor Threshold of 2 seconds.

Transcribe "Expect to Hear"

If your account has an active Velocity Plan subscription, you can auto-transcribe the Expect to Hear field when creating a Voice Test Case.

You can skip entering the Expect to Hear field and the transcriber will populate it for you. After transcription is complete, you can manually adjust it as necessary.

When you enable Velocity, the Test Case steps appear as shown below:

All the steps are similar to the Voice Test Case Steps described above. But if the Test Case is being developed for an existing IVR, you can skip entering the Expect to Hear, and it will be populated automatically by a transcriber.

When the Expect to Hear checkbox is selected and the step added, it will be passed to transcription. When the Expect to Hear checkbox is not selected, the Test Case will skip waiting during execution. In the user interface, all quality attributes will be disabled (Minor/Major Threshold, PSST, Confidence).

If the Reply With checkbox is unchecked, the Test Case execution will move on without performing any action. This is the same behavior as in the current version.

  1. After you specify the Reply Type, you do not need to enter the Expect To Hear. When you add this Voice Step, the transcriber will make a call and retrieve the text.
  2. Select the Expect to Hear checkbox. This will enable the field for entry.
    Do not enter any expected text, just leave it blank. Note that the Expect to Hear input will display ("Auto-Transcribe") hint.
  3. Enter the Reply With field.
  4. Specify the PSST field as described above.
  5. Click Add Voice Step.

    When you add this step, the Cyara Portal will display Transcribing... while audio retrieval and transcription is in progress.

    If there is an issue with audio retrieval or transcription, the Expect To Hear field will display an error icon with steps for how to resolve it.

    Once transcription is ready, it will be displayed in the Expect To Hear field of the relevant step.
    Note: The Expect To Hear transcription system will have configuration at the Account level to select the default Speaker. The Cyara Platform Administrator can add a new Speaker.

    Multiple transcriptions can take place at the same time.

    Once the step is added, you can add the next step. Meanwhile, the previous step will be transcribed.

    If you click the Expect To Hear field, the auto-transcribed text is displayed and you can click the Play button to listen to the audio. Note that the icon will switch to a Stop button while playing the audio.

    If a transcription cannot complete, the specific step is highlighted. The error can then be resolved manually. Click the Auto-Transcribe Again button to re-start the transcription. Once restarted, any auto transcription step will be transcribed again. If Expect to Hear is adjusted manually, it will not be overwritten.

    Audio recording of the transcription is available to fine-tune the Expect to Hear value.

Adding an Agent Step for Campaigns

If your Cyara installation has a license for either Velocity or Premium Pulse product functions, then you can add an Agent step to perform attached data validation on the agent side of the call. This allows you to perform Agent Experience (AX) verification to ensure that the correct information is presented to the agent during the call.

If it is necessary to run more than one Test Case with an agent step as part of your Campaign, use the unique calling line identifiers (CLI). See the Simulating CLI Numbers topic on how to specify CLI in the Called Number field.

To avoid hard-coding the numbers, it is possible to data-drive the CLI to ensure each call is unique within your Campaign.

Note: Cyara Cloud Accounts do not have access to this feature. Currently, it is limited to on-premises deployments with the purchased Velocity Plans.
To add a Verify Interaction Data Agent Step:
  1. Click the New Step button.
  2. Select the Agent Tab.
  3. Adjust the Step No if necessary.
  4. Enter a Description.
  5. Add expected agent attached data key-value pairs.
    1. To data-drive the Expected Value, use [Variable] syntax; see the Data Driven Test Cases topic for more information.
    2. It is possible to enter a regular expression to match several options or a substring of the attached data. To do so, use Regex (regular expression) syntax. See the regular expression language quick reference topic for more information.
    3. Use the Tab key to move to the next element of the table or Ctrl+Tab to move back.
    4. Use right-click to Copy, Move, Paste, and Delete the rows.
  6. Click Add Agent Step to add the step.

To run a Test Case with an Agent Step, you will have to:
  1. Configure an Agent Behavior with Verify Interaction Data from Test Case activity, as mentioned in the Voice Activity Types topic. Then, start an Agent Campaign to receive the calls.
  2. Run the Test Case within a Velocity Synchronized Campaign.

Adding a Cyara SMS/Web Step

Adding a SMS/Web Step enables setting up of an Omnichannel Test Case linkage to invoke dependent Test Cases on other channels. For example, a Voice Test Case querying an SMS Test Case to retrieve a PIN. Currently, the Voice channel can invoke Test Cases on SMS and Web channels.

The Service Step supports two service steps:
  1. Start Cyara Test Case
  2. Wait for Cyara Test Case Results

The first step is designed to kick off a Test Case validation on a different channel, the second one to retrieve the results. Note that the “wait” Step must always be after the “start” Step. The “wait” Step can be omitted, but then there will be no detailed result in the voice Test Case report details.

The inputs for a Start Cyara Test Case Step are:
  • Test Case
  • Any data-driven field from the selected Test Case, for example: Phone Number for Voice, SMS/URL for Web/Email

The input for Wait for Cyara Test Case Results is a reference to the Start Cyara Test Case Step.

The output for a Wait for Cyara Test Case Results Step is any variable used in the started Test Case (SMS or Web):
  • For Web
    1. Matches("selector", "value", "frame", "script") – the function can take advantage of Tags.
    2. Example for Expect field: "Matches(".account-no","Your new account number is {AlphaNum Length=10 $account}")
  • For SMS
    1. Example for Expect field: "Your PIN is {Number $pin}"
    2. List of Tags available in SMS, Web Test Cases to match and extract values:
      1. {AlphaNum Length=from-to $var} – matches and extracts a string of letters and numbers into $var variable
      2. {Number MaxDecimal=2 $var} – matches and extracts a number with two decimals into $var variable
      3. {URL Proto=HTTPS Host=DNS $var} – matches and extracts a URL that uses HTTPS protocol and is based on a DNS entry, not IP address into $var variable
To add an SMS/Web Step:
  1. To add an SMS Step, click the New Step button and select the SMS tab. A screen as shown below appears:

  2. Adjust the Step No if necessary.
  3. Enter a Description.
  4. Select the "Start Cyara Test Case" Service from the drop-down list.
  5. Click the Browse button and select the Test Case to run.
  6. If the selected Test Case has data-driven parameters, enter the values in the Input Parameter table.
  7. Specify the Timeout value if necessary.
  8. If the Step results are not essential for the next steps, select the Continue on Failure checkbox. If it is selected, the voice Test Case will continue even if the service call fails.
  9. Add the voice Steps as required.
  10. When you add a new Service Step and select the “Wait for Cyara Test Case Result” Service, the screen changes to:

    1. Specify the Variable for each output parameter. Variables use the $variablename syntax.
  11. Specify the Timeout value. For example, an SMS may take about a minute to arrive, so in this scenario, adjust the timeout to 60 seconds.
  12. Select Yes or No for the Continue on Failure option. If you need the Test Case to execute even if this Step has failed, then select Yes.
  13. After specifying all the details, click the Add Step button.
  14. To add a Web Step, click the New Step button and select the Web tab. A screen as shown below appears:

The steps are similar to adding the SMS steps.

Below is a sample Service Step to retrieve a PIN from SMS:

A Voice Test Case will execute and get results from an SMS Test Case:
  1. SMS number to wait for SMS is specified.
  2. PIN from the SMS contents will be extracted in output parameters.
  3. The retrieved PIN will be passed into the voice Step to generate a DTMF sequence for authentication.
As an alternative scenario, a Voice Test Case can execute and get results from a Web Test Case:
  1. The URL of the Website can be specified.
  2. The Contents of the Website can be extracted in the output parameters and used in other Service Steps.

Adding a Cyara Service Step

You can add a Cyara Service Step to a Test Case to invoke external services. Currently this functionality is limited to web-based services. To use a service, it must be defined in the Service Library. A service step has a name, input, and output parameters.

To add a Cyara Service Step, perform the following:

  1. Click the New Step button and select the Service tab. A screen as shown below appears:

  2. Adjust the Step No if necessary.
  3. Enter a Description.
  4. Click Browse and select the required Service from the "Choose Service" pop-up list:

  5. Based on the selected Service, the parameter list will be updated.
    1. Enter the Input Parameter values; you can use constants or variables.
    2. Enter the Output Parameter variable assignment. By default, the Test Case variable name will be the same as that of the output parameter. If desired, you can enter a new name.
  6. Specify the Timeout value. Use the Service Library Run Now feature to get a reasonable time the service should return a value. Most can take from 5-20 seconds to execute.
  7. Select Yes or No for the Continue on Failure option. If you need the Test Case to continue execution even if this Step has failed, then select Yes.
  8. After specifying all the Service details, click the Add Step button.
Was this article helpful?
0 out of 0 found this helpful
Return to top

Related articles

Comments

0 comments

Please sign in to leave a comment.