Skip to main content

Testing and validating XM Solutions

XM Solutions interact with many pieces of the Qualtrics platform, and allow you to write your own JavaScript.

If you would like to join the early access program for XM Solution Plugins, then please let us know by taking this survey.

Not generally available

This feature is in Early Access and is not yet generally available. Functionality, APIs, and documentation may change before general availability.

Since they're so complicated, it's important to understand how to effectively test your XM Solution. This page will discuss best practices for testing and validating a Solution.

Prerequisites​

Before starting, you should have some basic familiarity with XM Solutions by completing these pre-requisites:

Validation​

The first step to making sure your Solution is working, is to run the validation command. This command runs through all the steps of your Solution and validates the files along the way. It makes sure that your files are linked together properly and that each resource has the correct format. Validation runs through all the steps in the Solution without actually creating any resources, but to do so we must give it some information in order to properly validate the solution is working correctly. We provide information that acts as the user input to simulate the process of a user interacting with the Solution. This is done through the use of test files.

Test files​

Using test files to validate your solution is a great way to ensure that your Solution is working properly. In order to run validation on the test files correctly, you need to mock all the user inputs that you won't receive, as there will not be an actual user. This is done by creating mock answers that will be used in the validation process. We can use the Setup Assistant to create test files with answers created by you. Create your first test file by running solution generate-test. This will open up a window in the Setup Assistant UI that will help you create custom answers for your test files (You might need to log in). Go through each of the questions and answer them as you see fit. If the question is optional, it can be left blank or without a selection. After you finish answering all the questions, click on Finish and close the window. Go back to your terminal and you will see that under the 'generate-test' command there is a response telling you to run a command to retrieve the created test file.

solution generate-test
[SUCCESS] Launched session. Run `solution generate-test --session-id 3a4a5e3c-6661-45c5-87dc-930c2be2d3a1` when you've submitted your Setup Assistant to generate the test

After running the above command, a file will be created that looks something like this:

{
   "setup": {
      "answers": {
        "title": "example title",
        "question1": "answer1"
      }
   },
   "system": {
    "language": "EN",
    "user": {
      "userId": "",
      "brandId": "",
      "dc": ""
    }
  }
}

Test files are all stored in the tests directory in your plugin. A separate validation run will happen for every test file you have in that directory. The next section deals with running validation in your solution, but if you wish to understand all the types of questions you can include in your solution, read the types of questions available for XM Solutions. You can also modify the test file manually if you wish to do so. To better understand the created file, you can read the Understanding Setup Assistant forms section at the bottom of this page.

Running validation​

Validation is only run against plugins that have already been submitted. It is NOT run against the files on your computer. To run validation, you will need to run the following commands:

solution build
xmt submit extension
solution validate

You should repeat the build/submit/validate steps often as you develop your Solution.

Your test Setup Assistant answers should go in setup.answers in your test file. The answers object contains key/value pairs where each key is the elementId of a question in your Setup Assistant file and the value is a valid answer to that question:

{
  "setup": {
    "answers": {
      "elementId1": "answer1",
      "elementId2": {
        "someAnswerField": "answer2"
      },
      "elementId3": [
        "answer3a",
        "answer3b",
        "answer3c"
      ],
      "etc": "etc..."
    }
  }
}

Each question type's answer has a specific shape and sometimes multiple shapes depending on the question's configuration, so consult the list below to build a valid answer for each question. In the following sections, element will refer to one of the question objects in the elements array which will be in the pages array of your Setup Assistant file.

If element.configuration.required is true, then answers must have a valid answer for that question.

Manual testing​

Be sure to manually test your Solution often as well. Some easily overlooked things worth testing are:

  • Test in more than one brand, if applicable.
  • Make sure your Solution does not depend on any IDs or resources that only your brand has access to.
  • Verify all of your translations appear as expected for every language you support.
  • Make sure your Solution creates entries on the My Projects Page of the product properly if you expect it to do so.

Unit testing​

It's also smart to make sure your JavaScript/Typescript is well tested. All plugin templates come set up to use jest to test your code. However, you are welcome to use any testing tools you want. It's just JavaScript or Typescript, so there are many options available.

To get started with jest, check out their documentation here, and be sure to look at the example test files that come in all the plugin templates.