Regression Testing Using Postman Before Stable Version Release

Overview

This guide explains how to perform regression testing using Postman to verify that API responses remain unchanged before and after the stable version release.
While notifications are sent out in Slack communities if API responses change, you can use this method to check if there are any changes in the endpoints or how they are being used.

Prerequisites

• Primarily targeted at Project Managers, this test is to ensure that the API responses on client project sites have not changed from before.
• You will use the Postman application to manually execute tests from your local environment.
• Only the API for static access tokens (GET method) is targeted.

caution

• Postman stores test collection files and execution logs outside Japan (US or EU). Confirm with the client beforehand if testing can be conducted and ensure you have permission.
  • Security at Postman
• Do not target sites containing confidential or personal information. If necessary, create a separate testing environment with all such data replaced with dummy data.

Testing Flow

Follow the steps below to execute tests on the RC version just before the stable version release and confirm if there are any changes in the API responses.

1. Retrieve and save all response data from the stable version environment's API.
2. Switch to the RC version.
3. Request each API and compare the responses with those saved in step 1.

Preparations

First, sign up for Postman using the following URL.

• https://identity.getpostman.com/signup

Once registered, create a workspace in Postman.
Click on [Create Workspace].

Select [Empty Workspace] and proceed.

Enter a project name, ensure Internal (Private) is selected, and create the workspace.

Subsequently, download the Postman application from the following URL and use the desktop version of Postman.

• https://www.postman.com/downloads/

tip

You can also proceed with the browser version, but in that case, you need to follow the CORS policy. The following additions are required in Kuroco's CORS settings:

• Add https://projectname.postman.co to CORS_ALLOW_ORIGINS.
• Add the following to CORS_ALLOW_HEADERS:
  • X-RCMS-API-ACCESS-TOKEN
  • User-Agent
  • Cache-Control
  • Postman-Token

Creating Tests

Export Kuroco's API in OpenAPI Format

In the Kuroco admin panel, open the list of endpoints for the APIs to be tested and select [Export OpenAPI].

Select the output format as [JSON] and download the openapi.json (API definition file).

Creating a Postman Collection File

Using the exported API definition, create a Postman collection file (a file that contains tests to be executed).

We have prepared a script to create a Postman collection file from the API definition. In an environment where you can run Node.js scripts, install openapi-to-postmanv2@^5.0.0 and execute the following script.
https://gist.github.com/sakaguchi-diverta/2fbedcc430366126fea46d0f5a127a23

caution

The content and location of the script may change in the future.

Details of script execution are omitted.
You can also request work or explanations from the AI agent.
At our company, we generated the Postman collection file by requesting Devin to execute it.

Once the execution is complete, the Postman collection file postman_collection.json will be output.

Importing and Adjusting the Postman Collection

Return to the Postman workspace, select [Import], and upload the previously output postman_collection.json.

After the import is complete, the request information for each endpoint will be saved in folders labeled "サンプル保存" and "テスト".

In the "サンプル保存" folder, obtain and save the stable version response, and in the "テスト" folder, obtain the RC version response to compare with the stable version response. If there are endpoints you want to exclude from testing, remove them from the "テスト" folder.

For the detailed endpoint (details), the ID/Slug specification section becomes a variable. Replace the path variable with the ID of the endpoint to be tested and save it again (in both the "サンプル保存" and "テスト" folders).

If there are items you want to ignore during testing, select the endpoint in the "テスト" folders, add the item name (API response key) to the ignore_keys in the [Scripts] tab, and save it.

Example:

const ignore_keys = [
    "update_ymdhi",
    "ymd"
];

Saving the Access Token

Finally, save the API access token. Use the Postman Vault (an area for managing confidential information in a local environment) as the storage location.

Select [Vault] from the Postman footer menu and save the static API access token as RCMS_API_STATIC_TOKEN.

note

Please note that it is not X-RCMS-API-ACCESS-TOKEN.

How to Run the Test

Saving Samples in Kuroco Stable version

Switch the Kuroco site to the stable version for the test.

In Postman, select the [サンプル保存] folder and click [Run].

Next, click the orange run button to execute the sample save request.

Verify that the request is executed and a response of 200 is received.

After the execution is completed, select the top-level folder of the collection, and select the [Variables] tab to confirm that the API response obtained earlier is saved in the variables.

Saving Samples in Kuroco RC Version

Switch the site to the RC version for the test.

Similarly to saving samples, select the [テスト] folder and execute the request.

After the execution, check the results. If there is a change in the response, FAIL will be output; if there is no change, PASS will be output.

Compare the failed items with the response from the stable version.
In this case, it was found that the value of the temporary token attached to the image was different, resulting in a FAIL.

This concludes the explanation of the regression test before the stable version is reflected using Postman.