How to Run Integration Tests with Postman Collections on TeamCity: A Step-by-Step Guide

Postman collections are commonly used for integration, contract, and as a part of end-to-end testing. In this article, considering the popularity and practicality of Postman, I'll cover typical steps to run a Postman collection test set inside a TeamCity pipeline.

There will be two main sections. The first section is creating a sample Postman collection with tests and running them locally. The second one will contain running Postman tests inside a TeamCity pipeline.

The Postman Side

StackExchange has a public API that makes comprehensive content/page search possible for everyone. I'll use a simple HTTP GET request that calls the StackExchange API in this Postman UI application inside a Postman collection. The possible steps to take in this section are as follows.

• Create a Postman collection in the Postman UI application and create a GET request inside the collection. I used this URL that queries the StackExchange content which is tagged with the 'postman' keyword and gets lists of pages about Postman.

  

  I also created collection variables that will help us to make things more dynamic a few steps later.

  

• Click on the scripts tab, and create some tests with JavaScript code that uses the test library of Postman and tests the response against given cases. In my case, I created two simple tests. The first tests the response status code and the other one verifies if the response contains the desired data by checking the existence of a given tag in response content.

  

• To run tests, on the left menu, click the three dots icon on the collection name. Click the Generate Tests option. This will open the test window corresponding to the collection in which all collection tests are listed.

  

• Click the Run Collection button. This will open the test runner panel. Click Run Postman Tests to run tests.

  

• We can run the tests with different test data iteratively by providing data in the Postman test runner. For this, create a CSV file that contains values for collection variables we created and used earlier. In other words, our collection variables will act as placeholders for us to insert different test data to have a dynamic, iterative test experience.

• In my scenario, I will run two separate test data value sets which also means executing two separate iterations. This is what my CSV file looks like.

  (Note: For the first iteration, the search_text value will be postman and it will test the response content. The provided status_code = 200 will check if the HTTP response code is 200.

  In the second iteration, I'll again pass postman as search_text value and expect a passing test. But, for status_code with value 400, I'll intentionally create a failing test assuming that this endpoint always returns 200. This will help me see if we can properly see a failing test in test results.)

  

• It's time to add test data and run tests. Navigate to the Postman UI application. On the Runner window click Select File under Data label. Click on the Run PostmanTest button to run tests with the provided test data.

  (The button can appear with a different button text in your case depending on the collection name.)

  

• Test results are summarized as follows. All tests passed for the first iteration as expected (linked to the first line in the test_data.csv file). One of the tests fails while the other passes for the second iteration as designed intentionally.

  This is the green signal that our test data works in harmony with our test scripts.

  

• As our goal is to run Postman collection tests on TeamCity inside a pipeline, we should be able to run tests independently without using the Postman UI application. There's a CLI solution called Newman which is a command-line collection runner for Postman.

  Newman comes with different reporting options for test results. While we can create our custom reporters, in this guide, I'll use some present reporters listed below with other npm package URLs and references.

  1. newman => This is the main Postman runner. It comes with a CLI-based reporter when we install it.

  2. newman-reporter-htmlextra => Works in conjunction with the newman package and helps export test results in HTML format.

  3. newman-reporter-csv => Works in conjunction with the newman package and helps export test results in CSV format.

  4. newman-reporter-teamcity => This reporter is designed for TeamCity and helps export test results to the Tests tab in the TeamCity pipeline.

• We need to export the Postman collection run within the command line tool with the help of the newman. Navigate to the Postman application and export the collection by clicking the Export option. You can create a particular folder and save/copy the exported collection (JSON file) into that folder. You can also locate the test data CSV file in this same folder to ease the command line operations we'll do shortly.

  

• To run the Postman collection tests, open the command line and make sure you have the node and npm installed by checking its version.

  node --version

  npm --version

• Install newman with the below command and check the version to see if it's installed properly.

  npm install newman

  newman --version

• In the command line window, navigate to the folder that contains the exported Postman collection JSON file and the test data CSV file.

  

• Type the below newman command to run with the provided collection and test data configuration. The "--reporters cli" shows the test results in the command line window.

  newman run PostmanTest.postman_collection.json --reporters cli --iteration-data test_data.csv

• After running the command, the test results are summarized as shown below. One of the tests is failing out of four, which seems correct as designed intentionally.

  

  This shows us that when we create a TeamCity pipeline if we run a similar scenario with the same configuration we can expect to see the same results.

  The TeamCity Side

  In the TeamCity part of the guide, we'll create a TeamCity pipeline and execute the exported Postman collection with the test data inside a build agent. Differing from the previous section, we'll use various reporter types, and the main goal is to create test reports in different file formats in the pipeline artifacts while also getting the test results listed in the Tests tab of the TeamCity pipeline. We can continue with the following steps.

• Create a project in the TeamCity instance.

  (You can work with the TeamCity Cloud solution or host a TeamCity instance wherever possible. In this article, I'll move with the cloud option.)

  

• I created a public Bitbucket repository here and placed exported Postman collection and test data CSV files there. I'll be passing them into TeamCity as a version control source.

• Navigate to the TeamCity project and open the project configuration page.

  

• Click on the General Settings on the left menu and define a folder name to publish build artifacts in the Artifact path section. I used the name newman_artifacts.

  

• Move to Version Control Settings by clicking it on the left menu. Attach a VCS root by providing information about the repository that contains files and scripts we'll use when running the pipeline.

  I added my bitbucket repository as a file source to be used inside the pipeline.

  

• Click on the Build Steps link. We have to create a build step that will execute a script that will also run newman commands on the build agent and thus will execute Postman tests.

  

• Before adding a build step, we need a script that runs newman commands with the specified configuration when executed.

  For this, I created a .js file with the below content, named it run_newman.js, and saved it in Bitbucket in the same directory with Postman collection JSON and test data CSV files.

    const newman = require('newman');
  
    newman.run({
        collection: 'PostmanTest.postman_collection.json',
        iterationData: 'test_data.csv',
        reporters: ['htmlextra', 'csv', 'cli', 'teamcity', 'json-summary'],
        reporter : {
            "json-summary": {
                  summaryJsonExport :'newman_artifacts/_json_report.json'
        },
            "htmlextra": {
                  export: 'newman_artifacts/_html_report.html'
        },
            "csv": {
                  export: 'newman_artifacts/_csv_report.csv'
        }
      }
    }, process.exit);
  

  This script executes the Postman collection in JSON format (PostmanTest.postman_collection.json) which also has the test data dependency we created earlier in CSV format (test_data.csv).

  We can create reports in CLI, HTML, JSON, CSV, and TeamCity formats by defining them in the "reporters" line.

• We have to define a folder location and file name for the reporters to generate their test reports with their specific file formats. In my case, I export reports under the newman_artifacts folder which also refers to the folder I specified in the TeamCity project configuration.

• After creating and saving the script file my Bitbucket repository looks like this.

  

• We're ready to add a build step in TeamCity. Navigate to the TeamCity Build Steps page and click the + Add build step button.

  

• Choose Command Line from the runner list. Provide a name in the Step name field.

  Type the following commands in the Custom script box and Save the configuration.

  npm install newman

  npm install newman-reporter-htmlextra

  npm install newman-reporter-csv

  npm install newman-reporter-teamcity

  npm install newman-reporter-json-summary

  node run_newman.js

  

  When we run the pipeline, this command will first install newman with all the reporters and run the script file (run_newman.js) that contains the newman commands.

  The Build Steps page looks like this when the build step is saved.

  

• The pipeline project is ready to run. Click Projects on the top left of the page. Click on the project name and run the pipeline by clicking the Run button.

  See that a new run starts and shows up on the page.

  

• Click on the build when it's finished. Click the Tests tab. The newman-reporter-teamcity created the test results under the Tests tab. It shows that one of the executed tests failed as we intentionally wanted it to fail.

  

• Click the Build Log tab. Scroll down and see that the test report is written here in CLI format as we earlier executed in the local command window.

  

• Navigate back to the page where all builds are listed. See the newly executed build listed on the page.

  

• The blue layered icon that appears on the build list item represents the artifacts. This icon goes gray when the build has no files in the artifacts folder.

  Click on the blue artifacts icon, and explore the test reports by clicking on the file names.

  

  This is what the exported test report looks like in CSV format.

  

  The exported JSON file looks like this.

  

  Finally, the test report in HTML format is generated as follows.

  

This is the end of this article that we created a Postman collection with tests, ran them locally, created a TeamCity pipeline, ran Postman tests in the TeamCity pipeline, and got test results in various formats.

Thanks for following up!