Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Pytest marks with tests

Since 0.11.0, it is possible to ‘mark’ tests. This uses Pytest behind the scenes - see the pytest mark documentation for details on their implementation and prerequisites for use.

In short, marks can be used to:

An example of how these can be used:

---
test_name: Get server info from slow endpoint

marks:
  - slow

stages:
  - name: Get info
    request:
      url: "{host}/get-info-slow"
      method: GET
    response:
      status_code: 200
      json:
        n_users: 2048
        n_queries: 10000

---
test_name: Get server info from fast endpoint

marks:
  - fast

stages:
  - name: Get info
    request:
      url: "{host}/get-info"
      method: GET
    response:
      status_code: 200
      json:
        n_items: 2048
        n_queries: 5

Both tests get some server information from our endpoint, but one requires a lot of backend processing so we don’t want to run it on every test run. This can be selected like this:

$ py.test -m "not slow"

Conversely, if we just want to run all tests marked as ‘fast’, we can do this:

$ py.test -m "fast"

Marks can only be applied to a whole test, not to individual stages (with the exception of skip, see below).

Formatting marks

Marks can be formatted just like other variables:

---
test_name: Get server info from slow endpoint

marks:
  - "{specialmarker}"

This is mainly for combining with one or more of the special marks as mentioned below.

NOTE: Do not use the !raw token or rely on double curly brace formatting when formatting markers. Due to pytest-xdist, some behaviour with the formatting of markers is subtly different than other places in Tavern.

Special marks

There are 4 different ‘special’ marks from Pytest which behave the same as if they were used on a Python test.

NOTE: If you look in the Tavern integration tests, you may notice a _xfail key being used in some of the tests. This is for INTERNAL USE ONLY and may be removed in future without warning.

skip

To always skip a test, just use the skip marker:

...

marks:
  - skip

Separately from the markers, individual stages can be skipped by inserting the skip keyword into the stage:

stages:
  - name: Get info
    skip: True
    request:
      url: "{host}/get-info-slow"
      method: GET
    response:
      status_code: 200
      json:
        n_users: 2048
        n_queries: 10000
Skipping stages with simpleeval expressions

Stages can be skipped by using a skip key that contains a simpleeval expression. This allows for more complex conditional logic to determine if a stage should be skipped.

Example:

stages:
  - name: Skip based on variable value
    skip: "{v_int} > 50"
    request:
      url: "{host}/fake_list"
      method: GET
    response:
      status_code: 200

In this example, the stage will be skipped if v_int is greater than 50. Any valid simpleeval expression can be used.

skipif

Sometimes you just want to skip some tests, perhaps based on which server you’re using. Taking the above example of the ‘slow’ server, perhaps it is only slow when running against the live server at www.slow-example.com, but we still want to run it in our local tests. This can be achieved using skipif:

---
test_name: Get server info from slow endpoint

marks:
  - slow
  - skipif: "'slow-example.com' in '{host}'"

stages:
  - name: Get info
    request:
      url: "{host}/get-info-slow"
      method: GET
    response:
      status_code: 200
      json:
        n_users: 2048
        n_queries: 10000

skipif should be a mapping containing 1 key, a string that will be directly passed through to eval() and should return True or False. This string will be formatted first, so tests can be skipped or not based on values in the configuration. Because this needs to be a valid piece of Python code, formatted strings must be escaped as in the example above - using "'slow-example.com' in {host}" will raise an error.

xfail

If you are expecting a test to fail for some reason, such as if it’s temporarily broken, a test can be marked as xfail. Note that this is probably not what you want to ‘negatively’ check something like an API deprecation. For example, this is not recommended:

---
test_name: Get user middle name from endpoint on v1 api

stages:
  - name: Get from endpoint
    request:
      url: "{host}/api/v1/users/{user_id}/get-middle-name"
      method: GET
    response:
      status_code: 200
      json:
        middle_name: Jimmy

---
test_name: Get user middle name from endpoint on v2 api fails

marks:
  - xfail

stages:
  - name: Try to get from v2 api
    request:
      url: "{host}/api/v2/users/{user_id}/get-middle-name"
      method: GET
    response:
      status_code: 200
      json:
        middle_name: Jimmy

It would be much better to write a test that made sure that the endpoint just returned a 404 in the v2 api.

parametrize

A lot of the time you want to make sure that your API will behave properly for a number of given inputs. This is where the parametrize mark comes in:

---
test_name: Make sure backend can handle arbitrary data

marks:
  - parametrize:
      key: metadata
      vals:
        - 13:00
        - Reading: 27 degrees
        - 手机号格式不正确
        - ""

stages:
  - name: Update metadata
    request:
      url: "{host}/devices/{device_id}/metadata"
      method: POST
      json:
        metadata: "{metadata}"
    response:
      status_code: 200

This test will be run 4 times, as 4 separate tests, with metadata being formatted differently for each time. This behaves like the built in Pytest parametrize mark, where the tests will show up in the log with some extra data appended to show what was being run, eg Test Name[John], Test Name[John-Smythe John], etc.

The parametrize mark should be a mapping with key being the value that will be formatted and vals being a list of values to be formatted. Note that formatting of these values happens after checking for a skipif, so a skipif mark cannot rely on a parametrized value.

Multiple marks can be used to parametrize multiple values:

---
test_name: Test post a new fruit

marks:
  - parametrize:
      key: fruit
      vals:
        - apple
        - orange
        - pear
  - parametrize:
      key: edible
      vals:
        - rotten
        - fresh
        - unripe

stages:
  - name: Create a new fruit entry
    request:
      url: "{host}/fruit"
      method: POST
      json:
        fruit_type: "{edible} {fruit}"
    response:
      status_code: 201

This will result in 9 tests being run:

If you need to parametrize multiple keys but don’t want there to be a new test created for every possible combination, pass a list to key instead. Each item in val must then also be a list that is the same length as the key variable. Using the above example, perhaps we just want to test the server works correctly with the items “rotten apple”, “fresh orange”, and “unripe pear” rather than the 9 combinations listed above. This can be done like this:

---
test_name: Test post a new fruit

marks:
  - parametrize:
      key:
        - fruit
        - edible
      vals:
        - [ rotten, apple ]
        - [ fresh, orange ]
        - [ unripe, pear ]
        # NOTE: we can specify a nested list like this as well:
        # -
        #   - unripe
        #   - pear

stages:
  - name: Create a new fruit entry
    request:
      url: "{host}/fruit"
      method: POST
      json:
        fruit_type: "{edible} {fruit}"
    response:
      status_code: 201

This will result in only those 3 tests being generated.

This can be combined with the ‘simpler’ style of parametrisation as well - for example, to run the above test but also to specify whether the fruit was expensive or cheap:

---
test_name: Test post a new fruit and price

marks:
  - parametrize:
      key:
        - fruit
        - edible
      vals:
        - [ rotten, apple ]
        - [ fresh, orange ]
        - [ unripe, pear ]
  - parametrize:
      key: price
      vals:
        - expensive
        - cheap

stages:
  - name: Create a new fruit entry
    request:
      url: "{host}/fruit"
      method: POST
      json:
        fruit_type: "{price} {edible} {fruit}"
    response:
      status_code: 201

This will result in 6 tests:

Since 1.19.0 you can now also parametrize generic blocks of data instead of only strings. This can also be mixed and matched with items which are strings. If you do this, remember to use the force_format_include tag so it doesn’t come out as a string:

test_name: Test sending a list of list of keys where one is not a string

marks:
  - parametrize:
      key:
        - fruit
        - colours
      vals:
        - [ apple, [ red, green, pink ] ]
        - [ pear, [ yellow, green ] ]

stages:
  - name: Send fruit and colours
    request:
      url: "{host}/newfruit"
      method: POST
      json:
        fruit: "{fruit}"
        colours: !force_format_include "{colours}"

        # This sends:
        # {
        #   "fruit": "apple",
        #   "colours": [
        #     "red",
        #     "green",
        #     "pink"
        #   ]
        # }

The type of the ‘val’ does not need to be the same for each version of the test, and even external functions can be used to read values. For example this block will create 6 tests which sets the value_to_send key to a string, a list, or a dictionary:

---

test_name: Test parametrizing random different data types in the same test

marks:
  - parametrize:
      key: value_to_send
      vals:
        - a
        - [ b, c ]
        - more: stuff
        - yet: [ more, stuff ]
        - $ext:
            function: ext_functions:return_string
        - and: this
          $ext:
            function: ext_functions:return_dict

          # If 'return_dict' returns {"keys: ["a","b","c"]} this results in:
          # {
          #   "and": "this",
          #   "keys": [
          #     "a",
          #     "b",
          #     "c"
          #   ]
          # }

As see in the last example, if the $ext function returns a dictionary then it will also be merged with any existing data in the ‘val’. In this case, the return value of the function must be a dictionary or an error will be raised.

    # This would raise an error
    #- and: this
    #  $ext:
    #    function: ext_functions:return_string

NOTE: Due to implementation reasons it is currently impossible to parametrize the MQTT QoS parameter.

Using marks with fixtures

See the documentation on fixtures for the details on using Pytest fixtures with Tavern.

If you have a fixture that loads some information from a file or some other external data source, but the behaviour needs to change depending on which test is being run, this can be done by marking the test and accessing the test Node in your fixture to change the behaviour:

test_name: endpoint 1 test

marks:
  - endpoint_1
  - usefixtures:
       - read_uuid

stages:
    ...

---
test_name: endpoint 2 test

marks:
  - endpoint_2
  - usefixtures:
       - read_uuid

stages:
    ...

In the read_uuid fixture:

import pytest
import json

@pytest.fixture
def read_uuid(request):  # 'request' is a built in pytest fixture
    marks = request.node.own_markers
    mark_names = [m.name for m in marks]

    with open("stored_uuids.json", "r") as ufile:
        uuids = json.load(ufile)

    if "endpoint_1" in mark_names:
        return uuids["endpoint_1"]
    elif "endpoint_2" in mark_names:
        return uuids["endpoint_2"]
    else:
        pytest.fail("No marker found on test!")
Tavern
Core Concepts
Tavern
Strict key checking