Using Python functions in Tavern

Tavern provides several ways, documented below, to integrate Python code with your tests. Each approach has different strengths and is suited to different scenarios.

Quick Selection Guide:

• Need it for just one test? → External Functions or Tinctures

• Need pytest integration or session-wide data? → Fixtures

• Need per-stage execution with timing/wrapping? → Tinctures or Hooks

• Need it automatically for every test? → Hooks or Fixtures

Calling external functions¶

Not every response can be validated simply by checking the values of keys, so with Tavern you can call external functions to validate responses and save decoded data. You can write your own functions or use those built in to Tavern. Each function should take the response as its first argument, and you can pass extra arguments using the extra_kwargs key.

To make sure that Tavern can find external functions you need to make sure that it is in the Python path. For example, if utils.py is in the ‘tests’ folder, you will need to run your tests something like (on Linux):

$ PYTHONPATH=$PYTHONPATH:tests py.test tests/

Checking the response using external functions¶

The function(s) should be put into the verify_response_with block of a response (HTTP or MQTT):

  - name: Check friendly mess
    request:
      url: "{host}/token"
      method: GET
    response:
      status_code: 200
      verify_response_with:
        function: testing_utils:message_says_hello

# testing_utils.py
def message_says_hello(response):
    """Make sure that the response was friendly
    """
    assert response.json().get("message") == "hello world"

A list of functions can also be passed to verify_response_with if you need to check multiple things:

    response:
      status_code: 200
      verify_response_with:
        - function: testing_utils:message_says_hello
        - function: testing_utils:message_says_something_else
          extra_kwargs:
            should_say: hello

If an external function you are using raises any exception, the test will be considered failed. The return value from these functions is ignored.

Built-in validators¶

There are some external functions built in to Tavern to help assert common expectations

Validating JWT¶

validate_jwt takes the key of the returned JWT in the body as jwt_key, and additional arguments that are passed directly to the decode method in the PyJWT library. NOTE: Make sure the keyword arguments you are passing are correct or PyJWT will silently ignore them. In the future, this function will likely be changed to use a different library to avoid this issue.

# Make sure the response contains a key called 'token', the value of which is a
# valid jwt which is signed by the given key.
response:
  verify_response_with:
    function: tavern.helpers:validate_jwt
    extra_kwargs:
      jwt_key: "token"
      key: CGQgaG7GYvTcpaQZqosLy4
      options:
        verify_signature: true
        verify_aud: false

Validating with pykwalify¶

validate_pykwalify takes a pykwalify schema and verifies the body of the response against it.

# Make sure the response matches the given schema - a sequence of dictionaries,
# which has to contain a user name and may contain a user number.
response:
  verify_response_with:
    function: tavern.helpers:validate_pykwalify
    extra_kwargs:
      schema:
        type: seq
        required: True
        sequence:
          - type: map
            mapping:
              user_number:
                type: int
                required: False
              user_name:
                type: str
                required: True

Validating with a regex¶

validate_regex checks that the response body (or a specific header) matches the given regular expression. Optionally, a jmespath expression can be used to extract a string from the JSON body before matching. Named capture groups are returned and can be used in later requests.

response:
  verify_response_with:
    function: tavern.helpers:validate_regex
    extra_kwargs:
      expression: "^Bearer (?P<token>.+)$"
      header: "Authorization"

response:
  verify_response_with:
    function: tavern.helpers:validate_regex
    extra_kwargs:
      expression: "(?P<uuid>[0-9a-f\\-]{36})"
      in_jmespath: "data.id"

Validating content with JMESPath¶

validate_content checks values extracted from the response body using JMESPath expressions. Pass a list of comparisons, each specifying a jmespath, an operator, and an expected value.

response:
  verify_response_with:
    function: tavern.helpers:validate_content
    extra_kwargs:
      comparisons:
        - jmespath: "users[0].name"
          operator: "eq"
          expected: "Bob"
        - jmespath: "users | length(@)"
          operator: "gt"
          expected: 0

Checking a JMESPath match¶

check_jmespath_match asserts that a JMESPath query resolves to a truthy value in the response. Optionally, an expected value can be provided to assert the result matches it exactly. Without expected, it asserts the path resolves to a truthy (non-falsy) value—falsy values like [], "", 0, and False will be treated as failures.

response:
  verify_response_with:
    function: tavern.helpers:check_jmespath_match
    extra_kwargs:
      query: "items[?status == 'active']"
      expected:
        - name: "widget"
          status: "active"

Validating with a pydantic model¶

validate_pydantic validates the JSON response body against a Pydantic model. Pass the entry-point-style location of the model class as model_location.

Any extra can keyword arguments via extra_kwargs are passed to model_validate to control how the model is validated.

response:
  verify_response_with:
    function: tavern.helpers:validate_pydantic
    extra_kwargs:
      model_location: "myapp.models:UserResponse"
      extra: allow

To use this helper, Pydantic must be installed separately as it is an optional dependency.

Using external functions for other things¶

External functions can be used to inject arbitrary data into tests or to save data from the response.

An external function must return a dict where each key either points to a single value or to an object which is accessible using dot notation. The easiest way to do this is to return a Box object.

Note: Functions used with verify_response_with or save in the response block should always take the response as the first argument.

Injecting external data into a request¶

A use case for this is trying to insert some data into a response that is either calculated dynamically or fetched from an external source. If we want to generate some authentication headers to access our API for example, we can use an external function using the $ext key to calculate it dynamically (note as above that this function should not take any arguments):

# utils.py
from box import Box


def generate_bearer_token():
    token = sign_a_jwt()
    auth_header = {
        "Authorization": "Bearer {}".format(token)
    }
    return Box(auth_header)

This can be used as so:

- name: login
  request:
    url: http://server.com/login
    headers:
      x-my-header: abc123
      $ext:
        function: utils:generate_bearer_token
    json:
      username: test_user
      password: abc123
  response:
    status_code: 200

When an $ext function returns a mapping, its values are merged into the existing request block by default. The --tavern-merge-ext-function-values flag has been removed because this is now the default behaviour:

# ext_functions.py

def return_hello():
    return {"hello": "there"}

    request:
      url: "{host}/echo"
      method: POST
      json:
        goodbye: "now"
        $ext:
          function: ext_functions:return_hello

This will send both “hello” and “goodbye” in the request.

Saving data from a response¶

When using the $ext key in the save block there is special behaviour - each key in the returned object will be saved as if it had been specified separately in the save object. The function is called in the same way as a validator function, in the $ext key of the save object.

Say that we have a server which returns a response like this:

{
  "user": {
    "name": "John Smith",
    "id": "abcdef12345"
  }
}

If our test function extracts the key name from the response body (note as above that this function should take the response object as the first argument):

# utils.py
from box import Box


def test_function(response):
    return Box({"test_user_name": response.json()["user"]["name"]})

We would use it in the save object like this:

save:
  $ext:
    function: utils:test_function
  json:
    test_user_id: user.id

In this case, both {test_user_name} and {test_user_id} are available for use in later requests.

A more complicated example¶

For a more practical example, the built in validate_jwt function also returns the decoded token as a dictionary wrapped in a Box object, which allows dot-notation access to members. This means that the contents of the token can be used for future requests. Because Tavern will already be in the Python path (because you installed it as a library) you do not need to modify the PYTHONPATH.

For example, if our server saves the user ID in the ‘sub’ field of the JWT:

- name: login
  request:
    url: http://server.com/login
    json:
      username: test_user
      password: abc123
  response:
    status_code: 200
    verify_response_with:
      # Make sure a token exists
      function: tavern.helpers:validate_jwt
      extra_kwargs:
        jwt_key: "token"
        options:
          verify_signature: false
    save:
      # Saves a jwt token returned as 'token' in the body as 'jwt'
      # in the test configuration for use in future tests
      # Note the use of $ext again
      $ext:
        function: tavern.helpers:validate_jwt
        extra_kwargs:
          jwt_key: "token"
          options:
            verify_signature: false

- name: Get user information
  request:
    url: "http://server.com/info/{jwt.sub}"
    ...
  response:
    ...

Ideas for other helper functions which might be useful:

• Making sure that the response matches a database schema

• Making sure that an error returns the correct error text in the body

• Decoding base64 data to extract some information for use in a future query

• Validate templated HTML returned from an endpoint using an XML parser

• etc.

One thing to bear in mind is that data can only be saved for use within the same test - each YAML document is considered to be a separate test (not counting anchors as described below). If you need to use the data in multiple tests, you will either need to put it into another file which you then include, or perform the same request in each test to re-fetch the data.

Hooks¶

As well as fixtures as mentioned in the previous section, since version 0.28.0 there is a couple of hooks which can be used to extract more information from tests.

These hooks are used by defining a function with the name of the hook in your conftest.py that take the same arguments with the same names - these hooks will then be picked up at runtime and called appropriately.

NOTE: These hooks should be considered a ‘beta’ feature, they are ready to use but the names and arguments they take should be considered unstable and may change in a future release (and more may also be added).

More documentation for these can be found in the docstrings for the hooks in the tavern/testutils/pytesthook/newhooks.py file.

Before every test run¶

This hook is called after fixtures, global configuration, and plugins have been loaded, but before formatting is done on the test and the schema of the test is checked. This can be used to ‘inject’ extra things into the test before it is run, such as configurations blocks for a plugin, or just for some kind of logging.

Example usage:

import logging


def pytest_tavern_beta_before_every_test_run(test_dict, variables):
    logging.info("Starting test %s", test_dict["test_name"])

    variables["extra_var"] = "abc123"

After every test run¶

This hook is called after execution of each test, regardless of the test result. The hook can, for example, be used to perform cleanup after the test is run.

Example usage:

import logging


def pytest_tavern_beta_after_every_test_run(test_dict, variables):
    logging.info("Ending test %s", test_dict["test_name"])

After every response¶

This hook is called after every response for each stage - this includes HTTP responses, but also MQTT responses if you are using MQTT. This means if you are using MQTT it might be called multiple times for each stage!

Example usage:

def pytest_tavern_beta_after_every_response(expected, response):
    with open("logfile.txt", "a") as logfile:
        logfile.write("Got response: {}".format(response.json()))

Before every request¶

This hook is called just before each request with the arguments passed to the request “function”. By default, this is Session.request (from requests) for HTTP and Client.publish (from paho-mqtt) for MQTT.

Example usage:

import logging


def pytest_tavern_beta_before_every_request(request_args):
    logging.info("Making request: %s", request_args)

Tinctures¶

Another way of running functions at certain times is to use the ‘tinctures’ functionality:

# package/helpers.py

import logging
import time

logger = logging.getLogger(__name__)


def time_request(stage):
    t0 = time.time()
    yield
    t1 = time.time()
    logger.info("Request for stage %s took %s", stage, t1 - t0)


def print_response(_, extra_print="affa"):
    logger.info("STARTING:")
    (expected, response) = yield
    logger.info("Response is %s (%s)", response, extra_print)

---
test_name: Test tincture

tinctures:
  - function: package.helpers:time_request

stages:
  - name: Make a request
    tinctures:
      - function: package.helpers:print_response
        extra_kwargs:
          extra_print: "blooble"
    request:
      url: "{host}/echo"
      method: POST
      json:
        value: "one"

  - name: Make another request
    request:
      url: "{host}/echo"
      method: POST
      json:
        value: "two"

Tinctures can be specified on a per-stage level or a per-test level. When specified on the test level, the tincture is run for every stage in the test. In the above example, the time_request function will be run for both stages, but the ‘print_response’ function will only be run for the first stage.

Tinctures are similar to fixtures but are more similar to external functions. Tincture functions do not need to be annotated with a function like Pytest fixtures, and are referred to in the same way (path.to.package:function), and have arguments passed to them in the same way (extra_kwargs, extra_args) as external functions.

The first argument to a tincture is always a dictionary of the stage to be run.

If a tincture has a yield in the middle of it, during the yield the stage itself will be run. If a return value is expected from the yield (eg (expected, response) = yield in the example above) then the expected return values and the response object from the stage will be returned. This allows a tincture to introspect the response, and compare it against the expected, the same as the pytest_tavern_beta_after_every_response hook. This response object will be different for MQTT and HTTP tests!

If you need to run something before every stage or after every response in your test suite, look at using the hooks instead.

Pytest fixtures¶

There is some support for Pytest fixtures in Tavern tests. This is done by using the usefixtures mark (see the documentation about using Pytest marks for more information about marks). The return (or yielded) values of any fixtures will be available to use in formatting, using the name of the fixture.

An example of how this can be used in a test:

# conftest.py

import pytest
import logging
import time


@pytest.fixture
def server_password():
    with open("/path/to/password/file", "r") as pfile:
        password = pfile.read().strip()

    return password


@pytest.fixture(name="time_request")
def fix_time_request():
    t0 = time.time()

    yield

    t1 = time.time()

    logging.info("Test took %s seconds", t1 - t0)

---
test_name: Make sure server can handle a big query

marks:
  - usefixtures:
      - time_request
      - server_password

stages:
  - name: Do big query
    request:
      url: "{host}/users"
      method: GET
      params:
        n_items: 1000
      headers:
        authorization: "Basic {server_password}"
    response:
      status_code: 200
      json:
        ...

The above example will load basic auth credentials from a file, which will be used to authenticate against the server. It will also time how long the test took and log it.

usefixtures expects a list of fixture names which are then loaded by Pytest - look at their documentation to see how discovery etc. works.

There are some limitations on fixtures:

• Fixtures are per test, not per stage. The above example of timing a test will include the (small) overhead of doing validation on the responses, setting up the requests session, etc. If the test consists of more than one stage, it will time how long both stages took.

• Fixtures should be ‘function’ or ‘session’ scoped. ‘module’ scoped fixtures will raise an error and ‘class’ scoped fixtures may not behave as you expect.

• Parametrizing fixtures does not work - this is a limitation in Pytest.

Fixtures which are specified as autouse can also be used without explicitly using usefixtures in a test. This is a good way to essentially precompute a format variable without also having to use an external function or specify a usefixtures block in every test where you need it.

To do this, just pass the autouse=True parameter to your fixtures along with the relevant scope. Using ‘session’ will evalute the fixture once at the beginning of your test run and reuse the return value everywhere else it is used:

@pytest.fixture(scope="session", autouse=True)
def a_thing():
    return "abc"

---
test_name: Test autouse fixture

stages:
  - name: do something with fixture value
    request:
      url: "{host}/echo"
      method: POST
      json:
        value: "{a_thing}"