Strict key checking

‘Strict’ key checking can be enabled or disabled globally, per test, or per stage. ‘Strict’ key checking refers to whether extra keys in the response should be ignored or whether they should raise an error. With strict key checking enabled, all keys in dictionaries at all levels have to match or it will raise an error. With it disabled, Extra keys in the response will be ignored as long as the ones in your response block are present.

Strict key checking can be controlled individually for the response for the JSON body, the redirect query parameter, text, or the headers.

By default, strict key checking is disabled for headers and redirect query parameters in the response, but enabled for JSON (as well as when checking for JSON in an mqtt response). This is because although there may be a lot of ‘extra’ things in things like the response headers (such as server agent headers, cache control headers, etc), the expected JSON body will likely always want to be matched exactly.

Effect of different settings¶

This is best explained through an example. If we expect this response from a server:

{
  "first": 1,
  "second": {
    "nested": 2
  }
}

This is what we would put in our Tavern test:

...
response:
  json:
    first: 1
    second:
      nested: 2

The behaviour of various levels of ‘strictness’ based on the response:

Response	strict=on	strict=off
`{ "first": 1, "second": { "nested": 2 } }`	PASS	PASS
`{ "first": 1 }`	FAIL	PASS
`{ "first": 1, "second": { "another": 2 } }`	FAIL	FAIL
`{ "first": 1, "second": { "nested": 2, "another": 2 } }`	FAIL	PASS

Turning ‘strict’ off also means that extra items in lists will be ignored as long as the ones specified in the test response are present. For example, if the response from a server is [ 1, 2, 3 ] then strict being on - the default for the JSON response body - will match only [1, 2, 3].

With strict being turned off for the body, any of these in the test will pass:

[1, 2, 3]
[1]
[2]
[3]
[1, 2]
[2, 3]
[1, 3]

But not:

[2, 4] - ‘4’ not present in response from the server
[3, 1], [2, 1] - items present, but out of order

To match the last case you can use the special setting list_any_order. This setting can only be used in the ‘json’ key of a request, but will match list items in any order as long as they are present in the response.

Changing the setting¶

This setting can be controlled in 3 different ways, the order of priority being:

In the test/stage itself
Passed on the command line
Read from pytest config

This means that using the command line option will not override any settings for specific tests.

Each of these methods is done by passing a sequence of strings indicating which section (json/redirect_query_params/headers) should be affected, and optionally whether it is on or off.

json:off headers:on - turn off for the body, but on for the headers. redirect_query_params will stay default off.
json:off headers:off - turn body and header strict checking off
redirect_query_params:on json:on redirect parameters is turned on and json is kept on (as it is on by default), header strict matching is kept off (as default).

Leaving the ‘on’ or ‘off’ at the end of each setting will imply ‘on’ - ie, using json headers redirect_query_params as an option will turn them all on.

Command line¶

There is a command line argument, --tavern-strict, which controls the default global strictness setting.

# Enable strict checking for body and headers only
py.test --tavern-strict json:on headers:on redirect_query_params:off -- my_test_folder/

In the Pytest config file¶

This behaves identically to the command line option, but will be read from whichever configuration file Pytest is using.

[pytest]
tavern-strict = json:off headers:on

Per test¶

Strictness can also be enabled or disabled on a per-test basis. The strict key at the top level of the test should a list consisting of one or more strictness setting as described in the previous section.

---

test_name: Make sure the headers match what I expect exactly

strict:
  - headers:on
  - json:off

stages:
  - name: Try to get user
    request:
      url: "{host}/users/joebloggs"
      method: GET
    response:
      status_code: 200
      headers:
        content-type: application/json
        content-length: 20
        x-my-custom-header: chocolate
      json:
        # As long as "id: 1" is in the response, this will pass and other keys will be ignored
        id: 1

A special option that can be done at the test level (or at the stage level, as described in the next section) is just to pass a boolean. This will turn strict checking on or off for all settings for the duration of that test/stage.

test_name: Just check for one thing in a big nested dict

# completely disable strict key checking for this whole test
strict: False

stages:
  - name: Try to get user
    request:
      url: "{host}/users/joebloggs"
      method: GET
    response:
      status_code: 200
      json:
        q:
          x:
            z:
              a: 1

Per stage¶

Often you have a standard stage before other stages, such as logging in to your server, where you only care if it returns a 200 to indicate that you’re logged in. To facilitate this, you can enable or disable strict key checking on a per-stage basis as well.

Two examples for doing this - these examples should behave identically:

---

# Enable strict checking for this test, but disable it for the login stage

test_name: Login and create a new user

# Force re-enable strict checking, in case it was turned off globally
strict:
  - json:on

stages:
  - name: log in
    request:
      url: "{host}/users/joebloggs"
      method: GET
    response:
      # Disable all strict key checking just for this stage
      strict: False
      status_code: 200
      json:
        logged_in: True
        # Ignores any extra metadata like user id, last login, etc.

  - name: Create a new user
    request:
      url: "{host}/users/joebloggs"
      method: POST
      json: &create_user
        first_name: joe
        last_name: bloggs
        email: joe@bloggs.com
    response:
      status_code: 200
      # Because strict was set 'on' at the test level, this must match exactly
      json:
        <<: *create_user
        id: 1

Or if strict json key checking was enabled at the global level:

---

test_name: Login and create a new user

stages:
  - name: log in
    request:
      url: "{host}/users/joebloggs"
      method: GET
    response:
      strict:
        - json:off
      status_code: 200
      json:
        logged_in: True

  - name: Create a new user
    request: ...

Matching arbitrary return values in a response¶

Sometimes you want to just make sure that a value is returned, but you don’t know (or care) what it is. This can be achieved by using !anything as the value to match in the response block:

response:
  json:
    # Will assert that there is a 'returned_uuid' key, but will do no checking
    # on the actual value of it
    returned_block: !anything

This would match both of these response bodies:

returned_block: hello

returned_block:
  nested: value

Using the magic !anything value should only ever be used inside pre-defined blocks in the response block (for example, headers, params, and json for a HTTP response).

NOTE: Up until version 0.7.0 this was done by setting the value as null. This creates issues if you want to ensure that your server is actually returning a null value. Using null is still supported in the current version of Tavern, but will be removed in a future release, and should raise a warning.

Matching arbitrary specific types in a response¶

If you want to make sure that the key returned is of a specific type, you can use one of the following markers instead:

!anynumber: Matches any number (integer or float)
!anyint: Matches any integer
!anyfloat: Matches any float (note that this will NOT match integers!)
!anystr: Matches any string
!anybool: Matches any boolean (this will NOT match null)
!anylist: Matches any list
!anydict: Matches any dict/‘mapping’

Matching via a regular expression¶

Sometimes you know something will be a string, but you also want to make sure that the string matches some kind of regular expression. This can be done using external functions, but as a shorthand there is also the !re_ family of custom YAML tags that can be used to match part of a response. Say that we want to make sure that a UUID returned is a version 4 UUID, where the third block must start with 4 and the third block must start with 8, 9, “A”, or “B”.

  - name: Check that uuidv4 is returned
    request:
      url: { host }/get_uuid/v4
      method: GET
    response:
      status_code: 200
      json:
        uuid: !re_fullmatch "[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89AB][0-9a-f]{3}-[0-9a-f]{12}"

This is using the !re_fullmatch variant of the tag - this calls re.fullmatch under the hood, which means that the regex given needs to match the entire part of the response that is being checked for it to pass. There is also !re_search which will pass if it matches part of the thing being checked, or !re_match which will match part of the thing being checked, as long as it is at the beginning of the string. See the Python documentation for more details.

Another way of doing this is to use the builtin validate_regex helper function. For example if we want to get a version that is returned in a ‘meta’ key in the format v1.2.3-510c2665d771e1:

stages:
  - name: get a token by id
    request:
      url: "{host}/tokens/get"
      method: GET
      params:
        id: 456
    response:
      status_code: 200
      json:
        code: abc123
        id: 456
        meta:
          version: !anystr
          hash: 456
      save:
        $ext:
          function: tavern.helpers:validate_regex
          extra_kwargs:
            expression: "v(?P<version>[\d\.]+)-[\w\d]+"
            in_jmespath: "meta.version"

This is a more flexible version of the helper which can also be used to save values as in the example. If a named matching group is used as shown above, the saved values can then be accessed in subsequent stages by using the regex.<group-name> syntax, eg:

- name: Reuse thing specified in first request
  request:
    url: "{host}/get_version_info"
    method: GET
    params:
      version: "{regex.version}"
  response:
    status_code: 200
    json:
      simple_version: "v{regex.version}"
      made_on: "2020-02-21"

Type conversions¶

YAML has some magic variables that you can use to coerce variables to certain types. For example, if we want to write an integer but make sure it gets converted to a string when it’s actually sent to the server we can do something like this:

request:
  json:
    an_integer: !!str 1234567890

However, due to the way YAML is loaded this doesn’t work when you are using a formatted value. Because of this, Tavern provides similar special constructors that begin with a single exclamation mark that will work with formatted values. Say we want to convert a value from an included file to an integer:

request:
  json:
    # an_integer: !!int "{my_integer:d}" # Error
    an_integer: !int "{my_integer:d}" # Works

Because curly braces are automatically formatted, trying to send one in a string might cause some unexpected issues. This can be mitigated by using the !raw tag, which will not perform string formatting.

Note: This is just shorthand for replacing a { with a {{ in the string

request:
  json:
    # Sent as {"raw_braces": "{not_escaped}"}
    raw_braces: !raw "{not_escaped}"