Tutorial: Modeling an API¶

Note

This tutorial assumes that you’ve read the Overview and Usage documentation.

Throughout this tutorial we will be modeling an API with restcli, gradually adding to it as we learn new concepts, until we have a complete API client suite. While the final result will be pasted at the end, I encourage you to follow along and do it yourself as we go. This will give you many opportunities to experiment and learn things you may not have learned otherwise!

Debriefing¶

You have been commissioned to build an API for the notorious secret society, the Sons of Secrecy. You were told the following information, in hushed whispers:

New members can join by invite only.
Each member has a rank within the Society.
Your rank determines how many secrets you are told.
Only the highest ranking members, called Whisperers, have the ability to recruit and promote members through the ranks.

Your task is to create a membership service for the Whisperers to keep track of and manage their underlings. Using the service, Whisperers must be able to:

Invite new members.
Promote or demote members’ ranks.
Send “secrets” to members.

In addition, the service must be guarded by a secret key, and no requests should go through if they do not contain the key.

Let’s get started!

Requests¶

We’ll start by modeling the new member invitation service:

# secrecy.yaml
---
memberships:
    invite:
        method: post
        url: "{{ server }}/memberships/invite"
        headers:
            Content-Type: application/json
            X-Secret-Key: '{{ secret_key }}'
        body: |
            name: {{ member_name }}
            age: {{ member_age }}
            can_keep_secrets: true

We made a new Collection and saved it as secrecy.yaml. So far it has one Group called memberships with one Request called invite.

As requested, we’ve also added an X-Secret-Key header which holds the secret key. It’s parameterized so that each Whisperer can have their own personal key. This will be explained later in the templating section.

Request Parameters¶

Let’s zoom in a bit on Requests. While we’re at it, we’ll inspect our invite Request more closely as well.

method (string, required)

HTTP method to use. Case insensitive.

We chose POST as our method for invite since POST is generally used for creating resources. Also, per RFC 7231, the POST method should be used when the request is non-idempotent.

url (string, required, templating)

Fully qualified URL that will receive the request. Supports templating.

We chose to parameterize the scheme://host portion of the URL as {{ server }}. As we’ll see later, this makes it easy to change the host without a lot of labor, and makes it clear that the path portion of the URL, /memberships/invite, is the real subject of this Request.

We’ll learn more about template variables later, but for now we know that invitations happen at /send_invite.

headers (object, ~templating)

HTTP headers to add. Keys and values must all be strings. Values support templating, but keys don’t.

We’re using the standard Content-Type header as well as a custom, parameterized header called X-Secret-Key. We’ll inspect this further in the templating section.

body (string, templating)

The request body. It must be encoded as a string, to facilitate the full power of Jinja2 templating. You’ll probably want to read the section on YAML block style at some point.

The body string must contain valid YAML, which is converted to JSON before sending the request. Only JSON encoding is supported at this time.

Our body parameter has 3 fields, name, age, and can_keep_secrets. The first two are parameterized, but we just set the third to true since keeping secrets is pretty much required if you’re gonna join the Sons of Secrecy.

script (string)

A Python script to be executed after the request finishes and a response is received. Scripts can be used to dynamically update the Environment based on the response payload. We’ll learn more about this later in scripting.

Our invite Request doesn’t have a script.

Templating¶

restcli supports Jinja2 templates in the url, headers, and body Request Parameters. This is used to parameterize Requests with the help of Environments. Any template variables in these parameters, denoted by double curly brackets, will be replaced with concrete values from the given Environment before the request is executed.

During the Debriefing, were told that the Whisperers can move members up the ranks if they’re deemed worthy. Well it just so happens that Wanda, a fledgling member, has proven herself as a devout secret-keeper.

We’ll start by adding another Request to our memberships Group:

# secrecy.yaml
---
memberships:
    invite: ...

    bump_rank:
        method: patch
        url: '{{ server }}/memberships/{{ member_id }}'
        headers:
            Content-Type: application/json
            X-Secret-Key: '{{ secret_key }}'
        body: |
            title: '{{ titles[rank + 1] }}'
            rank: '{{ rank + 1 }}'

Whew, lots of variables! Let’s whip up an Environment file for Wanda. This strategy has the advantage that we can seamlessly move between different members without making any changes to the Collection.

# wanda.env.yaml
---
server: 'https://www.secrecy.org'
secret_key: sup3rs3cr3t
titles:
    - Loudmouth
    - Seeker
    - Keeper
    - Confidant
    - Spectre
member_id: UGK882I59
rank: 0
#new_secrets:
#    - secret basement room full of kittens
#    - turtles all the way down

Todo

add new_secrets below, remove from above.

Note

The env.yaml extension in wanda.env.yaml is just a convention to identify the file as an Environment. Any extension may be used.

We’re almost ready to run it, but let’s change server to something real so we don’t get any errors:

server: http://httpbin.org/anything

Now we’ll run the request:

$ restcli -c secrecy.yaml -e wanda.env.yaml run memberships bump_rank

Here’s what restcli does when we hit enter:

Load the Collection (secrecy.yaml) and locate the Request memberships.bump_rank.
Load the Environment (wanda.yaml).
Use the Environment to execute the contents of the url, headers, and body parameters as Jinja2 Templates,.
Run the resulting HTTP request.

If we could view the finalized Request object before running it in #4, this is what it would look like:

# secrecy.yaml

method: post
url: 'https://www.secrecy.org/memberships/12345/bump_rank'
headers:
    Content-Type: application/json
    X-Secret-Key: sup3rs3cr3t
body: |
    rank: 1
    title: Seeker

Here’s a piece-by-piece breakdown of what happened:

In the url section:
- {{ server }} was replaced with the value of Environment variable server.
- {{ member_id }} was replaced with the value of Environment variable member_id.
In the headers section, {{ secret_key }} was replaced with the value of Environment variable secret_key.
In the body section:
- {{ rank }} was replaced with the value of Environment variable rank, incremented by 1.
- {{ title }} was replaced by an item from the Environment variable titles, an array, by indexing it with the incremented rank value.

Note

When it gets a request, http://httpbin.org/anything echoes back the URL, headers, and request body in the response. You can use this to check your work. If something is off, be sure to fix it before we continue.

Congrats on your new rank Wanda!

What we just learned should cover most use cases, but if you need more power or just want to explore, there’s much more to templating than what we just covered! restcli supports the entire Jinja2 template language, so check out the official Template Designer Documentation for the whole scoop.

Scripting¶

Templating is a powerful feature that allows you to make modular, reusable Requests which encapsulate particular functions of your API without being tied to specifics. We demonstrated this by modeling a function to increase a member’s rank, and created an Environment file to use it on Wanda. If we wanted to do the same for another member, we’d simply create a new Environment.

However, what happens when it’s time for Wanda’s second promotion? We know her current rank is 1, but the Environment still says 0. If we ran the bump_rank Request on the same Environment again, we’d get the same result:

# secrecy.yaml

body: |
    rank: 1
    title: Seeker

We need a way to update the Environment automatically after we run the Request.

This is achieved through scripting. As mentioned earlier in Request Parameters, each Request supports an optional script parameter which contains Python code. It is evaluated after the request is ran, and can modify the current Environment.

Let’s add a script to our bump_rank Request:

# secrecy.yaml

bump_rank:
    ...
    script: |
        env['rank'] += 1

Now each time we run bump_rank it will update the Environment with the new value. Let’s run it again to see the changes in action:

$ restcli --save -c secrecy.yaml -e wanda.env.yaml run memberships bump_rank

Notice that we added the --save flag. Without this, changes to the Environment would not be saved to disk.

Open up your Environment file and make sure rank was updated successfully.

Note

All script examples were written for Python3.7, but most will probably work in Python3+. To get version info, including the Python version, use the --version flag:

$ restcli --version

Under the hood, scripts are executed with the Python builtin exec(), which is called with a code object containing the script as well as a globals dict containing the following variables:

response

A Response object from the Python requests library, which contains the status code, response headers, response body, and a lot more. Check out the Response API for a detailed list.

env

A Python dict which contains the entire hierarchy of the current Collection. It is mutable, and editing its contents may result in one or both of the following effects:

If running in interactive mode, any changes made will persist in the active Environment until the session ends.
If autosave is enabled, the changes will be saved to disk.

Any functions or variables imported in the lib section of the Config document will be available in your scripts as well. We’ll tackle the Config document in the next section.

Note

Since Python is whitespace sensitive, you’ll probably want to read the section on YAML block style.

The Config Document¶

So far our Collections have been composed of a single YAML document. restcli supports an optional second document per Collection as well, called the Config Document.

Note

If you’re not sure what “document” means in YAML, here’s a quick primer:

Essentially, documents allow you to have more than one YAML “file” (document) in the same file. Notice that --- that appears at the top of each example we’ve looked at? That’s how you tell YAML where your document begins.

Technically, the spec has more rules than that for documents but PyYAML, the library restcli uses, isn’t that strict. Here’s the spec anyway if you’re interested: http://yaml.org/spec/1.2/spec.html#id2800132

If present, the Config Document must appear before the Requests document. Breaking it down, a Collection must either:

contain exactly one document, the Requests document, or
contain exactly two documents; the Config Document and the Requests document, in that order.

Let’s add a Config Document to our Secretmasons Collection. We’ll take a look and then jump into explanations after:

# secrecy.yaml
---
defaults:
    headers:
        Content-Type: application/json
        X-Secret-Key: '{{ secret_key }}'
lib:
    - restcli.contrib.scripts

---
memberships:
    invite: ...

    upgrade: ...

Config Parameters¶

The Config Document is used for global configuration in general, so the parameters defined here don’t have much in common.

defaults (object)

Default values to use for each Request parameter when not specified in the Request. defaults has the same structure as a Request, so each parameters defined here must also be valid as a Request parameter.

lib (array)

lib is an array of Python module paths. Each module here must contain a function with the signature define(request, env, *args, **kwargs) which returns a dict. That dict will be added to the execution environment of any script that gets executed after a Request is completed.

restcli ships with a pre-baked lib module at restcli.contrib.scripts. It provides some useful utility functions to use in your scripts. It can also be used as a learning tool.

Appendix¶

A. YAML Block Style¶

Writing multiline strings for the body and script Request parameters without losing readability is easy with YAML’s block style. I recommend using literal style since it preserves whitespace and is the most readable. Adding to the example above:

body: |
    name: bar
    age: {{ foo_age }}
    attributes:
        fire_spinning: 32
        basket_weaving: 11

The vertical bar (|) denotes the start of a literal block, so newlines are preserved, as well as any additional indentation. In this example, the result is that the value of body is 5 lines of text, with the last two lines indented 4 spaces.

Note that it is impossible to escape characters within a literal block, so if that’s something you need you may have to try a different

Tutorial: Modeling an API¶

Debriefing¶

Requests¶

Request Parameters¶

Templating¶

Scripting¶

The Config Document¶

Config Parameters¶

Appendix¶

A. YAML Block Style¶

restcli

Navigation

Related Topics