API details

GitHub repository: kernelci-api

This guide describes the KernelCI API components such as data models and the Pub/Sub interface in detail. It also explains how to use the API directly for setting things up and issuing low-level queries.

Environment Variables

General instructions about the environment file are described on the local instance page. This section goes through all the environment variables used by the API.

Set ALGORITHM and ACCESS_TOKEN_EXPIRE_MINUTES in environment file

We need to specify an algorithm for JWT token encoding and decoding. ALGORITHM variable needs to be passed in the parameter for that. ALGORITHM is set default to HS256. We have used ACCESS_TOKEN_EXPIRE_MINUTES variable to set expiry time on generated jwt access token. ACCESS_TOKEN_EXPIRE_MINUTES is set default to None. If a user wants to change any of the above variables, they should be added to the .env file.

Configure Redis and Mongo

By default, API uses Redis and Database services specified in docker-compose.yaml. API is configured to use redis hostname redis and database service URL mongodb://db:27017 at the moment. In case of using different services or configurations, REDIS_HOST and MONGO_SERVICE variables should be added to .env file.

Users

This section describes API user accounts and various endpoints for user management.

Create an admin user

The very first admin user needs to be created with api.admin tool provided in the kernelci-api repository. Here is a guide to setup an admin user. We can use this admin user to create other user accounts.

Invite user (Admin only, required)

The recommended onboarding flow is invite-only:

1. Admin creates (or re-sends) an invite using POST /user/invite
2. User opens the invite link and sets a password (this also verifies the account)

Invite a new user (and return the token/link in the response for CLI usage):

$ curl -X 'POST' \
  'http://localhost:8001/latest/user/invite' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer <ADMIN-AUTHORIZATION-TOKEN>' \
  -d '{
  "username": "test",
  "email": "test@kernelci.org",
  "groups": [],
  "is_superuser": false,
  "send_email": false,
  "return_token": true,
  "resend_if_exists": false
}'

When send_email is false, no SMTP configuration is required and the response includes invite_url and token (when return_token is set) so you can deliver the link manually.

If a user already exists, set resend_if_exists to true and ensure the username and email match the existing user. Invites can only be resent to unverified users.

To preview which public URL will be used in invite links (admin-only):

$ curl -X 'GET' \
  'http://localhost:8001/latest/user/invite/url' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer <ADMIN-AUTHORIZATION-TOKEN>'

The public URL can be overridden via PUBLIC_BASE_URL in the environment.

To accept an invite and set the password (no authentication required):

$ curl -X 'POST' \
  'http://localhost:8001/latest/user/accept-invite' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "token": "<INVITE-TOKEN>",
  "password": "<new-password>"
}'

There is also a minimal web UI:

• Admin invite page: GET /user/invite
• Invite acceptance page: GET /user/accept-invite?token=<INVITE-TOKEN>

CLI-oriented user management

This section summarizes the minimum admin and user flows needed to implement a CLI. Use these endpoints verbatim and expect JSON responses unless noted.

Helper script:

• scripts/usermanager.py provides a small CLI for these endpoints.
• Optional config file (checked in order): ./usermanager.toml, ~/.config/kernelci/usermanager.toml
• Environment overrides: KCI_API_URL, KCI_API_TOKEN
• Optional instance selection: --instance or KCI_API_INSTANCE

Example config:

default_instance = "local"

[instances.local]
url = "http://localhost:8001/latest"
token = "<admin-or-user-token>"

[instances.staging]
url = "https://staging.kernelci.org/latest"
token = "<admin-or-user-token>"

Use --instance staging (or KCI_API_INSTANCE=staging) to select an instance.

Admin flow (requires admin bearer token):

• Create invite: POST /user/invite
  • Request fields: username, email, groups, is_superuser, send_email, return_token, resend_if_exists
  • Response fields: user, email_sent, public_base_url, accept_invite_url, invite_url (optional), token (optional)
  • Error cases: 400 if user exists and resend_if_exists is false, or if existing user is already verified, or if username/email mismatch
• Preview public link base: GET /user/invite/url
• List users: GET /users
• Get user by ID: GET /user/{id}
• Update user: PATCH /user/{id}
• Delete user: DELETE /user/{id}

User flow (no auth until invite accepted):

• Accept invite: POST /user/accept-invite
  • Request fields: token, password
  • Error cases: 400 invalid/expired token, inactive user, or already accepted invite; 404 user not found
• Get auth token: POST /user/login (form-encoded username, password)
• Who am I: GET /whoami
• Update own profile: PATCH /user/me
• Update password: POST /user/update-password
• Forgot/reset password: POST /user/forgot-password, then POST /user/reset-password

Invite link composition:

• Default base uses request host; PUBLIC_BASE_URL overrides.
• The acceptance URL is PUBLIC_BASE_URL + /user/accept-invite with a token query parameter.

Examples for a CLI:

Invite a user and return the token/link:

$ curl -X 'POST' \
  'http://localhost:8001/latest/user/invite' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer <ADMIN-AUTHORIZATION-TOKEN>' \
  -d '{
  "username": "alice",
  "email": "alice@example.org",
  "groups": ["kernelci"],
  "is_superuser": false,
  "send_email": false,
  "return_token": true,
  "resend_if_exists": false
}'

Sample response:

{
  "user": {
    "id": "6526448e7d140ee220971a0e",
    "email": "alice@example.org",
    "is_active": true,
    "is_superuser": false,
    "is_verified": false,
    "username": "alice",
    "groups": [{"id":"648ff894bd39930355ed16ad","name":"kernelci"}]
  },
  "email_sent": false,
  "public_base_url": "http://localhost:8001",
  "accept_invite_url": "http://localhost:8001/user/accept-invite",
  "invite_url": "http://localhost:8001/user/accept-invite?token=<INVITE-TOKEN>",
  "token": "<INVITE-TOKEN>"
}

Accept an invite:

$ curl -X 'POST' \
  'http://localhost:8001/latest/user/accept-invite' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "token": "<INVITE-TOKEN>",
  "password": "<new-password>"
}'

Get an auth token:

$ curl -X 'POST' \
  'http://localhost:8001/latest/user/login' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'username=alice&password=<new-password>'

Get authorization token

After successful user activation via an invite, the user can retrieve an authorization token to use certain API endpoints requiring user authorization.

$ curl -X 'POST' \
'http://localhost:8001/latest/user/login' \
-H 'accept: application/json' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-d 'username=test&password=test'

This will return an authorization bearer token.

Get all existing users

To get information of all added user accounts, user GET /users request.

$ curl -X 'GET' \
'http://localhost:8001/latest/users' \
-H 'accept: application/json' \
-H 'Authorization: <USER-AUTHORIZATION-TOKEN>'
{"items":[{"id":"6526448e7d140ee220971a0e","email":"admin@gmail.com","is_active":true,"is_superuser":true,"is_verified":true,"username":"admin","groups":[]},{"id":"615f30020eb7c3c6616e5ac3","email":"test-user@kernelci.org","is_active":true,"is_superuser":true,"is_verified":false,"username":"test-user","groups":[{"id":"648ff894bd39930355ed16ad","name":"kernelci"}]}],"total":2,"limit":50,"offset":0}

Get user account matching user ID

To get user by ID, use /user endpoint with user ID as a path parameter:

$ curl -X 'GET' \
'http://localhost:8001/latest/user/6526448e7d140ee220971a0e' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer <USER-AUTHORIZATION-TOKEN>'
{"id":"6526448e7d140ee220971a0e","email":"admin@gmail.com","is_active":true,"is_superuser":true,"is_verified":true,"username":"admin","groups":[]}

Reset password

A user can reset password for the account in case the password is forgotten or lost.

First, send request to POST /user/forgot-password endpoint with the user email address:

$ curl -X 'POST' \
  'http://localhost:8001/latest/user/forgot-password' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "email": "test@kernelci.org"
}'

The user will receive password reset token via email. The token should be sent to POST /user/reset-password request along with the new password to be set for the account:

$ curl -X 'POST' \T' \
  'http://localhost:8001/latest/user/reset-password' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "token": "PASSWORD-RESET-TOKEN-RECEIVED-BY-EMAIL",
  "password": "<new-password>"
}'

The user will receive an email confirming a successful password reset.

Update user password

A user can update password for the account with the request to /user/update-password endpoint. Please supply current password and new password along with the username for the account:

$ curl -X 'POST' \
'http://localhost:8001/latest/user/update-password' \
-H 'accept: application/json' \
-H 'Content-Type: application/x-www-form-urlencoded' \
-d 'username=test&password=<current-password>&new_password=<new-password>'

Update own user account

A user can update certain information for its own account, such as email, username, and password with a PATCH /user/me request. For example,

$ curl -X 'PATCH' \
'http://localhost:8001/latest/user/me' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer <USER-AUTHORIZATION-TOKEN>' \
-d '{
  "password": "<new-password-to-be-set>",
  "email": "<new-email-to-be-set>",
}'

Please note that user management fields such as is_useruser, is_verified, and is_active can not be updated by this request for security purposes. User group membership can only be updated by admin users.

Update an existing user account (Admin only)

Admin users can update other existing user accounts using a PATCH /user/<user-id> request.

For example, the below command will update an existing test user with a new email address and add it to kernelci user group.

$ curl -X 'PATCH' \
'http://localhost:8001/latest/user/615f30020eb7c3c6616e5ac3' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer <ADMIN-USER-AUTHORIZATION-TOKEN>' \
-d '{"email": "test-user@kernelci.org", "groups": ["kernelci"]}'

User groups and permissions

User groups are plain name strings stored in the usergroup collection. Group names must already exist before they can be assigned to users; otherwise the API returns 400.

User groups are plain name strings stored in the usergroup collection. You can manage them via the API endpoints below or directly with MongoDB tooling. Example with mongosh:

$ mongosh "mongodb://db:27017/kernelci"
> db.usergroup.insertOne({name: "runtime:lava-collabora:node-editor"})

Admin-only user group management endpoints are available:

• GET /user-groups (list; supports name filter)
• GET /user-groups/<group-id>
• POST /user-groups with {"name": "runtime:lava-collabora:node-editor"}
• DELETE /user-groups/<group-id> (fails with 409 if assigned to users)

Admin users can assign or remove groups via:

• POST /user/invite with a groups list
• PATCH /user/<user-id> with groups
• scripts/usermanager.py update-user --data '{"groups": [...]}'

To remove a group, send a groups list that omits it; the list replaces the existing groups.

Example using the helper script:

$ ./scripts/usermanager.py list-users
$ ./scripts/usermanager.py list-groups
$ ./scripts/usermanager.py create-group runtime:lava-collabora:node-editor
$ ./scripts/usermanager.py update-user 615f30020eb7c3c6616e5ac3 \
  --data '{"groups": ["runtime:lava-collabora:node-editor"]}'

Users cannot update their own groups; admin access is required.

Usermanager workflows (examples)

These examples use scripts/usermanager.py. It reads ./usermanager.toml or ~/.config/kernelci/usermanager.toml by default, and you can override with --api-url/--token or KCI_API_URL/KCI_API_TOKEN.

Common admin workflows:

• List users and capture IDs:

$ ./scripts/usermanager.py list-users
$ ./scripts/usermanager.py get-user <USER-ID>

• Invite a user (optionally add groups):

$ ./scripts/usermanager.py invite \
  --username alice \
  --email alice@example.org \
  --groups runtime:pull-labs-demo:node-editor \
  --return-token

• Accept an invite manually (useful for service accounts or testing):

$ ./scripts/usermanager.py accept-invite --token "<INVITE-TOKEN>"

• Login to get a bearer token:

$ ./scripts/usermanager.py login --username alice

• Deactivate or reactivate a user:

$ ./scripts/usermanager.py update-user <USER-ID> --inactive
$ ./scripts/usermanager.py update-user <USER-ID> --active

• Grant or revoke superuser:

$ ./scripts/usermanager.py update-user <USER-ID> --superuser
$ ./scripts/usermanager.py update-user <USER-ID> --no-superuser

• Mark a user verified or unverified (admin only):

$ ./scripts/usermanager.py update-user <USER-ID> --verified
$ ./scripts/usermanager.py update-user <USER-ID> --unverified

• Assign or remove groups:

$ ./scripts/usermanager.py update-user <USER-ID> \
  --add-group runtime:pull-labs-demo:node-editor
$ ./scripts/usermanager.py update-user <USER-ID> \
  --remove-group runtime:pull-labs-demo:node-editor
$ ./scripts/usermanager.py update-user <USER-ID> \
  --set-groups runtime:pull-labs-demo:node-editor,team-a

• Set a password (admin only, useful for service accounts):

$ ./scripts/usermanager.py update-user <USER-ID> --password "<new-password>"

• Manage user groups:

$ ./scripts/usermanager.py list-groups
$ ./scripts/usermanager.py create-group runtime:pull-labs-demo:node-editor
$ ./scripts/usermanager.py delete-group runtime:pull-labs-demo:node-editor

• Delete a user:

$ ./scripts/usermanager.py delete-user <USER-ID>

Permissions and node update rules

Node update permissions are determined by the user and the node being edited:

• Superusers can update any node.
• The node owner can update their own nodes.
• Users with group node:edit:any can update any node.
• Users with a group listed in the node’s user_groups can update that node.
• Users with runtime:<runtime>:node-editor or runtime:<runtime>:node-admin can update nodes whose data.runtime matches <runtime>.

Example: allow updates only for runtime pull-labs-demo:

$ mongosh "mongodb://db:27017/kernelci"
> db.usergroup.insertOne({name: "runtime:pull-labs-demo:node-editor"})

$ ./scripts/usermanager.py update-user <USER-ID> \
  --add-group runtime:pull-labs-demo:node-editor

To remove a user group definition entirely, delete it in MongoDB:

$ mongosh "mongodb://db:27017/kernelci"
> db.usergroup.deleteOne({name: "runtime:pull-labs-demo:node-editor"})

Delete user matching user ID (Admin only)

Only admin users can delete existing user account by providing user ID to DELETE /user/<user-id> endpoint:

$ curl -X 'DELETE' \
'http://localhost:8001/latest/user/658d1edecf0bce203d594f1c' \
-H 'Authorization: Bearer <ADMIN-USER-AUTHORIZATION-TOKEN>'

Nodes

Node objects form the basis of the API models to represent tests runs, kernel builds, regressions and other test-related entities and their relationships. See the model definitions in kernelci-core for details on the Node model and its subtypes. It’s possible to create new objects and retrieve them via the API.

Create a Node

To create a Node or a Node subtype object, for instance, a Checkout node, a POST request should be made along with the Node attributes. This requires an authentication token:

$ curl -X 'POST' \
  'http://localhost:8001/latest/node' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJib2IifQ.ci1smeJeuX779PptTkuaG1SEdkp5M1S1AgYvX8VdB20' \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "checkout",
    "kind": "checkout",
    "path": ["checkout"],
    "data": {
      "kernel_revision": {
        "tree": "mainline",
        "url": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
        "branch": "master",
        "commit": "2a987e65025e2b79c6d453b78cb5985ac6e5eb26",
        "describe": "v5.16-rc4-31-g2a987e65025e"
      }
    }
  }' | jq

{
  "id": "61bda8f2eb1a63d2b7152418",
  "kind": "checkout",
  "name": "checkout",
  "path": [
    "checkout"
  ],
  "group": null,
  "parent": null,
  "state": "running",
  "result": null,
  "artifacts": null,
  "data": {
    "kernel_revision": {
      "tree": "mainline",
      "url": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
      "branch": "master",
      "commit": "2a987e65025e2b79c6d453b78cb5985ac6e5eb26",
      "describe": "v5.16-rc4-31-g2a987e65025e"
    }
  },
  "created": "2024-02-01T09:58:28.479138",
  "updated": "2024-02-01T09:58:28.479142",
  "timeout": "2024-02-01T15:58:28.479145",
  "holdoff": null,
  "owner": "admin",
  "user_groups": []
}

Getting Nodes back

Reading Node doesn’t require authentication, so plain URLs can be used.

To get node by ID, use /node endpoint with node ID as a path parameter:

$ curl http://localhost:8001/latest/node/61bda8f2eb1a63d2b7152418 | jq

{
  "id": "61bda8f2eb1a63d2b7152418",
  "kind": "checkout",
  "name": "checkout",
  "path": [
    "checkout"
  ],
  "group": null,
  "parent": null,
  "state": "running",
  "result": null,
  "artifacts": null,
  "data": {
    "kernel_revision": {
      "tree": "mainline",
      "url": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
      "branch": "master",
      "commit": "2a987e65025e2b79c6d453b78cb5985ac6e5eb26",
      "describe": "v5.16-rc4-31-g2a987e65025e"
    }
  },
  "created": "2024-02-01T09:58:28.479000",
  "updated": "2024-02-01T09:58:28.479000",
  "timeout": "2024-02-01T15:58:28.479000",
  "holdoff": null,
  "owner": "admin",
  "user_groups": []
}

To get all the nodes as a list, use the /nodes API endpoint:

$ curl http://localhost:8001/latest/nodes
{
  "items": [
    {
      "id": "65a1355ee98651d0fe81e412",
      "kind": "node",
      "name": "time_test_cases",
      "path": [
        "checkout",
        "kunit-x86_64",
        "exec",
        "time_test_cases"
      ],
      "group": "kunit-x86_64",
      "parent": "65a1355ee98651d0fe81e40f",
      "state": "done",
      "result": null,
      "artifacts": null,
      "data": {
        "kernel_revision": {
          "tree": "mainline",
          "url": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
          "branch": "master",
          "commit": "70d201a40823acba23899342d62bc2644051ad2e",
          "describe": "v6.7-6264-g70d201a40823",
          "version": {
            "version": "6",
            "patchlevel": "7",
            "extra": "-6264-g70d201a40823"
          }
        }
      },
      "created": "2024-01-12T12:49:33.996000",
      "updated": "2024-01-12T12:49:33.996000",
      "timeout": "2024-01-12T18:49:33.996000",
      "holdoff": null,
      "owner": "admin",
      "user_groups": []
    },
    {
      "id": "65a1355ee98651d0fe81e413",
      "kind": "node",
      "name": "time64_to_tm_test_date_range",
      "path": [
        "checkout",
        "kunit-x86_64",
        "exec",
        "time_test_cases",
        "time64_to_tm_test_date_range"
      ],
      "group": "kunit-x86_64",
      "parent": "65a1355ee98651d0fe81e412",
      "state": "done",
      "result": "pass",
      "artifacts": null,
      "data": {
        "kernel_revision": {
          "tree": "mainline",
          "url": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
          "branch": "master",
          "commit": "70d201a40823acba23899342d62bc2644051ad2e",
          "describe": "v6.7-6264-g70d201a40823",
          "version": {
            "version": "6",
            "patchlevel": "7",
            "extra": "-6264-g70d201a40823"
          }
        }
      },
      "created": "2024-01-12T12:49:33.996000",
      "updated": "2024-01-12T12:49:33.996000",
      "timeout": "2024-01-12T18:49:33.996000",
      "holdoff": null,
      "owner": "admin",
      "user_groups": []
    },
    ...

To get nodes by providing attributes, use /nodes endpoint with query parameters. All the attributes except node ID can be passed to this endpoint. In case of ID, please use /node endpoint with node ID as described above.

$ curl 'http://localhost:8001/latest/nodes?kind=checkout&data.kernel_revision.tree=mainline' | jq

{
  "items": [
    {
      "id": "65a3982ee98651d0fe82b010",
      "kind": "checkout",
      "name": "checkout",
      "path": [
        "checkout"
      ],
      "group": null,
      "parent": null,
      "state": "done",
      "result": null,
      "artifacts": {
        "tarball": "https://kciapistagingstorage1.file.core.windows.net/staging/linux-mainline-master-v6.7-9928-g052d534373b7.tar.gz?sv=2022-11-02&ss=f&srt=sco&sp=r&se=2024-10-17T19:19:12Z&st=2023-10-17T11:19:12Z&spr=https&sig=sLmFlvZHXRrZsSGubsDUIvTiv%2BtzgDq6vALfkrtWnv8%3D"
      },
      "data": {
        "kernel_revision": {
          "tree": "mainline",
          "url": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
          "branch": "master",
          "commit": "052d534373b7ed33712a63d5e17b2b6cdbce84fd",
          "describe": "v6.7-9928-g052d534373b7",
          "version": {
            "version": "6",
            "patchlevel": "7",
            "extra": "-9928-g052d534373b7"
          }
        }
      },
      "created": "2024-01-14T08:15:42.454000",
      "updated": "2024-01-14T09:16:47.689000",
      "timeout": "2024-01-14T09:15:42.344000",
      "holdoff": "2024-01-14T08:46:39.040000",
      "owner": "admin",
      "user_groups": []
    },
    {
      "id": "65a3a545e98651d0fe82b4ed",
      "kind": "checkout",
      "name": "checkout",
      "path": [
        "checkout"
      ],
      "group": null,
      "parent": null,
      "state": "done",
      "result": null,
      "artifacts": {
        "tarball": "https://kciapistagingstorage1.file.core.windows.net/staging/linux-mainline-master-v6.7-9928-g052d534373b7.tar.gz?sv=2022-11-02&ss=f&srt=sco&sp=r&se=2024-10-17T19:19:12Z&st=2023-10-17T11:19:12Z&spr=https&sig=sLmFlvZHXRrZsSGubsDUIvTiv%2BtzgDq6vALfkrtWnv8%3D"
      },
      "data": {
        "kernel_revision": {
          "tree": "mainline",
          "url": "https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
          "branch": "master",
          "commit": "052d534373b7ed33712a63d5e17b2b6cdbce84fd",
          "describe": "v6.7-9928-g052d534373b7",
          "version": {
            "version": "6",
            "patchlevel": "7",
            "extra": "-9928-g052d534373b7"
          }
        }
      },
      "created": "2024-01-14T09:11:33.029000",
      "updated": "2024-01-14T10:11:48.092000",
      "timeout": "2024-01-14T10:11:32.922000",
      "holdoff": "2024-01-14T09:40:19.284000",
      "owner": "admin",
      "user_groups": []
    }
    ...

Attributes along with comparison operators are also supported for the /nodes endpoint. The attribute name and operator should be separated by __ i.e. attribute__operator. Supported operators are lt(less than), gt(greater than), lte(less than or equal to), gte(greater than or equal to) and re (regular expression matching).

$ curl 'http://localhost:8001/latest/nodes?kind=checkout&created__gt=2022-12-06T04:59:08.102000'

Note In order to support comparison operators in URL request parameters, models can not contain __ in the field name.

Additionally, the re operator offers some basic regular expression matching capabilities for query parameters. For instance:

$ curl 'http://localhost:8001/latest/nodes?kind=kbuild&name__re=x86'

returns all Kbuild nodes with the string “x86” in the node name.

API also supports multiple operator queries for the same field name. For example, date range queries can be triggered as below:

$ curl 'https://localhost:8001/nodes?created__gt=2024-02-26T13:21:55.301000&created__lt=2024-11-01&kind=checkout'

The above query will return all the checkout nodes in the specific date range provided, i.e., nodes created between 2024-02-26T13:21:55.301000 and 2024-11-01 date-time range.

Nodes with null fields can also be retrieved using the endpoint. For example, the below command will get all the nodes with parent field set to null:

$ curl 'http://localhost:8001/latest/nodes?parent=null'
"items":[{"_id":"63c549319fb3b62c7626e7f9","kind":"node","name":"checkout","path":["checkout"],"group":null,"data":{"kernel_revision":{"tree":"kernelci","url":"https://github.com/kernelci/linux.git","branch":"staging-mainline","commit":"1385303d0d85c68473d8901d69c7153b03a3150b","describe":"staging-mainline-20230115.1","version":{"version":6,"patchlevel":2,"sublevel":null,"extra":"-rc4-2-g1385303d0d85","name":null}}},"parent":null,"state":"available","result":null,"artifacts":{"tarball":"http://172.17.0.1:8002/linux-kernelci-staging-mainline-staging-mainline-20230115.1.tar.gz"},"created":"2023-01-16T12:55:13.879000","updated":"2023-01-16T12:55:51.780000","timeout":"2023-01-16T13:55:13.877000","holdoff":"2023-01-16T13:05:51.776000"},{"_id":"63c549329fb3b62c7626e7fa","kind":"node","name":"checkout","path":["checkout"],"group":null,"data":{"kernel_revision":{"tree":"kernelci","url":"https://github.com/kernelci/linux.git","branch":"staging-next","commit":"39384a5d7e2eb2f28039a92c022aed886a675fbf","describe":"staging-next-20230116.0","version":{"version":6,"patchlevel":2,"sublevel":null,"extra":"-rc4-5011-g39384a5d7e2e","name":null}}},"parent":null,"state":"available","result":null,"artifacts":{"tarball":"http://172.17.0.1:8002/linux-kernelci-staging-next-staging-next-20230116.0.tar.gz"},"created":"2023-01-16T12:55:14.706000","updated":"2023-01-16T12:56:30.886000","timeout":"2023-01-16T13:55:14.703000","holdoff":"2023-01-16T13:06:30.882000"}],"total":2,"limit":50,"offset":0}

Please make sure that the query parameter provided with the null value in the request exists in the Node schema. Otherwise, the API will behave unexpectedly and return all the nodes.

Update a Node

To update an existing node, use PUT request to node/{node_id} endpoint.

$ curl -X 'PUT' \
  'http://localhost:8001/latest/node/61bda8f2eb1a63d2b7152418' \
  -H 'accept: application/json' \
  -H 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJib2IifQ.ci1smeJeuX779PptTkuaG1SEdkp5M1S1AgYvX8VdB20' \
  -H 'Content-Type: application/json' \
  -d '{
  "name":"checkout-test",
  "data":{
    "kernel_revision":{
      "tree":"mainline",
      "url":"https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git",
      "branch":"master",
      "commit":"2a987e65025e2b79c6d453b78cb5985ac6e5eb26",
      "describe":"v5.16-rc4-31-g2a987e65025e"
    },
  },
  "created":"2022-02-02T11:23:03.157648"
}'
{"id":"61bda8f2eb1a63d2b7152418","kind":"node","name":"checkout-test","data":{"kernel_revision":{"tree":"mainline","url":"https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git","branch":"master","commit":"2a987e65025e2b79c6d453b78cb5985ac6e5eb26","describe":"v5.16-rc4-31-g2a987e65025e"}},"parent":null,"status":"pending","result":null, "created":"2022-02-02T11:23:03.157648", "updated":"2022-02-02T12:23:03.157648"}

Getting Nodes count

To get a count of all the nodes, use GET request to /count endpoint.

$ curl http://localhost:8001/latest/count
4

To get count of nodes matching provided attributes, use /count endpoint with query parameters. All the Node attributes except ID and timestamps(created, updated, timeout, and holdoff) can be passed to this endpoint.

$ curl http://localhost:8001/latest/count?name=checkout
3
$ curl http://localhost:8001/latest/count?data.kernel_revision.branch=staging-mainline
1

In case of providing multiple attributes, it will return count of nodes matching all the attributes.

$ curl 'http://localhost:8001/latest/count?name=checkout&artifacts.tarball=http://172.17.0.1:8002/linux-kernelci-staging-mainline-staging-mainline-20220927.0.tar.gz'
1

Same as /nodes, the /count endpoint also supports comparison operators for request query parameters.

$ curl 'http://localhost:8001/latest/count?name=checkout&created__lt=2022-12-06T04:59:08.102000'
3

To query the count of nodes with null attributes, use the endpoint with query parameters set to null.

$ curl 'http://localhost:8001/latest/count?result=null'
2

State diagram

The Node objects are governed by the following state machine:

graph LR
  running(Running) --> job{successful?}
  running -.-> |timeout| done(Done)
  job --> |no| done(Done)
  job --> |yes| available(Available)
  available -.-> |timeout| done
  available --> |holdoff expired| closing(Closing)
  closing -.-> |timeout| done
  closing --> |child nodes done| done

The state of the Node is kept in the state field in the Node models. The different state values are described below:

Running

The job has been scheduled and there is no result yet.

Available

The job was successful. Child nodes that depend on the job’s success can now be created.

Closing

The holdoff time has been reached, the node is now waiting for any closing child nodes to reach the Done state.

Done

The node has reached its final state.

Note The information whether the job succeeded or not, or if any tests passed or not is stored in a separate field result. The state machine doesn’t rely on the data field at all, it’s considered extra meta-data used for looking at the actual results. If a job fails, it goes straight from Running to Done and any service waiting for it to be Available will know it can’t be used (for example, runtime tests can’t be scheduled if a kernel build failed).

There are two fields that can cause time-driven state transitions:

holdoff

This is for the node to remain in the Available state for a minimum amount of time and allow other nodes that depend on its success to be created. Without this time constraint, there would be a race condition when the job is complete as without any child nodes it would go directly to Done.

timeout

This is the time when the node needs to go to the Done state regardless of its current state. The main reasons why this is needed are for when a node gets stuck and never completes its job, or while waiting for child nodes to complete.

Migrations

Migrations are required to propagate the changes made to API models to the database like updating or deleting a model field or a model.

pymongo-migrate package has been integrated to enable migration support in the API.

Use the below command to run migration from the api Docker container.

$ docker-compose exec api /bin/sh -c 'pymongo-migrate migrate -u mongodb://db/kernelci -m migrations -v'
2022-11-25 06:12:17,996 [DEBUG]  Migration target not specified, assuming upgrade
Command find#1957747793 STARTED
SON([('find', 'pymongo_migrate'), ('filter', {'name': '20221014061654_set_timeout'}), ('limit', 1), ('singleBatch', True), ('lsid', {'id': Binary(b'7\x18\xe4\xe7\xd0\x88Gc\xafq\x0f[\x8b\x8d\xec\xde', 4)}), ('$db', 'kernelci'), ('$readPreference', {'mode': 'primaryPreferred'})])
Command find#1957747793 SUCCEEDED in 215us
2022-11-25 06:12:17,998 [INFO ]  Running upgrade migration '20221014061654_set_timeout'
Command find#424238335 STARTED
SON([('find', 'node'), ('filter', {}), ('lsid', {'id': Binary(b'7\x18\xe4\xe7\xd0\x88Gc\xafq\x0f[\x8b\x8d\xec\xde', 4)}), ('$db', 'kernelci'), ('$readPreference', {'mode': 'primaryPreferred'})])
Command find#424238335 SUCCEEDED in 4404us
Command update#719885386 STARTED
SON([('update', 'node'), ('ordered', True), ('lsid', {'id': Binary(b'7\x18\xe4\xe7\xd0\x88Gc\xafq\x0f[\x8b\x8d\xec\xde', 4)}), ('$db', 'kernelci'), ('$readPreference', {'mode': 'primary'}), ('updates', [SON([('q', {'_id': ObjectId('63720d307d572b5aa15462f4')}), ('u', {'$set': {'timeout': datetime.datetime(2022, 11, 14, 15, 41, 4, 175000)}}), ('multi', False), ('upsert', False)])])])
Command update#719885386 SUCCEEDED in 989us
Command update#1649760492 STARTED
SON([('update', 'node'), ('ordered', True), ('lsid', {'id': Binary(b'7\x18\xe4\xe7\xd0\x88Gc\xafq\x0f[\x8b\x8d\xec\xde', 4)}), ('$db', 'kernelci'), ('$readPreference', {'mode': 'primary'}), ('updates', [SON([('q', {'_id': ObjectId('63720d317d572b5aa15462f5')}), ('u', {'$set': {'timeout': datetime.datetime(2022, 11, 14, 15, 41, 5, 454000)}}), ('multi', False), ('upsert', False)])])])
Command update#1649760492 SUCCEEDED in 151us

The above command will run the necessary upgrade for the migrations stored in migrations directory. If migration target is specified, it will run the necessary upgrade or downgrade to reach the target.

The status of migration can be found with the below command:

$ docker-compose exec api /bin/sh -c 'pymongo-migrate show -u mongodb://db/kernelci -m migrations -v'
Migration name            	Applied timestamp
Command find#1957747793 STARTED
SON([('find', 'pymongo_migrate'), ('filter', {'name': '20221014061654_set_timeout'}), ('limit', 1), ('singleBatch', True), ('lsid', {'id': Binary(b'\x1e\x0b"M\x1f\xb3N\xf4\xa3\x00\xaa\xff]c\xf8\xe5', 4)}), ('$db', 'kernelci'), ('$readPreference', {'mode': 'primaryPreferred'})])
Command find#1957747793 SUCCEEDED in 228us
20221014061654_set_timeout	2022-11-25T07:00:14.923000+00:00

If the upgrade has already been applied, downgrade can be run using the below command:

$ docker-compose exec api /bin/sh -c 'pymongo-migrate downgrade -u mongodb://db/kernelci -m migrations -v'
Command find#1957747793 STARTED
SON([('find', 'pymongo_migrate'), ('filter', {'name': '20221014061654_set_timeout'}), ('limit', 1), ('singleBatch', True), ('lsid', {'id': Binary(b'\x0c}X\x9d\xe9{O\xd5\x85\xb3\x904\xa2R\x89i', 4)}), ('$db', 'kernelci'), ('$readPreference', {'mode': 'primaryPreferred'})])
Command find#1957747793 SUCCEEDED in 237us
2022-11-25 07:07:21,857 [INFO ]  Running downgrade migration '20221014061654_set_timeout'
2022-11-25 07:07:21,858 [INFO ]  Execution time of '20221014061654_set_timeout': 1.049041748046875e-05 seconds
Command update#424238335 STARTED
SON([('update', 'pymongo_migrate'), ('ordered', True), ('lsid', {'id': Binary(b'\x0c}X\x9d\xe9{O\xd5\x85\xb3\x904\xa2R\x89i', 4)}), ('$db', 'kernelci'), ('$readPreference', {'mode': 'primary'}), ('updates', [SON([('q', {'name': '20221014061654_set_timeout'}), ('u', {'name': '20221014061654_set_timeout', 'applied': None}), ('multi', False), ('upsert', True)])])])
Command update#424238335 SUCCEEDED in 279us

Pub/Sub and CloudEvent

The API provides a publisher / subscriber interface so clients can listen to events and publish them too. All the events are formatted using CloudEvents.

Listen & Publish CloudEvent

The API provides different endpoints for publishing and listening to events.

For example, subscribe to a channel first:

curl -X 'POST' 'http://localhost:8001/latest/subscribe/abc' -H 'Authorization: Bearer TOKEN'
{"id":800,"channel":"abc","user":"bob"}

Use the subscription ID from the response to listen to the events:

$ curl -X 'GET' 'http://localhost:8001/latest/listen/800' -H 'Authorization: Bearer TOKEN'

Then in a second terminal publish CloudEvent message:

$ curl -X 'POST' 'http://localhost:8001/latest/publish/abc' -H 'Authorization: Bearer TOKEN' -H 'Content-Type: application/json' \
-d '{"data": {"sample_key":"sample_value"}}'

Other CloudEvent fields such as “type”, “source”, and “attributes” can also be sent to the request dictionary above.

You should see the message appear in the first terminal inside “data” dictionary:

$ curl -X 'GET' 'http://localhost:8001/latest/listen/800' -H 'Authorization: Bearer TOKEN'
{"type":"message","pattern":null,"channel":"abc","data":"{\"specversion\": \"1.0\", \"id\": \"9e67036c-650e-4688-b4dd-5b2eafd21f5f\", \"source\": \"https://api.kernelci.org/\", \"type\": \"api.kernelci.org\", \"time\": \"2024-01-04T10:48:39.974782+00:00\", \"data\": {\"sample_key\": \"sample_value\"}, \"owner\": \"bob\"}"}

Now, unsubscribe from the channel:

$ curl -X 'GET' 'http://localhost:8001/latest/unsubscribe/800' -H 'Authorization: Bearer TOKEN'

Meanwhile, something like this should be seen in the API logs:

$ docker-compose logs api | tail -4
kernelci-api | INFO:     127.0.0.1:35752 - "POST /subscribe/abc HTTP/1.1" 200 OK
kernelci-api | INFO:     127.0.0.1:35810 - "POST /publish/abc HTTP/1.1" 200 OK
kernelci-api | INFO:     127.0.0.1:35754 - "GET /listen/abc HTTP/1.1" 200 OK
kernelci-api | INFO:     127.0.0.1:36744 - "POST /unsubscribe/abc HTTP/1.1" 200 OK

Events

The /events endpoint provides access to the event history stored in MongoDB. Events are generated when nodes are created or updated, and are stored for a configurable retention period (default 7 days).

Event Structure

Each event contains the following fields:

The data object contains node information:

Query Parameters

The /events endpoint supports the following query parameters:

Note: When using recursive=true, the limit parameter is required and must be <= 1000.

Examples

Get all events (limited to default pagination):

$ curl http://localhost:8001/latest/events

Get events for completed jobs with passing results:

$ curl 'http://localhost:8001/latest/events?kind=job&state=done&result=pass'

Get recently created events (last hour):

$ curl 'http://localhost:8001/latest/events?op=created&from=2025-01-10T12:00:00'

Get events for a specific node path pattern (regex):

$ curl 'http://localhost:8001/latest/events?path=.*linux-next.*&limit=50'

Get events for a specific group:

$ curl 'http://localhost:8001/latest/events?group=kunit-x86_64&state=done'

Get events by owner:

$ curl 'http://localhost:8001/latest/events?owner=admin&kind=checkout'

Get events with full node data:

$ curl 'http://localhost:8001/latest/events?state=done&result=fail&recursive=true&limit=10'

Get events for a specific node ID:

$ curl 'http://localhost:8001/latest/events?node_id=65a1355ee98651d0fe81e412'

Combine multiple filters:

$ curl 'http://localhost:8001/latest/events?kind=test&state=done&result=fail&group=kselftest&from=2025-01-01&limit=100'

Sample Response

[
  {
    "id": "65a1355ee98651d0fe81e500",
    "timestamp": "2025-01-12T08:30:00.000000",
    "sequence_id": 12345,
    "channel": "node",
    "owner": "admin",
    "data": {
      "op": "updated",
      "id": "65a1355ee98651d0fe81e412",
      "kind": "test",
      "name": "kselftest-cpufreq",
      "path": ["checkout", "kbuild", "test", "kselftest-cpufreq"],
      "group": "kselftest",
      "state": "done",
      "result": "pass",
      "owner": "admin",
      "data": {
        "kernel_revision": {
          "tree": "mainline",
          "branch": "master",
          "commit": "abc123..."
        }
      },
      "is_hierarchy": false
    }
  }
]

Response with recursive=true

When recursive=true is specified, each event includes a node field with the full node object:

[
  {
    "id": "65a1355ee98651d0fe81e500",
    "timestamp": "2025-01-12T08:30:00.000000",
    "data": {
      "op": "updated",
      "id": "65a1355ee98651d0fe81e412",
      "kind": "test",
      "name": "kselftest-cpufreq",
      "state": "done",
      "result": "pass"
    },
    "node": {
      "id": "65a1355ee98651d0fe81e412",
      "kind": "test",
      "name": "kselftest-cpufreq",
      "path": ["checkout", "kbuild", "test", "kselftest-cpufreq"],
      "group": "kselftest",
      "parent": "65a1355ee98651d0fe81e400",
      "state": "done",
      "result": "pass",
      "artifacts": {...},
      "data": {...},
      "created": "2025-01-12T08:00:00.000000",
      "updated": "2025-01-12T08:30:00.000000",
      "owner": "admin",
      "user_groups": []
    }
  }
]