Building a CLI for Jev and Using It to Assess Pull Request Risk in GitHub Actions

I recently started looking into Jev. TypeSafe provides official TypeScript and Python SDKs,¹ but I could not find an official command-line tool, so I decided to build one.

https://github.com/stefafafan/jev

Design principles of stefafafan/jev

I designed the interface around reading values from standard input and writing the result to standard output, making it easy to combine with commands such as jq. Since the CLI is written in Go, it does not use the official SDKs.

Here is an example of how it can be used:

Shell

git diff | jev noul "Are there any breaking changes in this diff?" | jq .

TypeSafe AI was not accepting new users when I tried, which left me unable to use it directly. However, Jev is apparently available through Cloudflare Workers AI and Vercel AI Gateway, so I added support for both.

The CLI can select the corresponding provider when the relevant environment variables are set, or you can specify one explicitly with --provider, as shown below.

Shell

export TYPESAFE_API_KEY=...
git diff | jev --provider typesafe noul "Are there any breaking changes in this diff?" | jq .

Shell

export CLOUDFLARE_API_TOKEN=...
export CLOUDFLARE_ACCOUNT_ID=...
git diff | jev --provider cloudflare noul "Are there any breaking changes in this diff?" | jq .

Using it in GitHub Actions

I had already been thinking that pull request reviews are hard, and that it would be useful to have Jev (reportedly fast and inexpensive) make a rough first pass. So I also created an Action that simply installs the CLI above in GitHub Actions.

https://github.com/stefafafan/setup-jev

As its name suggests, it only handles setup: internally, it installs the binary matching the operating system and requested version.

YAML

- uses: stefafafan/setup-jev@68157056af19eac3d6d649efa641d1e9662b5186 # v1.0.1
  with:
    version: v0.1.2

Trying it in a GitHub workflow

I am now running a workflow that assesses pull request risk in the stefafafan/jev repository.

It runs for pull requests in certain states, passes the output of gh pr diff to jev choice, and asks it to select one of risk-low, risk-medium, or risk-high. The core of the workflow looks like this:

Bash

result=$(gh pr diff "$PR_NUMBER" --repo "$GITHUB_REPOSITORY" | \
  jev --provider cloudflare choice \
    --option risk-low \
    --option risk-medium \
    --option risk-high \
    "$instructions")

risk=$(jq -er '.answers.result.choice' <<< "$result")

The rest of the workflow adds and removes labels based on the result. When I tried it, the classification step finished in roughly 1–2 seconds despite calling an external API, which seems useful.

One concern is that the current setup relies on a token issued in advance, which could be abused if leaked. Ideally, I would use a short-lived token valid only for the workflow run, but that does not seem feasible yet. For now, the practical mitigations seem to be using the narrowest permissions and shortest expiration periods available, and setting a budget cap.

https://github.com/typesafe-ai ↩

Design principles of stefafafan/jev

Using it in GitHub Actions

Trying it in a GitHub workflow

Footnotes