Running as a scheduled job on GitHub Actions
Docker distribution of Napkin makes it easy to deploy it in unattended environments, such as CI/CD platform. In the examples below, we will execute Napkin Spec on daily basis using GitHub Actions. This approach can be easily adapted by the user for a CI/CD platform of choice.
Some Napkin features require the metadata store to be persisted between Spec runs. We recommend configuring a Postgres database as a metadata store. Alternatively, one needs to persist Sqlite metadata store between job runs.
BigQuery
In order to pass DB credentials to unattended job, you need to generated credentials file as described in Connecting to the database document. Next, you need to add it as NAPKIN_CREDS secret in GitHub repository settings.
name: Napkin
on:
push:
branches:
- master
schedule:
# trigger the workflow daily at 3 am
# * is a special character in YAML so you have to quote this string
- cron: '0 3 * * *'
jobs:
napkin:
runs-on: ubuntu-20.04
container:
image: soostone/napkin-exe:v0.5.10
steps:
- uses: actions/checkout@v2
- name: Get branch name
id: branch-name
uses: tj-actions/branch-names@v5
- shell: bash
env:
NAPKIN_CREDS: ${{ secrets.NAPKIN_CREDS }}
run: echo "$NAPKIN_CREDS" > creds.json
- run: napkin run --log-format Server -C creds.json --arg branch=${{ steps.branch-name.outputs.current_branch }}
Postgres/Redshift
Credentials to the database have to be provided, so Napkin can connect to the database. In this example, we assume that the full connection string is present in the YAML Spec except for the DB password. When you add it as PGPASSWORD secret in GitHub repository settings it will be reexported as PGPASSWORD variable that will be consumed by the Postgres client.
name: Napkin
on:
push:
branches:
- master
schedule:
# trigger the workflow daily at 3 am
# * is a special character in YAML so you have to quote this string
- cron: '0 3 * * *'
jobs:
napkin:
runs-on: ubuntu-20.04
container:
image: soostone/napkin-exe:v0.5.10
steps:
- uses: actions/checkout@v2
- name: Get branch name
id: branch-name
uses: tj-actions/branch-names@v5
- shell: bash
env:
PGPASSWORD: ${{ secrets.PGPASSWORD }}
- run: napkin run --log-format Server --arg branch=${{ steps.branch-name.outputs.current_branch }}