IDE Support

Visual Studio Code is the recommended IDE to develop data processing pipelines with Napkin.

Visual Studio Code

However, some configuration is required to get the best experience. In this document, we describe what features can increase productivity and how they can be configured.

We recommend installing the following extensions:

• YAML
• Advanced meta programming support
  • Haskell
• SQL editing
  • Postgres
  • SQL (BigQuery)
• Devcontainer support
  • Remote - Containers
  • Docker

YAML Schema

Editing Specs in YAML format can be assisted by IDE with instant validation and autocomplete. Note that schema validation does not replace the napkin validate command which performs a dry-run check and evaluates all referenced programs and queries. Once YAML extension is installed you need to configure Napkin Spec schema. If you have used napkin init the appropriate line has been already added to spec.yaml. Otherwise, you need to add the following line to your spec.yaml.

spec.yaml

# yaml-language-server: $schema=https://soostone-napkin-public.s3.us-east-1.amazonaws.com/schema/schema.json

Alternatively, one may configure the schema path in the VSCode configuration file (.vscode/settings.json or in global settings):

.vscode/settings.json

"yaml.schemas": {
  "https://soostone-napkin-public.s3.us-east-1.amazonaws.com/schema/schema.json": "spec*.yaml",
}

The above snippets will use Spec schema for the last release of Napkin. In some cases, the use of the latest and greatest schema may not be desired. We also publish tagged schema versions at https://soostone-napkin-public.s3.us-east-1.amazonaws.com/schema/app-version/X.Y.Z/schema.json.

Alternatively, one can refer local schema file and check it in the Git repository. The schema file can be either downloaded or generated by the napkin yaml-schema spec-schema.json command. The schema file is also already in Docker image at "/usr/share/napkin/spec-schema.json.

Tasks

We provide a .vscode/tasks.json for the most frequent tasks: validate, validate (interactive), dump, and run commands.

.vscode/tasks.json

{
  // See https://go.microsoft.com/fwlink/?LinkId=733558
  // for the documentation about the tasks.json format
  "version": "2.0.0",
  "tasks": [
    {
      "label": "dump",
      "type": "shell",
      "command": "napkin dump -o dump"
    },
    {
      "label": "validate",
      "type": "shell",
      "command": "napkin validate"
    },
    {
      "label": "validate interactive",
      "type": "shell",
      "command": "napkin validate --interactive",
      "isBackground": true
    },
    {
      "label": "run",
      "type": "shell",
      "command": "napkin run"
    },
  ]
}

Metaprogramming: Haskell Language Server

Metaprogramming in Haskell can be supported by Haskell Language Server and Haskell extension. Haskell Language Server will make the feedback loop shorter by displaying errors and warnings, as well as presenting information about types in tooltips. It will also speed up Spec development by providing autocompletion. All necessary binaries are included in Docker and Nix distributions of Napkin. However, the hie.yaml file has to be created in the project root directory. If you have used the napkin init command to create your project the necessary file is already there.

hie.yaml

cradle:
  bios:
    shell: napkin hie-bios --spec-file specs/spec.yaml

Other Editors

Both YAML and Haskell language servers provide support for numerous editors. YAML schema support can be enabled according to YAML Language Server documentation. Alternatively, Spec YAMLs can be validated with yajsv. For metaprogramming assistance, please refer to Haskell Language Server docs for further information.