Get Started
Beginner Tutorial
First-Time Setup
Connect to Existing Instance
Language Clients
Learn
Key Features
Target Audience
Basic Concepts
Intro to Data Versioning
Intro to Pipelines
Intro to Console
Developer Workflow
Diagrams
Glossary
Set Up
Cloud Deploy
Local Deploy
On-Prem Deploy
Pachctl
Pachctl Auto-completion
Unified Deployment
Authentication & IdP Connectors
Authorization (RBAC)
Connection
Environment Variables
Kubernetes RBAC
Import a Kubernetes Context
Log Aggregation (Loki)
Non-Default Namespaces
Enterprise Edition
Enterprise Server (ES)
S3 Gateway API
TLS (SSL, HTTPS)
Tracing (Jaeger)
Blob/Object Storage
Global Config
Manage
Helm Chart Values (HCVs)
S3 Gateway
Backup & Restore
Upgrade
PachCTL Shell
Check IdP User
Supported Releases & Features
PostgresSQL Fine-Tuning
Cluster Access
Deactivate Authorization
GPUs
Log In via IdP
Revoke User Access
Scaling Limits (CE)
Secrets
Sidecar S3 Gateway
Storage Optimization
Usage Metrics
Monitor with Prometheus
Uninstall
Prepare Data
Datum Batching
Defer Processing via Staging Branch
Ingest Data
Mount Volumes
Skip Failed Datums
Time-Windowed Data
Transactions
Build Pipelines & DAGs
Pipeline Specification (PPS)
👉 Pipeline Ops
Project Ops
Branch Ops
Datum Ops
Provenance Ops
Tutorials
Export Data
Egress To An SQL Database
Export via Egress
Export via PachCTL
Mount a Repo Locally
S3 Gateway Operations
Integrate
Determined
Google BigQuery
JupyterLab
Label Studio
Superb AI
VS Code Auto-Completion
Weights and Biases
Run Commands
pachctl
pachctl auth
pachctl auth activate
pachctl auth check
pachctl auth check project
pachctl auth check repo
pachctl auth deactivate
pachctl auth get
pachctl auth get cluster
pachctl auth get enterprise
pachctl auth get project
pachctl auth get repo
pachctl auth get-config
pachctl auth get-groups
pachctl auth get-robot-token
pachctl auth login
pachctl auth logout
pachctl auth revoke
pachctl auth roles-for-permission
pachctl auth rotate-root-token
pachctl auth set
pachctl auth set cluster
pachctl auth set enterprise
pachctl auth set project
pachctl auth set repo
pachctl auth set-config
pachctl auth use-auth-token
pachctl auth whoami
pachctl buildinfo
pachctl check
pachctl check status
pachctl completion
pachctl completion bash
pachctl completion zsh
pachctl config
pachctl config delete
pachctl config delete context
pachctl config get
pachctl config get active-context
pachctl config get active-enterprise-context
pachctl config get context
pachctl config get metrics
pachctl config import-kube
pachctl config list
pachctl config list context
pachctl config set
pachctl config set active-context
pachctl config set active-enterprise-context
pachctl config set context
pachctl config set metrics
pachctl config update
pachctl config update context
pachctl connect
pachctl copy
pachctl copy file
pachctl create
pachctl create branch
pachctl create defaults
pachctl create pipeline
pachctl create project
pachctl create repo
pachctl create secret
pachctl debug
pachctl debug analyze
pachctl debug binary
pachctl debug dump
pachctl debug local
pachctl debug log-level
pachctl debug profile
pachctl debug template
pachctl delete
pachctl delete all
pachctl delete branch
pachctl delete commit
pachctl delete commitV2
pachctl delete defaults
pachctl delete file
pachctl delete job
pachctl delete pipeline
pachctl delete project
pachctl delete repo
pachctl delete secret
pachctl delete transaction
pachctl diff
pachctl diff file
pachctl draw
pachctl draw pipeline
pachctl edit
pachctl edit pipeline
pachctl enterprise
pachctl enterprise deactivate
pachctl enterprise get-state
pachctl enterprise heartbeat
pachctl enterprise pause
pachctl enterprise pause-status
pachctl enterprise register
pachctl enterprise sync-contexts
pachctl enterprise unpause
pachctl exit
pachctl find
pachctl find commit
pachctl finish
pachctl finish commit
pachctl finish transaction
pachctl fsck
pachctl get
pachctl get file
pachctl glob
pachctl glob file
pachctl idp
pachctl idp create-client
pachctl idp create-connector
pachctl idp delete-client
pachctl idp delete-connector
pachctl idp get-client
pachctl idp get-config
pachctl idp get-connector
pachctl idp list-client
pachctl idp list-connector
pachctl idp set-config
pachctl idp update-client
pachctl idp update-connector
pachctl inspect
pachctl inspect branch
pachctl inspect cluster
pachctl inspect commit
pachctl inspect datum
pachctl inspect defaults
pachctl inspect file
pachctl inspect job
pachctl inspect pipeline
pachctl inspect project
pachctl inspect repo
pachctl inspect secret
pachctl inspect transaction
pachctl kube-events
pachctl license
pachctl license activate
pachctl license add-cluster
pachctl license delete-all
pachctl license delete-cluster
pachctl license get-state
pachctl license list-clusters
pachctl license update-cluster
pachctl list
pachctl list branch
pachctl list commit
pachctl list datum
pachctl list file
pachctl list job
pachctl list pipeline
pachctl list project
pachctl list repo
pachctl list secret
pachctl list transaction
pachctl logs
pachctl loki
pachctl mount
pachctl port-forward
pachctl put
pachctl put file
pachctl rerun
pachctl rerun pipeline
pachctl restart
pachctl restart datum
pachctl resume
pachctl resume transaction
pachctl run
pachctl run cron
pachctl run pfs-load-test
pachctl run pps-load-test
pachctl shell
pachctl squash
pachctl squash commit
pachctl squash commitV2
pachctl start
pachctl start commit
pachctl start pipeline
pachctl start transaction
pachctl stop
pachctl stop job
pachctl stop pipeline
pachctl stop transaction
pachctl subscribe
pachctl subscribe commit
pachctl unmount
pachctl update
pachctl update defaults
pachctl update pipeline
pachctl update project
pachctl update repo
pachctl validate
pachctl validate pipeline
pachctl version
pachctl wait
pachctl wait commit
pachctl wait job
Debug
Common Issues
Debug Pipelines
Troubleshooting Deployments
View Audit Logs
View Kubernetes Logs
Pachyderm SDK
Client Initialization (Start Here)
Method Mapping
First Project
Examples
Reference Docs
Contribute
Coding Conventions
Contributor Setup
Developing on Windows with VSCode
Documentation Style Guide
GitHub

Pipeline Ops

Learn how to create, delete, and update pipelines using PachCTL and jsonnet templating.

April 4, 2024