draforge

DRAForge - Optional DigitalOcean Showcase Runbook

This runbook covers one optional, provider-specific showcase on a 2-node DigitalOcean Kubernetes (DOKS) cluster. DRAForge itself is provider-neutral; ordinary installation and release validation do not require DigitalOcean, DOKS, or DOCR.

Non-production exposure: task demo:up deliberately enables unauthenticated HTTP through values-showcase-docr.yaml. Use it only for short-lived demonstrations with non-sensitive cluster metadata, then run task demo:down.

Setup Requirements

Before launching the showcase:

Ensure the doctl, terraform, and helm command-line tools are installed on the host.
Verify you have a valid DigitalOcean API token.
Configure a .env file at the root of the repository containing:
```
DIGITALOCEAN_TOKEN="your-digitalocean-api-token"
```

Bring Up the Showcase

To completely build and deploy the showcase platform, run:

task demo:up

What `task demo:up` performs:

Cost & Safety Audit: Executes scripts/audit-cloud-resources.sh to ensure no active worker nodes or standalone Droplets exceed budget limits. Runs scripts/validate-plan.py to confirm the Terraform plan complies with the maximum 2-worker node policy.
Infrastructure Provisioning: Initializes and runs terraform apply to spin up the Dedicated VPC, DOKS cluster, and private DigitalOcean Container Registry (DOCR).
Private Registry Integration: Configures Kubeconfig registry secrets in the namespaces.
Remote Image Builds: Submits rootless Kubernetes Kaniko jobs to build the server, controller, and sim-driver container images directly on the DOKS cluster.
Helm Chart Installation: Deploys with values-showcase-docr.yaml, using private DOCR component tags, the registry-draforge pull secret, wildcard demo CORS, and an explicitly enabled insecure HTTP Gateway.
Scenario Seeding: Applies the custom simulated device pool configurations (e.g. examples/scenarios/basic-gpu.yaml).
Exposes Endpoint: Retrieves the LoadBalancer IP of the Cilium Gateway and outputs the live dashboard URL.

The normal chart defaults remain on GHCR, render no pull secret, and create no Gateway or Ingress. The showcase override is intentionally applied only by task demo:up after scripts/remote-build.sh reconciles the DOCR secret in both draforge-ci and draforge-system.

Showcase Walkthrough

Once the dashboard is online:

Explore the Graph: Navigate to the Graph tab. You should see the interactive layout displaying the two worker nodes, their simulated GPU resource slices, and the active devices.
Cluster Health Diagnostics: Go to the Doctor tab to review the 5 passing checks (DRA API availability, node compatibility, and slice consistency).
Observe Resource Claims: Apply a test pod requesting a GPU (e.g., kubectl apply -f examples/scenarios/insufficient-capacity.yaml or a claim template). The dashboard will dynamically stream the allocation event.
Simulate a Fault: Inject a device failure using the CLI:
```
draforge inject-fault --pool basic-gpu-pool --type unhealthy
```
Watch the graph update in real-time. The affected devices will show warning states, and the doctor check will highlight the failure.

Tear Down the Showcase

To avoid unnecessary DigitalOcean billings, clean up all provisioned resources:

task demo:down

This will uninstall the Helm release, delete all scenario custom resources, and execute terraform destroy to tear down the DOKS cluster, VPC, and registry.

Optional DOKS Smoke Testing

The manual workflow in .github/workflows/e2e.yml is a provider-specific showcase adapter that runs the tagged smoke package on an existing DOKS cluster. It is not the general release gate. Provider-neutral release validation is performed by the credential-free install E2E matrix described in End-to-End Testing.

E2E-DOKS Environment Approval Flow

GitHub Environment: The workflow is gated by the e2e-doks GitHub Environment.
Required Approvals: Direct repository maintainer approval is required to trigger execution in this environment.
Safety Phrase: When triggering the workflow manually via workflow_dispatch, you must enter run-e2e-doks in the confirm input field.

Secret Configuration & Token Scope

The workflow requires a GitHub Actions repository secret named DIGITALOCEAN_TOKEN.

Use a custom-scope token with kubernetes:read and kubernetes:access_cluster; Droplet, VPC, registry, and Kubernetes create/update/delete API scopes are not required.
The token is used only by doctl to locate the existing cluster and retrieve a one-hour kubeconfig credential.
Never expose the token or generated kubeconfig in inputs, logs, artifacts, or committed files.
Only maintainers who can approve the protected e2e-doks environment should run the workflow.

The workflow does not provision DigitalOcean infrastructure. It consumes CPU and memory on the existing DOKS workers, which remain billable until the cluster is destroyed through the separate showcase teardown flow.

Manual E2E Run Checklist

Before running the remote E2E workflow:

Confirm that the target DOKS cluster (default: draforge-cluster) exists and is active on DigitalOcean.
Verify that your DIGITALOCEAN_TOKEN secret is configured in the repository settings under Secrets and variables -> Actions.
Go to the Actions tab, select the Optional DOKS E2E workflow, and click Run workflow.
Enter run-e2e-doks as confirmation and specify the correct cluster name.

Cleanup Behavior after Failure/Cancellation

Automatic Cleanup Job: The workflow has a dedicated cleanup job that runs always() (even if the main test job fails or is cancelled).
Run-scoped cleanup: The script trap and cleanup job delete the remote E2E Job, ServiceAccount, ClusterRole, and ClusterRoleBinding selected by the GitHub run ID. The shared draforge-ci Namespace, ResourceQuota, and LimitRange remain for later remote jobs. The existing Helm/scenario cleanup remains best-effort. The DOKS cluster itself is never deleted by this workflow.

Common Failure Modes & Troubleshooting

Error: DIGITALOCEAN_TOKEN secret is not configured
- Cause: The secret was not added or is named incorrectly.
- Fix: Check repository secrets and ensure DIGITALOCEAN_TOKEN is defined.
Error: cluster ... not found
- Cause: The cluster name provided in the workflow input does not exist in the DigitalOcean account.
- Fix: Ensure the cluster is created first or double-check the spelling in the cluster_name input.
Error: Unauthorized / Forbidden API calls
- Cause: The token is expired or lacks kubernetes:read / kubernetes:access_cluster.
- Fix: Generate a custom-scope token with those two scopes and update the GitHub secret.

This site is open source. Improve this page.