Cài đặt cluster

Cluster installation

Install the following on every cluster node (control plane and workers). GPU nodes additionally need NVIDIA drivers and the container toolkit.

Base packages (all nodes)

sudo apt-get update && sudo apt-get install -y \
  curl wget jq snapd ca-certificates gnupg lsb-release \
  open-iscsi util-linux

# Optional: convenience aliases
echo 'alias kubectl="microk8s kubectl"' >> ~/.bashrc
echo 'alias helm="microk8s helm"' >> ~/.bashrc

GPU nodes only — NVIDIA driver & container toolkit

# Install NVIDIA driver (example — use your distro's recommended method)
sudo apt-get install -y nvidia-driver-550

# Install nvidia-container-toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit

# Verify driver
nvidia-smi

Run these steps on every node in the cluster. Pick one node as the initial control-plane (master).

# Install MicroK8s (pin channel if needed, e.g. 1.30/stable)
sudo snap install microk8s --classic --channel=1.30/stable

# Add your user to the microk8s group (log out/in afterward)
sudo usermod -a -G microk8s $USER
sudo chown -f -R $USER ~/.kube
newgrp microk8s

# Wait until the node is ready (on the first / master node)
microk8s status --wait-ready

# Verify
microk8s kubectl get nodes

High availability (optional)

For a multi-master HA cluster, enable HA on the first node, then join additional control-plane nodes:

# On the first master node only
microk8s enable ha-cluster

# Generate join token for additional masters
microk8s add-node --token-ttl 3600

Run the printed microk8s join … command on each additional master node.

After MicroK8s is installed on worker nodes, join them to the cluster from the master node.

On the master node

# Generate a join command (default TTL 24h)
microk8s add-node

# Example output:
# microk8s join 10.10.126.127:25000/abc123…

On each worker node

# Paste the join command from the master
microk8s join 10.10.126.127:25000/<token>

Verify cluster membership

# On the master
microk8s kubectl get nodes -o wide

# All nodes should show Ready
# HA clusters show multiple control-plane nodes in microk8s status

Run on the master node (addons apply cluster-wide). Wait for each addon to become ready before continuing.

# Core addons required by DeepOps
microk8s enable rbac            # Role-based access control (often pre-enabled)
microk8s enable dns             # CoreDNS
microk8s enable helm3           # Helm 3 package manager
microk8s enable registry        # Local registry at localhost:32000
microk8s enable metrics-server  # Required for kubectl top / monitor sidecar
microk8s enable ingress         # Ingress controller (Traefik or nginx class)
microk8s enable cert-manager      # TLS certificate management

# Verify
microk8s status

DeepOps uses cert-manager with a cluster-wide ACME issuer defined in letsencrypt-cert-issuer.yaml at the repository root. The issuer uses the HTTP-01 challenge via the ingress controller (ingressClassName: nginx). Complete this section after enabling the ingress and cert-manager addons (step 4).

Prerequisites

• microk8s enable ingress and microk8s enable cert-manager are active
• Your hub domain (e.g. hub.example.com) has a public DNS A record pointing to the cluster ingress IP
• Port 80 is reachable from the internet (required for HTTP-01 validation)

Step 1 — Edit the ClusterIssuer

Open letsencrypt-cert-issuer.yaml and set your contact email for Let's Encrypt expiry notices:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@your-domain.com          # <-- change this
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          ingressClassName: nginx          # matches dohub ingress.className

Step 2 — Apply the ClusterIssuer

cd /path/to/DeepOps

# Wait for cert-manager pods to be ready first
microk8s kubectl wait --for=condition=Available \
  deployment/cert-manager -n cert-manager --timeout=120s
microk8s kubectl wait --for=condition=Available \
  deployment/cert-manager-webhook -n cert-manager --timeout=120s

# Apply the issuer
microk8s kubectl apply -f letsencrypt-cert-issuer.yaml

# Confirm the issuer is ready
microk8s kubectl get clusterissuer letsencrypt-prod
# STATUS should become Ready

Step 3 — Annotate the Dohub ingress

Before deploying Dohub, add the cert-manager annotation so the ingress shim automatically requests a certificate into the TLS secret tls-dohub-secret (already referenced in dohub/values.yaml).

# dohub/values.yaml
ingress:
  enabled: true
  className: "nginx"
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "600"
  hosts:
    - host: hub.example.com          # your hub domain
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: tls-dohub-secret
      hosts:
        - hub.example.com

Step 4 — Verify certificate issuance

After running build-and-deploy.sh (step 10), cert-manager creates a Certificate resource and stores the issued cert in the secret:

# Certificate and challenge status
microk8s kubectl get certificate,certificaterequest,order,challenge -n dohub

# TLS secret populated by cert-manager
microk8s kubectl get secret tls-dohub-secret -n dohub

# Test HTTPS (should NOT use -k once cert is valid)
curl -vI https://hub.example.com/

Workspace subdomains

Workspace ingresses use per-user hostnames such as <slug>-<username>.<DOMAIN_NAME> and reference the same tls-dohub-secret for sidecar routes on the hub domain. The HTTP-01 issuer in letsencrypt-cert-issuer.yaml issues one certificate per hostname listed in an ingress TLS block. For many dynamic workspace subdomains you may need a wildcard certificate (DNS-01 solver) or additional Certificate resources — extend the ClusterIssuer beyond the provided file if required.

DirectPV provides the directpv-min-io StorageClass used by Dohub and all workspace PVCs. The repo includes a values file tuned for MicroK8s kubelet paths.

# From the DeepOps repository root
cd /path/to/DeepOps

# Add the MinIO Helm repository
microk8s helm repo add minio https://charts.min.io/
microk8s helm repo update

# Install DirectPV (uses directpv-install.yaml from this repo)
microk8s helm upgrade --install directpv minio/directpv \
  -f directpv-install.yaml \
  --namespace directpv \
  --create-namespace

# Verify pods
microk8s kubectl get pods -n directpv
microk8s kubectl get storageclass directpv-min-io

Values file reference (directpv-install.yaml)

# MicroK8s kubelet path (required)
kubeletDirPath: "/var/snap/microk8s/common/var/lib/kubelet/"

# Driver image — plugin version must match this tag
images:
  directpv: quay.io/minio/directpv:v5.1.1

The CLI plugin version must match the DirectPV driver installed in the cluster (v5.1.1). Install on every machine that runs kubectl directpv commands (admin workstation and/or nodes where Dohub executes discover/init).

Option A — download release binary (recommended)

DIRECTPV_VERSION="v5.1.1"
ARCH="linux_amd64"   # or linux_arm64

curl -fLo kubectl-directpv \
  "https://dl.min.io/aistor/directpv/release/${DIRECTPV_VERSION#v}/linux-amd64/kubectl-directpv"

sudo install -m 0755 kubectl-directpv /usr/local/bin/kubectl-directpv

# Verify version matches the cluster driver
kubectl-directpv --version
# Expected: directpv version v5.1.1

Option B — copy from repository

The repo ships a binary at ./kubectl-directpv (embedded in the Dohub Docker image). Ensure it matches directpv-install.yaml before use.

sudo install -m 0755 ./kubectl-directpv /usr/local/bin/kubectl-directpv
kubectl directpv --version

DirectPV must discover raw drives on each node, then initialize the selected devices. Initialization formats drives and erases all data.

Step 7.1 — Discover drives

# Discover block devices and write results to a YAML file
kubectl directpv discover --output-file /tmp/directpv-drives.yaml

# Review discovered drives (node, device, size, status)
kubectl directpv list drives

Step 7.2 — Select drives to initialize

Edit /tmp/directpv-drives.yaml and set selected: true on each drive you want DirectPV to manage. Alternatively, use the Dohub admin UI (Cluster → DirectPV discover) to select drives and save.

Step 7.3 — Initialize selected drives

# WARNING: irreversible — formats selected drives
kubectl directpv init /tmp/directpv-drives.yaml --dangerous

# Confirm drives are available
kubectl directpv list drives
kubectl get directpvdrives.directpv.min.io

HAMi requires nvidia-container-runtime as the default containerd runtime on GPU nodes. On MicroK8s, edit the containerd template (not the generated containerd.toml, which is overwritten on restart).

Perform these steps on each GPU node:

Step 8.1 — Add the nvidia runtime to containerd template

TEMPLATE="/var/snap/microk8s/current/args/containerd-template.toml"

# Append nvidia runtime definition (if not already present)
sudo tee -a "$TEMPLATE" <<'EOF'

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
      runtime_type = "io.containerd.runc.v2"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
        BinaryName = "/usr/bin/nvidia-container-runtime"
EOF

Step 8.2 — Set nvidia as the default runtime

# Change default_runtime_name from "${RUNTIME}" to "nvidia-container-runtime"
sudo sed -i 's/default_runtime_name = "\${RUNTIME}"/default_runtime_name = "nvidia-container-runtime"/' \
  /var/snap/microk8s/current/args/containerd-template.toml

# Verify the change
grep default_runtime_name /var/snap/microk8s/current/args/containerd-template.toml

Step 8.3 — Restart MicroK8s on GPU nodes

# Full restart (recommended)
sudo microk8s stop
sudo microk8s start

# Or restart containerd only
sudo snap restart microk8s.daemon-containerd

# Confirm node rejoins Ready
microk8s kubectl get nodes

Step 8.4 — Verify GPU access from a test pod

microk8s kubectl run gpu-test --rm -it --restart=Never \
  --image=nvidia/cuda:12.4.0-base-ubuntu22.04 \
  -- nvidia-smi

HAMi (formerly k8s-vGPU-scheduler) enables fractional GPU sharing. DeepOps workspace templates request nvidia.com/gpu and optionally nvidia.com/gpumem (MiB) via the codehub chart.

Step 9.1 — Label GPU nodes

# List nodes and identify GPU workers
microk8s kubectl get nodes -o wide

# Label each GPU node (required — HAMi ignores unlabeled nodes)
microk8s kubectl label nodes <gpu-node-name> gpu=on

# Example
microk8s kubectl label nodes worker-gpu-01 gpu=on

Step 9.2 — Check Kubernetes version

microk8s kubectl version --short
# Note the server version, e.g. v1.35.0

Step 9.3 — Install HAMi Helm chart

microk8s helm repo add hami-charts https://project-hami.github.io/HAMi/
microk8s helm repo update

# Set scheduler image tag to match your Kubernetes server version
K8S_VERSION=$(microk8s kubectl version -o json | jq -r '.serverVersion.gitVersion')

microk8s helm upgrade --install hami hami-charts/hami \
  --namespace kube-system \
  --create-namespace \
  --set scheduler.kubeScheduler.imageTag="${K8S_VERSION}"

# Verify
microk8s kubectl get pods -n kube-system | grep hami

Both hami-device-plugin and hami-scheduler pods should reach Running state.

Example GPU workspace resources (codehub chart)

resources:
  limits:
    nvidia.com/gpu: 1
    nvidia.com/gpumem: 10240   # 10 GiB vGPU memory (optional)
  requests:
    nvidia.com/gpu: 1
    nvidia.com/gpumem: 10240

Step 10.1 — Configure secrets and values

cd /path/to/DeepOps

# Environment secrets (GitHub OAuth, admin users, domain)
cp dohub/secrets/.env.example dohub/secrets/.env
nano dohub/secrets/.env

# Key variables:
#   GITHUB_CLIENT_ID / GITHUB_CLIENT_SECRET
#   ADMIN_USERS=your-github-username
#   DOMAIN_NAME=hub.example.com
#   NAMESPACE=dohub
#   DJANGO_SECRET_KEY=…

Edit hub configuration and ingress host:

• dohub/configmap/config.yaml — storage class, server templates, ingress controller mode
• dohub/values.yaml — ingress host, TLS secret, image repository

Step 10.2 — Build images and deploy

From the repository root on a machine with Docker access to the MicroK8s registry:

chmod +x build-and-deploy.sh

# Build all images, push to local registry, deploy Helm chart
./build-and-deploy.sh -t 1.0.0 -r localhost:32000/dohub -p all

This script builds and optionally pushes:

• localhost:32000/dohub:1.0.0 — Django hub (includes kubectl, helm, kubectl-directpv, codehub chart)
• localhost:32000/ssh-bridge:1.0.0 — SSH-over-HTTPS sidecar
• localhost:32000/port-tunnel:1.0.0 — TCP port tunnel sidecar
• localhost:32000/monitor-sidecar:1.0.0 — Resource monitor sidecar

Step 10.3 — GitHub OAuth callback

Register a GitHub OAuth App with callback URL:

https://<your-ingress-host>/github-callback

Step 10.4 — TLS

TLS is configured with Let's Encrypt via cert-manager — see the Configure TLS with Let's Encrypt section. In short:

1. Edit the email in letsencrypt-cert-issuer.yaml and apply it to the cluster
2. Add cert-manager.io/cluster-issuer: letsencrypt-prod to dohub/values.yaml ingress annotations
3. Ensure DNS for your hub domain points at the ingress controller before deploying

cert-manager will populate tls-dohub-secret automatically when the Dohub ingress is created.

# Cluster health
microk8s kubectl get nodes
microk8s kubectl get pods -A | grep -Ev 'Running|Completed'

# DirectPV
microk8s kubectl get storageclass directpv-min-io
kubectl directpv list drives

# HAMi
microk8s kubectl get pods -n kube-system -l app.kubernetes.io/name=hami
microk8s kubectl describe node <gpu-node> | grep -A5 'Capacity'

# Dohub
microk8s kubectl get pods,ingress,pvc -n dohub

# TLS (cert-manager)
microk8s kubectl get clusterissuer letsencrypt-prod
microk8s kubectl get certificate -n dohub
curl -I https://<your-domain>/

Log in via GitHub, create a server from a template, and start it. A workspace pod should schedule, bind a DirectPV PVC, and become reachable at:

https://<workspace-slug>-<username>.<DOMAIN_NAME>/

Troubleshooting