SSH Tunnel Configuration

Many HPC sites give compute nodes no outbound internet access, and many Kubernetes clusters have no publicly reachable ingress. The wstunnel shadow needs both: the workload dials out to an ingress and runs a wstunnel client there. On an air-gapped site it cannot.

The SSH shadow inverts the direction. Instead of the compute node dialing out, the cluster dials in to the site's SSH login node and forwards each exposed port from the compute node the job landed on:

browser → Ingress / Service → shadow pod (ssh -L) → HPC login node → compute node (Jupyter)

The only requirement is outbound SSH from the cluster to the login node. The offloaded pod runs nothing on its side: no wstunnel client, no WireGuard configuration, no pre-exec injection.

info

The SSH shadow replaces wstunnel, not full mesh. It exposes the offloaded pod's ports to the cluster; it does not give the pod access back into the cluster, so an offloaded workload still cannot reach in-cluster object storage or message buses. Combining the two is rejected at startup and tracked in #548.

---

Before you start: what the login node has to allow

The shadow is an ordinary SSH client, so everything it needs is decided by the login node's sshd_config.

1. AllowTcpForwarding yes for the account the shadow logs in as. This is the one that catches people out: plenty of HPC sites set AllowTcpForwarding no globally and re-enable it only for a subset of users, often only those who authenticate with MFA. Check it before anything else:

   # from your own machine, with the same key/principal the shadow will use
   ssh -N -L 19999:<login node>:22 <user>@<login node> &
   nc -z 127.0.0.1 19999 && head -c 40 < /dev/tcp/127.0.0.1/19999

   If the site refuses, the shadow still starts and still reports Ready — the failure only shows up when traffic arrives, as a connection reset at the client and this line in kubectl logs <shadow pod> -c ssh-forward:

   channel 1: open failed: administratively prohibited: open failed

   If the site will not allow it, switch to ForwardMode: exec, which relays through a command on the login node instead and needs no forwarding privilege.

2. A route from the login node to the compute nodes, on the ports the pod exposes. ssh -L resolves and connects to the compute node from the login node, using whatever name the plugin reported. Verify with the name the plugin actually reports (hostname -f on the compute node for the Slurm plugin), not the short name:

   ssh <user>@<login node> "curl -sv http://<compute node fqdn>:<port>/"

3. Outbound SSH from the cluster to the login node's port, from the namespaces shadows are created in. Network policies that restrict pod egress have to allow it.

ssh -L carries TCP only. UDP ports on the offloaded pod are skipped, with a warning in the virtual kubelet log.

---

Which shadow does what

	Direction	Trigger	Workload must run
wstunnel (default)	cluster → pod	EnableTunnel + exposed ports	wstunnel client, dials out
ssh	cluster → pod	EnableTunnel + exposed ports	nothing
full mesh	bidirectional	FullMesh: true, every pod	mesh.sh (slirp4netns + WireGuard)

---

Configuration

Virtual Kubelet

# VirtualKubeletConfig.yaml
Network:
  EnableTunnel: true
  ShadowMode: ssh
  SSH:
    LoginHost: login.hpc.example.org
    User: alice
    KeySecret: hpc-ssh-key

EnableTunnel turns shadow pods on for offloaded pods with exposed ports; ShadowMode picks which shadow is rendered. Both are needed.

Helm

virtualNode:
  network:
    enableTunnel: true
    shadowMode: ssh
    ssh:
      loginHost: login.hpc.example.org
      user: alice
      keySecret: hpc-ssh-key

See examples/ssh_tunnel.yaml in the chart repository for a complete deployment.

Options

Option	Default	Description
LoginHost	—	SSH login node to forward through (required)
User	—	Login name on that node (required)
Port	22	Login node's SSH port
Image	ghcr.io/interlink-hq/interlink/ssh-tunnel:<version>	Image the shadow runs. Needs an ssh client, plus kinit for Kerberos
Auth	publickey	publickey or kerberos
ForwardMode	portforward	portforward (ssh -L) or exec (relay through a command on the login node)
ExecConnectCommand	nc	Command run on the login node in exec mode, invoked as <command> <node> <port>
KeySecret	—	Secret holding the private key (publickey)
KeySecretKey	id_ed25519	Key inside KeySecret
KeytabSecret	—	Secret holding the keytab (kerberos)
KeytabSecretKey	user.keytab	Key inside KeytabSecret
Principal	—	Kerberos principal (kerberos)
Krb5ConfigMap	—	ConfigMap with a krb5.conf, mounted at /etc/krb5.conf
KnownHostsConfigMap	—	ConfigMap with a known_hosts file
ReplicateCredentials	true	Copy the credential into each shadow's namespace
NodeWaitTimeout	2h	How long the shadow waits for the compute node
ExtraOptions	[]	Extra ssh options, each passed as -o <option>

Misconfiguration fails at startup, not on the first offloaded pod, so a missing LoginHost or a kerberos block without a Principal is caught immediately.

---

Forward modes

The default, portforward, is one ssh -L per exposed port. It is the cheapest option and depends on nothing but the ssh client, but it requires AllowTcpForwarding yes on the login node.

exec is for sites that do not grant it. It gives each exposed port a local listener in the shadow and relays every accepted connection through a command run on the login node — by default nc <compute node> <port>. Nothing asks sshd to forward anything, so the only privileges involved are the ones the shadow already uses to log in.

Network:
  ShadowMode: "ssh"
  SSH:
    LoginHost: "login.hpc.example.org"
    User: "alice"
    KeySecret: "hpc-ssh-key"
    ForwardMode: "exec"
    ExecConnectCommand: "nc"   # ncat, or "nc -q 1" on a netcat that does not half-close

The two are interchangeable above the socket: the Service still fronts the same container port, and the offloaded workload still only binds 0.0.0.0. Nothing on either side can tell them apart.

	portforward	exec
Login node must allow	TCP forwarding	running a command, and have a netcat
Connections	one long-lived ssh	one ssh channel per TCP connection
Cost per connection	none	one multiplexed channel open
Failure surfaces	at connect time	at connect time

The relays share a single SSH connection, opened as a control master before the listeners start. Without it every connection would pay a full handshake — a notebook UI opens dozens — and the login node would see a session per connection. Because they share one TCP connection they also share its head-of-line blocking, which is fine for a UI and less so for bulk transfer.

Two things to check when picking exec:

• Which netcat. nc varies. netcat-traditional does not propagate a half-close, so a client that closes its write side can leave the connection hanging; nc -q 1 or ncat avoids it. Set ExecConnectCommand accordingly.
• Process limits. Even multiplexed, the login node runs a netcat per connection. Sites with per-user process caps will notice a busy shadow.

---

Authentication

Public key

ssh-keygen -t ed25519 -f ./hpc_key -N ''
ssh-copy-id -i ./hpc_key.pub alice@login.hpc.example.org

kubectl create secret generic hpc-ssh-key -n interlink \
  --from-file=id_ed25519=./hpc_key

SSH:
  Auth: publickey
  KeySecret: hpc-ssh-key

Kerberos

For GSSAPI sites (GSSAPIAuthentication yes on the login node):

kubectl create secret generic hpc-keytab -n interlink \
  --from-file=user.keytab=/path/to/user.keytab

SSH:
  Auth: kerberos
  KeytabSecret: hpc-keytab
  Principal: alice@EXAMPLE.ORG
  Krb5ConfigMap: krb5-config   # if the image's /etc/krb5.conf lacks your realm

The shadow runs kinit in an init container and refreshes the ticket from a sidecar for the life of the tunnel, so long-running notebooks survive ticket expiry.

Host key verification

Without KnownHostsConfigMap the shadow uses StrictHostKeyChecking=accept-new, which trusts whatever key the login node presents on first contact. That is a small MITM window on every new shadow pod. For production, pin the key:

ssh-keyscan login.hpc.example.org > known_hosts
kubectl create configmap hpc-known-hosts -n interlink --from-file=known_hosts

SSH:
  KnownHostsConfigMap: hpc-known-hosts

This switches the shadow to StrictHostKeyChecking=yes.

---

How the compute node is discovered

The shadow is created when the pod is, which is long before the batch system has scheduled anything — so at creation time there is no node to forward to yet.

1. interLink creates a ConfigMap, <shadow>-node, holding an empty compute-node key.
2. The plugin reports the allocated node as PodStatus.NodeName once the job runs.
3. interLink patches that value into the ConfigMap. It is mounted, not injected into the pod spec, so kubelet refreshes it in place — restarting the shadow would change its pod IP, which Kubernetes has already been told is the offloaded pod's IP.
4. The shadow's wait-for-node init container polls the mounted file, then ssh-forward starts the tunnel.

While the job is queued the shadow sits in Init:0/1. That is the expected state, not a failure — it reads as "waiting for the allocation" rather than a crash-looping tunnel. After NodeWaitTimeout the init container fails with an explicit message.

If the job is later requeued onto a different host, the plugin reports the new one, interLink republishes it, and the shadow rebuilds its forwarders in place:

compute node changed from node07.hpc.example.org to node11.hpc.example.org, rebuilding the tunnel

The container is not restarted, so the shadow keeps its pod IP — the one Kubernetes has already been told is the offloaded pod's IP.

warning

This requires a plugin that reports PodStatus.NodeName. A plugin that does not will leave the shadow waiting until the timeout expires.

What a plugin may report

The reported name ends up as an argument to ssh, so interLink accepts only an RFC 1123 hostname or an IP literal and refuses anything else rather than trying to escape it:

Refusing to publish compute node "node07 -oProxyCommand=..." for ns/pod:
not a hostname or IP address. The shadow passes this to ssh, so it is rejected
rather than escaped.

A name carrying whitespace would otherwise split into an extra ssh option, and one carrying a quote or $( would break out of the relay command in exec mode. Plugins should report what the node calls itself — hostname -f for the Slurm plugin — and nothing else.

---

Credential replication

Shadow pods follow the offloaded pod. With interlink.eu/shadow-same-ns: "true" — the usual choice for multi-tenant setups such as Kubeflow notebooks — that means the shadow lands in a per-user namespace, where the SSH credential does not exist.

By default interLink copies the credential Secret, and any Krb5ConfigMap / KnownHostsConfigMap, from its own namespace into the shadow's namespace.

Copies are marked with interlink.eu/replicated-from, and only objects carrying that annotation are overwritten on a later pass. If an object of the same name is already there and holds something else, pod creation fails rather than replacing it:

refusing to overwrite secret alice/hpc-ssh-key: it already exists, holds different
content and was not created by interLink

That matters with shadow-same-ns, where the target is somebody's own namespace and a name collision would otherwise destroy their Secret. An unmarked copy whose content already matches is adopted instead of refused, so upgrading from a version that predates the marker does not break existing deployments.

warning

Replication makes the credential readable by anyone who can read Secrets in those namespaces. If that is not acceptable, set ReplicateCredentials: false and provision the Secret in each namespace yourself, through a secrets operator or a replication controller.

---

Pod configuration

Nothing special is required — an offloaded pod that exposes a port gets a shadow:

apiVersion: v1
kind: Pod
metadata:
  name: notebook
  annotations:
    # Keep the shadow in this pod's namespace, so it can mount credentials
    # replicated here rather than into a separate <ns>-shadow namespace.
    interlink.eu/shadow-same-ns: "true"
spec:
  nodeSelector:
    kubernetes.io/hostname: hpc-vk
  tolerations:
  - key: virtual-node.interlink/no-schedule
    operator: Exists
    effect: NoSchedule
  containers:
  - name: jupyter
    image: docker://jupyter/scipy-notebook
    ports:
    - containerPort: 8888
      name: notebook

The shadow's Service carries the same ports, so anything that can reach the Service — an Ingress, an Istio VirtualService, kubectl port-forward — reaches the notebook.

interlink.eu/wstunnel-extra-ports works here too, for ports not declared in the container spec.

note

ssh -L forwards TCP only. UDP ports are skipped, with a warning in the virtual kubelet log.

---

Troubleshooting

Shadow stuck in Init:0/1. Normal while the job is queued. If it persists, check that the job is actually running and that the plugin reports the node:

kubectl get configmap <shadow>-node -o jsonpath='{.data.compute-node}'

Empty means no plugin has reported one yet.

ssh-forward crash-loops with a permission error. The credential Secret is missing from the shadow's namespace, or KeySecretKey does not match the key inside it. With ReplicateCredentials: false the Secret must already exist there.

Connection refused through the tunnel. The tunnel reached the compute node but nothing is listening on that port. Services often bind 127.0.0.1 by default; bind 0.0.0.0 instead — for a notebook, --ip=0.0.0.0.

Connection reset, and channel N: open failed: administratively prohibited in the ssh-forward log. The login node refuses port forwarding for this account. Either have the site allow it, or switch to ForwardMode: exec.

Host key verification failed. KnownHostsConfigMap does not contain the login node's current key. Re-run ssh-keyscan and update the ConfigMap.

Refusing to publish compute node ... in the virtual kubelet log, and the shadow stays in Init:0/1. The plugin reported something that is not a hostname or an IP address, so it was not passed to ssh. See what a plugin may report; the plugin needs fixing, and the value it produced is quoted in the message.

Pod creation fails with refusing to overwrite secret .... The shadow's namespace already contains a different object under the credential's name. Rename the credential, or set ReplicateCredentials: false and provision it yourself.

An nginx wstunnel Deployment appears instead of the SSH shadow. A custom template file is overriding the built-in one. The virtual kubelet prefers Network.WstunnelTemplatePath over its own templates; the Helm chart stops mounting it in ssh mode, but a hand-written config may still point at one.

---

References

• Discussion #535 — the original proposal
• Issue #545 — in-cluster SSH deployment
• Wstunnel Configuration
• Mesh Network Configuration