Network policies

Harthelm isolates teams from each other and from the platform with NetworkPolicies. They only work on a CNI that enforces them, with endPort; prove that with the self-check below.

The release namespaceCopy Anchor Link

With networkPolicy.enabled (the default) everything in the release namespace is denied, then exactly this is allowed:

Who counts as the edge, the scrapers and the operator:

Egress to the outsideCopy Anchor Link

Destinations outside the cluster come from networkPolicy.egress: per destination a list of NetworkPolicy peers (to) and TCP ports.

Narrow each to your addresses; hart admin doctor warns about every rule without to. A rule with neither to nor ports allows all egress, every port to every address; the doctor names it with "all ports" Since 1.0.3.

networkPolicy:
  egress:
    apiServer: {to: [{ipBlock: {cidr: 10.0.0.1/32}}], ports: [443, 6443]}
    idp: {to: [{ipBlock: {cidr: 192.0.2.30/32}}], ports: [443]}

networkPolicy.egress.extra appends raw egress rules to core's policy, networkPolicy.egress.router to the router's (for forward targets named in router.forward.allowHosts, which a NetworkPolicy cannot match by name).

Team namespacesCopy Anchor Link

Core renders the policies of every team namespace itself:

• pods talk to each other inside the namespace, never to another team's;
• DNS only to the cluster DNS pods (teams.network.dns);
• core only on its API and git proxy ports, never the router or the edge;
• the internet on every port except TCP 25, but no private address: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 100.64.0.0/10, 169.254.0.0/16 (which includes cloud metadata endpoints), fc00::/7 and fe80::/10.

Mirrors and other services on private addresses that workspaces must reach go in teams.network.mirrors.

Forward services (a workspace service that forwards to an outside host) go through the router, which reaches public addresses while router.forward.allowPublic is on, and the ranges in router.forward.allowCIDRs.

CiliumCopy Anchor Link

Cilium cannot select the API server by address in a NetworkPolicy. Give the platform pods their API access with CiliumNetworkPolicies through extraObjects, and give core the rule for the per-team backstop it renders when it detects Cilium (a CiliumClusterwideNetworkPolicy that denies the private ranges, the API server and SMTP even if someone adds a policy in a team namespace):

extraObjects:
  - apiVersion: cilium.io/v2
    kind: CiliumNetworkPolicy
    metadata: {name: harthelm-platform-apiserver, namespace: harthelm-system}
    spec:
      endpointSelector:
        matchExpressions:
          - {key: app.kubernetes.io/name, operator: In, values: [harthelm]}
          - {key: app.kubernetes.io/component, operator: In, values: [core, router]}
      egress:
        - toEntities: [kube-apiserver, host, remote-node]
          toPorts: [{ports: [{port: "6443", protocol: TCP}, {port: "443", protocol: TCP}]}]
  # The preflight Job runs as a hook before the other objects exist, so its policy is a hook too.
  - apiVersion: cilium.io/v2
    kind: CiliumNetworkPolicy
    metadata:
      name: harthelm-preflight-apiserver
      namespace: harthelm-system
      annotations:
        helm.sh/hook: pre-install,pre-upgrade
        helm.sh/hook-weight: "-20"
        helm.sh/hook-delete-policy: before-hook-creation
        argocd.argoproj.io/hook: PreSync
        argocd.argoproj.io/sync-wave: "-2"
        argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
    spec:
      endpointSelector:
        matchLabels: {app.kubernetes.io/name: harthelm, app.kubernetes.io/component: preflight}
      egress:
        - toEntities: [kube-apiserver, host, remote-node]
          toPorts: [{ports: [{port: "6443", protocol: TCP}, {port: "443", protocol: TCP}]}]
  - apiVersion: cilium.io/v2
    kind: CiliumNetworkPolicy
    metadata: {name: harthelm-db-apiserver, namespace: harthelm-system}
    spec:
      endpointSelector: {matchLabels: {cnpg.io/cluster: harthelm-db}}
      egress:
        - toEntities: [kube-apiserver, host, remote-node]
          toPorts: [{ports: [{port: "6443", protocol: TCP}, {port: "443", protocol: TCP}]}]
rbac:
  extraClusterRules:
    - apiGroups: [cilium.io]
      resources: [ciliumclusterwidenetworkpolicies]
      verbs: [get, list, watch, create, update, patch, delete]

harthelm-db is <release>-db; change it if your release is not called harthelm.

Since 1.5.0 these values are a supported starting point: each release renders them with the chart and checks them against the Cilium schemas, so a chart change that breaks them is caught before it ships. What each part is for:

• harthelm-platform-apiserver gives core and the router the API; harthelm-db-apiserver gives the bundled database's instance manager the API. Leave the latter out with an external database. Every other pod of the release has no API token and needs none.
• harthelm-preflight-apiserver is a hook in the preflight Job's own wave. The Job is a Helm pre-install and pre-upgrade hook and an Argo CD PreSync hook, so it runs before the ordinary objects of the release exist (on a first sync, before any of them); an ordinary policy would not be there yet.
• rbac.extraClusterRules lets core manage ciliumclusterwidenetworkpolicies. With it core renders the backstop per team namespace and the policy that lets the managed PostgreSQL pods reach the API server; without it the capability probe reports cilium-policies as absent. The admission guard limits core to objects labelled harthelm.dev/managed=true.
• The ports are the API server's: 6443 on kubeadm clusters, 443 on many managed ones. Keep the one yours listens on.

Keep them in a values file of their own, cilium.yaml, and pass it after yours (-f my-values.yaml -f cilium.yaml). Helm replaces lists, it does not merge them: if your own values set extraObjects or rbac.extraClusterRules too, merge the entries into one list by hand. The names assume the release harthelm in the namespace harthelm-system. Afterwards helm test harthelm and hart admin doctor --capabilities (network-policies, cilium-policies) show that the policies hold. For an Ingress controller on host ports without a load balancer, see Without a load balancer.

The self-checkCopy Anchor Link

helm test harthelm -n harthelm-system

A probe pod connects to a probe server on three ports: one allowed by a single port, two allowed by an endPort range, and one that must be blocked. The test fails when a policy is not enforced or the range is ignored. Under Argo CD a report-only copy runs after every sync. The doctor's network-policy check and the Capabilities page show the newest result, so run the test again after every CNI change. tests.networkPolicy.enabled switches the probe off.