nodeup: load ip_set module and disable firewalld on RHEL10 - #18478

Summary

Two related fixes for kops nodeup on the ForceNftables() distros (RHEL 10+, Rocky 10+):

1. Load the ip_set kernel module alongside nf_tables / nf_conntrack in loadKernelModules.
2. Stop and mask firewalld.service via a new step on FirewallBuilder, gated on Distribution.ForceNftables().

Together these unblock the failing Calico cells in the e2e-kops-grid-calico-{rhel10,rocky10}* grid, which produce two distinct failure modes today.

Problem

Mode 1 — Felix ipsetsManager panics (arm64 cells)

Calico v3.31's Felix unconditionally starts an ipsetsManager that shells out to ipset list -name during dataplane resync, even when NFTablesMode: Enabled is in effect. On RHEL 10-family arm64 kernels the ip_set module isn't auto-loaded, so the netlink request returns EINVAL and Felix panics in a tight loop, taking calico-node with it. Every node reports CrashLoopBackOff, validation fails after ~25 min, Up returns exit 1.

The relevant Felix log on a failing node:

[ERROR] felix/ipsets.go:758: Bad return code from 'ipset list -name'. stderr="ipset v7.11: Kernel error received: Invalid argument"
[PANIC] felix/ipsets.go:416: Failed to update IP sets after multiple retries.

lsmod on the same node confirms ip_set is absent while nf_tables and nf_conntrack (already modprobed by kops) are present.

Example failing jobs:

• AWS Calico rhel10arm64 k8s 1.33 kops 1.35
• AWS Calico rhel10arm64 k8s 1.33 kops master
• AWS Calico rocky10arm64 k8s 1.32 kops 1.35

Mode 2 — BGP keepalives killed by firewalld interaction (GCE x86_64 cells)

On the GCE-optimized Rocky Linux 10 image, firewalld.service starts at boot: Starting firewalld.service - firewalld - dynamic firewall daemon..., and the resulting nft ruleset contains a populated table inet firewalld with default-reject filter_INPUT and filter_FORWARD policies plus a ct state invalid drop rule. firewalld's periodic-reload behavior plus that ct state invalid drop race against Calico's BPF-mode INPUT chain rule:

meta mark & 0x05000000 == 0x05000000 meta l4proto tcp ... reject with tcp reset

which TCP-RSTs any BGP keepalive that arrives after the session's conntrack entry has been dropped. BIRD then logs Error: Hold timer expired on all peers simultaneously and the mesh never recovers. Cluster validation succeeds but e2e tests time out at the 60-minute kubetest2 cap.

This pattern persists even with networking.calico.nftablesMode: Enabled set (the field added in #18452), confirming the flap isn't caused by Felix's nft_compat usage. the host-level firewalld is the remaining disturbance.

Example failing builds:

• GCE Calico rocky10 k8s 1.33 kops 1.35
• GCE Calico rocky10 k8s 1.35 kops 1.35 (the original report, pre-nftablesMode)

Why only the GCE image

Empirical comparison from this investigation's own artifacts:

The CIQ-built GCE-optimized spin ships firewalld active; the Red Hat AWS AMIs and the upstream Rocky GenericCloud qcow2 don't. Disabling firewalld in nodeup is therefore a no-op on the AWS path and a fix on the GCE path.

Fix

upup/pkg/fi/nodeup/command.go — loadKernelModules: append ip_set to the ForceNftables() modprobe list. modprobe already logs warnings on failure rather than aborting nodeup, so the call is safe on hosts where the module package isn't installed.

nodeup/pkg/model/firewall.go — new disableFirewalld(c) step: mirrors the existing disableNMCloudSetup pattern. it writes a marker file under /etc/kops/ and uses OnChangeExecute to run systemctl disable --now firewalld.service and systemctl mask firewalld.service, both wrapped in bash -c '... 2>/dev/null; true' so they're safe no-ops on images where firewalld isn't installed. Gated on b.Distribution.ForceNftables().

Precedent for unconditionally disabling firewalld

• Calico requirements doc: "If your Linux distribution comes with installed Firewalld or another iptables manager it should be disabled."
• RKE2 known issues: "firewalld should be disabled on systems running RKE2."
• OpenShift: disables firewalld on every RHEL worker (openshift-ansible #11824).
• AWS RHEL/Rocky 10 image team: strips firewalld from the AMI.