{"id":235,"date":"2024-10-09T13:24:37","date_gmt":"2024-10-09T11:24:37","guid":{"rendered":"https:\/\/blog.openshift.one\/?p=235"},"modified":"2024-10-09T15:16:39","modified_gmt":"2024-10-09T13:16:39","slug":"scaling-openshift-compute-worker-baremetal-nodes-part-2","status":"publish","type":"post","link":"https:\/\/blog.openshift.one\/index.php\/2024\/10\/09\/scaling-openshift-compute-worker-baremetal-nodes-part-2\/","title":{"rendered":"Scaling OpenShift compute (worker) BareMetal nodes – Part 2"},"content":{"rendered":"\n

In the first post from this series – Scaling OpenShift compute (worker) BareMetal nodes – Part 1<\/a> – I explained the fully automated process of OpenShift<\/a> baremetal node bootstrap using out-of-band-management (OOBM) interface and virtual media functionality. While it is very convenient and time efficient, sometimes it does not fit to the environment where the cluster is running. For an instance there may be a firewall between the cluster and OOBM network which prevents communication between OpenShift<\/a>‘s Metal3<\/a> components and baremetal servers’ management consoles. In such case Metal3<\/a> duties in regard to servers power management are delegated to… a human being \ud83d\ude09<\/p>\n\n\n\n

Luckily OpenShift can provide valuable help in getting new node provisioned even manually. However the documentation about it is a bit unstructured and somewhat hard to follow, unless you know what you’re looking for. And for that reason I decided to gather all the information I gained and put it into a single blog post. <\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Few assumptions.<\/h2>\n\n\n\n

I assume the cluster is installed with Agent Based Installer (https:\/\/docs.openshift.com\/container-platform\/4.16\/installing\/installing_with_agent_based_installer\/preparing-to-install-with-agent-based-installer.html<\/a>) with BareMetal platform type. Clusters with user-provisioned infrastructure require additional validation and configuration to use the Machine API.<\/p>\n\n\n\n

Clusters with the infrastructure platform type none<\/code> cannot use the Machine API. This limitation applies even if the compute machines that are attached to the cluster are installed on a platform that supports the feature. To view the platform type for your cluster, run the following command:<\/p>\n\n\n\n

# oc get infrastructure cluster -o jsonpath='{.status.platform}'\nBareMetal<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
I also assume the environment is connected to Internet, however performing this exercise in disconnected\/air-gapped environment should be similar.<\/p>\n\n\n\n
The last but not least I need to get coreos-installer<\/code> package installed:<\/p>\n\n\n\n
# dnf install -y coreos-installer<\/code><\/pre>\n\n\n\n
Scaling-out the compute nodes using manually crafted RHCOS ISO image.<\/h2>\n\n\n\n
This scenario covers the case where fully automated node provisioning cannot be performed. The Red Hat CoreOS ISO bootstrap image can be written on DVD, USB or simply mounted with VirtualMedia if your OOBM interface allows to boot the server from it. In my lab I use Sushy tools<\/a> to emulate Redfish interface for baremetal management. <\/p>\n\n\n\n
The initial state of the cluster looks as follow:<\/p>\n\n\n\n
# oc get nodes,machines,machinesets,bmh\nNAME            STATUS   ROLES                         AGE   VERSION\nnode\/master-0   Ready    control-plane,master,worker   3d   v1.30.4\nnode\/master-1   Ready    control-plane,master,worker   3d   v1.30.4\nnode\/master-2   Ready    control-plane,master,worker   3d   v1.30.4\n\nNAME                                               PHASE     TYPE   REGION   ZONE   AGE\nmachine.machine.openshift.io\/ocp4-l2k5k-master-0   Running                          3d\nmachine.machine.openshift.io\/ocp4-l2k5k-master-1   Running                          3d\nmachine.machine.openshift.io\/ocp4-l2k5k-master-2   Running                          3d\n\nNAME                                                  DESIRED   CURRENT   READY   AVAILABLE   AGE\nmachineset.machine.openshift.io\/ocp4-l2k5k-worker-0   0         0                             3d\n\nNAME                               STATE       CONSUMER              ONLINE   ERROR   AGE\nbaremetalhost.metal3.io\/master-0   unmanaged   ocp4-l2k5k-master-0   true             3d\nbaremetalhost.metal3.io\/master-1   unmanaged   ocp4-l2k5k-master-1   true             3d\nbaremetalhost.metal3.io\/master-2   unmanaged   ocp4-l2k5k-master-2   true             3d<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
As you can see it is three nodes compact cluster where control plane and workloads are hosted on the same nodes. This deployment type allows you to benefit from highly available platform with minimal hardware footprint. <\/p>\n\n\n\n
<\/p>\n\n\n\n
Setting network configuration for the new node<\/h3>\n\n\n\n
If your node has a single NIC and the IP configuration is obtained via DHCP that\u2019s easy and you can skip to the next paragraph. However for the most of scenarios that\u2019s not the case. Very often there are multiple NICs which are supposed to be configured as bonds, use of the tagged VLANs or there is no DHCP in place and IP configuration has to be statically applied. When you were installing your cluster using Agent Based Installer, you probably came across agent-config.yaml<\/a> file which contains configuration of all nodes you\u2019re deploying at the cluster installation time, including network configuration written in a NMState format. However now, since deployment is already done, the file isn\u2019t useful anymore and you have to provide network configuration individually for the each node the other way. Luckily BareMetalHost<\/code> resource allows to reference Secret with NMState configuration for the new node.<\/p>\n\n\n\n
The full list of options including examples you can find at the official NMState project page: https:\/\/nmstate.io\/<\/a>. For the demo purposes I will configure bond interface with two physical NICs, using active-backup (Mode 1) bonding which does not require any integration at the switch side. Additionaly IP configuration is static, therefore each new node will require individual Secret with NMState configuration<\/strong> (WARNING: overlapping IP addresses ahead!<\/mark><\/strong>). For more information about bonding and OpenShift, please refer to: https:\/\/docs.openshift.com\/container-platform\/4.16\/networking\/k8s_nmstate\/k8s-nmstate-updating-node-network-config.html#virt-example-bond-nncp_k8s_nmstate-updating-node-network-config<\/a><\/p>\n\n\n\n
Bellow is the content of Secret containing NMState configuration for my new baremetal node:<\/p>\n\n\n\n
apiVersion: v1
kind: Secret
metadata:
  name: compute-0-network-config-secret
  namespace: openshift-machine-api
type: Opaque
stringData:
  nmstate: |
    interfaces:
      - name: bond0
        type: bond
        state: up
        link-aggregation:
          mode: active-backup
          options:
            primary: enp1s0
          port:
            - enp1s0
            - enp2s0
        ipv4:
          dhcp: false
          enabled: true
          address:
            - ip: 192.168.232.103
              prefix-length: 24
      - name: enp1s0
        type: ethernet
        state: up
        ipv4:
          enabled: false
          dhcp: false
      - name: enp2s0
        type: ethernet
        state: up
        ipv4:
          enabled: false
          dhcp: false
      - name: enp3s0
        type: ethernet
        state: up
        ipv4:
          enabled: false
          dhcp: false
      - name: enp4s0
        type: ethernet
        state: up
        ipv4:
          enabled: false
          dhcp: false
    routes:
      config:
        - destination: 0.0.0.0\/0
          next-hop-address: 192.168.232.1
          next-hop-interface: bond0
          table-id: 254
    dns-resolver:
      config:
        server:
          - 192.168.232.1

<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
The command to create it is as follow:<\/p>\n\n\n\n
# oc -n openshift-machine-api create -f compute-0.nmstate.secret.yaml\nsecret\/compute-0-network-config-secret created<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
Creating BareMetalHost resource (and labeling it)<\/h3>\n\n\n\n
The BareMetalHost resource defines a physical host and its properties. For more information about it please refer to Bare-metal configuration<\/a> documentation. The resource definition looks as follow:<\/p>\n\n\n\n
apiVersion: metal3.io\/v1alpha1\nkind: BareMetalHost\nmetadata:\n  name: compute-0\n  namespace: openshift-machine-api\nspec:\n  automatedCleaningMode: metadata\n  bmc:\n    address: \"\"\n    credentialsName: \"\"\n  bootMACAddress: de:ad:be:ef:66:04\n  bootMode: UEFI\n  customDeploy:\n    method: install_coreos\n  externallyProvisioned: true\n  hardwareProfile: unknown\n  online: true\n  userData:\n    name: worker-user-data-managed\n    namespace: openshift-machine-api<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
The comprehensive description of BareMetalHost resource can be found at The BareMetalHost spec<\/a>, but for now please check the following:<\/p>\n\n\n\n