Version: 2.4.15

Using an External Ceph Driver

These instructions are about using the external Ceph driver in an RKE2 cluster. If you are using RKE, additional steps are required. For details, refer to this section.

Requirements
Using the Ceph Driver with RKE
Installing the ceph-csi driver on an RKE2 cluster
Install the ceph-csi driver using Helm
Creating RBD Ceph Resources
Configure RBD Ceph Access Secrets
- User Account
- Admin Account
Create RBD Testing Resources
Tested Versions
Troubleshooting

Requirements

Make sure ceph-common and xfsprogs packages are installed on SLE worker nodes.

Using the Ceph Driver with RKE

The resources below are fully compatible with RKE based clusters, but there is a need to do an additional kubelet configuration for RKE.

On RKE clusters, the kubelet component is running in a Docker container and doesn't have access to the host's kernel modules as rbd and libceph by default.

To solve this limitation, you can either run modprobe rbd on worker nodes, or configure the kubelet containers to automatically mount the /lib/modules directory from the host into the container.

For the kubelet configuration, put the following lines into the cluster.yml file prior to RKE cluster provisioning. You can also modify the cluster.yml later in the Rancher UI by clicking on Edit Cluster > Edit as YAML and restarting the worker nodes.

services:  kubelet:    extra_binds:      - '/lib/modules:/lib/modules:ro'

For more information about the extra_binds directive, refer to this section.

Installing the ceph-csi driver on an RKE2 cluster

> Note: These steps are needed for dynamic RBD provisioning only.

For more information about the ceph-csi-rbd chart, refer to this page.

To get details about your SES cluster, run:

ceph mon dump

Read its output:

dumped monmap epoch 3epoch 3fsid 79179d9d-98d8-4976-ab2e-58635caa7235last_changed 2021-02-11T10:56:42.110184+0000created 2021-02-11T10:56:22.913321+0000min_mon_release 15 (octopus)0: [v2:10.85.8.118:3300/0,v1:10.85.8.118:6789/0] mon.a1: [v2:10.85.8.123:3300/0,v1:10.85.8.123:6789/0] mon.b2: [v2:10.85.8.124:3300/0,v1:10.85.8.124:6789/0] mon.c

Later you'll need the fsid and mon addresses values.

Install the ceph-csi Driver Using Helm

Run these commands:

helm repo add ceph-csi https://ceph.github.io/csi-chartshelm repo updatehelm search repo ceph-csi -lhelm inspect values ceph-csi/ceph-csi-rbd \> ceph-csi-rbd-values.yaml

Modify the ceph-csi-rbd-values.yaml file and keep there only the required changes:

# ceph-csi-rbd-values.yamlcsiConfig:  - clusterID: "79179d9d-98d8-4976-ab2e-58635caa7235"    monitors:      - "10.85.8.118:6789"      - "10.85.8.123:6789"      - "10.85.8.124:6789"provisioner:  name: provisioner  replicaCount: 2

Make sure the ceph monitors are reachable from the RKE2 cluster, for example, by ping.

kubectl create namespace ceph-csi-rbdhelm install --namespace ceph-csi-rbd ceph-csi-rbd ceph-csi/ceph-csi-rbd --values ceph-csi-rbd-values.yamlkubectl rollout status deployment ceph-csi-rbd-provisioner -n ceph-csi-rbdhelm status ceph-csi-rbd -n ceph-csi-rbd

in case you'd like to modify the configuration directly via Helm, you may adapt the ceph-csi-rbd-values.yaml file and call:

helm upgrade \  --namespace ceph-csi-rbd ceph-csi-rbd ceph-csi/ceph-csi-rbd --values ceph-csi-rbd-values.yaml

Creating RBD Ceph Resources

# Create a ceph pool:ceph osd pool create myPool 64 64
# Create a block device pool:rbd pool init myPool
# Create a block device image:rbd create -s 2G myPool/image
# Create a block device user and record the key:ceph auth get-or-create-key client.myPoolUser mon "allow r" osd "allow class-read object_prefix rbd_children, allow rwx pool=myPool" | tr -d '\n' | base64QVFDZ0R5VmdyRk9KREJBQTJ5b2s5R1E2NUdSWExRQndhVVBwWXc9PQ==
# Encode the ceph user myPoolUser into a bash64 hash:echo "myPoolUser" | tr -d '\n' | base64bXlQb29sVXNlcg==
# Create a block device admin user and record the key:ceph auth get-or-create-key client.myPoolAdmin mds 'allow *' mgr 'allow *' mon 'allow *' osd 'allow * pool=myPool' | tr -d '\n' | base64QVFCK0hDVmdXSjQ1T0JBQXBrc0VtcVhlZFpjc0JwaStIcmU5M3c9PQ==
# Encode the ceph user myPoolAdmin into a bash64 hash:echo "myPoolAdmin" | tr -d '\n' | base64bXlQb29sQWRtaW4=

Configure RBD Ceph Access Secrets

User Account#

For static RBD provisioning (the image within the ceph pool must exist), run these commands:

cat \> ceph-user-secret.yaml \<\< EOFapiVersion: v1kind: Secretmetadata:  name: ceph-user  namespace: defaulttype: kubernetes.io/rbddata:  userID: bXlQb29sVXNlcg==  userKey: QVFDZ0R5VmdyRk9KREJBQTJ5b2s5R1E2NUdSWExRQndhVVBwWXc9PQ==EOF
kubectl apply -f ceph-user-secret.yaml

Admin Account#

For dynamic RBD provisioning (used for automatic image creation within a given ceph pool), run these commands:

cat \> ceph-admin-secret.yaml \<\< EOFapiVersion: v1kind: Secretmetadata:  name: ceph-admin  namespace: defaulttype: kubernetes.io/rbddata:  userID: bXlQb29sQWRtaW4=  userKey: QVFCK0hDVmdXSjQ1T0JBQXBrc0VtcVhlZFpjc0JwaStIcmU5M3c9PQ==EOF
kubectl apply -f ceph-admin-secret.yaml

Create RBD Testing Resources

Using RBD in Pods#

# podcat \> ceph-rbd-pod-inline.yaml \<\< EOFapiVersion: v1kind: Podmetadata:  name: ceph-rbd-pod-inlinespec:  containers:  - name: ceph-rbd-pod-inline    image: busybox    command: ["sleep", "infinity"]    volumeMounts:    - mountPath: /mnt/ceph_rbd      name: volume  volumes:  - name: volume    rbd:      monitors:      - 10.85.8.118:6789      - 10.85.8.123:6789      - 10.85.8.124:6789      pool: myPool      image: image      user: myPoolUser      secretRef:        name: ceph-user      fsType: ext4      readOnly: falseEOF kubectl apply -f ceph-rbd-pod-inline.yamlkubectl get podkubectl exec pod/ceph-rbd-pod-inline -- df -k | grep rbd

Using RBD in Persistent Volumes#

# pod-pvc-pvcat \> ceph-rbd-pod-pvc-pv-allinone.yaml \<\< EOFapiVersion: v1kind: PersistentVolumemetadata:  name: ceph-rbd-pvspec:  capacity:    storage: 2Gi  accessModes:    - ReadWriteOnce  rbd:    monitors:    - 10.85.8.118:6789    - 10.85.8.123:6789    - 10.85.8.124:6789    pool: myPool     image: image    user: myPoolUser     secretRef:      name: ceph-user    fsType: ext4    readOnly: false---kind: PersistentVolumeClaimapiVersion: v1metadata:  name: ceph-rbd-pvcspec:  accessModes:  - ReadWriteOnce  resources:    requests:      storage: 2Gi---apiVersion: v1kind: Podmetadata:  name: ceph-rbd-pod-pvc-pvspec:  containers:  - name: ceph-rbd-pod-pvc-pv    image: busybox    command: ["sleep", "infinity"]    volumeMounts:    - mountPath: /mnt/ceph_rbd      name: volume  volumes:  - name: volume    persistentVolumeClaim:      claimName: ceph-rbd-pvcEOF kubectl apply -f ceph-rbd-pod-pvc-pv-allinone.yamlkubectl get pv,pvc,podkubectl exec pod/ceph-rbd-pod-pvc-pv -- df -k | grep rbd

Using RBD in Storage Classes#

This example is for dynamic provisioning. The ceph-csi driver is needed.

# pod-pvc-sccat \> ceph-rbd-pod-pvc-sc-allinone.yaml \<\<EOFapiVersion: storage.k8s.io/v1kind: StorageClassmetadata:  name: ceph-rbd-sc  annotations:    storageclass.kubernetes.io/is-default-class: "true"provisioner: rbd.csi.ceph.comparameters:   clusterID: 79179d9d-98d8-4976-ab2e-58635caa7235   pool: myPool   imageFeatures: layering   csi.storage.k8s.io/provisioner-secret-name: ceph-admin   csi.storage.k8s.io/provisioner-secret-namespace: default   csi.storage.k8s.io/controller-expand-secret-name: ceph-admin   csi.storage.k8s.io/controller-expand-secret-namespace: default   csi.storage.k8s.io/node-stage-secret-name: ceph-admin   csi.storage.k8s.io/node-stage-secret-namespace: defaultreclaimPolicy: DeleteallowVolumeExpansion: truemountOptions:   - discard---kind: PersistentVolumeClaimapiVersion: v1metadata:  name: ceph-rbd-sc-pvcspec:  accessModes:    - ReadWriteOnce  resources:    requests:      storage: 2Gi  storageClassName: ceph-rbd-sc---    apiVersion: v1kind: Podmetadata:  name: ceph-rbd-pod-pvc-scspec:  containers:  - name:  ceph-rbd-pod-pvc-sc    image: busybox    command: ["sleep", "infinity"]    volumeMounts:    - mountPath: /mnt/ceph_rbd      name: volume  volumes:  - name: volume    persistentVolumeClaim:      claimName: ceph-rbd-sc-pvcEOF kubectl apply -f ceph-rbd-pod-pvc-sc-allinone.yamlkubectl get pv,pvc,sc,podkubectl exec pod/ceph-rbd-pod-pvc-sc -- df -k | grep rbd

RKE2 Server/Master Provisioning#

sudo sucurl -sfL https://get.rke2.io | sh -systemctl enable --now rke2-server
cat \> /root/.bashrc \<\< EOFexport PATH=$PATH:/var/lib/rancher/rke2/bin/export KUBECONFIG=/etc/rancher/rke2/rke2.yamlEOF
cat /var/lib/rancher/rke2/server/node-tokentoken: K10ca0c38d4ff90d8b80319ab34092e315a8b732622e6adf97bc9eb0536REDACTED::server:ec0308000b8a6b595da000efREDACTED

RKE2 Agent/Worker provisioning#

mkdir -p /etc/rancher/rke2/
cat \> /etc/rancher/rke2/config.yaml \<\< EOFserver: https://10.100.103.23:9345token: K10ca0c38d4ff90d8b80319ab34092e315a8b732622e6adf97bc9eb0536REDACTED::server:ec0308000b8a6b595da000efREDACTEDEOF
curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="agent" sh -systemctl enable --now rke2-agent.service

To import the cluster into Rancher, click ☰ > Cluster Management. Then on the Clusters page, click Import Existing. Then run the provided kubectl command on the server/master node.

Tested Versions

OS for running RKE2 nodes: JeOS SLE15-SP2 with installed kernel-default-5.3.18-24.49

kubectl versionClient Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.4", GitCommit:"c96aede7b5205121079932896c4ad89bb93260af", GitTreeState:"clean", BuildDate:"2020-06-22T12:00:00Z", GoVersion:"go1.13.11", Compiler:"gc", Platform:"linux/amd64"}Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7+rke2r1", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-20T01:50:52Z", GoVersion:"go1.15.5b5", Compiler:"gc", Platform:"linux/amd64"}
helm versionversion.BuildInfo{Version:"3.4.1", GitCommit:"c4e74854886b2efe3321e185578e6db9be0a6e29", GitTreeState:"clean", GoVersion:"go1.14.12"}

Kubernetes version on RKE2 cluster: v1.19.7+rke2r1

Troubleshooting

In case you are using SUSE's ceph-rook based on SES7, it might be useful to expose the monitors on hostNetwork by editing rook-1.4.5/ceph/cluster.yaml and setting spec.network.hostNetwork=true.

Also for operating the ceph-rook cluster, it is useful to deploy a toolbox on the Kubernetes cluster where ceph-rook is provisioned by kubectl apply -f rook-1.4.5/ceph/toolbox.yaml Then all the ceph related commands can be executed in the toolbox pod, for example, by running kubectl exec -it -n rook-ceph rook-ceph-tools-686d8b8bfb-2nvqp -- bash

Operating with the ceph - basic commands:

ceph osd pool statsceph osd pool delete myPool myPool --yes-i-really-really-mean-itrbd list -p myPool\> csi-vol-f5d3766c-7296-11eb-b32a-c2b045952d38\> image

Delete the image: rbd rm csi-vol-f5d3766c-7296-11eb-b32a-c2b045952d38 -p myPool

CephFS commands in rook toolbox:

ceph -sceph fs lsceph fs fail cephfsceph fs rm cephfs --yes-i-really-mean-itceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-itceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it

To prepare a cephfs filesystem, you can run this command on a rook cluster:

kubectl apply -f rook-1.4.5/ceph/filesystem.yaml