# Rook

## 官方网站

{% embed url="<https://rook.io/>" %}

## 项目地址

{% embed url="<https://github.com/rook/rook>" %}

## 快速上手

### 安装集群

准备osd存储介质

| 硬盘符号 | 大小   | 作用           |
| ---- | ---- | ------------ |
| sdb  | 50GB | OSD Data     |
| sdc  | 50GB | OSD Data     |
| sdd  | 50GB | OSD Data     |
| sde  | 50GB | OSD Metadata |

{% hint style="info" %}
安装前使用命令`lvm lvs`,`lvm vgs`和`lvm pvs`检查上述硬盘是否已经被使用，若已经使用需要删除，且确保硬盘上不存在分区和文件系统
{% endhint %}

确保开启内核rbd模块并安装lvm2

```
modprobe rbd
yum install -y lvm2
```

安装operator

```
git clone --single-branch --branch release-1.2 https://github.com/rook/rook.git
cd rook/cluster/examples/kubernetes/ceph
kubectl create -f common.yaml
kubectl create -f operator.yaml
```

安装ceph集群

```
---
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
  name: rook-ceph
  namespace: rook-ceph
spec:
  cephVersion:
    image: ceph/ceph:v14.2.5
    allowUnsupported: false
  dataDirHostPath: /var/lib/rook
  skipUpgradeChecks: false
  mon:
    count: 3
    allowMultiplePerNode: true
  mgr:
    modules:
    - name: pg_autoscaler
      enabled: true
  dashboard:
    enabled: true
    ssl: true
  monitoring:
    enabled: false
    rulesNamespace: rook-ceph
  network:
    hostNetwork: false
  rbdMirroring:
    workers: 0
  annotations:
  resources:
  removeOSDsIfOutAndSafeToRemove: false
    useAllNodes: false
    useAllDevices: false
    config:
    nodes:
    - name: "minikube"
      devices:
      - name: "sdb"
      - name: "sdc"
      - name: "sdd"
      config:
        storeType: bluestore
        metadataDevice: "sde"
        databaseSizeMB: "1024"
        journalSizeMB: "1024"
        osdsPerDevice: "1"
  disruptionManagement:
    managePodBudgets: false
    osdMaintenanceTimeout: 30
    manageMachineDisruptionBudgets: false
    machineDisruptionBudgetNamespace: openshift-machine-api
```

安装命令行工具

```
kubectl create -f toolbox.yaml
```

在toolbox中使用命令`ceph -s`查看集群状态

{% hint style="info" %}
在重装ceph集群时需要清理rook数据目录（默认：/var/lib/rook）
{% endhint %}

为ceph-dashboard服务添加ingress路由

```
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: rook-ceph-mgr-dashboard
  namespace: rook-ceph
  annotations:
    kubernetes.io/ingress.class: "nginx"
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/server-snippet: |
      proxy_ssl_verify off;
spec:
  tls:
   - hosts:
     - rook-ceph.minikube.local
     secretName: rook-ceph.minikube.local
  rules:
  - host: rook-ceph.minikube.local
    http:
      paths:
      - path: /
        backend:
          serviceName: rook-ceph-mgr-dashboard
          servicePort: https-dashboard
```

获取访问dashboard所需的admin账号密码

```
kubectl get secret rook-ceph-dashboard-password -n rook-ceph -o jsonpath='{.data.password}'|base64 -d
```

将域名rook-ceph.minikube.local加入/etc/hosts后通过浏览器访问

{% embed url="<https://rook-ceph.minikube.local/>" %}

![](/files/-Ly3NdwehTaActYg9AOa)

### 使用rbd存储

创建rbd存储池

```
---
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  failureDomain: osd
  replicated:
    size: 3
```

{% hint style="info" %}
由于仅有一个节点和三个OSD，因此采用osd作为故障域
{% endhint %}

创建完成后在rook-ceph-tools中使用指令`ceph osd pool ls`可以看到新建了以下存储池

* replicapool

以rbd为存储介质创建storageclass

```
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
  clusterID: rook-ceph
  pool: replicapool
  imageFormat: "2"
  imageFeatures: layering
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
  csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
```

使用statefulset测试通过storageclass挂载rbd存储

```
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: storageclass-rbd-test
  namespace: default
  labels:
    app: storageclass-rbd-test
spec:
  replicas: 2
  selector:
    matchLabels:
      app: storageclass-rbd-test
  template:
    metadata:
      labels:
        app: storageclass-rbd-test
    spec:
      restartPolicy: Always
      containers:
        - name: storageclass-rbd-test
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: data
              mountPath: /data
          image: 'centos:7'
          args:
            - 'sh'
            - '-c'
            - 'sleep 3600'
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: rook-ceph-block
```

### 使用cephfs存储

创建mds服务与cephfs文件系统

```
---
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: myfs
  namespace: rook-ceph
spec:
  metadataPool:
    failureDomain: osd
    replicated:
      size: 3
  dataPools:
    - failureDomain: osd
      replicated:
        size: 3
  preservePoolsOnDelete: true
  metadataServer:
    activeCount: 1
    activeStandby: true
    placement:
    annotations:
    resources:
```

创建完成后在rook-ceph-tools中使用指令`ceph osd pool ls`可以看到新建了以下存储池

* myfs-metadata
* myfs-data0

以cephfs为存储介质创建storageclass

```
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-cephfs
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
  clusterID: rook-ceph
  fsName: myfs
  pool: myfs-data0
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
reclaimPolicy: Delete
mountOptions:
```

使用deployment测试通过storageclass挂载cephfs共享存储

```
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: data-storageclass-cephfs-test
  namespace: default
  labels:
    app: storageclass-cephfs-test
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-cephfs
  volumeMode: Filesystem
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: storageclass-cephfs-test
  namespace: default
  labels:
    app: storageclass-cephfs-test
spec:
  replicas: 2
  selector:
    matchLabels:
      app: storageclass-cephfs-test
  template:
    metadata:
      labels:
        app: storageclass-cephfs-test
    spec:
      restartPolicy: Always
      containers:
        - name: storageclass-cephfs-test
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: data
              mountPath: /data
          image: 'centos:7'
          args:
            - 'sh'
            - '-c'
            - 'sleep 3600'
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: data-storageclass-cephfs-test
```

### 使用s3存储

创建对象存储网关

```
---
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: my-store
  namespace: rook-ceph
spec:
  metadataPool:
    failureDomain: osd
    replicated:
      size: 3
  dataPool:
    failureDomain: osd
    replicated:
      size: 3
  preservePoolsOnDelete: false
  gateway:
    type: s3
    sslCertificateRef:
    port: 80
    securePort:
    instances: 1
    placement:
    annotations:
    resources:
```

创建完成后在rook-ceph-tools中使用指令`ceph osd pool ls`可以看到新建了以下存储池

* .rgw\.root
* my-store.rgw\.buckets.data
* my-store.rgw\.buckets.index
* my-store.rgw\.buckets.non-ec
* my-store.rgw\.control
* my-store.rgw\.log
* my-store.rgw\.meta

为ceph-rgw服务添加ingress路由

```
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: rook-ceph-rgw
  namespace: rook-ceph
  annotations:
    kubernetes.io/ingress.class: "nginx"
    kubernetes.io/tls-acme: "true"
spec:
  tls:
   - hosts:
     - rook-ceph-rgw.minikube.local
     secretName: rook-ceph-rgw.minikube.local
  rules:
  - host: rook-ceph-rgw.minikube.local
    http:
      paths:
      - path: /
        backend:
          serviceName: rook-ceph-rgw-my-store
          servicePort: http
```

将域名rook-ceph-rgw\.minikube.local加入/etc/hosts后通过浏览器访问

{% embed url="<https://rook-ceph-rgw.minikube.local/>" %}

![](/files/-Ly3Ocb5Ey11QzZ8TU5A)

#### 使用S3用户

添加对象存储用户

```
---
apiVersion: ceph.rook.io/v1
kind: CephObjectStoreUser
metadata:
  name: my-user
  namespace: rook-ceph
spec:
  store: my-store
  displayName: "my display name"
```

创建对象存储用户的同时会生成以`{{.metadata.namespace}}-object-user-{{.spec.store}}-{{.metadata.name}}`为命名规则的secret，其中保存了该S3用户的AccessKey和SecretKey

获取AccessKey

```
kubectl get secret rook-ceph-object-user-my-store-my-user -n rook-ceph -o jsonpath='{.data.AccessKey}'|base64 -d
```

获取SecretKey

```
kubectl get secret rook-ceph-object-user-my-store-my-user -n rook-ceph -o jsonpath='{.data.SecretKey}'|base64 -d
```

根据上述步骤获取到的信息，使用S3客户端进行连接即可使用该S3用户

![](/files/-Lxz_0uISBvB_WY4JLl0)

#### 使用S3存储桶

创建以s3为存储的storageclass

```
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rook-ceph-delete-bucket
provisioner: ceph.rook.io/bucket
reclaimPolicy: Delete
parameters:
  objectStoreName: my-store
  objectStoreNamespace: rook-ceph
  region: default
```

{% hint style="info" %}
目前不支持以s3存储创建pvc，仅可用于创建存储桶
{% endhint %}

为storageclass创建对应的存储桶资源申请

```
apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: ceph-delete-bucket
spec:
  generateBucketName: ceph-bkt
  storageClassName: rook-ceph-delete-bucket
```

存储桶创建后会生成与桶资源申请同名的secret，其中保存着用于连接该存储桶的AccessKey和SecretKey

获取AccessKey

```
kubectl get secret ceph-delete-bucket -n rook-ceph -o jsonpath='{.data.AWS_ACCESS_KEY_ID}'|base64 -d
```

获取SecretKey

```
kubectl get secret ceph-delete-bucket -n rook-ceph -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}'|base64 -d
```

{% hint style="info" %}
使用该方式获取的s3用户已经做了配额限制只能使用一个存储桶
{% endhint %}


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://viva.gitbook.io/project/cloud-native/rook.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.