When the piano was invented, musicians noticed that it had become much easier to produce a sound with the instrument. But pianists were expected to play many more sounds than a classic instrument. Complexity was just moving to another place.
Kubernetes makes application deployment easier, yet application features require an increasing number of components and requirements. A good example of this is the IBM blockchain network, in which the trust on the transaction can be distributed between organizations.
Four organizations — R1, R2, R3 and R4 — have jointly decided and entered an agreement to set up and exploit a Hyperledger Fabric network.
Without entering all the details, this application is complex to backup and restore. It creates :
- A least 11 PVCs that must be backed up and restored in the right order.
- More than 12 deployments if we include the client application.
- Multiple secrets, configmaps, service accounts, services and routes.
- Multiple IBP custom resources managed by an operator that creates the deployment and PVCs.
IBM features an IBP console that helps with deploying peers, oredererers and ca nodes on the network. It also helps to manage identities and channels, and visualize the transaction on the ledger. This UI is very handy, but constantly creates new PVCs, deployments and route services as the network grows, all of which must be backed up regularly. Additionally, the operator must also be managed by the Lifecycle Operator on OpenShift, which creates even more complexity.
Without a tool such as Kasten K10 by Veeam to protect and restore such applications, backup and restore processes are challenging. In this post, we’ll demonstrate the power of Kasten K10 using this scenario as an example.
Assumptions
In this tutorial, we assume that you have installed Kasten on OpenShift. (If that’s not the case, read this blog post.)
My base domain on OpenShift will be michael-1.aws.kasten.io.
We will restore the application without the Operator Lifecycle Manager, to demonstrate restoration is possible on a cluster that does not exist on vanilla Kubernetes.
Step 1: Create a Blockchain Network
The first step is to create the blockchain namespace:
oc new-project my-blockchain Next, we install the blockchain operator, browse to the Red Hat Marketplace and log in or create a new account. Then, we will register the cluster to the Red Hat marketplace.
Now, we’ll go to the Operator Hub. In the search bar, type “blockchain” to load the blockchain tile.
Step 2: Apply the Security Context Constraint
The next step is to copy and save the following security context constraint object to the local system as ibp-scc.yaml:
cat <<EOF | oc apply -f -
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: true
allowPrivilegedContainer: true
allowedCapabilities:
- NET_BIND_SERVICE
- CHOWN
- DAC_OVERRIDE
- SETGID
- SETUID
- FOWNER
apiVersion: security.openshift.io/v1
defaultAddCapabilities: []
fsGroup:
type: RunAsAny
groups:
- system:serviceaccounts:my-blockchain
kind: SecurityContextConstraints
metadata:
name: my-blockchain
readOnlyRootFilesystem: false
requiredDropCapabilities: []
runAsUser:
type: RunAsAny
seLinuxContext:
type: RunAsAny
supplementalGroups:
type: RunAsAny
volumes:
- "*"
EOF Then, we will run the following commands to add the file to the cluster, and add the constraint to the project:
oc adm policy add-scc-to-user my-blockchain system:serviceaccounts:my-blockchain When the command is successful, we will see a response that is similar to the following example:
securitycontextconstraints.security.openshift.io/my-blockchain created
scc "blockchain-project" added to: ["system:serviceaccounts:my-blockchain"] Step 3: Deploy the IBM Blockchain Platform console
There are four instances available listed under “Provided APIs”:
- IBP CA (Advanced users): Deploys an instance of an IBM Blockchain Platform CA.
- IBP Console: The IBM Blockchain Platform console UI, or “console”, is an award-winning user interface for building your blockchain network.
- IBP Orderer (Advanced users): Deploys an instance of an IBM Blockchain Platform ordering service.
- IBP Peer (Advanced users): Deploys an instance of an IBM Blockchain Platform peer.
Step 4: Create the Instance on the IBPConsole Tile
apiVersion: ibp.com/v1beta1
kind: IBPConsole
metadata:
name: ibpconsole
namespace: my-blockchain
labels:
app.kubernetes.io/name: "ibp"
app.kubernetes.io/instance: "ibp"
app.kubernetes.io/managed-by: "ibm-ibp"
spec:
email: michael@kasten.io
password: ultrasecurepassword
imagePullSecrets:
- regcred
registryURL: cp.icr.io/cp
license:
accept: true
networkinfo:
domain: apps.michael-1.aws.kasten.io
storage:
console:
class: ''
size: 5Gi
serviceAccountName: ibm-blockchain
version: 2.5.1 See https://cloud.ibm.com/docs/blockchain-sw-251?topic=blockchain-sw-251-deploy-ocp-rhm#console-deploy-ocp-rhm-advanced for more advanced options.
Step 5: Verify the Console Installation and Log In
oc get deployment -n my-blockchain
NAME READY UP-TO-DATE AVAILABLE AGE
ibp-operator 1/1 1 1 34m
ibpconsole 1/1 1 1 16m All deployment are in the ready state:
oc get po
NAME READY STATUS RESTARTS AGE
ibp-operator-7f896d6644-sh2mx 1/1 Running 0 36m
ibpconsole-6f94ddc6f9-t9khx 4/4 Running 0 18moc get route
NAME HOST/PORT
PATH SERVICES PORT TERMINATION WILDCARD
ibpconsole-console
my-blockchain-ibpconsole-console.apps.michael-1.aws.kasten.io
ibpconsole optools passthrough None
ibpconsole-proxy
my-blockchain-ibpconsole-proxy.apps.michael-1.aws.kasten.io
ibpconsole optools passthrough None We connect to https://my-blockchain-ibpconsole-console.apps.michael-1.aws.kasten.io
and use our email and password provided in the ibpconsole michael@kasten.io and ultrasecurepassword. We then change the password as requested by the UI and reconnect:
Step 6: Create a Blockchain Network
We will create a minimal network for testing purposes following this tutorial.
At this point, we can run backup and recovery on the first two blocks:
But it’s more interesting to have transactions to make sure we are able to retrieve them upon restore.
We can use this tutorial to create a nodejs basic smartcontract
Here’s a summary of the steps:
- First we install the VS Code extension for the IBM Blockchain Platform.
- Next, we follow the basic tutorial in VS Code to create a smartcontract in nodejs call js-contract.
- Then we export the contract in .cds format, which is compatible with the 1.4-compatible channel created in the previous steps.
- Finally, we install and instantiate the js-contract on channel 1.
All these operations create some transactions on the ledger that we can use now to test the restoration process:
It could also be interesting to connect from VS Code to the platform and create some transactions, but this is beyond the scope of this blog post.
What’s Available for Backup and Restore?
At this point, we have these pods:
oc get po
NAME READY STATUS RESTARTS AGE
chaincode-execution-8d91fc09-df12-4c91-8b1b-6239b7947e23 1/1 Running 0 64mibp-operator-7f896d6644-sh2mx 1/1 Running 0 22h
ibpconsole-6f94ddc6f9-t9khx 4/4 Running 0 22h
orderingserviceca-b64685fc9-2q6dl 1/1 Running 0 15h
orderingservicenode1-f7bb5f9f-2w29b 2/2 Running 0 14h
org1ca-7fd68c8cc-ggvpq 1/1 Running 0 16h
peerorg1-58b48f49f9-mwp2j 4/4 Running 0 15h We also have these deployments:
oc get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
ibp-operator 1/1 1 1 22h
ibpconsole 1/1 1 1 22h
orderingserviceca 1/1 1 1 15h
orderingservicenode1 1/1 1 1 14h
org1ca 1/1 1 1 16h
peerorg1 1/1 1 1 15h Note that the chaincode-execution pod is not linked to any deployment.
Here are the PVCs…
oc get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ibpconsole-pvc Bound pvc-07638d8f-1e9d-4fb8-a4d7-5dc3a7a05840 5Gi RWO gp2-csi 22h
orderingserviceca-pvc Bound pvc-b71c1051-d538-4eff-9e0f-937497125dec 20Gi RWO gp2-csi 15h
orderingservicenode1-pvc Bound pvc-3748a0cd-995e-4701-bbb9-484634345b5f 100Gi RWO gp2-csi 14h
org1ca-pvc Bound pvc-d46a10b3-f9d7-4bbb-8a71-d4574aa05baa 20Gi RWO gp2-csi 16h
peerorg1-pvc Bound pvc-e5575337-064f-4e0c-8e8f-1745655b8529 100Gi RWO gp2-csi 15h
peerorg1-statedb-pvc Bound pvc-e18e0783-8608-4668-9640-7f411726c6bd 10Gi RWO gp2-csi 15h …and the corresponding CRD mapping to ibp objects:
oc get ibpca,ibpconsole,ibporderer,ibppeer
NAME READY
ibpca.ibp.com/orderingserviceca
ibpca.ibp.com/org1ca
NAME READY
ibpconsole.ibp.com/ibpconsole
NAME READY
ibporderer.ibp.com/orderingservice
ibporderer.ibp.com/orderingservicenode1
NAME READY
ibppeer.ibp.com/peerorg1 Here are the Operator Lyfecycle Manager elements provided to install the global blockchain application:
oc get operatorgroup,subscription,csv
NAME AGE
operatorgroup.operators.coreos.com/my-blockchain-jh4jr 23h
NAME PACKAGE SOURCE CHANNEL
subscription.operators.coreos.com/ibm-blockchain ibm-blockchain ibm-operator-catalog v2.5
NAME DISPLAY VERSION REPLACES PHASE
clusterserviceversion.operators.coreos.com/ibm-blockchain.v2.5.1 IBM Blockchain 2.5.1 Succeeded The Operator Lifecycle Manager created the clusterrole and clusterrolebinding during install, to let the ibp-operator manage pods, deployment and IBP custom resources. These need to be captured at cluster resource scope level:
oc get clusterrole | grep ibp
ibpcas.ibp.com-v1beta1-admin 2021-03-29T14:30:16Z
ibpcas.ibp.com-v1beta1-crdview 2021-03-29T14:30:17Z
ibpcas.ibp.com-v1beta1-edit 2021-03-29T14:30:16Z
ibpcas.ibp.com-v1beta1-view 2021-03-29T14:30:16Z
ibpconsoles.ibp.com-v1beta1-admin 2021-03-29T14:30:17Z
ibpconsoles.ibp.com-v1beta1-crdview 2021-03-29T14:30:17Z
ibpconsoles.ibp.com-v1beta1-edit 2021-03-29T14:30:17Z
ibpconsoles.ibp.com-v1beta1-view 2021-03-29T14:30:17Z
ibporderers.ibp.com-v1beta1-admin 2021-03-29T14:30:17Z
ibporderers.ibp.com-v1beta1-crdview 2021-03-29T14:30:17Z
ibporderers.ibp.com-v1beta1-edit 2021-03-29T14:30:17Z
ibporderers.ibp.com-v1beta1-view 2021-03-29T14:30:17Z
ibppeers.ibp.com-v1beta1-admin 2021-03-29T14:30:17Z
ibppeers.ibp.com-v1beta1-crdview 2021-03-29T14:30:17Z
ibppeers.ibp.com-v1beta1-edit 2021-03-29T14:30:17Z
ibppeers.ibp.com-v1beta1-view 2021-03-29T14:30:17Z
oc get clusterrole | grep ibm
ibm-blockchain.v2.5.2-6bdc5f6d8
oc get clusterrolebinding | grep ibm
ibm-blockchain.v2.5.2-6bdc5f6d8 Restoration Constraints
- Topology constraint
To accomplish high resiliency, the blockchain operator decides on a very precise topology breakdown of the pods around zones. When needed, the operator can create a zone label on the PVC that corresponds to the zone of the actual PVC:
oc get pvc -L zone
NAME STATUS VOLUME ZONE
ibpconsole-pvc Bound pvc-a08bd96f-bc41-432f-ac96-26c4850b81ce
orderingserviceca-pvc Bound pvc-f3ca0d28-012a-4a5a-a793-474045bac3ce eu-west-1a
orderingservicenode1-pvc Bound pvc-161921ca-3fcf-490a-b49b-5ab88ee6bc41
org1ca-pvc Bound pvc-308f93b1-eb45-439a-8808-1e3d79550b46 eu-west-1a
peerorg1-pvc Bound pvc-d0318054-af5f-44a4-b49c-7f1cd5d4e56e eu-west-1b
peerorg1-statedb-pvc Bound pvc-2ac27a9c-a571-488a-a28e-d38f90054a42 eu-west-1b To reinforce this, an affinity rule is also set up on the deployment itself:
oc get deploy peerorg1 -o jsonpath='{.spec.template.spec.affinity}' | jq
{
"nodeAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": {
"nodeSelectorTerms": [
{
"matchExpressions": [
{
"key": "topology.kubernetes.io/zone",
"operator": "In",
"values": [
"eu-west-1b"
]
},
{
"key": "topology.kubernetes.io/region",
"operator": "In",
"values": [
"eu-west-1"
]
}
]
},
{
"matchExpressions": [
{
"key": "failure-domain.beta.kubernetes.io/zone",
"operator": "In",
"values": [
"eu-west-1b"
]
},
{
"key": "failure-domain.beta.kubernetes.io/region",
"operator": "In",
"values": [
"eu-west-1"
]
}
]
}
]
}
},
"podAntiAffinity": {
"preferredDuringSchedulingIgnoredDuringExecution": [
{
"podAffinityTerm": {
"labelSelector": {
"matchExpressions": [
{
"key": "orgname",
"operator": "In",
"values": [
"org1msp"
]
}
]
},
"topologyKey": "kubernetes.io/hostname"
},
"weight": 100
}
]
}
} That means that when restoring the PVI, we must make sure we restore it to the right zone.
- Data Consistency Constraint
To understand the data consistency constraint, we must first understand the transaction flow:
Every peer pod holds 2 PVCs:
- The ledger PVC (100Gi), or the materialised blockchain, made of block and transactions inside the blocks that have been validated.
- The couchdb PVC (10Gi) that represents the state of the ledger with not yet validated transactions. There are also some transactions validated, but not yet committed to the ledger.
It’s important to make sure that the couchdb PVC does not have validated transactions that the ledger does not have. IBM recommends taking a snapshot of the couchdb PVC at 3:00 a.m. and a snapshot of the ledger PVC at 3:05 a.m.
Every orderer pod has a ledger PVC, but contrary to a public blockchain that uses proof of work to validate the blockchain, private blockchains use orderers to order the validated block.
We must be sure that peers do not have validated transactions that the orderer does not have. This time, we are not speaking about consistency inside a pod, but consistency inside a wide network. IBM recommends taking a snapshot of the PVC orderer at 5:00 a.m. (2 hours after the snapshot of the peers).
There are also CA pods and an IBP Console pod with a single PVC. These should be backed up each time the network topology changes. To make it simpler, let’s back them up at 5:00 a.m. with the orderer.
Here’s a summary of the scheduled backups:
|
PEER COUCHDB PVC |
3:00 A.M. |
|
Peer Ledger PVC |
3:05 a.m. |
|
Orderer Ledger PVC |
5:00 a.m. |
|
CA and IBPConsole PVC |
5:00 a.m. |
Implementing the Backup Strategy
We can implement the backup strategy easily by creating:
- One daily policy that backs up the entire namespace at 3:00 a.m. and 5:00 a.m.
- One daily policy that backs up the entire namespace at 3:05 a.m.
- One weekly cluster resource policy (to capture clusterrole and clusterrolebinding).
With this schedule, we have everything we need to restore the application consistently.
Restoring the Application After a Disaster
Let’s remove the my-blockchain application to emulate the disaster:
oc delete ns my-blockchain Step 1: Recreate the PVC from the Different Restore Points
First, we’ll recreate the empty namespace:
oc create ns my-blockchain We’ll use the partial restore capacity of Kasten K10 to pick up the different PVCs for the various restore points:
|
RESTOREPOINT |
PVC |
|
3:00 |
peerorg1-pvc |
|
3:05 |
peerorg1-statedb-pvc |
|
5:00 |
orderingservicenode1-pvc |
|
|
The console pvc and the rest of CA pvc ibpconsole-pvc orderingserviceca-pvc org1ca-pvc |
This image shows how to restore only the peerorg1-pvc from the 3:00 a.m. restore point:
Step 2: ReCreate the PVC in the Right Zone
Before launching the restore, we need to apply a transform to make sure the PVC is recreated in the right zone. When the blockchain operator defines precisely in which zone the PVC has been created, it add a “zone” label to the PVC and an affinity constraint to the deployment using this PVC:
oc get pvc peerorg1-pvc -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
labels:
...
zone: eu-west-1b
name: peerorg1-pvc
...oc get deploy peerorg1 -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
...
name: peerorg1
namespace: my-blockchain
….
spec:
....
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
- eu-west-1b
- key: topology.kubernetes.io/region
operator: In
values:
- eu-west-1
- matchExpressions:
- key: failure-domain.beta.kubernetes.io/zone
operator: In
values:
- eu-west-1b
- key: failure-domain.beta.kubernetes.io/region
operator: In
values:
- eu-west-1
...
We will use this label to recreate the PVC in the right zone. Then, we’ll create a storageclass per zone:
oc get sc eu-west-1a -o yaml
allowVolumeExpansion: true
allowedTopologies:
- matchLabelExpressions:
- key: topology.ebs.csi.aws.com/zone
values:
- eu-west-1a
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: eu-west-1a
parameters:
encrypted: "true"
type: gp2
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
... We’ll use a transform to copy the zone name in the storageClassName in the PVC spec:
Then, we’ll apply this transform for each of the three restore points.
Step 3: Restore the Rest of the namespace (with Some Exclusions)
Now we have the namespace with only the PVCs. We must bring back the rest of the namespace, but without:
- The PVCs that we already got back
- The chaincode pods (they we’ll be recreated at smartcontract invoke)
- The configmap ibp-operator-lock
- The elements of the Operator Lifecycle Manager:
- ClusterServiceVersion
- Subscription
- OperatorGroup
- InstallPlan
We also need to scale down all the deployments. This is because Kasten K10 will restore the CRD after all deployments are up and running. However, if those deployments rely on this CRD to work, it’s best to scale down everything so that Kasten K10 considers everything successful and moves to CRD restoration. We’ll scale up afterwards.
Step 4: Restore the clusterrole and clusterrolebinding from the Cluster Restore Point
When we deleted the namespace, the Operator Lifecycle Manager automatically deleted the clusterrole and clusterrolebinding created for the IBM Blockchain operator:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
creationTimestamp: "2021-03-31T23:09:32Z"
labels:
olm.owner: ibm-blockchain.v2.5.1
olm.owner.kind: ClusterServiceVersion
olm.owner.namespace: my-blockchain
operators.coreos.com/ibm-blockchain.my-blockchain: ""
name: ibm-blockchain.v2.5.1-6bdc5f6d8
rules:
- apiGroups:
- apiextensions.k8s.io
resources:
- persistentvolumeclaims
- persistentvolumes
verbs:
- '*'
... The olm.owner label has OLM to remove this object if the ClusterServiceVersion is removed, and we decide to restore without the OLM objects. To achieve our restore, we need to remove the labels section.
We retrieve the clusterrole and clusterrolebinding from the cluster restore point and apply a transform to remove the labels section:
If we were restoring in a new cluster, we would have to restore the other my-blockhain clusterrole and rolebinding created by the deployment of the operator by OLM, as well:
Restart the Deployment
We can now restart the deployment and check that everything is recovered and that all the pods are back:
for dep in $(oc get deploy -o name); do oc scale $dep --replicas=1; doneoc get pods
NAME READY STATUS RESTARTS AGE
ibp-operator-7dd4bfb76f-s89rx 1/1 Running 0 2d8h
ibpconsole-cbf76d57d-2xwx9 4/4 Running 0 2d8h
orderingserviceca-78fb95747c-dvk4k 1/1 Running 0 2d8h
orderingservicenode1-646dd846bc-94fwm 2/2 Running 0 2d8h
org1ca-75845db8bf-ts6tf 1/1 Running 0 2d8h
peerorg1-7758946784-kqcrp 4/4 Running 0 2d8h The best way to check that your data is recovered is to check the block transactions:
We can verify that we got back all the blocks and the transaction. We were also able to restart the whole blockchain application, within the data consistency and topology constraints.
Conclusion
With this article, we’ve shown how you can more easily perform a complex backup and restoration process with Kasten K10. Kasten has all the necessary features to make your backup and restore possible, as long as you understand how your application works.
Try Kasten K10 for free today.