Certain services within the Broadleaf ecosystem utilize the concept of a cluster singleton pattern to ensure that you have exactly one instance of a particular node in a cluster responsible for executing a specific piece of functionality.
Some examples that employ this pattern include:
Broadleaf’s Messaging Resiliency: See the Durable Send characteristics that are included with Broadleaf’s Messaging Components for more details.
Scheduled Job Co-ordination: To ensure a trigger event is only emitted once, the Scheduled Job Service should be configured to operate as a singleton in a cluster. See the Clustering Components section of the Scheduled Job Service for more details.
In order to accomplish this cluster singleton pattern, Broadleaf employs the use of Apache’s CamelClusterService allowing you to configure a number of backing Cluster SPI implementations out-of-the-box (e.g. Kubernetes, Zookeeper, File-based, etc…)
Broadleaf provides example configuration and support for the following implementations:
org.apache.camel.component.file.cluster.FileLockClusterService
org.apache.camel.component.kubernetes.cluster.KubernetesClusterService
See the following classes for more details:
The manifest file can now be used to establish your desired cluster service implementation type, per environment if desired.
schema:
version: '1.0'
project:
groupId: com.example.microservices
packageName: com.example.microservices
starterParentVersion: 2.1.0-GA
version: 1.0.0-SNAPSHOT
clusterServiceType: (1)
local: file
docker: file
cloud: kubernetesThe clusterServiceType elements allows you to specify the implementation type for any or all of local, docker, and cloud environments. Valid values for each environment are file, zookeeper, and kubernetes.
ClusterServiceType may be omitted altogether, or any environment component may be omitted. Any omission results in the use of the default file type (unless otherwise declared via environment variable - see below).
zookeeper is new as of 2.0.2-GA, 2.1.0-GA. When this type is used in the manifest, the manifest lifecycle (e.g. mvn flex:generate) will also maintain any appropriate zk ACL credentials needed by generated flexpackages to allow those runtimes to successfully authenticate to the secured zookeeper instance. The relevant credentials are automatically added to the secure yaml files sourced for the config server. This maintenance is a convenience of using the manifest-based starter, but is not required.
If not using a manifest lifecycle, Zookeeper (and friends) can be manually specified using the explicit approaches defined below as well.
Broadleaf services by default have the FileLockClusterService implementation configured. This is suitable for LOCAL development outside the use of Kubernetes.
When this implementation is enabled, you will see INFO logs emitted similar to below:
o.a.c.c.f.cluster.FileLockClusterView : Lock on file /var/folders/.../.../T/transition-request-supporting_REBASE acquired (lock=sun.nio.ch.FileLockImpl[0:1 exclusive valid])
Lock on file /var/folders/.../.../T/single-index-request-supporting_SINGLE_INDEX_REQUEST acquired (lock=sun.nio.ch.FileLockImpl[0:1 exclusive valid])
Lock on file /var/folders/.../.../T/transition-request-supporting_DEPLOY acquired (lock=sun.nio.ch.FileLockImpl[0:1 exclusive valid])
...You may also otice the following WARNING emitted to your logs if the system is still configured to use file on a non-local environment:
***** WARNING - broadleaf.messaging.cluster-service-implementation-type is set to `file`. This prevents cluster detection beyond a single node, which is only appropriate for local development. This value should be changed to `kubernetes` or `zookeeper` (usually `kubernetes` via manifest environment variable when deploying to k8s). *****|
Important
|
As of Broadleaf 1.5.x, we have deprecated the use of org.apache.camel.component.jgroups.cluster.JGroupsLockClusterService as the default (in favor of the FileLockClusterService). While JGroups might be an adequate solution for some setups, there is some significant configuration that would need to be done to fully support it. JGroups utilizes UDP Multi-casting to achieve communication between the nodes. This mechanism is not fully supported in a lot of cloud environments without additional networking configuration and setup. Instead, we recommend configuring the KubernetesClusterService or the ZooKeeperClusterService for non-local deployments
|
As we provide a lot of support for deploying the Microservices stack to a Kubernetes cluster, it makes sense to utilize Kubernetes’s built in APIs to co-ordinate leader/member setups within replicas in a cluster using Apache’s default org.apache.camel.component.kubernetes.cluster.KubernetesClusterService.
In order to tell Broadleaf to change from a File-based implementation to a Kubernetes-based implementation, you can pass in the following ENV overrides:
BROADLEAF_MESSAGING_CLUSTERSERVICEIMPLEMENTATIONTYPE: kubernetes
BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE: <unique name for each deployment> (e.g. auth, browse, cart, processing, or supporting if deploying the Balanced Flex Package Composition or cart, catalog, campaign, etc… if deploying the Granular Composition)
this property broadleaf.messaging.clusterK8sAppLabelValueworks in conjunction withbroadleaf.messaging.clusterK8sAppLabelwhich is defined here - the default value isk8s-app. This is the name of the pod label used to group common service instances allowing the system to determine which pods belong to which "group" (e.g. pod-1 and pod-2 have the Labels:k8s-app=cartwhereas pod-3 and pod-4 have the Labels:k8s-app=browse). Broadleaf’s Helm Charts will by default create pods with the appropriate labels and values depending on deployed Flex Package. If you are not using Broadleaf’s Helm Charts, you must create pods with the appropriate labels (i.e.broadleaf.messaging.clusterK8sAppLabel) and values (i.e.broadleaf.messaging.clusterK8sAppLabelValue) manually yourself in order for theKubernetesClusterServiceto determine leadership properly.
LOGGING_LEVEL_ORG_APACHE_CAMEL_COMPONENT_KUBERNETES_CLUSTER_LOCK: DEBUG (optional, but helpful to set the log level here to DEBUG initially to diagnose any connectivity issues)
To get this to work on your Kubernetes Cluster, you need to have a service account that specifically has access to the Kubernetes Lease access.
This service account will need permissions to read and write to the lease objects by creating ClusterRole and ClusterRoleBinding 's:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: lease-access
rules:
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- get
- create
- update
- list
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- listapiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: lease-access
roleRef:
kind: ClusterRole
name: lease-access
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: {{ .Values.serviceAccount.name }}
namespace: {{ .Values.serviceAccount.namespace }}Our example Helm chart accelerators (i.e. devops-helm-charts) contains reference to a base chart that has the above resources defined (cluster-singleton). This is referenced in the blc-initialization umbrella chart. See the file blc-initialization/Chart.yaml for more details:
dependencies:
...
- name: cluster-singleton
version: [MY VERSION]
repository: https://repository.broadleafcommerce.com/repository/helm-private
the default values assume deployment to the defaultkubernetes namespace and use of thedefaultservice account. You can override these when installing the chart for your purposes.
Our example Helm chart accelerators (i.e. devops-helm-charts) showcase enabling the kubernetes camel configuration already defined for a Balanced deployment. See the file blc-environments/cloudexample/blc-balanced-flex-demo-cloudexample.yaml for more details.
flex-balanced-browse:
...
extraEnvs:
- name: BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE
value: browse
flex-balanced-cart:
...
extraEnvs:
- name: BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE
value: cart
flex-balanced-common-env:
data:
...
BROADLEAF_MESSAGING_CLUSTERSERVICEIMPLEMENTATIONTYPE: kubernetes
LOGGING_LEVEL_ORG_APACHE_CAMEL_COMPONENT_KUBERNETES_CLUSTER_LOCK: DEBUG
flex-balanced-processing:
...
extraEnvs:
- name: BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE
value: processing
flex-balanced-supporting:
...
extraEnvs:
- name: BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE
value: supportingAnd the same for the auth service. See blc-environments/cloudexample/blc-auth-demo-cloudexample.yaml for more details.
auth:
...
extraEnvs:
...
- name: BROADLEAF_MESSAGING_CLUSTERSERVICEIMPLEMENTATIONTYPE
value: kubernetes
- name: BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE
value: auth
- name: LOGGING_LEVEL_ORG_APACHE_CAMEL_COMPONENT_KUBERNETES_CLUSTER_LOCK
value: DEBUGWith all the configuration in place, you can run the following command against your Kubernetes cluster to verify that leases have been provisioned as expected: kubectl get leases
NAME HOLDER AGE
leaders-batch-index-request-processing-batch-index-request processing-5b4d98c557-428vh 136m
leaders-batch-index-request-supporting-batch-index-request supporting-67f8946db9-mzx9s 138m
leaders-cartnotification-cart-cart-pending-payment-failed-event cart-85bb888d-vzj6f 138m
leaders-cartnotification-cart-cart-rejection-event cart-85bb888d-vzj6f 138m
leaders-cartnotification-cart-checkout-completion cart-85bb888d-vzj6f 138m
leaders-cartnotification-cart-checkout-rollback-event cart-85bb888d-vzj6f 138m
leaders-changesummary-supporting-change supporting-67f8946db9-mzx9s 138m
leaders-completion-supporting-transition-completion supporting-67f8946db9-mzx9s 138m
leaders-import-batch-request-processing-batch-request processing-5b4d98c557-428vh 136m
leaders-import-delete-sandbox-processing-import-delete-sandbox-request processing-5b4d98c557-428vh 136m
leaders-import-process-request-processing-process-import-request processing-5b4d98c557-428vh 136m
...
if you have a rather large topology (e.g. deploying multiple replicas of a full granulardeployment) and you are seeing substantial CPU usage on the Kubernetes API Master Nodes/Control Plane, you may find it beneficial to tune the Apache Camel Kubernetes Config Options, such as therenew-deadline,jitter-factor, andlease-duration.
Using Zookeeper as a cluster service lock source can be interesting for implementations not using kubernetes, or for expansive granular deployments in which the kubernetes lease maintenance for a large number of cluster service locks may be onerous. Since Zookeeper is a first-class member of the reference stack, it is another option to consider.
Support for cluster service via zookeeper can be configured via environment variables alone.
| Property | Value | Default | Notes |
|---|---|---|---|
|
|
||
|
|
|
Whether the system should attempt authentication when connecting to zookeeper |
|
|
The valid username for zookeeper auth |
|
|
The valid password for zookeeper auth. This should be supplied via a secure mechanism. |
Executing the 4-letter command dump against the zookeeper admin should generally be enough to confirm zookeeper is being successfully used for cluster service locking via the /camel root. If you have the default configuration for zookeeper with the admin running in the container, you can usually create a terminal to the container and check from there. For example, after establishing a terminal session, curl http(s)//localhost:[8080]/commands/dump. You will be looking for the presence of one or more sessions starting at the /camel root.
Underlying Kubernetes Client used by Camel Kubernetes: Fabric.io Kubernetes Client (This is a good resoure to validate Kubernetes Cluster Version compatibility)