Certain services within the Broadleaf ecosystem utilize the concept of a cluster singleton pattern to ensure that you have exactly one instance of a particular node in a cluster responsible for executing a specific piece of functionality.
Some examples that employ this pattern include:
Broadleaf’s Messaging Resiliency: See the Durable Send characteristics that are included with Broadleaf’s Messaging Components for more details.
Scheduled Job Co-ordination: To ensure a trigger event is only emitted once, the Scheduled Job Service should be configured to operate as a singleton in a cluster. See the Clustering Components section of the Scheduled Job Service for more details.
Camel Cluster Service
In order to accomplish this cluster singleton pattern, Broadleaf employs the use of Apache’s CamelClusterService allowing you to configure a number of backing Cluster SPI implementations out-of-the-box (e.g. Kubernetes, Zookeeper, File-based, etc…)
Broadleaf provides example configuration and support for the following implementations:
com.broadleafcommerce.common.messaging.environment.MessagingProperties which is documented here
com.broadleafcommerce.scheduledjob.service.autoconfigure.ClusterProperties which is documented here
Broadleaf services by default have the FileLockClusterService implementation configured. This is suitable for LOCAL development outside the use of Kubernetes.
When this implementation is enabled, you will see INFO logs emitted similar to below:
o.a.c.c.f.cluster.FileLockClusterView : Lock on file /var/folders/.../.../T/transition-request-supporting_REBASE acquired (lock=sun.nio.ch.FileLockImpl[0:1 exclusive valid])
Lock on file /var/folders/.../.../T/single-index-request-supporting_SINGLE_INDEX_REQUEST acquired (lock=sun.nio.ch.FileLockImpl[0:1 exclusive valid])
Lock on file /var/folders/.../.../T/transition-request-supporting_DEPLOY acquired (lock=sun.nio.ch.FileLockImpl[0:1 exclusive valid])
You may also otice the following WARNING emitted to your logs if the system is still configured to use file on a non-local environment:
***** WARNING - broadleaf.messaging.cluster-service-implementation-type is set to `file`. This prevents cluster detection beyond a single node, which is only appropriate for local development. This value should be changed to `kubernetes` or `zookeeper` (usually `kubernetes` via manifest environment variable when deploying to k8s). *****
As of Broadleaf 1.5.x, we have deprecated the use of org.apache.camel.component.jgroups.cluster.JGroupsLockClusterService as the default (in favor of the FileLockClusterService). While JGroups might be an adequate solution for some setups, there is some significant configuration that would need to be done to fully support it. JGroups utilizes UDP Multi-casting to achieve communication between the nodes. This mechanism is not fully supported in a lot of cloud environments without additional networking configuration and setup. Instead, we recommend configuring the KubernetesClusterService or the ZooKeeperClusterService for non-local deployments
As we provide a lot of support for deploying the Microservices stack to a Kubernetes cluster, it makes sense to utilize Kubernetes’s built in APIs to co-ordinate leader/member setups within replicas in a cluster using Apache’s default org.apache.camel.component.kubernetes.cluster.KubernetesClusterService.
In order to tell Broadleaf to change from a File-based implementation to a Kubernetes-based implementation, you can pass in the following ENV overrides:
BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE: <unique name for each deployment> (e.g. auth, browse, cart, processing, or supporting if deploying the Balanced Flex Package Composition or cart, catalog, campaign, etc… if deploying the Granular Composition)
this property broadleaf.messaging.clusterK8sAppLabelValue works in conjunction with broadleaf.messaging.clusterK8sAppLabel which is defined here - the default value is k8s-app. This is the name of the pod label used to group common service instances allowing the system to determine which pods belong to which "group" (e.g. pod-1 and pod-2 have the Labels: k8s-app=cart whereas pod-3 and pod-4 have the Labels: k8s-app=browse). Broadleaf’s Helm Charts will by default create pods with the appropriate labels and values depending on deployed Flex Package. If you are not using Broadleaf’s Helm Charts, you must create pods with the appropriate labels (i.e. broadleaf.messaging.clusterK8sAppLabel) and values (i.e. broadleaf.messaging.clusterK8sAppLabelValue) manually yourself in order for the KubernetesClusterService to determine leadership properly.
LOGGING_LEVEL_ORG_APACHE_CAMEL_COMPONENT_KUBERNETES_CLUSTER_LOCK: DEBUG (optional, but helpful to set the log level here to DEBUG initially to diagnose any connectivity issues)
To get this to work on your Kubernetes Cluster, you need to have a service account that specifically has access to the Kubernetes Lease access.
This service account will need permissions to read and write to the lease objects by creating ClusterRole and ClusterRoleBinding 's:
Our example Helm chart accelerators (i.e. devops-helm-charts) contains reference to a base chart that has the above resources defined (cluster-singleton). This is referenced in the blc-initialization umbrella chart. See the file blc-initialization/Chart.yaml for more details:
the default values assume deployment to the default kubernetes namespace and use of the default service account. You can override these when installing the chart for your purposes.
Balanced Helm Chart Example
Our example Helm chart accelerators (i.e. devops-helm-charts) showcase enabling the kubernetes camel configuration already defined for a Balanced deployment. See the file blc-environments/cloudexample/blc-balanced-flex-demo-cloudexample.yaml for more details.
if you have a rather large topology (e.g. deploying multiple replicas of a full granular deployment) and you are seeing substantial CPU usage on the Kubernetes API Master Nodes/Control Plane, you may find it beneficial to tune the Apache Camel Kubernetes Config Options, such as the renew-deadline, jitter-factor, and lease-duration.