Broadleaf Microservices
  • v1.0.0-latest-prod

Camel Cluster Service

Overview

Certain services within the Broadleaf ecosystem utilize the concept of a cluster singleton pattern to ensure that you have exactly one instance of a particular node in a cluster responsible for executing a specific piece of functionality.

Some examples that employ this pattern include:

  • Broadleaf’s Messaging Resiliency: See the Durable Send characteristics that are included with Broadleaf’s Messaging Components for more details.

  • Scheduled Job Co-ordination: To ensure a trigger event is only emitted once, the Scheduled Job Service should be configured to operate as a singleton in a cluster. See the Clustering Components section of the Scheduled Job Service for more details.

Camel Cluster Service

In order to accomplish this cluster singleton pattern, Broadleaf employs the use of Apache’s CamelClusterService allowing you to configure a number of backing Cluster SPI implementations out-of-the-box (e.g. Kubernetes, Zookeeper, File-based, etc…​)

Broadleaf provides example configuration and support for the following implementations:

  • org.apache.camel.component.file.cluster.FileLockClusterService

  • org.apache.camel.component.kubernetes.cluster.KubernetesClusterService

See the following classes for more details:

  • com.broadleafcommerce.common.messaging.environment.MessagingProperties which is documented here

  • com.broadleafcommerce.scheduledjob.service.autoconfigure.ClusterProperties which is documented here

FileLockClusterService (Local)

Broadleaf services by default have the FileLockClusterService implementation configured. This is suitable for LOCAL development outside the use of Kubernetes.

When this implementation is enabled, you will see INFO logs emitted similar to below:

o.a.c.c.f.cluster.FileLockClusterView    : Lock on file /var/folders/.../.../T/transition-request-supporting_REBASE acquired (lock=sun.nio.ch.FileLockImpl[0:1 exclusive valid])
Lock on file /var/folders/.../.../T/single-index-request-supporting_SINGLE_INDEX_REQUEST acquired (lock=sun.nio.ch.FileLockImpl[0:1 exclusive valid])
Lock on file /var/folders/.../.../T/transition-request-supporting_DEPLOY acquired (lock=sun.nio.ch.FileLockImpl[0:1 exclusive valid])
...

You may also otice the following WARNING emitted to your logs if the system is still configured to use file on a non-local environment:

***** WARNING - broadleaf.messaging.cluster-service-implementation-type is set to `file`. This prevents cluster detection beyond a single node, which is only appropriate for local development. This value should be changed to `kubernetes` or `zookeeper` (usually `kubernetes` via manifest environment variable when deploying to k8s). *****
Important
 As of Broadleaf 1.5.x, we have deprecated the use of org.apache.camel.component.jgroups.cluster.JGroupsLockClusterService as the default (in favor of the FileLockClusterService). While JGroups might be an adequate solution for some setups, there is some significant configuration that would need to be done to fully support it. JGroups utilizes UDP Multi-casting to achieve communication between the nodes. This mechanism is not fully supported in a lot of cloud environments without additional networking configuration and setup. Instead, we recommend configuring the KubernetesClusterService or the ZooKeeperClusterService for non-local deployments

KubernetesClusterService (Non-Local)

As we provide a lot of support for deploying the Microservices stack to a Kubernetes cluster, it makes sense to utilize Kubernetes’s built in APIs to co-ordinate leader/member setups within replicas in a cluster using Apache’s default org.apache.camel.component.kubernetes.cluster.KubernetesClusterService.

In order to tell Broadleaf to change from a File-based implementation to a Kubernetes-based implementation, you can pass in the following ENV overrides:

  • BROADLEAF_MESSAGING_CLUSTERSERVICEIMPLEMENTATIONTYPE: kubernetes

  • BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE: <unique name for each deployment> (e.g. auth, browse, cart, processing, or supporting if deploying the Balanced Flex Package Composition or cart, catalog, campaign, etc…​ if deploying the Granular Composition)

Important
 this property broadleaf.messaging.clusterK8sAppLabelValue works in conjunction with broadleaf.messaging.clusterK8sAppLabel which is defined here - the default value is k8s-app. This is the name of the pod label used to group common service instances allowing the system to determine which pods belong to which "group" (e.g. pod-1 and pod-2 have the Labels: k8s-app=cart whereas pod-3 and pod-4 have the Labels: k8s-app=browse). Broadleaf’s Helm Charts will by default create pods with the appropriate labels and values depending on deployed Flex Package. If you are not using Broadleaf’s Helm Charts, you must create pods with the appropriate labels (i.e. broadleaf.messaging.clusterK8sAppLabel) and values (i.e. broadleaf.messaging.clusterK8sAppLabelValue) manually yourself in order for the KubernetesClusterService to determine leadership properly.

  • LOGGING_LEVEL_ORG_APACHE_CAMEL_COMPONENT_KUBERNETES_CLUSTER_LOCK: DEBUG (optional, but helpful to set the log level here to DEBUG initially to diagnose any connectivity issues)

Kubernetes Configuration

To get this to work on your Kubernetes Cluster, you need to have a service account that specifically has access to the Kubernetes Lease access.

This service account will need permissions to read and write to the lease objects by creating ClusterRole and ClusterRoleBinding 's:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: lease-access
rules:
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - get
  - create
  - update
  - list
- apiGroups:
    - ""
  resources:
    - pods
  verbs:
    - get
    - list
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: lease-access
roleRef:
  kind: ClusterRole
  name: lease-access
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: {{ .Values.serviceAccount.name }}
  namespace: {{ .Values.serviceAccount.namespace }}
Cluster Singleton Helm Chart Example

Our example Helm chart accelerators (i.e. devops-helm-charts) contains reference to a base chart that has the above resources defined (cluster-singleton). This is referenced in the blc-initialization umbrella chart. See the file blc-initialization/Chart.yaml for more details:

dependencies:
  ...
  - name: cluster-singleton
    version: [MY VERSION]
    repository: https://repository.broadleafcommerce.com/repository/helm-private
Note
 the default values assume deployment to the default kubernetes namespace and use of the default service account. You can override these when installing the chart for your purposes.
Balanced Helm Chart Example

Our example Helm chart accelerators (i.e. devops-helm-charts) showcase enabling the kubernetes camel configuration already defined for a Balanced deployment. See the file blc-environments/cloudexample/blc-balanced-flex-demo-cloudexample.yaml for more details.

flex-balanced-browse:
  ...
  extraEnvs:
    - name: BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE
      value: browse

flex-balanced-cart:
  ...
  extraEnvs:
    - name: BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE
      value: cart

flex-balanced-common-env:
  data:
    ...
    BROADLEAF_MESSAGING_CLUSTERSERVICEIMPLEMENTATIONTYPE: kubernetes
    LOGGING_LEVEL_ORG_APACHE_CAMEL_COMPONENT_KUBERNETES_CLUSTER_LOCK: DEBUG

flex-balanced-processing:
  ...
  extraEnvs:
    - name: BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE
      value: processing

flex-balanced-supporting:
  ...
  extraEnvs:
    - name: BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE
      value: supporting

And the same for the auth service. See blc-environments/cloudexample/blc-auth-demo-cloudexample.yaml for more details.

auth:
  ...
  extraEnvs:
    ...
    - name: BROADLEAF_MESSAGING_CLUSTERSERVICEIMPLEMENTATIONTYPE
      value: kubernetes
    - name: BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE
      value: auth
    - name: LOGGING_LEVEL_ORG_APACHE_CAMEL_COMPONENT_KUBERNETES_CLUSTER_LOCK
      value: DEBUG
Validation

With all the configuration in place, you can run the following command against your Kubernetes cluster to verify that leases have been provisioned as expected: kubectl get leases

NAME                                                                                    HOLDER                        AGE
leaders-batch-index-request-processing-batch-index-request                              processing-5b4d98c557-428vh   136m
leaders-batch-index-request-supporting-batch-index-request                              supporting-67f8946db9-mzx9s   138m
leaders-cartnotification-cart-cart-pending-payment-failed-event                         cart-85bb888d-vzj6f           138m
leaders-cartnotification-cart-cart-rejection-event                                      cart-85bb888d-vzj6f           138m
leaders-cartnotification-cart-checkout-completion                                       cart-85bb888d-vzj6f           138m
leaders-cartnotification-cart-checkout-rollback-event                                   cart-85bb888d-vzj6f           138m
leaders-changesummary-supporting-change                                                 supporting-67f8946db9-mzx9s   138m
leaders-completion-supporting-transition-completion                                     supporting-67f8946db9-mzx9s   138m
leaders-import-batch-request-processing-batch-request                                   processing-5b4d98c557-428vh   136m
leaders-import-delete-sandbox-processing-import-delete-sandbox-request                  processing-5b4d98c557-428vh   136m
leaders-import-process-request-processing-process-import-request                        processing-5b4d98c557-428vh   136m
...
Note
 if you have a rather large topology (e.g. deploying multiple replicas of a full granular deployment) and you are seeing substantial CPU usage on the Kubernetes API Master Nodes/Control Plane, you may find it beneficial to tune the Apache Camel Kubernetes Config Options, such as the renew-deadline, jitter-factor, and lease-duration.

Additional Resources