Broadleaf Microservices
  • v1.0.0-latest-prod

Camel Cluster Service

Overview

Certain services within the Broadleaf ecosystem utilize the concept of a cluster singleton pattern to ensure that you have exactly one instance of a particular node in a cluster responsible for executing a specific piece of functionality.

Some examples that employ this pattern include:

  • Broadleaf’s Messaging Resiliency: See the Durable Send characteristics that are included with Broadleaf’s Messaging Components for more details.

  • Scheduled Job Co-ordination: To ensure a trigger event is only emitted once, the Scheduled Job Service should be configured to operate as a singleton in a cluster. See the Clustering Components section of the Scheduled Job Service for more details.

Camel Cluster Service

In order to accomplish this cluster singleton pattern, Broadleaf employs the use of Apache’s CamelClusterService allowing you to configure a number of backing Cluster SPI implementations out-of-the-box (e.g. Kubernetes, Zookeeper, File-based, etc…​)

Broadleaf provides example configuration and support for the following implementations:

  • org.apache.camel.component.file.cluster.FileLockClusterService

  • org.apache.camel.component.kubernetes.cluster.KubernetesClusterService

See the following classes for more details:

  • com.broadleafcommerce.common.messaging.environment.MessagingProperties which is documented here

  • com.broadleafcommerce.scheduledjob.service.autoconfigure.ClusterProperties which is documented here

Manifest-Based Projects (since 2.0.2-GA, 2.1.0-GA)

The manifest file can now be used to establish your desired cluster service implementation type, per environment if desired.

schema:
  version: '1.0'
project:
  groupId: com.example.microservices
  packageName: com.example.microservices
  starterParentVersion: 2.1.0-GA
  version: 1.0.0-SNAPSHOT
  clusterServiceType: (1)
    local: file
    docker: file
    cloud: kubernetes
  1. The clusterServiceType elements allows you to specify the implementation type for any or all of local, docker, and cloud environments. Valid values for each environment are file, zookeeper, and kubernetes.

ClusterServiceType may be omitted altogether, or any environment component may be omitted. Any omission results in the use of the default file type (unless otherwise declared via environment variable - see below).

zookeeper is new as of 2.0.2-GA, 2.1.0-GA. When this type is used in the manifest, the manifest lifecycle (e.g. mvn flex:generate) will also maintain any appropriate zk ACL credentials needed by generated flexpackages to allow those runtimes to successfully authenticate to the secured zookeeper instance. The relevant credentials are automatically added to the secure yaml files sourced for the config server. This maintenance is a convenience of using the manifest-based starter, but is not required.

If not using a manifest lifecycle, Zookeeper (and friends) can be manually specified using the explicit approaches defined below as well.

FileLockClusterService (Local)

Broadleaf services by default have the FileLockClusterService implementation configured. This is suitable for LOCAL development outside the use of Kubernetes.

When this implementation is enabled, you will see INFO logs emitted similar to below:

o.a.c.c.f.cluster.FileLockClusterView    : Lock on file /var/folders/.../.../T/transition-request-supporting_REBASE acquired (lock=sun.nio.ch.FileLockImpl[0:1 exclusive valid])
Lock on file /var/folders/.../.../T/single-index-request-supporting_SINGLE_INDEX_REQUEST acquired (lock=sun.nio.ch.FileLockImpl[0:1 exclusive valid])
Lock on file /var/folders/.../.../T/transition-request-supporting_DEPLOY acquired (lock=sun.nio.ch.FileLockImpl[0:1 exclusive valid])
...

You may also otice the following WARNING emitted to your logs if the system is still configured to use file on a non-local environment:

***** WARNING - broadleaf.messaging.cluster-service-implementation-type is set to `file`. This prevents cluster detection beyond a single node, which is only appropriate for local development. This value should be changed to `kubernetes` or `zookeeper` (usually `kubernetes` via manifest environment variable when deploying to k8s). *****
Important
As of Broadleaf 1.5.x, we have deprecated the use of org.apache.camel.component.jgroups.cluster.JGroupsLockClusterService as the default (in favor of the FileLockClusterService). While JGroups might be an adequate solution for some setups, there is some significant configuration that would need to be done to fully support it. JGroups utilizes UDP Multi-casting to achieve communication between the nodes. This mechanism is not fully supported in a lot of cloud environments without additional networking configuration and setup. Instead, we recommend configuring the KubernetesClusterService or the ZooKeeperClusterService for non-local deployments

KubernetesClusterService (Non-Local)

As we provide a lot of support for deploying the Microservices stack to a Kubernetes cluster, it makes sense to utilize Kubernetes’s built in APIs to co-ordinate leader/member setups within replicas in a cluster using Apache’s default org.apache.camel.component.kubernetes.cluster.KubernetesClusterService.

In order to tell Broadleaf to change from a File-based implementation to a Kubernetes-based implementation, you can pass in the following ENV overrides:

  • BROADLEAF_MESSAGING_CLUSTERSERVICEIMPLEMENTATIONTYPE: kubernetes

  • BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE: <unique name for each deployment> (e.g. auth, browse, cart, processing, or supporting if deploying the Balanced Flex Package Composition or cart, catalog, campaign, etc…​ if deploying the Granular Composition)

Important
this property broadleaf.messaging.clusterK8sAppLabelValue works in conjunction with broadleaf.messaging.clusterK8sAppLabel which is defined here - the default value is k8s-app. This is the name of the pod label used to group common service instances allowing the system to determine which pods belong to which "group" (e.g. pod-1 and pod-2 have the Labels: k8s-app=cart whereas pod-3 and pod-4 have the Labels: k8s-app=browse). Broadleaf’s Helm Charts will by default create pods with the appropriate labels and values depending on deployed Flex Package. If you are not using Broadleaf’s Helm Charts, you must create pods with the appropriate labels (i.e. broadleaf.messaging.clusterK8sAppLabel) and values (i.e. broadleaf.messaging.clusterK8sAppLabelValue) manually yourself in order for the KubernetesClusterService to determine leadership properly.
  • LOGGING_LEVEL_ORG_APACHE_CAMEL_COMPONENT_KUBERNETES_CLUSTER_LOCK: DEBUG (optional, but helpful to set the log level here to DEBUG initially to diagnose any connectivity issues)

Kubernetes Configuration

To get this to work on your Kubernetes Cluster, you need to have a service account that specifically has access to the Kubernetes Lease access.

This service account will need permissions to read and write to the lease objects by creating ClusterRole and ClusterRoleBinding 's:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: lease-access
rules:
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - get
  - create
  - update
  - list
- apiGroups:
    - ""
  resources:
    - pods
  verbs:
    - get
    - list
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: lease-access
roleRef:
  kind: ClusterRole
  name: lease-access
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: {{ .Values.serviceAccount.name }}
  namespace: {{ .Values.serviceAccount.namespace }}
Cluster Singleton Helm Chart Example

Our example Helm chart accelerators (i.e. devops-helm-charts) contains reference to a base chart that has the above resources defined (cluster-singleton). This is referenced in the blc-initialization umbrella chart. See the file blc-initialization/Chart.yaml for more details:

dependencies:
  ...
  - name: cluster-singleton
    version: [MY VERSION]
    repository: https://repository.broadleafcommerce.com/repository/helm-private
Note
the default values assume deployment to the default kubernetes namespace and use of the default service account. You can override these when installing the chart for your purposes.
Balanced Helm Chart Example

Our example Helm chart accelerators (i.e. devops-helm-charts) showcase enabling the kubernetes camel configuration already defined for a Balanced deployment. See the file blc-environments/cloudexample/blc-balanced-flex-demo-cloudexample.yaml for more details.

flex-balanced-browse:
  ...
  extraEnvs:
    - name: BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE
      value: browse

flex-balanced-cart:
  ...
  extraEnvs:
    - name: BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE
      value: cart

flex-balanced-common-env:
  data:
    ...
    BROADLEAF_MESSAGING_CLUSTERSERVICEIMPLEMENTATIONTYPE: kubernetes
    LOGGING_LEVEL_ORG_APACHE_CAMEL_COMPONENT_KUBERNETES_CLUSTER_LOCK: DEBUG

flex-balanced-processing:
  ...
  extraEnvs:
    - name: BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE
      value: processing

flex-balanced-supporting:
  ...
  extraEnvs:
    - name: BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE
      value: supporting

And the same for the auth service. See blc-environments/cloudexample/blc-auth-demo-cloudexample.yaml for more details.

auth:
  ...
  extraEnvs:
    ...
    - name: BROADLEAF_MESSAGING_CLUSTERSERVICEIMPLEMENTATIONTYPE
      value: kubernetes
    - name: BROADLEAF_MESSAGING_CLUSTERK8SAPPLABELVALUE
      value: auth
    - name: LOGGING_LEVEL_ORG_APACHE_CAMEL_COMPONENT_KUBERNETES_CLUSTER_LOCK
      value: DEBUG
Validation

With all the configuration in place, you can run the following command against your Kubernetes cluster to verify that leases have been provisioned as expected: kubectl get leases

NAME                                                                                    HOLDER                        AGE
leaders-batch-index-request-processing-batch-index-request                              processing-5b4d98c557-428vh   136m
leaders-batch-index-request-supporting-batch-index-request                              supporting-67f8946db9-mzx9s   138m
leaders-cartnotification-cart-cart-pending-payment-failed-event                         cart-85bb888d-vzj6f           138m
leaders-cartnotification-cart-cart-rejection-event                                      cart-85bb888d-vzj6f           138m
leaders-cartnotification-cart-checkout-completion                                       cart-85bb888d-vzj6f           138m
leaders-cartnotification-cart-checkout-rollback-event                                   cart-85bb888d-vzj6f           138m
leaders-changesummary-supporting-change                                                 supporting-67f8946db9-mzx9s   138m
leaders-completion-supporting-transition-completion                                     supporting-67f8946db9-mzx9s   138m
leaders-import-batch-request-processing-batch-request                                   processing-5b4d98c557-428vh   136m
leaders-import-delete-sandbox-processing-import-delete-sandbox-request                  processing-5b4d98c557-428vh   136m
leaders-import-process-request-processing-process-import-request                        processing-5b4d98c557-428vh   136m
...
Note
if you have a rather large topology (e.g. deploying multiple replicas of a full granular deployment) and you are seeing substantial CPU usage on the Kubernetes API Master Nodes/Control Plane, you may find it beneficial to tune the Apache Camel Kubernetes Config Options, such as the renew-deadline, jitter-factor, and lease-duration.

ZookeeperClusterService (since 2.0.2-GA, 2.1.0-GA)

Using Zookeeper as a cluster service lock source can be interesting for implementations not using kubernetes, or for expansive granular deployments in which the kubernetes lease maintenance for a large number of cluster service locks may be onerous. Since Zookeeper is a first-class member of the reference stack, it is another option to consider.

Zookeeper Configuration

Support for cluster service via zookeeper can be configured via environment variables alone.

Property Value Default Notes

BROADLEAF_MESSAGING_CLUSTERSERVICEIMPLEMENTATIONTYPE

zookeeper

BROADLEAF_LOCK_ZKACL_ENABLED

true or false

true

Whether the system should attempt authentication when connecting to zookeeper

BROADLEAF_LOCK_ZKACL_USERNAME

readonly-user

The valid username for zookeeper auth

BROADLEAF_LOCK_ZKACL_PASSWORD

The valid password for zookeeper auth. This should be supplied via a secure mechanism.

Validation

Executing the 4-letter command dump against the zookeeper admin should generally be enough to confirm zookeeper is being successfully used for cluster service locking via the /camel root. If you have the default configuration for zookeeper with the admin running in the container, you can usually create a terminal to the container and check from there. For example, after establishing a terminal session, curl http(s)//localhost:[8080]/commands/dump. You will be looking for the presence of one or more sessions starting at the /camel root.

Additional Resources