Broadleaf Microservices
  • v1.0.0-latest-prod

Operational Recipes

Important
The following recipes are applicable to initializr/manifest-based projects (Release Train 1.8.4+) and requires credentials to access various Broadleaf resources referenced in Getting Started Locally. Please read this section and have the appropriate local environment setup configured before proceeding.

The following recipes are catered to mature DevOps operational teams looking to manage and support a Broadleaf implementation themselves. This section provides operational recipes and guides for common tasks related to setting up, managing, and operating a Broadleaf Microservices ecosystem.

For those that would like support with DevOps and operational concerns related to working with a Broadleaf implementation, Broadleaf provides a fully managed cloud offering called Broadleaf Cloud as well as DevOps Managed Support Services. These offerings provide many benefits including a fully managed pipeline catered to building out a Broadleaf ecosystem. More details on other capabilities that we can provide in this area can be found here.

Generate New Security Artifacts

  • Audience: Architects, DevOps, SRE Teams

  • Stage: Environment Setup

  • Background: Broadleaf provides a standalone security tool to help operation teams manage the necessary security artifacts that are generated for an Initializr-based Microservices project

  • Why: It is recommended that you generate a new set of security artifacts for every environment that Broadleaf is deployed to (e.g. Dev, QA, Production)

  • Assumptions:

    • You’ve authenticated with Broadleaf’s Nexus to obtain the necessary tools and artifacts as identified in the Getting Started Locally guide

    • You are working with an Initializr-based Broadleaf project structure containing a manifest

    • You plan to use Broadleaf’s Spring Cloud Config Server for distributed configuration property management

    • Optional but highly recommended - You have access to a Secrets Vault or Secrets Manager (e.g either a cloud-native offering like Google’s Secrets Manager or Azure Key Vault or a third-party solution like HashiCorp Vault) to store and maintain the source security artifacts for safe keeping

Reference Local Workspace Setup

Below is a recommended local workspace setup for an operations team member. This setup supports management of multiple security artifacts for different environments. We’ll be using and referencing this setup in the following recipes.

Reference Workspace Setup

For organizational purposes, we recommend creating a separate working directory for each environment (e.g. Dev,QA, and Production etc…​) to store and generate your security artifacts

The below instructions will outline the steps necessary to prepare each of these working directories to generate and manage the security artifacts for that environment.

Download Correct Version of Security Tool

In order to generate the correct properties and artifacts that apply to your implementation, you will need to download the correct Security Tool version that is applicable to the starter base and release train versions defined for your project. To find this version you can execute the following Maven Command in the manifest directory of your main Broadleaf project.

./mvnw -q \
  org.apache.maven.plugins:maven-help-plugin:3.2.0:evaluate \
  -Dexpression=blc.starter.version \
  -DforceStdout

Keep note of this version as you’ll need to use this to download the correct tool version.

Important
As you continually upgrade and update your Broadleaf implementation to reference new starter bases dependencies and new release trains, you’ll want to make sure that you download the appropriate tool version that align with the updates. You can find that congruent version by running the command above.

To download the tool, create a separate working directory for the environment you want to generate security artifacts for (e.g. my-dev.com) and take note of the path to this directory.

Next, cd into the manifest directory of your main Broadleaf project and run the following Maven dependency copy command. Make sure to replace the [replace_with_correct_version] and the [/my/local/path/to/my-dev.com] parts with the appropriate values based on what you noted from the steps above.

./mvnw org.apache.maven.plugins:maven-dependency-plugin:3.6.0:copy \
  -Dartifact=com.broadleafcommerce.microservices:broadleaf-microservices-starter-security-tool:[replace_with_correct_version]:jar \
  -DartifactId=broadleaf-microservices-starter-security-tool-downloader \
  -DgroupId=com.broadleafcommerce.microservices \
  -DoutputDirectory=[/my/local/path/to/my-dev.com] \
  -DstripVersion=true

Copy Current Manifest Artifact

Next, you’ll want to copy over the current Manifest JAR built for your project to your environment working directory. You can find this artifact if you cd into your manifest directory of your primary Broadleaf project. Next cd into the target directory. Note: If you do not have a target directory, you can always execute a new build by running ./mvnw clean install which should generate a new target directory with several build artifacts inside. Inside this target directory, you should find a manifest JAR (e.g. microservice-flexpackage-manifest-1.0.0-SNAPSHOT.jar). Copy this file into your environment’s working directory. For example:

cp target/microservice-flexpackage-manifest-1.0.0-SNAPSHOT.jar /my/local/path/to/my-dev.com

Generate New Security Artifacts and Config Distribution

You should now have both a security-tool.jar and a manifest.jar in your environment’s working directory (e.g. /my/local/path/to/my-dev.com). With this in place, you are now ready to generate new security artifacts by running the following Java command (replacing the name of the security-tool.jar with the actual name of the JAR file in your directory):

java -jar security-tool.jar generate

With that command executed, you should now see new security and config folders generated in your current environment working directory.

HINT: you can always run java -jar security-tool.jar help to view additional command options available to you when executing this tool

While this is optional, it is highly recommended that you now store these generated artifacts in the security folder in a shared secure location (i.e. NOT in a source control system) preferably a secrets manager or vault. This allows you (or another operations team member with the appropriate access) to create a shared "source of truth" that correspond with the latest versions of these artifacts - this will help if and when you need to make changes and updates to the properties over time.

Configure Config Server with a Git Backend

  • Audience: Architects, DevOps, SRE Teams

  • Stage: Environment Setup

  • Why: If you are using Broadleaf’s recommended Config Service (which is built on Spring Cloud Config), it is recommended that you configure that service with a Git backend allowing you to achieve a GitOps style operational setup. This allows both developers and operations teams to manage properties through source control. When these properties change, the config service can be configured to pick up these changes, propagate them, and make sure these updated properties are available to all the concerned microservices within the ecosystem

  • Assumptions:

    • You’ve authenticated with Broadleaf’s Nexus to obtain the necessary tools and artifacts as identified in the Getting Started Locally guide

    • You are working with an Initializr-based Broadleaf project structure containing a manifest

    • You plan to use Broadleaf’s Spring Cloud Config Server for distributed configuration property management

    • You have the ability to create a Git Repository (that is accessible to the Config Service) e.g. GitHub

Create Git Repos & READONLY User

It is recommended that you create at least 2 repositories, one for secure encrypted properties and one for general non-secure properties. These repositories can then be configured with the appropriate team access according to your security posture. In our examples and reference, we’ve called the the Git repos, configserver-[ENV]-secure and configserver-[ENV]-insecure.

You will also want to create a READONLY user that has access to pull the properties from these corresponding repos. Take note of the credentials needed for this user as it will be set and injected into the configuration for Broadleaf’s Spring Cloud Config service.

Spring Cloud Config Properties

You can pass in one or more backing Git datastore configurations for your properties using ENV properties. If you are using the helm charts that are generated for you using the helm:generate plugin, you will notice that you can configure these properties using the following helm properties (i.e. secrets.environmentRepositoryCreds.keyValuePairs.*) that are overridable in the values.yaml file.

Your overrides, might look something like this:

secrets:
  environmentRepositoryCreds:
    keyValuePairs:
      SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_TYPE: "git"
      SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_DEFAULT_LABEL: "develop"
      SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_URI: "https://github.com/MyOrg/configserver-dev-insecure"
      SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_USERNAME: "setme"
      SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_PASSWORD: "setme"
      SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_TYPE: "git"
      SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_DEFAULT_LABEL: "develop"
      SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_URI: "https://github.com/MyOrg/configserver-dev-secure"
      SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_USERNAME: "setme"
      SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_PASSWORD: "setme"

Configure Push Notifications [Optional]

Many source code repositories allow notification of changes to a repository based on a configured webhook. You can setup webhooks following the directions identified by your code repository provider. For example, GitHub uses a POST to any configured webhooks with a JSON body containing a list of commits. Once the webhook is configured, Spring Cloud Config natively handles the processing and consumption of those events. More details on this can be found on Spring’s official documentation.

Note
If you choose not to enable push notification integrations, you can always manually restart your config service, which during the start-up process will pull the latest properties and configurations from the configured Git repository

Encrypt New Secure Properties for Config Server

  • Audience: DevOps, SRE Teams

  • Stage: Environment Setup + Maintenance

  • Why: If you are using Broadleaf’s recommended Config Service, you may need to add and manage additional secure properties for your implementation (outside of the default Broadleaf properties)

  • Assumptions:

    • You’ve authenticated with Broadleaf’s Nexus to obtain the necessary tools and artifacts as identified in the Getting Started Locally guide

    • You are working with an Initializr-based Broadleaf project structure containing a manifest

    • You plan to use Broadleaf’s Spring Cloud Config Server for distributed configuration property management

    • You’ve already generated a set of security and config artifacts following the recipe "Generate New Security Artifacts" above, for each environment that you would like to update

Modifying credentials-report.env

The main mechanic for managing secure and encrypted properties within a Broadleaf ecosystem leveraging a Config Server is to manage these properties using the generated credentials-report.env which you will find in the generated security folder for your environment.

If you are adding/updating an existing credentials-report.env that is already applied to an environment, you will need to pull down the original "source" (presumably stored in your Secure Vault/Secrets Manager) in order to update it. You can then add arbitrary properties to this report using a special syntax which will be described below.

Special Property Extension Syntax

Suppose you have a new secure property that you wish to be encrypted and stored in the Config Server (e.g. MY_SECRET_PROPERTY=1234). In order to create the appropriate Spring Application context files with the encrypted values, for example:

my:
  secret:
    property: '{cipher}{key:version_1}AQCQx...'

I can run the same security-tool.jar as mentioned in the recipe "Generate New Security Artifacts" against my updated credentials-report.env with my new property added.

The key difference is that (since Broadleaf does not have any knowledge of what this property is) you must define this property in your credentials-report.env file with a special syntax that matches the following form:

[PROPERTY_KEY]_EXTENSION_[PROFILE]_[PROFILE]...
  • [PROPERTY_KEY] is the fully qualified property using Spring’s Relaxed Binding underscore notation

  • followed by the constant _EXTENSION_

  • followed by one ore more [PROFILE] 's which define which microservice this specific property is applicable to

For example, let’s say the MY_SECRET_PROPERTY that I am trying to add is only applicable to the Auth microservice, in that case, I could add the following new plain-text property in my credentials-report.env

MY_SECRET_PROPERTY_EXTENSION_AUTH=1234

After modifying the security/credentials-report.env with the new secret using the special syntax, run in your current working environments working directory:

java -jar security-tool.jar encrypt

After running the encrypt command, you should now have newly encrypted config properties in your config/secure directory for that environment. If you have this pointed to a backing Git repo, you can now check in these changes directly to source control.

Targeting Specific Components using Profiles

To determine the "Profile Keys" that you can use in your property syntax, you can download the component-profiles-map.yml which will give you a file that contains the different profiles the security tool uses when creating the appropriate encrypted configuration files.

Determine Bootstrap Library Starter Version

Similar to downloading the security-tool.jar you will want to obtain the correct version of the component-profiles-map.yml that applies to your installation. In the manifest directory of your main Broadleaf project, run the following Maven command:

./mvnw -q \
  org.apache.maven.plugins:maven-help-plugin:3.2.0:evaluate \
  -Dexpression=blc.starter.version \
  -DforceStdout

Take note of the emitted version as you will use this in the next step.

Download the Component Profiles Map

Run the following Maven command in the manifest directory of your main Broadleaf project. Take note to replace the [replace_with_correct_version] and [/my/local/path/to/my-dev.com] placeholders in the below command.

./mvnw org.apache.maven.plugins:maven-dependency-plugin:3.6.0:unpack \
  -Dartifact=com.broadleafcommerce.microservices:broadleaf-microservices-starter-bootstrap:[replace_with_correct_version]:jar \
  -DartifactId=broadleaf-microservices-starter-bootstrap \
  -DgroupId=com.broadleafcommerce.microservices \
  -Dincludes=component-profiles-map.yml \
  -DoutputDirectory=[/my/local/path/to/my-dev.com] \
  -DstripVersion=true

This should download the component-profiles-map.yml file to your environment’s working directory. If you inspect the contents of this file, you can find the different profile keys that can be used to target specific microservice components.

Configure an Externally Managed Datasource for Broadleaf Microservices

  • Audience: DevOps, SRE Teams

  • Stage: Environment Setup

  • Why: It is common to configure Broadleaf Microservices to point to a externally managed datasource (e.g. a Cloud Managed SQL instance). The following recipe will guide you through the configuration needed to support this.

  • Assumptions:

    • You’ve authenticated with Broadleaf’s Nexus to obtain the necessary tools and artifacts as identified in the Getting Started Locally guide

    • You are working with an Initializr-based Broadleaf project structure containing a manifest

    • You plan to use Broadleaf’s Spring Cloud Config Server for distributed configuration property management

    • You’ve already generated a set of security and config artifacts following the recipe "Generate New Security Artifacts" above, for each environment that you would like to update

    • This will build on top of the recipe "Encrypt New Secure Properties for Config Server", so please review and be familiar with that recipe before proceeding

For this recipe, we will configure and store encrypted connection info and credential information in Broadleaf’s Config Service.

To do this, we will override and re-encrypt properties on an existing credentials-report.env and then sync that to our backing Config Service datastore (e.g. Git). Following the patterns described in the recipe "Encrypt New Secure Properties for Config Server", we can change and add the following properties to our security/credentials-report.env file for my current environment’s working directory:

# Database Data
BROADLEAF_DATASOURCE_PASSWORD=<set me>

# Cloud Database Connection Info
BROADLEAF_COMPOSITE_DATASOURCE_URL_EXTENSION_PROCESSINGFLEX_SUPPORTINGFLEX_BROWSEFLEX_CARTFLEX_DATA=jdbc:postgresql://<my-ip>:5432/broadleaf
SPRING_DATASOURCE_URL_EXTENSION_AUTH_DATA=jdbc:postgresql://<my-ip>:5432/broadleaf?currentSchema=auth

To summarize:

  • update the property BROADLEAF_DATASOURCE_PASSWORD to our new Cloud SQL DB instance password.

  • using the "Special Config Syntax", we can update the Datasource URL and target this property to the following microservice flex packages and applications: Processing, Supporting, Browse, Cart, Data, and Auth

After modifying the credentials-report.env with the data store information, you can now run the following Java command in your environment’s working directory:

java -jar security-tool.jar encrypt

With this run, you should now have newly encrypted config properties in your config/secure directory which you can now sync with your Config Service

Update Security Artifacts Based on Manifest Changes

  • Audience: DevOps, SRE Teams

  • Stage: Initial Environment Setup

  • Why: When you’re setting up a new environment and you need to make a change to the manifest.yml that informs the creation of new security artifacts, then you will need to run the generate security-tool command against the new manifest JAR in order to produce new security artifacts.

  • Example Use Case: You plan to deploy certain services like Kafka to another namespace in Kuberentes (instead of default) . If you do this, you’ll need to change/update the "alternate domains" in your manifest.yml file which will inform the creation process of all self-signed certificates. Specifically, we may need to inform the generation process of the SAN’s fully qualified domains that need to be added to the trust stores when generating those security artifacts (i.e. all applicable Kubernetes DNS paths that may contain my new namespace).

  • Assumptions:

    • You’ve authenticated with Broadleaf’s Nexus to obtain the necessary tools and artifacts as identified in the Getting Started Locally guide

    • You are working with an Initializr-based Broadleaf project structure containing a manifest

    • You plan to use Broadleaf’s Spring Cloud Config Server for distributed configuration property management

    • You’ve already generated a set of security and config artifacts following the recipe "Generate New Security Artifacts" above, for each environment that you would like to update

Let’s say you wish to deploy Kafka to a new Kubernetes namespace called broadleafsupporting instead of deploying it to the default namespace. With this example, you will need to update the alternates domains in your manifest.yml to include/change the alternate domains to specify that specific k8 namespace:

- name: broker
  platform: kafka
  descriptor: snippets/docker-kafka.yml
  enabled: true
  domain:
    alternates:
    - kafkacluster-0.kafkacluster-headless.broadleafsupporting.svc.cluster.local
    - kafkacluster-0.kafkacluster-headless.broadleafsupporting.svc.cluster
    - kafkacluster-0.kafkacluster-headless.broadleafsupporting.svc
    - kafkacluster-0.kafkacluster-headless.broadleafsupporting
    - kafkacluster-0.kafkacluster-headless
    - kafkacluster-1.kafkacluster-headless.broadleafsupporting.svc.cluster.local
    - kafkacluster-1.kafkacluster-headless.broadleafsupporting.svc.cluster
    - kafkacluster-1.kafkacluster-headless.broadleafsupporting.svc
    - kafkacluster-1.kafkacluster-headless.broadleafsupporting
    - kafkacluster-1.kafkacluster-headless
    - kafkacluster-2.kafkacluster-headless.broadleafsupporting.svc.cluster.local
    - kafkacluster-2.kafkacluster-headless.broadleafsupporting.svc.cluster
    - kafkacluster-2.kafkacluster-headless.broadleafsupporting.svc
    - kafkacluster-2.kafkacluster-headless.broadleafsupporting
    - kafkacluster-2.kafkacluster-headless

Once you’ve saved your new manifest.yml. Generate a new manifest.jar and place the security-tool.jar and the project’s updated manifest.jar in the same environment working directory.

Run: java -jar security-tool.jar generate

Output: will produce updated security artifacts (e.g. kafka trustore and creds) in the security directory of the same working directory

Important
if there are existing certificates and truststore artifacts already present in the security directory, and you want to replace them, you will need to delete the old ones first and then run generate to generate new artifacts

Updating Seed Data with Liquibase Params

<property name="blcAdminPasswordBcrypt" value="todo-set" />
<changeSet author="broadleaf" id="auth-user-7" labels="auth,required">
        <preConditions onFail="MARK_RAN">
            <sqlCheck expectedResult="0">SELECT COUNT(*) FROM blc_user WHERE id='-2';</sqlCheck>
        </preConditions>
        <insert tableName="blc_user">
            <column name="id" value="-2" />
            ...
            <column name="PASSWORD" value="${blcAdminPasswordBcrypt}" />
            ...
        </insert>
</changeSet>

With this convention in place, you may notice a couple pre-generated ENV properties for you in your security/credentials-report.env, for example:

SPRING_LIQUIBASE_PARAMETERS_BLCADMINPASSWORDBCRYPT=$2a$10...
SPRING_LIQUIBASE_PARAMETERS_BLCADMIN2PASSWORDBCRYPT_EXTENSION_LIQUIBASE=$2a$10...
SPRING_LIQUIBASE_PARAMETERS_BLCADMINAAAPASSWORDBCRYPT_EXTENSION_LIQUIBASE=$2a$10...

These properties can be manually changed prior to an initial installation , re-encrypted using the security-tool.jar as mentioned in the recipe: "Encrypt New Secure Properties for Config Server" and synced with the config server.

During an initial installation, the data module will pull these values from the config server and run the appropriate changelogs applying any Liquibase Params overrides as necessary to your empty datastore.

Important Considerations About Already Applied Changelogs

Liquibase changelogs are applied once and marked as "run" in a databasechangelog table. This is typically done during initial installation when running the data module (or via the job-data helm chart in kubernetes) for the first time. The initializr required changelogs contain pre-conditions "Mark Ran as Failed" if an existing record already exists in the datastore.

What does this mean? From an operational perspective, if (as an example) you wish to change the master admin password and bcrypt value and you’ve already applied the changelog with a previously set password, then changing credentials in the credentials report and re-generating encrypted values for the config server will have NO IMPACT on the data since that record has already been inserted and applied.

Options to change a "seed data" record if already applied:

  • manually update the data in your datastore (ignoring what’s specified in the credentials report)

  • If you would like to have the credentials report and associated config server encrypted "in sync" with the values in the data store, then you will need to manually remove the database changelog and associated applied record and then re-run the data module with the new seed property applied from the configserver.

Debug by Enabling the Broadleaf Environment Report

  • Audience: Developers, Architects, DevOps, SRE Teams

  • Stage: Application Debugging

  • Why: It’s often times useful to understand what actual properties the Broadleaf Flex Applications are resolving and using during startup and runtime. Broadleaf provides a configurable startup report that will emit all resolved properties to help troubleshoot any property-related questions.

By default, for security purposes, the Broadleaf Environment Report is disabled, however configuring the following property for an application can turn it on for debugging purposes.

Important
We recommend that this be DISABLED for production environments
broadleaf:
  environment:
    report:
      disabled: 'false'