ImportantThe following recipes are applicable to initializr/manifest-based projects (Release Train 1.8.4+) and requires credentials to access various Broadleaf resources referenced in Getting Started Locally. Please read this section and have the appropriate local environment setup configured before proceeding.
The following recipes are catered to mature DevOps operational teams looking to manage and support a Broadleaf implementation themselves. This section provides operational recipes and guides for common tasks related to setting up, managing, and operating a Broadleaf Microservices ecosystem.
For those that would like support with DevOps and operational concerns related to working with a Broadleaf implementation, Broadleaf provides a fully managed cloud offering called Broadleaf Cloud as well as DevOps Managed Support Services. These offerings provide many benefits including a fully managed pipeline catered to building out a Broadleaf ecosystem. More details on other capabilities that we can provide in this area can be found here.
Audience: Architects, DevOps, SRE Teams
Stage: Environment Setup
Background: Broadleaf provides a standalone security tool to help operation teams manage the necessary security artifacts that are generated for an Initializr-based Microservices project
Why: It is recommended that you generate a new set of security artifacts for every environment that Broadleaf is deployed to (e.g. Dev
, QA
, Production
)
Assumptions:
You’ve authenticated with Broadleaf’s Nexus to obtain the necessary tools and artifacts as identified in the Getting Started Locally guide
You are working with an Initializr-based Broadleaf project structure containing a manifest
You plan to use Broadleaf’s Spring Cloud Config Server for distributed configuration property management
Optional but highly recommended - You have access to a Secrets Vault or Secrets Manager (e.g either a cloud-native offering like Google’s Secrets Manager or Azure Key Vault or a third-party solution like HashiCorp Vault) to store and maintain the source security artifacts for safe keeping
Below is a recommended local workspace setup for an operations team member. This setup supports management of multiple security artifacts for different environments. We’ll be using and referencing this setup in the following recipes.
For organizational purposes, we recommend creating a separate working directory for each environment (e.g. Dev
,QA
, and Production
etc…) to store and generate your security artifacts
The below instructions will outline the steps necessary to prepare each of these working directories to generate and manage the security artifacts for that environment.
In order to generate the correct properties and artifacts that apply to your implementation, you will need to download the correct Security Tool
version that is applicable to the starter base and release train versions defined for your project. To find this version you can execute the following Maven Command in the manifest
directory of your main Broadleaf project.
./mvnw -q \ org.apache.maven.plugins:maven-help-plugin:3.2.0:evaluate \ -Dexpression=blc.starter.version \ -DforceStdout
Keep note of this version as you’ll need to use this to download the correct tool version.
ImportantAs you continually upgrade and update your Broadleaf implementation to reference new starter bases dependencies and new release trains, you’ll want to make sure that you download the appropriate tool version that align with the updates. You can find that congruent version by running the command above.
To download the tool, create a separate working directory for the environment you want to generate security artifacts for (e.g. my-dev.com
) and take note of the path to this directory.
Next, cd
into the manifest
directory of your main Broadleaf project and run the following Maven dependency copy
command. Make sure to replace the [replace_with_correct_version]
and the [/my/local/path/to/my-dev.com]
parts with the appropriate values based on what you noted from the steps above.
./mvnw org.apache.maven.plugins:maven-dependency-plugin:3.6.0:copy \ -Dartifact=com.broadleafcommerce.microservices:broadleaf-microservices-starter-security-tool:[replace_with_correct_version]:jar \ -DartifactId=broadleaf-microservices-starter-security-tool-downloader \ -DgroupId=com.broadleafcommerce.microservices \ -DoutputDirectory=[/my/local/path/to/my-dev.com] \ -DstripVersion=true
Next, you’ll want to copy over the current Manifest JAR built for your project to your environment working directory. You can find this artifact if you cd
into your manifest
directory of your primary Broadleaf project. Next cd
into the target
directory. Note: If you do not have a target
directory, you can always execute a new build by running ./mvnw clean install
which should generate a new target
directory with several build artifacts inside. Inside this target
directory, you should find a manifest JAR (e.g. microservice-flexpackage-manifest-1.0.0-SNAPSHOT.jar
). Copy this file into your environment’s working directory. For example:
cp target/microservice-flexpackage-manifest-1.0.0-SNAPSHOT.jar /my/local/path/to/my-dev.com
You should now have both a security-tool.jar
and a manifest.jar
in your environment’s working directory (e.g. /my/local/path/to/my-dev.com
). With this in place, you are now ready to generate new security artifacts by running the following Java command (replacing the name of the security-tool.jar
with the actual name of the JAR file in your directory):
java -jar security-tool.jar generate
With that command executed, you should now see new security
and config
folders generated in your current environment working directory.
HINT: you can always run
java -jar security-tool.jar help
to view additional command options available to you when executing this tool
While this is optional, it is highly recommended that you now store these generated artifacts in the security
folder in a shared secure location (i.e. NOT in a source control system) preferably a secrets manager or vault. This allows you (or another operations team member with the appropriate access) to create a shared "source of truth" that correspond with the latest versions of these artifacts - this will help if and when you need to make changes and updates to the properties over time.
Audience: Architects, DevOps, SRE Teams
Stage: Environment Setup
Why: If you are using Broadleaf’s recommended Config Service (which is built on Spring Cloud Config), it is recommended that you configure that service with a Git backend allowing you to achieve a GitOps style operational setup. This allows both developers and operations teams to manage properties through source control. When these properties change, the config service can be configured to pick up these changes, propagate them, and make sure these updated properties are available to all the concerned microservices within the ecosystem
Assumptions:
You’ve authenticated with Broadleaf’s Nexus to obtain the necessary tools and artifacts as identified in the Getting Started Locally guide
You are working with an Initializr-based Broadleaf project structure containing a manifest
You plan to use Broadleaf’s Spring Cloud Config Server for distributed configuration property management
You have the ability to create a Git Repository (that is accessible to the Config Service) e.g. GitHub
READONLY
UserIt is recommended that you create at least 2 repositories, one for secure encrypted properties and one for general non-secure properties. These repositories can then be configured with the appropriate team access according to your security posture. In our examples and reference, we’ve called the the Git repos, configserver-[ENV]-secure
and configserver-[ENV]-insecure
.
You will also want to create a READONLY
user that has access to pull the properties from these corresponding repos. Take note of the credentials needed for this user as it will be set and injected into the configuration for Broadleaf’s Spring Cloud Config service.
You can pass in one or more backing Git datastore configurations for your properties using ENV
properties. If you are using the helm
charts that are generated for you using the helm:generate
plugin, you will notice that you can configure these properties using the following helm
properties (i.e. secrets.environmentRepositoryCreds.keyValuePairs.*
) that are overridable in the values.yaml
file.
Your overrides, might look something like this:
secrets: environmentRepositoryCreds: keyValuePairs: SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_TYPE: "git" SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_DEFAULT_LABEL: "develop" SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_URI: "https://github.com/MyOrg/configserver-dev-insecure" SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_USERNAME: "setme" SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_0_PASSWORD: "setme" SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_TYPE: "git" SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_DEFAULT_LABEL: "develop" SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_URI: "https://github.com/MyOrg/configserver-dev-secure" SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_USERNAME: "setme" SPRING_CLOUD_CONFIG_SERVER_COMPOSITE_1_PASSWORD: "setme"
Many source code repositories allow notification of changes to a repository based on a configured webhook. You can setup webhooks following the directions identified by your code repository provider. For example, GitHub uses a POST to any configured webhooks with a JSON body containing a list of commits. Once the webhook is configured, Spring Cloud Config natively handles the processing and consumption of those events. More details on this can be found on Spring’s official documentation.
NoteIf you choose not to enable push notification integrations, you can always manually restart your config service, which during the start-up process will pull the latest properties and configurations from the configured Git repository
Audience: DevOps, SRE Teams
Stage: Environment Setup + Maintenance
Why: If you are using Broadleaf’s recommended Config Service, you may need to add and manage additional secure properties for your implementation (outside of the default Broadleaf properties)
Assumptions:
You’ve authenticated with Broadleaf’s Nexus to obtain the necessary tools and artifacts as identified in the Getting Started Locally guide
You are working with an Initializr-based Broadleaf project structure containing a manifest
You plan to use Broadleaf’s Spring Cloud Config Server for distributed configuration property management
You’ve already generated a set of security
and config
artifacts following the recipe "Generate New Security Artifacts"
above, for each environment that you would like to update
credentials-report.env
The main mechanic for managing secure and encrypted properties within a Broadleaf ecosystem leveraging a Config Server is to manage these properties using the generated credentials-report.env
which you will find in the generated security
folder for your environment.
If you are adding/updating an existing credentials-report.env
that is already applied to an environment, you will need to pull down the original "source" (presumably stored in your Secure Vault/Secrets Manager) in order to update it. You can then add arbitrary properties to this report using a special syntax which will be described below.
Suppose you have a new secure property that you wish to be encrypted and stored in the Config Server (e.g. MY_SECRET_PROPERTY=1234
). In order to create the appropriate Spring Application context files with the encrypted values, for example:
my: secret: property: '{cipher}{key:version_1}AQCQx...'
I can run the same security-tool.jar
as mentioned in the recipe "Generate New Security Artifacts"
against my updated credentials-report.env
with my new property added.
The key difference is that (since Broadleaf does not have any knowledge of what this property is) you must define this property in your credentials-report.env
file with a special syntax that matches the following form:
[PROPERTY_KEY]_EXTENSION_[PROFILE]_[PROFILE]...
[PROPERTY_KEY]
is the fully qualified property using Spring’s Relaxed Binding underscore notation
followed by the constant _EXTENSION_
followed by one ore more [PROFILE]
's which define which microservice this specific property is applicable to
For example, let’s say the MY_SECRET_PROPERTY
that I am trying to add is only applicable to the Auth
microservice, in that case, I could add the following new plain-text property in my credentials-report.env
MY_SECRET_PROPERTY_EXTENSION_AUTH=1234
After modifying the security/credentials-report.env
with the new secret using the special syntax,
run in your current working environments working directory:
java -jar security-tool.jar encrypt
After running the encrypt
command, you should now have newly encrypted config
properties in your config/secure
directory for that environment. If you have this pointed to a backing Git repo, you can now check in these changes directly to source control.
To determine the "Profile Keys" that you can use in your property syntax, you can download the component-profiles-map.yml
which will give you a file that contains the different profiles the security tool uses when creating the appropriate encrypted configuration files.
Similar to downloading the security-tool.jar
you will want to obtain the correct version of the component-profiles-map.yml
that applies to your installation. In the manifest
directory of your main Broadleaf project, run the following Maven command:
./mvnw -q \ org.apache.maven.plugins:maven-help-plugin:3.2.0:evaluate \ -Dexpression=blc.starter.version \ -DforceStdout
Take note of the emitted version as you will use this in the next step.
Run the following Maven command in the manifest
directory of your main Broadleaf project. Take note to replace the [replace_with_correct_version]
and [/my/local/path/to/my-dev.com]
placeholders in the below command.
./mvnw org.apache.maven.plugins:maven-dependency-plugin:3.6.0:unpack \ -Dartifact=com.broadleafcommerce.microservices:broadleaf-microservices-starter-bootstrap:[replace_with_correct_version]:jar \ -DartifactId=broadleaf-microservices-starter-bootstrap \ -DgroupId=com.broadleafcommerce.microservices \ -Dincludes=component-profiles-map.yml \ -DoutputDirectory=[/my/local/path/to/my-dev.com] \ -DstripVersion=true
This should download the component-profiles-map.yml
file to your environment’s working directory. If you inspect the contents of this file, you can find the different profile keys that can be used to target specific microservice components.
Audience: DevOps, SRE Teams
Stage: Environment Setup
Why: It is common to configure Broadleaf Microservices to point to a externally managed datasource (e.g. a Cloud Managed SQL instance). The following recipe will guide you through the configuration needed to support this.
Assumptions:
You’ve authenticated with Broadleaf’s Nexus to obtain the necessary tools and artifacts as identified in the Getting Started Locally guide
You are working with an Initializr-based Broadleaf project structure containing a manifest
You plan to use Broadleaf’s Spring Cloud Config Server for distributed configuration property management
You’ve already generated a set of security
and config
artifacts following the recipe "Generate New Security Artifacts"
above, for each environment that you would like to update
This will build on top of the recipe "Encrypt New Secure Properties for Config Server"
, so please review and be familiar with that recipe before proceeding
For this recipe, we will configure and store encrypted connection info and credential information in Broadleaf’s Config Service.
To do this, we will override and re-encrypt properties on an existing credentials-report.env
and then sync that to our backing Config Service datastore (e.g. Git). Following the patterns described in the recipe "Encrypt New Secure Properties for Config Server"
, we can change and add the following properties to our security/credentials-report.env
file for my current environment’s working directory:
# Database Data BROADLEAF_DATASOURCE_PASSWORD=<set me> # Cloud Database Connection Info BROADLEAF_COMPOSITE_DATASOURCE_URL_EXTENSION_PROCESSINGFLEX_SUPPORTINGFLEX_BROWSEFLEX_CARTFLEX_DATA=jdbc:postgresql://<my-ip>:5432/broadleaf SPRING_DATASOURCE_URL_EXTENSION_AUTH_DATA=jdbc:postgresql://<my-ip>:5432/broadleaf?currentSchema=auth
To summarize:
update the property BROADLEAF_DATASOURCE_PASSWORD
to our new Cloud SQL DB instance password.
using the "Special Config Syntax", we can update the Datasource URL and target this property to the following microservice flex packages and applications: Processing
, Supporting
, Browse
, Cart
, Data
, and Auth
After modifying the credentials-report.env
with the data store information, you can now run the following Java command in your environment’s working directory:
java -jar security-tool.jar encrypt
With this run, you should now have newly encrypted config properties in your config/secure
directory which you can now sync with your Config Service
Audience: DevOps, SRE Teams
Stage: Initial Environment Setup
Why: When you’re setting up a new environment and you need to make a change to the manifest.yml
that informs the creation of new security artifacts, then you will need to run the generate
security-tool
command against the new manifest JAR in order to produce new security artifacts.
Example Use Case: You plan to deploy certain services like Kafka
to another namespace in Kuberentes (instead of default
) . If you do this, you’ll need to change/update the "alternate domains" in your manifest.yml
file which will inform the creation process of all self-signed certificates. Specifically, we may need to inform the generation process of the SAN’s fully qualified domains that need to be added to the trust stores when generating those security artifacts (i.e. all applicable Kubernetes DNS paths that may contain my new namespace).
Assumptions:
You’ve authenticated with Broadleaf’s Nexus to obtain the necessary tools and artifacts as identified in the Getting Started Locally guide
You are working with an Initializr-based Broadleaf project structure containing a manifest
You plan to use Broadleaf’s Spring Cloud Config Server for distributed configuration property management
You’ve already generated a set of security
and config
artifacts following the recipe "Generate New Security Artifacts"
above, for each environment that you would like to update
Let’s say you wish to deploy Kafka
to a new Kubernetes namespace called broadleafsupporting
instead of deploying it to the default
namespace. With this example, you will need to update the alternates
domains in your manifest.yml
to include/change the alternate domains to specify that specific k8 namespace:
- name: broker platform: kafka descriptor: snippets/docker-kafka.yml enabled: true domain: alternates: - kafkacluster-0.kafkacluster-headless.broadleafsupporting.svc.cluster.local - kafkacluster-0.kafkacluster-headless.broadleafsupporting.svc.cluster - kafkacluster-0.kafkacluster-headless.broadleafsupporting.svc - kafkacluster-0.kafkacluster-headless.broadleafsupporting - kafkacluster-0.kafkacluster-headless - kafkacluster-1.kafkacluster-headless.broadleafsupporting.svc.cluster.local - kafkacluster-1.kafkacluster-headless.broadleafsupporting.svc.cluster - kafkacluster-1.kafkacluster-headless.broadleafsupporting.svc - kafkacluster-1.kafkacluster-headless.broadleafsupporting - kafkacluster-1.kafkacluster-headless - kafkacluster-2.kafkacluster-headless.broadleafsupporting.svc.cluster.local - kafkacluster-2.kafkacluster-headless.broadleafsupporting.svc.cluster - kafkacluster-2.kafkacluster-headless.broadleafsupporting.svc - kafkacluster-2.kafkacluster-headless.broadleafsupporting - kafkacluster-2.kafkacluster-headless
Once you’ve saved your new manifest.yml
. Generate a new manifest.jar
and place the security-tool.jar
and the project’s updated manifest.jar
in the same environment working directory.
Run: java -jar security-tool.jar generate
Output: will produce updated security artifacts (e.g. kafka trustore and creds) in the security
directory of the same working directory
Importantif there are existing certificates and truststore artifacts already present in the security directory, and you want to replace them, you will need to delete the old ones first and then run generate
to generate new artifacts
Audience: Architects, DevOps, SRE Teams
Stage: Initial Environment Setup
Why: Broadleaf supports the use of "configurable seed data" through the use of Liquibase Params
. Liquibase Params
support the idea of a "one-time" initial load property mechanism that can be overriden via ENV
properties e.g. SPRING_LIQUIBASE_PARAMETERS_*
How: Broadleaf leverages the Liquibase Params
concept to support overriding seed data properties that are typically changed or updated across implementations. For example, I may want to explicitly define the bcrypt
password for the master admin (instead of using the generated password).
Passing in Liquibase Parameters replaces placeholder values that are defined in the framework’s liquibase changelog files:
<property name="blcAdminPasswordBcrypt" value="todo-set" /> <changeSet author="broadleaf" id="auth-user-7" labels="auth,required"> <preConditions onFail="MARK_RAN"> <sqlCheck expectedResult="0">SELECT COUNT(*) FROM blc_user WHERE id='-2';</sqlCheck> </preConditions> <insert tableName="blc_user"> <column name="id" value="-2" /> ... <column name="PASSWORD" value="${blcAdminPasswordBcrypt}" /> ... </insert> </changeSet>
With this convention in place, you may notice a couple pre-generated ENV
properties for you in your security/credentials-report.env
, for example:
SPRING_LIQUIBASE_PARAMETERS_BLCADMINPASSWORDBCRYPT=$2a$10... SPRING_LIQUIBASE_PARAMETERS_BLCADMIN2PASSWORDBCRYPT_EXTENSION_LIQUIBASE=$2a$10... SPRING_LIQUIBASE_PARAMETERS_BLCADMINAAAPASSWORDBCRYPT_EXTENSION_LIQUIBASE=$2a$10...
These properties can be manually changed prior to an initial installation , re-encrypted using the security-tool.jar
as mentioned in the recipe: "Encrypt New Secure Properties for Config Server"
and synced with the config server.
During an initial installation, the data
module will pull these values from the config server and run the appropriate changelogs applying any Liquibase Params
overrides as necessary to your empty datastore.
Liquibase changelogs are applied once and marked as "run" in a databasechangelog
table. This is typically done during initial installation when running the data
module (or via the job-data
helm chart in kubernetes) for the first time. The initializr required changelogs contain pre-conditions "Mark Ran as Failed" if an existing record already exists in the datastore.
What does this mean? From an operational perspective, if (as an example) you wish to change the master admin password and bcrypt value and you’ve already applied the changelog with a previously set password, then changing credentials in the credentials report and re-generating encrypted values for the config server will have NO IMPACT on the data since that record has already been inserted and applied.
Options to change a "seed data" record if already applied:
manually update the data in your datastore (ignoring what’s specified in the credentials report)
If you would like to have the credentials report and associated config server encrypted "in sync" with the values in the data store, then you will need to manually remove the database changelog and associated applied record and then re-run the data
module with the new seed property applied from the configserver.
Audience: Developers, Architects, DevOps, SRE Teams
Stage: Application Debugging
Why: It’s often times useful to understand what actual properties the Broadleaf Flex Applications are resolving and using during startup and runtime. Broadleaf provides a configurable startup report that will emit all resolved properties to help troubleshoot any property-related questions.
By default, for security purposes, the Broadleaf Environment Report is disabled, however configuring the following property for an application can turn it on for debugging purposes.
ImportantWe recommend that this be DISABLED for production environments
broadleaf: environment: report: disabled: 'false'