Broadleaf Microservices
  • v1.0.0-latest-prod

Enhanced Environment Configuration (Since 1.7.7-GA)

Basic environment configuration is provided for the application via standard Spring practice. Yaml files in the classpath of the application serve as sources of property name to value pairs. The yaml files, based on file name convention, are further discriminated based on active Spring profile at the time of application start. Moreover, as best practice, these properties are marshalled into @ConfigurationProperties annotated beans during early application startup lifecycle. The application then injects and reads environment configuration values from these beans during startup, as well as the remainder of the application lifecycle. The topic is covered in more detail here: https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.external-config.typesafe-configuration-properties.java-bean-binding.

Building on this base, Broadleaf is providing several new features related to environment configuration that leverage existing Spring Framework functionality, as well as introduce enhancements based on Broadleaf-specific behavior.

  1. Spring Cloud Config Server support with runtime property refresh

  2. Property discrimination based on Broadleaf application and tenant context

Spring Cloud Config Server Support

Spring Cloud Config Server enhances basic environment configuration by distributing property acquisition, and is especially well suited for microservice architectures. With many microservices in play, it can be prohibitive to apply new configuration across those many services, and it is desirable to have a central source of truth for configuration from which those services can derive their configuration. This is the main purpose of Spring Cloud Config, as all microservices will now be bootstrapped to use the Spring Cloud Config instance as a new property source at startup. Furthermore, changes made at the Spring Cloud Config Server level (e.g. in a GIT repo holding the property yaml files), can trigger notification via Spring Cloud Bus to all interested microservices, causing those microservices to pull updated versions of the properties at runtime without requiring application restart.

Broadleaf now supports Spring Cloud Config Server via a set of opinionated artifacts that can be leveraged by client implementations to achieve this level of distributed configuration.

  • broadleaf-config-server-client - library used by a microservice to listen for Spring Cloud Bus notifications from the Spring Cloud Config Server instance - triggering refresh of the application environment configuration.

  • broadleaf-config-server-eventhubs - Docker image available to Broadleaf registered clients that launches a Spring Cloud Config Server instance with pre-defined dependencies for Spring Cloud Bus support of Azure EventHubs.

  • broadleaf-config-server-kafka - Docker image available to Broadleaf registered clients that launches a Spring Cloud Config Server instance with pre-defined dependencies for Spring Cloud Bus support of Apache Kafka.

  • broadleaf-config-server-kinesis - Docker image available to Broadleaf registered clients that launches a Spring Cloud Config Server instance with pre-defined dependencies for Spring Cloud Bus support of Amazon Kinesis.

  • broadleaf-config-server-pubsub - Docker image available to Broadleaf registered clients that launches a Spring Cloud Config Server instance with pre-defined dependencies for Spring Cloud Bus support of Google PubSub.

The docker images are suitable for local demonstration, as well as production deployment in Kubernetes. These are also multi-platform Docker images that support both x64 and arm64 processor architectures.

Refer to Spring’s docs for more details on the features and functionality of Spring Cloud Config: https://spring.io/projects/spring-cloud-config

Standard Setup

Most commonly, Spring Cloud Config will be backed by a GIT repository, although many other backing sources are supported. See https://docs.spring.io/spring-cloud-config/docs/current/reference/html/#_environment_repository for a comprehensive list of supported environment repositories.

General Requirements:

  • Containerized environment, such as Docker or Kubernetes

  • Working Broadleaf Microservices implementation >= release train 1.7.5-GA for the most recent Spring Cloud dependencies

  • Running message broker, such as Apache Kafka (or other if using cloud native broker).

  • Valid credentials for pulling from the Broadleaf docker registry

Application Setup

Note
Since Release Train 1.8.3 - if utilizing an Initializr-based project, this configuration is automatically configured when enabling the config component in your manifest.yml
  • Add broadleaf-config-server-client as a dependency for your project. This is as simple as declaring the dependency in your implementation’s root pom.xml.

    ...
    <dependency>
        <groupId>com.broadleafcommerce.microservices</groupId>
        <artifactId>broadleaf-config-server-client</artifactId>
        <version>1.0.0-GA</version>
    </dependency>
</dependencies>
Note
The version element is not required if you’re on release train 1.7.7-GA, or above, as broadleaf-config-server-client will already be included in Broadleaf’s common dependencies BOM.
  • In your application resources directory where your basic application.yml files exist today should be a bootstrap.yml file (create one if it doesn’t exist). This will hold the bootstrap configuration required to connect your environment initialization to the remote Spring Cloud Config Server instance.

  • Edit bootstrap.yml to include:

spring:
  cloud:
    config:
      uri: http://config:8888/config
      fail-fast: false
      retry:
        max-attempts: 50
    bus:
      enabled: true
Warning
This is an insecure configuration. Refer to the Securing The Installation section later for details on how to setup your installation in a more secure way.
  • Review all application-xxx.yml files and make sure the property spring.cloud.bus.enabled is not being set to false anywhere.

  • (Usually in application-default.yml) declare your bindings on the springCloudBus channel:

spring:
  cloud:
    stream:
      bindings:
        ...
        springCloudBusInput:
          destination: springCloudBus
        springCloudBusOutput:
          destination: springCloudBus

Config Server Setup

  • If using the standard docker-compose approach for launching supporting services, you can include Broadleaf’s config server as a container in this environment. This sample config will leverage our opinionated Kafka supported image and a git-based environment repository:

services:
  ...
  config:
    env_file:
      - common-environment.env
    environment:
      SPRING_CLOUD_CONFIG_SERVER_GIT_URI: '[git repo url]'
      SPRING_CLOUD_CONFIG_SERVER_GIT_USERNAME: [git username]
      SPRING_CLOUD_CONFIG_SERVER_GIT_PASSWORD: [git password]
      SPRING_CLOUD_CONFIG_SERVER_GIT_CLONEONSTART: true
      SPRING_CLOUD_CONFIG_SERVER_GIT_DEFAULTLABEL: master [or whatever branch to use]
    image: repository.broadleafcommerce.com:5001/broadleaf/broadleaf-config-server-kafka:1.0.0-GA
    networks:
      - kafkanet
      - backend
    ports:
      - '8888:8888'
    tty: true
  ...

Note this is running in the same network as kafka (i.e. kafkanet) and is exposed on port 8888.

Warning
This example docker-compose yaml file edit is meant to be illustrative only. You should supply environment variables for git credentials in a secure way - not declared in plaintext in a yaml file. This is generally achieved in production by injecting kubernetes secrets as environment variables for your container.
Note
This configuration can be adapted for other containerized environments (e.g. kubernetes provisioned via Helm)
  • Review the contents of common-environment.env (assuming that you have such a file). It should contain similar properties to these, and you can add them here if missing. The values should match the ports exposed by your instances of Kafka and Zookeeper running in Docker as well:

SPRING_CLOUD_STREAM_KAFKA_BINDER_BROKERS=localkafka:29092
SPRING_CLOUD_STREAM_KAFKA_BINDER_ZKNODES=zk:2181

In this document, we primarily demonstrate configuration for connectivity to Apache Kafka. For other message brokers, refer to the Spring Cloud Stream documentation for the other binders to understand the appropriate configuration for connecting to those brokers. https://spring.io/projects/spring-cloud-stream.

Application Startup

Note
Since Release Train 1.8.3 - if utilizing an Initializr-based project, this configuration is automatically configured when enabling the config component in your manifest.yml
  • In whatever fashion your instance(s) of Broadleaf microservices are started, you will need to include an additional environment variable (or system property will work as well).

    • As environment variable: SPRING_CLOUD_BOOTSTRAP_ENABLED: true

    • As system property: -Dspring.cloud.bootstrap.enabled=true

This will cause Spring to launch the application with bootstrap support so your changes in bootstrap.yml above will be consumed at startup.

Note
At this point, your application and environment are minimally configured to pull configuration from the github repository as a new property source. Furthermore, the /monitor endpoint is available on the Broadleaf config server instance. POST requests against this endpoint will engage a refresh notification over the bus to the running microservice instances according to the stipulations noted at: https://docs.spring.io/spring-cloud-config/docs/current/reference/html/#_push_notifications_and_spring_cloud_bus. For continuous delivery of environment changes, it is advantageous to hook the /monitor endpoint into the event hook mechanism of the source control system, such as GitHub webhook.

Demo

Broadleaf’s config server images come with additional spring profiles that allow the full lifecycle of environment acquisition and refresh to be demonstrated locally without requiring a remote GIT repository. The docker-compose setup for the config server should be edited as follows:

services:
  ...
  config:
    env_file:
      - common-environment.env
    environment:
      SPRING_PROFILES_ACTIVE: sample,composite
    image: repository.broadleafcommerce.com:5001/broadleaf/broadleaf-config-server-kafka:1.0.0-GA
    networks:
      - kafkanet
      - backend
    ports:
      - '8888:8888'
    tty: true
  ...

This will cause the config server to load a property file from its classpath that specifies broadleaf.cart.validation.cart.maxNumberOfUniqueCartItems to have a value of 9. If your implementation uses this validation, you should be able to launch your application (presumably on your local dev computer) and see the system prevent you at runtime from adding more than 9 items to your cart.

Refresh the value

  • Create a temporary file called application.yml and set its contents to:

broadleaf:
  cart:
    validation:
      cart:
        maxNumberOfUniqueCartItems: 3
  • We are going to transfer this file to the docker container running the Broadleaf config server. First, we need to identify the container. Execute docker ps and get the container id for the config container.

  • Copy the new application.yml file to the Broadleaf config server:

docker cp ./application.yml [container id]:/BOOT-INF/classes/clientconfigs/application.yml

In this case, the config server is monitoring the directory inside the container, so your copy command is enough to engage the refresh and does not require a call to /monitor. You should see something similar to this in your application logs:

2022-11-07 14:29:52.480  INFO 52192 --- [container-0-C-1] o.s.cloud.bus.event.RefreshListener      : Received remote refresh request.
2022-11-07 14:29:53.408  INFO 52192 --- [container-0-C-1] c.c.c.ConfigServicePropertySourceLocator : Fetching config from server at : http://localhost:8888/config
2022-11-07 14:29:53.440  INFO 52192 --- [container-0-C-1] c.c.c.ConfigServicePropertySourceLocator : Located environment: name=application, profiles=[default], label=null, version=null, state=null
2022-11-07 14:29:53.440  INFO 52192 --- [container-0-C-1] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-configClient'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/clientconfigs/application.yml'}]
2022-11-07 14:29:53.608  INFO 52192 --- [container-0-C-1] o.s.boot.SpringApplication               : No active profile set, falling back to 1 default profile: "default"
2022-11-07 14:29:53.727  INFO 52192 --- [container-0-C-1] o.s.boot.SpringApplication               : Started application in 1.231 seconds (JVM running for 315.606)
2022-11-07 14:30:00.536  INFO 52192 --- [nio-8447-exec-9] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2022-11-07 14:30:00.541  INFO 52192 --- [nio-8447-exec-9] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2022-11-07 14:30:00.555  INFO 52192 --- [nio-8447-exec-9] o.s.web.servlet.DispatcherServlet        : Completed initialization in 14 ms
2022-11-07 14:30:01.162  INFO 52192 --- [container-0-C-1] o.s.cloud.bus.event.RefreshListener      : Keys refreshed [broadleaf.cart.validation.cart.maxNumberOfUniqueCartItems]

At this point, if you repeat the cart item use case, the system should now block you if you attempt to add more than 3 items to a cart, in accordance with the new property value.

Securing The Installation

The setup so far has not taken security into account. The security related to config server falls primarily into 3 areas:

  1. HTTPS - communication with the config server over the network should utilize TLS.

  2. Authentication - calling the config server API should require an authenticated call (e.g. basic authentication)

  3. Encryption - Some properties are sensitive and should be stored encrypted at rest. This is the most prevalent when storing sensitive properties in property files in source control.

Securing the installation starts with activating the secure spring profile on the container via the SPRING_PROFILES_ACTIVE environment variable. However, some additional setup is needed to fulfill the requirements of enacting this profile.

  1. username and password for basic auth

  2. keystore file and password to use for TLS

  3. keystore file, password, and key alias to use for encryption/decryption

services:
  ...
  config:
    env_file:
      - common-environment.env
    environment:
      SPRING_PROFILES_ACTIVE: secure
      SPRING_SECURITY_USER_NAME: [config server user name]
      SPRING_SECURITY_USER_PASSWORD: [config server user password]
      SERVER_SSL_KEYSTORE: '/var/upload/tls.keystore'
      SERVER_SSL_KEYSTOREPASSWORD: [TLS keystore password]
      ENCRYPT_KEYSTORE_LOCATION: '/var/upload/decrypt.keystore'
      ENCRYPT_KEYSTORE_PASSWORD: [decrypt keystore password]
      ENCRYPT_KEYSTORE_ALIAS: [decrypt keystore alias to key]
      SPRING_CLOUD_CONFIG_SERVER_GIT_URI: '[git repo url]'
      SPRING_CLOUD_CONFIG_SERVER_GIT_USERNAME: [git username]
      SPRING_CLOUD_CONFIG_SERVER_GIT_PASSWORD: [git password]
      SPRING_CLOUD_CONFIG_SERVER_GIT_CLONEONSTART: true
      SPRING_CLOUD_CONFIG_SERVER_GIT_DEFAULTLABEL: master [or whatever branch to use]
    image: repository.broadleafcommerce.com:5001/broadleaf/broadleaf-config-server-kafka:1.0.0-GA
    networks:
      - kafkanet
      - backend
    ports:
      - '8888:8888'
    tty: true
    volumes:
      - ${PWD}/tls.keystore:/var/upload/tls.keystore
      - ${PWD}/decrypt.keystore:/var/upload/decrypt.keystore
  ...

To speak to the secured installation, the client side configuration must also be updated. Building on the bootstrap.yml example from above:

spring:
  cloud:
    config:
      uri: https://localhost:8888/config
      username: [config server user name]
      password: [config server user password]
      fail-fast: false
      retry:
        max-attempts: 50
    bus:
      enabled: true

The uri has changed to an https url, and we have included the username and password for the config server authentication.

Warning
These example yaml file edits are meant to be illustrative only. You should supply environment variables for credentials in a secure way - not declared in plaintext in a yaml file. This is generally achieved in production by injecting kubernetes secrets as environment variables for your container.
Note
Please refer to https://docs.spring.io/spring-cloud-config/docs/current/reference/html/#_encryption_and_decryption for detailed instructions on key generation, encrypting properties using the key, etc…​ While generating your own keystore for encryption/decryption makes sense, doing the same (self-signed cert) does not usually make sense for TLS. Rather, a cert from a trusted authority is preferable. It is difficult to cause the spring cloud config client to ignore cert validation, and you also lose the ability for the source control webhook to call the `/monitor' endpoint on the config server without an appropriate cert.

Additional Customization

If the opinionated Broadleaf config server Docker images available do not meet your needs, or you require additional code customization measures to achieve a specialized goal, it is possible to start from a template project, customize, and build your own Docker image. Each of the 4 broker types Broadleaf supports out-of-the-box has a zip assembly of the original source for the project available from Broadleaf maven repository. Each can be downloaded at one of the following locations (requires registered client maven repository credentials):

Kafka

https://repository.broadleafcommerce.com/repository/microservices/com/broadleafcommerce/microservices/broadleaf-config-server-kafka/1.0.0-GA/broadleaf-config-server-kafka-1.0.0-GA.zip

EventHubs

https://repository.broadleafcommerce.com/repository/microservices/com/broadleafcommerce/microservices/broadleaf-config-server-eventhubs/1.0.0-GA/broadleaf-config-server-eventhubs-1.0.0-GA.zip

Kinesis

https://repository.broadleafcommerce.com/repository/microservices/com/broadleafcommerce/microservices/broadleaf-config-server-kinesis/1.0.0-GA/broadleaf-config-server-kinesis-1.0.0-GA.zip

PubSub

https://repository.broadleafcommerce.com/repository/microservices/com/broadleafcommerce/microservices/broadleaf-config-server-pubsub/1.0.0-GA/broadleaf-config-server-pubsub-1.0.0-GA.zip

Each project, when unzipped, can be edited to add/change dependencies, customize code or Spring behavior, etc…​ A New Docker image can be created by executing the following command in the root directory of the project. This image will be deployed to your local Docker registry and is suitable for local testing.

mvn clean install -Pdocker -DbuildLocalDockerOnly=true

If you generate a new image as part of CI (or other), including the -Pdocker profile and registry url (and omitting the buildLocalDockerOnly system property) will cause an appropriate Docker image to be deployed to your target docker registry during a maven deploy or release (as long as the process is properly authenticated to the remote docker registry).

mvn deploy -Pdocker -DdockerDeploymentRepository=my.docker.registry:5001
Note
Broadleaf provides Spring Cloud Config Server implementations for several different broker types as a convenience. While broadleaf-config-server-client library is a requirement for the e-commerce application because of lower-level Spring Cloud Stream/Bus specifics, the Broadleaf config server portion is not required and any valid Spring Cloud Config Server instance is applicable. This opens the door for using a Broadleaf implementation with an existing Spring Cloud Config Server (rather than rolling a new one from scratch).

Caveats

When the application Spring environment is refreshed, any properties accessed from the environment directly, as well as properties accessed from @ConfigurationProperties annotated beans, are immediately updated with the latest values. However, other components controlled by Spring are not automatically refreshed (unless annotated with @RefreshScope). This means other components leveraging environment properties at initialization, but not during normal request flows, will not benefit from the environment refresh (since they no longer user the properties post-initialization). This represents a subset of the available environment properties in the system. For example, the property controlling connection pool size is only used during connection pool creation, and not thereafter. It is useful to note this limitation when considering what properties will actually be refreshed in a discernible way upon change. On the other hand, upon application restart, all components will be initialized with the latest. Therefore, restarting the application is one strategy to realize configuration update without having to re-deploy the application.

Broadleaf does not employ @RefreshScope in the framework on any beans. It is hard to predict the consequences of the destroy/initialize bean lifecycle on the runtime performance of the application. For example, some beans control resources like async thread pools that take some amount of time to shutdown. And, since all request threads that need to access the API of the refreshing component will block until it finishes initialization, there is a possibility of negative resource suffocation consequences, like Tomcat server request thread or DB connection pool starvation.

In the event a property is only used during bean initialization, and is required for behavior change at runtime refresh, there are several strategies that can be employed:

  1. Override the Broadleaf bean and override the flow where the altered property should be used and change the behavior to leverage the changed property.

  2. Or, override the Broadleaf bean and add the @RefreshScope annotation to force Spring to re-initialize the bean on environment refresh.

(1) is always preferable to (2).

Property Discrimination

In addition to distributed property resolution through Spring Cloud Config, Broadleaf also offers enhanced property discrimination based on Broadleaf tenant context. In a given request flow, the HTTP request header is consulted for Broadleaf context request information, including tenant and application id. This information is made available to the application via DiscriminatedPropertyContext. Moreover, Broadleaf will create a proxy for @ConfigurationProperties annotated beans at startup and cause any calls to the bean’s accessor API to check with the environment (rather than the POJO’s value for the field) for qualified properties based on the context using the following precedence:

  1. Check for Application discriminated property first

  2. Check for Tenant discriminated property second

  3. Check for undiscriminated property last

In order to leverage discrimination, property names should use the following syntax. While discriminated properties certainly may be declared at the Broadleaf config server level, they need not be. Property discrimination can be entirely leveraged without any distribution of the configuration environment.

Application Property

my.property.prefix.application.[application id].name

Tenant Property

my.property.prefix.tenant.[tenant id].name

Undiscriminated Property

my.property.prefix.name

Currently, the DiscriminatedPropertyContext is setup in an HTTP filter, and since discrimination relies on this context, only REST endpoint flows currently benefit. However, any flow that can be enhanced with DiscriminatedPropertyContext can theoretically take advantage of discrimination.

Note
See /shared-concepts/multi-tenancy for more information on tenant and application concepts.

Standard Setup

The feature requires a Broadleaf Microservices implementation leveraging at least version 1.4.13-GA of the broadleaf-common-extension library. Note, this is included as part of the 1.7.7-GA release train. However, earlier release trains >= 1.7.5-GA can explicitly declare the broadleaf-common-extension dependency at version 1.4.13-GA to realize the feature without upgrading overall to 1.7.7-GA.

The feature is disabled by default. Enable the @ConfigurationProperties annotated bean proxy behavior by setting the property com.broadleafcommerce.environment.property.discrimination.enabled to true.

The application of the proxy behavior is further limited by inclusion/exclusion filters. Out-of-the-box, Broadleaf only includes Broadleaf scoped @ConfigurationProperties beans (ignoring Spring and other third party library configuration beans). To turn off the default Broadleaf scoping and include everything, set the com.broadleafcommerce.environment.property.discrimination.broadleaf.filters.enabled property to false. Most of the time, this is not necessary and the implementation can register additional inclusions/exclusions by adding new filter beans (see below).

Fine Tuning Inclusion/Exclusion

For inclusion filtering, declare a bean in your Spring @Configuration class of type com.broadleafcommerce.common.extension.environment.DiscriminatedPropertyIncludeFilterSupplier. For the opposite effect, declare a bean of type com.broadleafcommerce.common.extension.environment.DiscriminatedPropertyExcludeFilterSupplier. The bean should return a Regex pattern to match the desired portion of the property prefix. The regex need only match a portion of the property name, but should not try to match more than what is exposed by the prefix param of the @ConfigurationProperties annotation. For example, given the following @ConfigurationProperties bean:

@ConfigurationProperties("broadleaf.notification.account-invite")
@Data
public class AccountNotificationProperties {

    private String domainPattern = "https://%s:8456";

    ...

A valid inclusion declaration for this @ConfigurationProperties bean would be:

@Bean
public DiscriminatedPropertyIncludeFilterSupplier inclusionFilterSupplier() {
    return () -> "^broadleaf.notification";
}

This is valid, since it matches some portion of the prefix param of the @ConfigurationProperties annotation, but not beyond. On the other hand, this would NOT be valid:

@Bean
public DiscriminatedPropertyIncludeFilterSupplier inclusionFilterSupplier() {
    return () -> "^broadleaf.notification.account-invite.domainPattern";
}

Caveats

Discrimination requires the DiscriminatedPropertyContext be set. In cases where this is not set, discrimination will not be active and the undiscriminated property will be returned (even if the @ConfigurationProperties bean has been proxied). The most notable flow where this will occur is during async message consumption, since tenant information is currently not set here. For message driven flows that require tenant property discrimination, extension of Broadleaf to modify the message creation and consumption is currently required. Generally, this pattern would be followed:

  1. Extend Broadleaf where the message is created in order to set tenant and application id values on the message header. Note, mileage will vary, as tenant information may not always be known or available at this point.

  2. Extend Broadleaf where the message is consumed. Review the message header, if applicable, and set the DiscriminatedPropertyContext.

It is often useful to visualize the properties actually discriminated in a request flow at runtime. This can be used as part of troubleshooting @ConfigurationProperties bean proxy behavior, or as part of determining if a discrminated property is being used. Setting the com.broadleafcommerce.environment.property.discrimination.report.count.enabled property to true will enable reporting of all discriminated properties resolved during a request thread, and will do so at the end of the request lifecycle in the standard application log. A sample of the output looks like this:

2022-11-07 21:39:19.953  INFO 93872 --- [nio-8447-exec-3] c.b.c.e.e.DiscriminatedPropertyContext   : Reporting all property keys and access count in the request lifecycle using DefaultDiscriminatedPropertyAccessor for request POST : /carts
key: broadleaf.cart.notifyAsync | count: 2
key: broadleaf.cart.validation.cart.application.2.maxNumberOfUniqueCartItems | count: 2
key: broadleaf.cart.validation.cart.shouldLimitNumberOfUniqueCartItems | count: 2
key: broadleaf.vendor-authentication-privilege.ignoreVendorRestrictionInApplicationContext | count: 4

2022-11-07 21:39:19.959  INFO 93872 --- [nio-8447-exec-9] c.b.c.e.e.DiscriminatedPropertyContext   : Reporting all property keys and access count in the request lifecycle using DefaultDiscriminatedPropertyAccessor for request POST : /cart
key: broadleaf.cartoperation.cartprovider.cartsUri | count: 2
key: broadleaf.cartoperation.cartprovider.serviceClient | count: 2
key: broadleaf.cartoperation.cartprovider.url | count: 2
key: broadleaf.cartoperation.catalogprovider.fieldMappingUri | count: 2
key: broadleaf.cartoperation.catalogprovider.productsUri | count: 2
key: broadleaf.cartoperation.catalogprovider.serviceClient | count: 3
key: broadleaf.cartoperation.catalogprovider.url | count: 3
key: broadleaf.cartoperation.mapping.cartitem.productMappings | count: 2
key: broadleaf.cartoperation.mapping.cartitem.variantMappings | count: 2
key: broadleaf.cartoperation.offerprovider.applyUri | count: 2
key: broadleaf.cartoperation.offerprovider.serviceClient | count: 2
key: broadleaf.cartoperation.offerprovider.url | count: 2
key: broadleaf.cartoperation.paymentprovider.paymentsUri | count: 2
key: broadleaf.cartoperation.paymentprovider.serviceClient | count: 2
key: broadleaf.cartoperation.paymentprovider.url | count: 2
key: broadleaf.cartoperation.pricingprovider.priceInfosUri | count: 3
key: broadleaf.cartoperation.pricingprovider.serviceClient | count: 3
key: broadleaf.cartoperation.pricingprovider.url | count: 3