- v1.0.0-latest-prod
|
Important
|
Data Exchange functionality is in a pre-release stage and not generally available. The functionality as well as this documentation is subject to change. |
Broadleaf Data Exchange Services (DES) is a service developed for use by clients that need an integration point that facilitates orchestration between a caller and multiple back-end Broadleaf services, usually for the purpose of master data exchange. In other words, it takes the complexity of calling the many Broadleaf services available, and provides a single API to encapsulate the orchestration of the calls to those services. Broadleaf Commerce Microservices Edition uses multiple services to store, retrieve, and process data. These services include, for example, Catalog Service, Pricing Service, Inventory Service, and Translations (multiple services). In order to create/update a Product via the admin console, multiple services or service APIs are invoked and this orchestration can be complicated for a client. DES was designed to provide a synchronous API that accepts a rich, unified payload of data, orchestrates calls to all the requisite services (including fetching existing data, and inserting or updating data), and returns a unified response to the client. The synchronicity of this avoids the need for callbacks or webhooks to allow the calling client to determine the eventual state of the provided data that often has to be used when asynchronous messaging is used and eventual consistency is assumed.
While DES is synchronous from the client’s perspective, multi-threading is employed internally to allow parallel processing of batches of entities and their constituents. Broadleaf has made use of Java’s java.util.concurrent.CompletableFuture components to handle the orchestration and parallelization.
DES, by default, accepts a JSON payload of entities (in batch with a default max size of 50) via a REST API endpoint. Those entities are broken into constituent entities, assisted by a thread-safe component called BatchContext. Necessary services are called (by default, by RESTful web request), and coordinated by a composition of CompletableFuture. This ensures that things that must happen synchronously or in a particular order do so, and that things that can happen in parallel do so as well, and that the client’s request thread otherwise blocks and returns when all of the individual calls complete and the response has been composed.
DES is ultimately stateless in its own right. It provides pass-through calls to various services, aggregates and returns results including errors, and does not store or access any data from its own data store.
It is not expected that customer-facing or admin-facing applications will use DES. Rather, it is anticipated that this will be used by 1st and 3rd party integrations or middleware to reduce their exposure to Broadleaf’s API surface area.
Currently, DES provides two endpoints:
Create Products
Update Products
Each of these service endpoints accepts a batch (List) of ComprehensiveProduct. This ComprehensiveProduct contains references to Translations, Variants, Prices, Variant Prices, Variant Translations, Inventory, Assets, Asset Translations, etc. Each of these is based on the projection domain provided by their respective resource service. For example, ComprehensiveProduct is based on Product from Catalog Service. However, some components (e.g. Translations) have been simplified for use by the client to avoid the need to know details of Broadleaf’s internal workings.
Following is a high level diagram of the usage of DES:
When making dozens or hundreds of calls to external services in a batch context, a single error can potentially jeopardize the processing of an entire batch. This is a stateless request/response paradigm. Broadleaf makes extensive use of Spring’s RetryTemplate to ensure a best effort to recover from temporary problems such as network glitches, bursts of heavy load, or service re-starts or re-deployments. Since the effort to process each entity individually (in parallel) has been made, errors occuring during the process of any one entity will not necessarily affect the processing of another.
Since the processing of individual entities and their constituents is handled asynchronously (via multiple threads), it’s quite easy to collect individual exceptions or errors. These are assigned to their respective entities in the response. When an error occurs for a particular entity processing stops for that entity and its constituents. For example, if there is a validation exception associated with a product, no attempt will be made to process translations, variants, assets, prices, etc. As mentioned above, errors associated with one entity will not necessarily affect the processing of another.
Individual errors will be returned in a list in the response object. The data type is DataExchangeError and will have:
HTTP status code indicating the status code of the REST call to create, update, or delete a particular entity
Message indicating the info known about the exception or error, or the name of the Java exception if no message was provided
JSON path to the object that recieved the exception (e.g. $[3].variants[2].translations[0], indicating that the 1st translation in a list for the 3rd variant associated with the 4th product, where the $, or root, is the list of returned entities or ComprehensiveProduct)
|
Note
|
There are currently no rollback capabilities across entities. This means that if you successfully create or update an object, but not another object in a batch (including child objects), the successful state is persisted and maintained while the failed state is reported in the form of errors described, above. Consider making all changes in a sandbox to avoid partial success states in production. Use the DataExchangeError list to determine if errors occurred and if so, which entities were affected. |
DES components can be extended or augmented just like any other Broadleaf Microservice component. Because DES does not, by default, contain business logic (it’s an integration service) the pre-defined domain (e.g. ComprehensiveProduct, and any of its constituents such as Variant) can be extended and registered with Broadleaf’s TypeFactory or, because it’s a JSON mapping, it will accept unknown properties as part of a generic Map (using @JsonAnySetter and @JsonAnyGetter) and pass those details through to the resource service. As long as the names and structures of the extensions match those of the resource service, everything will pass through without additional work or extension.
If you extend an existing DES domain/projection (e.g. ComprehensiveProduct) and need to augment it with relationships to other, complex objects (e.g. objects that are not Strings, numbers, dates, etc.) it’s best to define the new complex objects as new domain, add those to the TypeFactory, and create an explicit mapping or relationship to the existing object. This is especially true if you need to process those entities separately (i.e. they are not simply passed through as part of the call). Otherwise, new, unmapped objects get deserialized into a Map<String,Object> and are set using the @JsonAnySetter. This means that it will get passed along to resource tier. This is fine if the resource tier expects such an object to be passed along. However, if the new object requires a separate API call, and/or the resource tier does not expect to receive such an object as a property of the existing parent, then errors or other undefined behavior may occur. For example, if you are extending ComprehensiveProduct to pass along a new field that you’ve added to Product in the Catalog Service (e.g. a field called popularity with a Double value), then no extra work is required as this new field will be accepted via the @JsonAnySetter and passed along to the Catalog Service like this: "popularity": 12.93. However, consider the following hypothetical object:
// Note that this should be registered with Broadleaf's TypeFactory
@Data
public class Foo extends EntityExchangeObject {
private String bar;
private int baz;
}This should be added as an explicit mapping to ComprehensiveProduct, for example:
// Note that this should be registered with Broadleaf's TypeFactory
@Data
public class MyComprehensiveProduct extends ComprehensiveProduct {
private List<Foo> foos;
}or like this:
// Note that this should be registered with Broadleaf's TypeFactory
@Data
public class MyComprehensiveProduct extends ComprehensiveProduct {
private Foo foo;
}By doing this, any Foo objects passed in will be registered in the BatchContext with a correlationId. You’ll be able to register errors/exceptions for each of these objects.
Presumably, Foo objects, in this case would be processed separately using a custom endpoint.This leads us to the fact that clients may wish to call other services or endpoints that are not provided out-of-box. They can do this by simply implementing the Broadleaf Spring Bean component and re-defining it through Spring configuration, which will override the default implementation. Alternatively, you may wish to implement com.broadleafcommerce.dataexchange.service.AbstractBatchItemHandler. In this case, you can simply register a FooBatchItemHandler:
public class FooBatchItemHandler<T extends Foo> extends AbstractBatchItemHandler<T,T> {
@Override
public boolean canHandle(ExchangeObject exchangeObject) {
return (exchangeObject instanceof Foo);
}
// Additional methods to implement...
// See Javadocs for com.broadleafcommerce.dataexchange.service.BatchItemHandler
}The above code will allow you to register this FooBatchItemHandler as a Spring Bean. It will be picked up and added to a list that will be used to process any Foo entities that are encountered.
|
Note
|
It is important to properly compose any additional java.util.concurrent.CompletableFuture instances so that they operate efficiently, block when necessary, and complete only when all instances complete properly. Please check Javadocs and online tutorials as these are "native" Java components. Consider implementing com.broadleafcommerce.dataexchange.service.BatchItemHandler or com.broadleafcommerce.dataexchange.service.AbstractBatchItemHandler to assist in proper composition. BatchContext is available to most methods and that can be used to save a reference to state for the duration of the execution of the batch. BatchContext is a thread-safe component for storing results, errors, and arbitrary state of a given batch. It’s an in-memory component so it does not persist any data.
|
We assume that the client will need to get an OAuth token from the Auth service prior to calling DES. We also assume that the client will pass a valid ContextInfo. This will define the use of a sandbox for updating entities. If there are failures, it will not affect production data or state. Corrections can be applied, as needed. This can be done by providing the correct ContextInfo when submitting a request to DES. In order to promote to production, additional APIs need to be invoked or an administrator needs to manually do this via the admin app.
DES uses a shared thread pool to process each of the provided entities. Since the majority of the work in this case is I/O, this should not heavily affect CPU. However, it’s important to note that this is something that can be adjusted using the property, broadleaf.dataexchange.async.taskExecutorThreadPoolSize. The default value is 10 (threads). It is recommended that DES be deployed either as part of the Processing FlexPackage or as a stand-alone service (certainly not as part of a customer-facing deployment). It’s also recommended that the Admin Gateway be responsible for routing to this service since that is not typically customer-facing.
broadleaf.dataexchange.maxBatchsize (default: 50)
Maximum top level entities (e.g. Products) in a batch
broadleaf.dataexchange.connectionTimeoutMillis (default: 5000)
Timeout in milliseconds for the web client to connect to a resource service
broadleaf.dataexchange.readTimeoutMillis (default: 2000)
Timeout in milliseconds for the web client to read from the socket
broadleaf.dataexchange.writeTimeoutMillis (default: 2000)
Timeout in milliseconds for the web client to write to the socket
broadleaf.dataexchange.async.taskExecutorQueueCapacity (default: 2000)
Capacity of the bounded queue to add tasks for asynchronous processing
broadleaf.dataexchange.async.maxQueueAddTimeMillis (default: 1000)
Maximum amount of time to wait for space to become available if the task queue is full
broadleaf.dataexchange.async.maxTries (default: 3)
Maximum number of tries on a per-call basis (e.g. 1 try and 2 retries) in the event that failures occur in calling resource services
broadleaf.dataexchange.async.initialDelayMillis (default: 10)
The initial delay, in milliseconds, before the first retry in the event that a failure occurs
broadleaf.dataexchange.async.delayMultiplier (default: 2)
The multiplier for calculating an exponential backoff time for retries
broadleaf.dataexchange.async.maxDelayMillis (default: 80)
The maximum delay time between retries when using an exponential backoff policy