Broadleaf Microservices
  • v1.0.0-latest-prod

Adding a New Export Implementation

You can customize any microservice to use the Broadleaf Common Data Export Library and add an export target implementation for any specific data type.

For a high-level overview of the Broadleaf Data Export Common Library and its components, visit this documentation.

This tutorial will demonstrate each step for adding a new implementation of this library.

Initial Setup

Important
The following steps are only needed for a service that does not have an Out-Of-Box dependency of Data Export.

Adding the Common Data Export dependency

To be able to use the Data Export library, add a new dependency in the pom.xml file of the intended microservice:

<dependency>
    <groupId>com.broadleafcommerce.microservices</groupId>
    <artifactId>broadleaf-common-dataexport</artifactId>
    <!--Replace this with the latest version as necessary -->
    <version>2.0.1-GA</version>
</dependency>

Extending the JpaAutoConfiguration

Since the export data concept is being introduced to the service, add a new DemoOrderJpaAutoConfiguration that will run before the original OrderJpaAutoConfiguration. Its purpose is to specify the ExportDataRouteSupporting.class as a supportingRouteType.

@Configuration
@AutoConfigureBefore(OrderJpaAutoConfiguration.class)
@JpaDataRoute(boundPropertiesType = OrderProperties.class,
        routePackage = ORDER_ROUTE_PACKAGE, routeKey = ORDER_ROUTE_KEY,
        supportingRouteTypes = {
                // Include all the routes from OrderJpaAutoConfiguration
                TrackingDataRouteSupporting.class,
                MessagingDataRouteSupporting.class,
                ApplicationDataRouteSupporting.class,
                // Add the route for export
                ExportDataRouteSupporting.class})
public class DemoOrderJpaAutoConfiguration {}
Important
Make sure you use the @AutoConfigureBefore annotation.
Tip
To learn more about supporting route types, visit the Data Routing Documentation

Schema Changes to support Data Export

Once the broadleaf-common-dataexport library is added as a dependency in the project, it will introduce new tables into the database. To add the changesets responsible for updating the database schema, run UtilitiesIT. After doing so, verify that new changesets are added to your DB changelog, including items such as creating the blc_export table.

Specification

Define a ExportSpecification that maps the desired fields to their designated headers in the export document.

public class OrderExportSpecification implements ExportSpecification {

    @Getter
    private final LinkedHashMap<String, String> fieldConfigMap = new LinkedHashMap<>();

    public OrderExportSpecification() {
        initSpecification();
    }

    @Override
    public void initSpecification() {
        getFieldConfigMap().put(Fields.NAME, Headers.NAME);
        getFieldConfigMap().put(Fields.CART_TYPE, Headers.CART_TYPE);
        getFieldConfigMap().put(Fields.STATUS, Headers.STATUS);
        getFieldConfigMap().put(Fields.CUSTOMER_ID, Headers.CUSTOMER_ID);
        getFieldConfigMap().put(Fields.ORDER_NUMBER, Headers.ORDER_NUMBER);
        getFieldConfigMap().put(Fields.SUBMIT_DATE, Headers.SUBMIT_DATE);
        getFieldConfigMap().put(Fields.ORDER_ITEMS, Headers.ORDER_ITEMS);
        getFieldConfigMap().put(Fields.ATTRIBUTES, Headers.ATTRIBUTES);
    }

    @UtilityClass
    public static class Headers {
        public static final String NAME = "Name";
        public static final String CART_TYPE = "Cart Type";
        public static final String STATUS = "Status";
        public static final String CUSTOMER_ID = "Customer ID";
        public static final String ORDER_NUMBER = "orderNumber";
        public static final String SUBMIT_DATE = "submitDate";
        public static final String ORDER_ITEMS = "orderItems";
        public static final String ATTRIBUTES = "attributes";
    }

    @UtilityClass
    public static class Fields {
        public static final String NAME = "name";
        public static final String CART_TYPE = "cartType";
        public static final String STATUS = "status";
        public static final String CUSTOMER_ID = "customerId";
        public static final String ORDER_NUMBER = "orderNumber";
        public static final String SUBMIT_DATE = "submitDate";
        public static final String ORDER_ITEMS = "orderItems";
        public static final String ATTRIBUTES = "attributes";
    }

    @Override
    public boolean canModifyBeanSerializer(Class<?> beanClass) {
        return Order.class.isAssignableFrom(beanClass)
                || Phone.class.isAssignableFrom(beanClass);
    }
}

Converter

Add an extension of AbstractExportRowConverter to process each row for the designated record.

public class OrderRowConverter extends AbstractExportRowConverter<Order> {

    public OrderRowConverter(ExportSpecification specification, ObjectMapper baseMapper) {
        super(specification, baseMapper);
    }

    @Override
    public Map<String, String> convert(Order source) {
        Map<String, String> result = getMapper().convertValue(source,
                new TypeReference<Map<String, String>>() {});
        Optional.ofNullable(source.getAttributes())
                .ifPresent(attribute -> {
                    if (!attribute.isEmpty()) {
                        result.putAll(getMapper().convertValue(attribute,
                                new TypeReference<Map<String, String>>() {}));
                    }
                });

        return result;
    }

    @Override
    protected SimpleModule getSpecificationExportModule(ExportSpecification specification) {
        SimpleModule simpleModule = super.getSpecificationExportModule(specification);
        // The serializer added here is defined below
        simpleModule.addSerializer(new OrderAttributesExportSerializer());

        return simpleModule;
    }
}

Complex Attributes Serializer

If necessary, add an extension of Jackson’s StdSerializer in order to serialize complex fields.

public class OrderAttributesExportSerializer
        extends StdSerializer<Map<String, Object>> {
    private final String separator;
    private final String valueSeparator;
    private final String keyValueSeparator;

    /**
     * Construct the serializer with default separators.
     */
    public OrderAttributesExportSerializer() {
        this("|", ",", ":");
    }

    public OrderAttributesExportSerializer(String separator,
            String valueSeparator,
            String keyValueSeparator) {
        super(Map.class, false);
        this.separator = separator;
        this.valueSeparator = valueSeparator;
        this.keyValueSeparator = keyValueSeparator;
    }

    @Override
    @SuppressWarnings("unchecked")
    public void serialize(Map<String, Object> value, JsonGenerator gen, SerializerProvider provider)
            throws IOException {
        if (value != null) {
            String result = value.entrySet().stream()
                    .map(entry -> {
                        Object attrValue = entry.getValue();
                        String stringValue;
                        if (attrValue instanceof Collection) {
                            stringValue = ((Collection<?>) attrValue)
                                    .stream()
                                    .map(String::valueOf)
                                    .collect(Collectors.joining(valueSeparator));
                        } else {
                            stringValue = String.valueOf(attrValue);
                        }
                        return String.join(keyValueSeparator, entry.getKey(), stringValue);
                    }).collect(Collectors.joining(separator));
            gen.writeString(result);
        }
    }
}

Export Processor

Define a DefaultExportTarget to be utilized by the Processor, and later used to specify the type of export when initiated by the endpoint.

public enum DefaultExportTarget {
    ORDER
}

Define an OrderExportProcessor that implements ExportProcessor and overrides its methods. This component is responsible for processing the Export to be generated from the request, and it may be extended with additional logic for a custom export.

@CommonsLog
@RequiredArgsConstructor
@DataRouteByExample(Order.class)
public class OrderExportProcessor implements ExportProcessor<Order> {

    private final OrderService<Order> orderService;
    private final TypeFactory typeFactory;
    private final FilterParser<Node> filterParser;
    private final OrderRowConverter orderRowConverter;
    private final ContextRequestHydrator hydrator;

    @Override
    public boolean canHandle(Export export) {
        return StringUtils.equals(DefaultExportTarget.ORDER.name(),
                export.getTarget());
    }

    @Override
    public LinkedHashSet<String> getHeaders() {
        return orderRowConverter.getHeaders();
    }

    @Override
    public ReadRecordsResponse<Order> readRecordsToProcess(Export export) {
        ContextInfo contextInfo = buildQueryContextInfo(export);

        Node filters = getFilters(export);
        final Stream<Order> result;

        // If the target implementation is intended to support Export#inclusions and Export#exclusions, filtration on those should be handled here. This example does not support those filters.
        filters = filters == null ? new EmptyNode() : filters;

        // For demonstration purposes in this tutorial, we will utilize the following naive implementation that will eagerly load all records into memory
        result = orderService.readAll(filters, contextInfo).stream();

        return ReadRecordsResponse.success(result);
    }

    @Override
    public RowGenerationResponse generateRows(List<Order> batchToProcess, Export export) {
        return RowGenerationResponse.success(batchToProcess.stream()
                .map(orderRowConverter::convert).collect(Collectors.toList()));
    }

    protected Node getFilters(Export export) {
        if (StringUtils.isBlank(export.getFilterString())) {
            return null;
        }

        return filterParser.parse(export.getFilterString());
    }

    protected ContextInfo buildQueryContextInfo(Export export) {
        ContextRequest contextRequest = hydrator.hydrate(buildContextRequestFromExport(export));
        if (StringUtils.isEmpty(contextRequest.getCustomerContextId())) {
            contextRequest.setCustomerContextId(contextRequest.getTenantId());
        }
        ContextInfo contextInfo = new ContextInfo(OperationType.READ);
        contextInfo.setAuthor(export.getAuthor());
        contextInfo.setContextRequest(contextRequest);

        return contextInfo;
    }

    protected ContextRequest buildContextRequestFromExport(Export export) {
        ContextRequest contextRequest = typeFactory.get(ContextRequest.class);
        contextRequest.setApplicationId(export.getExportingApplicationId());
        contextRequest.setTenantId(export.getTenantId());
        contextRequest.setSandboxId(export.getExportingSandboxId());
        contextRequest.setCatalogId(export.getExportingCatalogId());
        contextRequest.setTenantId(export.getTenantId());
        contextRequest.setCustomerContextId(export.getExportingCustomerContextId());
        return contextRequest;
    }
}
Important
A production implementation should instead leverage a query method that will actually stream records incrementally from the datastore. For examples of this, see BroadleafPagingStreams#streamBuilder from the DataTracking library.
Tip
Visit the Export domain’s inclusions and exclusions Javadocs for more information

Spring Configuration

In order services, add a new AutoConfiguration class to register the OrderExportSpecification, OrderRowConverter, and OrderExportProcessor beans.

@Configuration
public class DemoOrderExportAutoConfiguration {

    @Configuration
    public static class Specifications {

        @Bean
        public OrderExportSpecification orderExportSpecification() {
            return new OrderExportSpecification();
        }
    }

    @Configuration
    public static class Converters {

        @Bean
        public OrderRowConverter orderRowConverter(
                OrderExportSpecification orderExportSpecification,
                ObjectMapper objectMapper) {
            return new OrderRowConverter(orderExportSpecification, objectMapper);
        }
    }

    @Bean
    public OrderExportProcessor orderExportProcessor(
            OrderService<Order> orderService,
            TypeFactory typeFactory,
            FilterParser<Node> filterParser,
            OrderRowConverter orderRowConverter,
            ContextRequestHydrator hydrator) {
        return new OrderExportProcessor(orderService,
                typeFactory,
                filterParser,
                orderRowConverter,
                hydrator);
    }
}
Note
You may need to enable your auto-configuration class via spring.factories as discussed in this documentation.

Export Endpoint

Add a custom endpoint and inject ExportManager, ExportService, and ExportDownloadService. For this example, add a GET endpoint to read an existing export record. Also, add a POST endpoint to generate a new record. This endpoint is responsible for passing an exportRequest into exportManager.initiateExport() to trigger the workflow in the Export Common Library. Finally, add a GET endpoint that will call the ExportDownloadService to retrieve an already existing export record as a file.

@RestController
@DataRouteByExample(Order.class)
@RequiredArgsConstructor
public class OrderExportEndpoint {

    private final ExportManager exportManager;
    private final ExportService<Export> exportService;
    private final ExportDownloadService exportDownloadService;

    @GetMapping("/exports/orders/{id}")
    public ResponseEntity<?> readExportById(
            @PathVariable(value = "id") String exportId,
            @ContextOperation(OperationType.READ) ContextInfo contextInfo) {
        Export export = exportService.readById(exportId);
        return ResponseEntity.ok()
                .body(export);
    }

    @PostMapping(value = "/exports/orders",
            consumes = MediaType.APPLICATION_JSON_VALUE)
    public Export exportOrders(
            @RequestBody ExportRequest exportRequest,
            @ContextOperation(value = OperationType.READ) ContextInfo context) {
        return exportManager.initiateExport(exportRequest,
                DefaultExportTarget.ORDER.name(), context);
    }

    @GetMapping("/exports/orders/{id}/download")
    public ResponseEntity<StreamingResponseBody> downloadOrderExport(
            @PathVariable("id") String exportId,
            @ContextOperation(value = OperationType.READ) ContextInfo context,
            HttpServletResponse response) {
        Export export = exportService.readById(exportId);
        if (!CommonExportUtils.isExportBelongsToCurrentApplication(export, context) ||
                !CommonExportUtils.isExportFinished(export)) {
            return ResponseEntity.notFound().build();
        }

        Optional<Resource> exportFile = exportDownloadService.getExportFile(exportId);
        response.setCharacterEncoding(StandardCharsets.UTF_8.name());
        return ResponseEntity.ok()
                .contentType(MediaType.TEXT_PLAIN)
                .header(HttpHeaders.CONTENT_DISPOSITION,
                        String.format("attachment; filename=%s",
                                CommonExportUtils.getUserFriendlyFilenameForExport(export)))
                .body(outputStream -> IOUtils.copy(exportFile.get().getInputStream(),
                        outputStream));
    }
}

The above demonstrates fundamental use of the export library. In your implementation, you may want to add other endpoints to read a filtered collection of exports, initiate an export for a specific list of entities, return a paged response, implement custom permissions, etc.

Note

As of Export Common 2.0.1, the old exportManager.initiateExport() method that obtains the export filter string from the endpoint’s request parameter was deprecated. The export filter string is now expected to be a part of exportRequest which is the request body. If you wish to run with an older version of the Broadleaf Data Export Common Library and/or use the parameter based approach, the endpoint to initiate exports should include the filterString request parameter as shown below, with matching implementations in the Metadata that shall be discussed further in this tutorial.

    @PostMapping(value = "/exports/orders",
            consumes = MediaType.APPLICATION_JSON_VALUE)
    public Export exportOrders(
            @RequestParam(value = "cq", defaultValue = "", required = false) String filterString,
            @RequestBody ExportRequest exportRequest,
            @ContextOperation(value = OperationType.READ) ContextInfo context) {
        return exportManager.initiateExport(exportRequest,
                DefaultExportTarget.ORDER.name(), filterString, context);
    }

Web Auto Configuration

Register the endpoint adding a configuration class that calls EnableBroadleafControllers with the @ComponentScan annotation specifying the new endpoint class.

@Configuration
@ComponentScan(basePackageClasses = OrderExportEndpoint.class)
@EnableConfigurationProperties(OrderWebProperties.class)
@AutoConfigureAfter(OrderWebAutoConfiguration.class)
public class DemoOrderWebAutoConfiguration {}
Note
You may need to enable your auto-configuration class via spring.factories as discussed in this documentation.

Messaging

Add the following properties to your flex package’s configuration file (e.g. min’s application-defaults.yml).

spring:
  cloud:
    stream:
      bindings:
        processExportRequestInput:
          group: order-export
          destination: exportRequest
        processExportRequestOutput:
          destination: exportRequest
        cloneProductOutput:
          destination: cloneProduct
broadleaf:
  export:
    process-export-request:
      retry:
        retry-cluster-service-namespace: process-export-request-order

Expose the Endpoint in the Admin using Metadata

For this tutorial, we will extend the already existing Orders grid in the admin using metadata.

To do this, find the augmentation key for the view (in this case, order:orders:browse), and get the component you wish to customize (in this case, the default grid). Last, configure the export modal by calling .exportGridAction() as seen below:

@Configuration
@RequiredArgsConstructor
@EnableConfigurationProperties(OrderMetadataProperties.class)
@AutoConfigureAfter(OrderServicesMetadataAutoConfiguration.class)
public class OrderMetadataAutoConfiguration {
    @UtilityClass
    public class OrderProps {
        public final String ORDER_AUG_KEY = "order:orders:browse";
        public final String ORDER_EXPORTS = "/order/exports/orders";
        public final String ORDER_EXPORT = ORDER_EXPORTS + "/${exportId}";
        public final String ORDER_EXPORT_DOWNLOAD = "/api" + ORDER_EXPORT + "/download";
    }

    @Bean
    public ComponentSource addExportImplementationToGrid() {
        return registry -> {
            EntityBrowseView<?> browseView =
                    (EntityBrowseView<?>) registry.get(OrderProps.ORDER_AUG_KEY);

            EntityGridView<?> grid = browseView.getDefaultGrid();

            grid.exportGridAction(CustomerScopes.CUSTOMER, exportAction -> exportAction
                    .startExportEndpoint(startEndpoint -> startEndpoint
                            .uri(OrderProps.ORDER_EXPORTS))
                    .readExportEndpoint(readEndpoint -> readEndpoint
                            .uri(OrderProps.ORDER_EXPORT))
                    .downloadExportUri(OrderProps.ORDER_EXPORT_DOWNLOAD)
                    .nameField("ORDER")
                    .label("Export Orders"));
        };
    }
}
Note

As of Export Common 2.0.1, the old exportManager.initiateExport() method that obtains the export filter string from the endpoint’s request parameter was deprecated. The export filter string is now expected to be a part of exportRequest which is the request body. Following this flow, as of Metadata Common 2.0.4, there is a passFilterStringAsParam attribute in ExportGridAction that is set to false by default when using an ExportGrid. This attribute dictates if the filter string is sent as a request parameter (true) or as part of the exportRequest request body (false) when initiating an export. If you wish to run with an older version of the Broadleaf Data Export Common Library and/or use the parameter based approach, the Metadata configuration for the ExportGridAction should include the setting of the passFilterStringAsParam attribute to true, as seen below.

    @Bean
    public ComponentSource addExportImplementationToGrid() {
        return registry -> {
            EntityBrowseView<?> browseView =
                    (EntityBrowseView<?>) registry.get(OrderProps.ORDER_AUG_KEY);

            EntityGridView<?> grid = browseView.getDefaultGrid();

            grid.exportGridAction(CustomerScopes.CUSTOMER, exportAction -> exportAction
                    .startExportEndpoint(startEndpoint -> startEndpoint
                            .uri(OrderProps.ORDER_EXPORTS))
                    .readExportEndpoint(readEndpoint -> readEndpoint
                            .uri(OrderProps.ORDER_EXPORT))
                    .downloadExportUri(OrderProps.ORDER_EXPORT_DOWNLOAD)
                    .passFilterStringAsParam(true)
                    .nameField("ORDER")
                    .label("Export Orders"));
        };
    }
Note
You may need to enable your auto-configuration class via spring.factories as discussed in this documentation.
Tip
Follow these tutorials to learn more about how to customize the admin using Metadata.

Testing in the Broadleaf Admin

Let’s test out our new export in the admin.

  • Visit the Orders page. You will see the newly added export button in the grid.

New Export Button
  • Click on 'Export'. You will see the export orders modal, where you may overwrite the pre-populated file name field and initiate a new export.

Export Orders Modal
  • Click on 'Submit'.

  • Once the export has been created successfully, you will see two notifications. Download the export file.

Download File
Tip
This link hits the previously added downloadOrderExport endpoint. You may customize this behavior to fit your needs (e.g. adding a page of exports with a download option for each record).
  • Take a look at the CSV file generated from your orders!

CSV File Preview