Broadleaf Microservices
  • v1.0

Adding a Custom Import Batch Handler

Many clients use the Import Service primarily for its baseline importing pipeline and in most cases will customize the actual ingestion schemas to match the complexity of the product feeds coming in from their source system.

In this tutorial, we’ll walk through creating a custom Import Batch Handler to handle importing Products using a custom CSV flat file.

The custom CSV format that we will create a handler for will look like this:

Part Nbr,Name,Short Description,Long Description,Model Code,Class Code,SRP,Is Stocking Online,Image URL
"UNIQUE-ID-001","My Unique Product 1","Short Description 1","Long Description 1","CAT1","SUBCAT1","7.44","true","https://www.broadleafcommerce.com/cmsstatic/blc-nav-logo.svg"
"UNIQUE-ID-002","My Unique Product 2","Short Description 2","Long Description 2","CAT1","SUBCAT2","88.58","true","https://www.broadleafcommerce.com/cmsstatic/blc-nav-logo.svg"
"UNIQUE-ID-003","My Unique Product 3","Short Description 3","Long Description 3","CAT2","SUBCAT3","130.12","true","https://www.broadleafcommerce.com/cmsstatic/blc-nav-logo.svg"
"UNIQUE-ID-004","My Unique Product 4","Short Description 4","Long Description 4","CAT2","SUBCAT4","155.82","false","https://www.broadleafcommerce.com/cmsstatic/blc-nav-logo.svg"

Creating a Custom Product Import in the Catalog Service

Utility Classes

As we start creating a variety of different import handlers, it’s sometimes beneficial to abstract some common functions into helper classes. Below you will find a few classes that may be beneficial for your ingestion process across different types of imports.

First, we will create an ImportUtil class in the Catalog Service

package com.broadleafdemo.catalog.dataimport;

import org.apache.commons.lang3.CharUtils;
import org.springframework.lang.Nullable;
import org.springframework.util.StringUtils;

import com.broadleafcommerce.common.dataimport.messaging.BatchCompletionRecord;
import com.broadleafcommerce.common.dataimport.messaging.BatchCompletionRecordStatus;
import com.broadleafcommerce.common.dataimport.messaging.BatchRequest;
import com.broadleafcommerce.data.tracking.core.service.BulkPersistenceResponse;
import com.broadleafcommerce.data.tracking.core.type.TrackingLevel;

import java.util.List;
import java.util.Map;
import java.util.function.Function;
import java.util.stream.Collectors;

import lombok.experimental.UtilityClass;

@UtilityClass
public class ImportUtil {

    /**
     * Utility method to retrieve all failed records from a {@link BulkPersistenceResponse}.
     * <p>
     * Example usage:
     * <p>
     *
     * <pre>
     *     {@code
     *        getFailedRecords(response, correlationIdMap, Product::getId);
     *     }
     * </pre>
     *
     * @param persistenceResponse The persistence response.
     * @param correlationIdMap The correlation ID map.
     * @param correlationIdFunction The function that resolves to a key in the correlationIdMap.
     * @param <T> The domain type of the BulkPersistenceResponse
     * @return A collection of failed BatchCompletionRecords for the persistence response.
     */
    public static <T> List<BatchCompletionRecord> getFailedRecords(
            BulkPersistenceResponse<T> persistenceResponse,
            Map<String, String> correlationIdMap,
            Function<T, String> correlationIdFunction) {
        return persistenceResponse
                .getFailures()
                .stream()
                .map(failure -> {
                    String correlationId = correlationIdMap
                            .get(correlationIdFunction.apply(failure.getFailedEntity()));
                    String failureMsg = failure.getType() + (failure.getValidationErrors() != null
                            ? "-" + failure.getValidationErrors().toString()
                            : "");
                    return new BatchCompletionRecord(correlationId,
                            BatchCompletionRecordStatus.ERROR,
                            failureMsg,
                            null);
                })
                .collect(Collectors.toList());
    }

    /**
     * Utility method to retrieve all successfully persisted records from a
     * {@link BulkPersistenceResponse}.
     * <p>
     * Example usage:
     * <p>
     *
     * <pre>
     *     {@code
     *        getSuccessfulRecords(response, correlationIdMap, Product::getId);
     *     }
     * </pre>
     *
     * @param persistenceResponse The persistence response.
     * @param correlationIdMap The correlation ID map.
     * @param correlationIdFunction A function that resolves to a key in the correlationIdMap.
     * @param <T> The domain type of the BulkPersistenceResponse
     * @return A collection of successful BatchCompletionRecords for the persistence response.
     */
    public static <T> List<BatchCompletionRecord> getSuccessfulRecords(
            BulkPersistenceResponse<T> persistenceResponse,
            Map<String, String> correlationIdMap,
            Function<T, String> correlationIdFunction) {
        return getSuccessfulRecords(
                persistenceResponse,
                correlationIdMap,
                correlationIdFunction,
                null);
    }

    /**
     * Utility method to retrieve all successfully persisted records from a
     * {@link BulkPersistenceResponse}.
     * <p>
     * Example usage:
     * <p>
     *
     * <pre>
     *     {@code
     *        getSuccessfulRecords(response, correlationIdMap, Product::getId);
     *     }
     * </pre>
     *
     * @param persistenceResponse The persistence response.
     * @param correlationIdMap The correlation ID map.
     * @param correlationIdFunction A function that resolves to a key in the correlationIdMap.
     * @param <T> The domain type of the BulkPersistenceResponse
     * @return A collection of successful BatchCompletionRecords for the persistence response.
     */
    public static <T> List<BatchCompletionRecord> getSuccessfulRecords(
            BulkPersistenceResponse<T> persistenceResponse,
            Map<String, String> correlationIdMap,
            Function<T, String> correlationIdFunction,
            @Nullable Function<T, String> idFunction) {
        return persistenceResponse
                .getSuccessfullyPersisted()
                .stream()
                .map(success -> new BatchCompletionRecord(
                        correlationIdMap.get(correlationIdFunction.apply(success)),
                        BatchCompletionRecordStatus.SUCCESS,
                        idFunction == null ? null : idFunction.apply(success)))
                .collect(Collectors.toList());
    }

    public static String getUrlStr(String urlStr) {
        StringBuilder sanitizedUrl = new StringBuilder();
        urlStr = urlStr.toLowerCase();
        urlStr = StringUtils.trimTrailingWhitespace(urlStr);
        char[] url = urlStr.toCharArray();
        for (char character : url) {
            if (CharUtils.isAsciiAlphanumeric(character) || character == '-') {
                sanitizedUrl.append(character);
            }
            if (character == ' ') {
                sanitizedUrl.append("-");
            }
        }
        String result = sanitizedUrl.toString().replaceAll("\\-{2,}", "\\-");
        return "/" + result;
    }

    public static boolean isSandboxImport(BatchRequest batchRequest) {
        return !StringUtils.isEmpty(batchRequest.getContext().getSandboxId());
    }

    public static long getCatalogLevel(BatchRequest batchRequest) {
        return isSandboxImport(batchRequest) ? TrackingLevel.USER.getLevel()
                : TrackingLevel.PRODUCTION.getLevel();
    }

}

Next, we can create a CustomImportContextHelper

package com.broadleafdemo.catalog.dataimport;

import com.broadleafcommerce.common.dataimport.messaging.BatchRequest;
import com.broadleafcommerce.data.tracking.core.context.ContextInfo;
import com.broadleafcommerce.data.tracking.core.context.ContextRequest;
import com.broadleafcommerce.data.tracking.core.type.OperationType;
import com.broadleafcommerce.data.tracking.core.web.ContextRequestHydrator;

import java.util.Optional;

import lombok.NonNull;
import lombok.RequiredArgsConstructor;

@RequiredArgsConstructor
public class CustomImportContextHelper {

    private final ContextRequestHydrator contextRequestHydrator;

    public ContextInfo getContext(@NonNull BatchRequest batch) {
        return getContext(batch, OperationType.UNKNOWN);
    }

    public ContextInfo getContext(@NonNull BatchRequest batch,
            @NonNull OperationType operationType) {
        ContextInfo contextInfo = new ContextInfo(operationType);
        contextInfo.setContextRequest(getContextRequest(batch));
        contextInfo.setAuthor(batch.getContext().getAuthor());
        return contextInfo;
    }

    public ContextRequest getContextRequest(BatchRequest batchRequest) {
        Optional<BatchRequest.BatchContext> batchContext = Optional.of(batchRequest)
                .map(BatchRequest::getContext);
        Optional<String> applicationId = batchContext
                .map(BatchRequest.BatchContext::getApplicationId);
        Optional<String> catalogId = batchContext
                .map(BatchRequest.BatchContext::getCatalogId);
        Optional<String> tenantId = batchContext
                .map(BatchRequest.BatchContext::getTenantId);

        ContextRequest contextRequest = new ContextRequest()
                .withApplicationId(applicationId.orElse(null))
                .withCatalogId(catalogId.orElse(null))
                .withTenantId(tenantId.orElse(null))
                .withCatalogLevel(ImportUtil.getCatalogLevel(batchRequest))
                .withSandBoxId(batchRequest.getContext().getSandboxId());

        return contextRequestHydrator.hydrate(contextRequest);
    }
}

Extend Services to Support External System Identifier

Typically, implementers will utilize import as a way to ingest and sync data from an external system. We recommend using the EXTERNAL_ID field on the target entities as a means to store that Unique Identifer that matches across systems.

In some cases, you may need to extend Broadleaf’s existing business layer service to support that lookup (or to lookup by any extended attribute that is relevant for your implementation).

In this example, we’ll extend the default ProductService and the CategoryService to support lookups by External ID

package com.broadleafdemo.catalog.service;

import org.apache.commons.collections4.CollectionUtils;
import org.springframework.lang.Nullable;
import org.springframework.util.Assert;

import com.broadleafcommerce.catalog.domain.CategoryProduct;
import com.broadleafcommerce.catalog.domain.category.Category;
import com.broadleafcommerce.catalog.domain.product.Product;
import com.broadleafcommerce.catalog.repository.category.CategoryRepository;
import com.broadleafcommerce.catalog.service.CategoryProductService;
import com.broadleafcommerce.catalog.service.DefaultCategoryService;
import com.broadleafcommerce.catalog.service.product.ProductService;
import com.broadleafcommerce.catalog.service.rsql.RSQLEvaluationService;
import com.broadleafcommerce.common.extension.cache.CacheStateManager;
import com.broadleafcommerce.data.tracking.core.Trackable;
import com.broadleafcommerce.data.tracking.core.context.ContextInfo;
import com.broadleafcommerce.data.tracking.core.filtering.fetch.FilterParser;
import com.broadleafcommerce.data.tracking.core.filtering.fetch.rsql.EmptyNode;
import com.broadleafcommerce.data.tracking.core.service.RsqlCrudEntityHelper;

import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.Optional;
import java.util.StringJoiner;

import cz.jirutka.rsql.parser.RSQLParser;
import cz.jirutka.rsql.parser.ast.Node;

public class MyCategoryService<P extends Category> extends DefaultCategoryService<P> {

    private static final Node EMPTY_NODE = new EmptyNode();

    private final RSQLParser rsqlParser;

    public MyCategoryService(final CategoryRepository<Trackable> repository,
            final RsqlCrudEntityHelper helper,
            final ProductService<Product> productService,
            final CategoryProductService<CategoryProduct> categoryProductService,
            final RSQLEvaluationService rsqlEvaluationService,
            @Nullable final CacheStateManager cacheStateManager,
            final FilterParser<Node> parser) {
        super(repository, helper, productService, categoryProductService, rsqlEvaluationService,
                cacheStateManager, parser);
        this.rsqlParser = new RSQLParser();
    }

    public Optional<P> readByExternalId(final String externalId,
            @Nullable final ContextInfo contextInfo) {
        Assert.hasText(externalId, "ExternalId cannot be blank");
        final List<Trackable> categoryDomain =
                getRepository().findAll(buildExternalIdFilter(externalId), contextInfo);

        if (categoryDomain.isEmpty()) {
            return Optional.empty();
        }

        if (categoryDomain.size() > 1) {
            final String msg =
                    String.format("Expected one result for external ID %s, but there were %s",
                            externalId, categoryDomain.size());
            throw new IllegalStateException(msg);
        }

        return Optional.of(getHelper().getMapper().process(categoryDomain.get(0), contextInfo));
    }

    public List<Category> readAllByExternalIds(@lombok.NonNull final Collection<String> externalIds,
            @Nullable final ContextInfo contextInfo) {
        final Node externalIdFilter = buildExternalIdFilter(externalIds);
        final List<Trackable> categories = getRepository().findAll(externalIdFilter, contextInfo);
        return getHelper().getMapper().process(categories, contextInfo);
    }

    private Node buildExternalIdFilter(final Collection<String> externalIds) {
        if (CollectionUtils.isEmpty(externalIds)) {
            return EMPTY_NODE;
        }

        final StringJoiner externalIdsString = new StringJoiner(",");
        externalIds.forEach(externalIdsString::add);

        return rsqlParser.parse("externalId=in=(" + externalIdsString.toString() + ")");
    }

    private Node buildExternalIdFilter(final String externalId) {
        return buildExternalIdFilter(Collections.singletonList(externalId));
    }
}

Finally, here’s the extension of the DefaultProductService

package com.broadleafdemo.catalog.service;

import org.apache.commons.collections4.CollectionUtils;
import org.springframework.lang.Nullable;
import org.springframework.util.Assert;

import com.broadleafcommerce.catalog.domain.CategoryProduct;
import com.broadleafcommerce.catalog.domain.product.Product;
import com.broadleafcommerce.catalog.domain.product.Variant;
import com.broadleafcommerce.catalog.repository.product.ProductRepository;
import com.broadleafcommerce.catalog.service.CategoryProductService;
import com.broadleafcommerce.catalog.service.product.DefaultProductService;
import com.broadleafcommerce.catalog.service.product.VariantService;
import com.broadleafcommerce.common.extension.TypeFactory;
import com.broadleafcommerce.common.extension.cache.CacheStateManager;
import com.broadleafcommerce.data.tracking.core.Trackable;
import com.broadleafcommerce.data.tracking.core.context.ContextInfo;
import com.broadleafcommerce.data.tracking.core.filtering.fetch.FilterParser;
import com.broadleafcommerce.data.tracking.core.filtering.fetch.rsql.EmptyNode;
import com.broadleafcommerce.data.tracking.core.service.RsqlCrudEntityHelper;

import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.Optional;
import java.util.StringJoiner;

import cz.jirutka.rsql.parser.RSQLParser;
import cz.jirutka.rsql.parser.ast.Node;

public class MyProductService<P extends Product> extends DefaultProductService<P> {

    private static final Node EMPTY_NODE = new EmptyNode();

    private final RSQLParser rsqlParser;

    public MyProductService(ProductRepository<Trackable> repository,
            RsqlCrudEntityHelper helper,
            VariantService<Variant> variantService,
            CategoryProductService<CategoryProduct> categoryProductService,
            @Nullable CacheStateManager cacheStateManager,
            FilterParser<Node> parser,
            TypeFactory typeFactory) {
        super(repository, helper, variantService, categoryProductService, cacheStateManager,
                parser, typeFactory);
        this.rsqlParser = new RSQLParser();
    }

    public Optional<P> readByExternalId(final String externalId,
            @Nullable final ContextInfo contextInfo) {
        Assert.hasText(externalId, "ExternalId cannot be blank");
        final List<Trackable> productDomain =
                getRepository().findAll(buildExternalIdFilter(externalId), contextInfo);

        if (productDomain.isEmpty()) {
            return Optional.empty();
        }

        if (productDomain.size() > 1) {
            final String msg =
                    String.format("Expected one result for external ID %s, but there were %s",
                            externalId, productDomain.size());
            throw new IllegalStateException(msg);
        }

        return Optional.of(getHelper().getMapper().process(productDomain.get(0), contextInfo));
    }

    public List<Product> readAllByExternalIds(@lombok.NonNull final Collection<String> externalIds,
            @Nullable final ContextInfo contextInfo) {
        final Node externalIdFilter = buildExternalIdFilter(externalIds);
        final List<Trackable> products = getRepository().findAll(externalIdFilter, contextInfo);
        return getHelper().getMapper().process(products, contextInfo);
    }

    private Node buildExternalIdFilter(final Collection<String> externalIds) {
        if (CollectionUtils.isEmpty(externalIds)) {
            return EMPTY_NODE;
        }

        final StringJoiner externalIdsString = new StringJoiner(",");
        externalIds.forEach(externalIdsString::add);

        return rsqlParser.parse("externalId=in=(" + externalIdsString.toString() + ")");
    }

    private Node buildExternalIdFilter(final String externalId) {
        return buildExternalIdFilter(Collections.singletonList(externalId));
    }

}

Creating the Custom Product Batch Handler

With the above classes in place, we should now be ready to create our CustomProductBatchHandler

Let’s first create an Interface that we can use for other custom import handlers.

package com.broadleafdemo.catalog.dataimport;

import org.springframework.lang.NonNull;

import com.broadleafcommerce.common.dataimport.ImportBatchHandler;
import com.broadleafcommerce.common.dataimport.messaging.BatchCompletionRecord;
import com.broadleafcommerce.common.dataimport.messaging.BatchCompletionRecordStatus;
import com.broadleafcommerce.common.dataimport.messaging.BatchRequest;
import com.broadleafcommerce.common.jpa.JpaConstants;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

/**
 * Custom importer interface that bypasses the need to implement a custom
 * com.broadleafcommerce.dataimport.processor.spec.Specification
 * <p>
 * A custom specification is still required if default Specification behavior is not sufficient.
 * <p>
 * To implement an import batch handler, simply implement this interface,
 * {@link CustomBatchHandler#getImportType}, and {@link ImportBatchHandler#handle(BatchRequest)} as
 * normal.
 */
public interface CustomBatchHandler extends ImportBatchHandler {

    @NonNull
    String getImportType();

    default boolean canHandle(BatchRequest batchRequest) {
        return getImportType().equals(batchRequest.getType());
    }

    /**
     * Combines multiple BatchCompletionRecords with the same correlationId into a single record.
     * This is to support creation of multiple records for a single import row.
     *
     * @param completions All completions, including duplicates.
     * @return A new list of batch completion records, with duplicate records combined into a single
     *         record.
     */
    default List<BatchCompletionRecord> combineRecords(List<BatchCompletionRecord> completions) {
        Map<String, BatchCompletionRecord> completionRecords = new HashMap<>();
        for (BatchCompletionRecord record : completions) {
            String correlationId = record.getCorrelationId();
            if (!completionRecords.containsKey(correlationId)) {
                completionRecords.put(correlationId, record);
            } else {
                completionRecords.put(correlationId,
                        combineRecords(completionRecords, record, correlationId));
            }
        }
        return new ArrayList<>(completionRecords.values());
    }

    default BatchCompletionRecord combineRecords(
            Map<String, BatchCompletionRecord> completionRecords,
            BatchCompletionRecord duplicateRecord,
            String correlationId) {
        BatchCompletionRecord firstRecord = completionRecords.get(correlationId);
        BatchCompletionRecordStatus status = firstRecord.getStatus();
        String identifier = firstRecord.getResourceTierIdentifier() + "|"
                + duplicateRecord.getResourceTierIdentifier();
        // The field is limited to 36 characters
        if (identifier.length() > JpaConstants.CONTEXT_ID_LENGTH) {
            identifier = identifier.substring(0, JpaConstants.CONTEXT_ID_LENGTH);
        }
        String errorMessage = null;
        if (isFailedRecord(firstRecord, duplicateRecord)) {
            status = BatchCompletionRecordStatus.ERROR;
            String firstRecordError =
                    firstRecord.getErrorMessage() == null ? "" : firstRecord.getErrorMessage();
            String duplicateRecordError = duplicateRecord.getErrorMessage() == null ? ""
                    : duplicateRecord.getErrorMessage();
            errorMessage = firstRecordError + "; " + duplicateRecordError;
        }
        // Updating existing BatchCompletionRecords is not allowed because [arg0], so we need to
        // create new ones.
        return new BatchCompletionRecord(correlationId, status, errorMessage, identifier);
    }

    default boolean isFailedRecord(BatchCompletionRecord firstRecord,
            BatchCompletionRecord duplicateRecord) {
        return BatchCompletionRecordStatus.ERROR.equals(firstRecord.getStatus()) ||
                BatchCompletionRecordStatus.ERROR.equals(duplicateRecord.getStatus());
    }
}

Next, we can create the implementation of our CustomBatchHandler

package com.broadleafdemo.catalog.dataimport;

import org.apache.commons.lang3.StringUtils;
import org.springframework.lang.Nullable;
import org.springframework.util.CollectionUtils;

import com.broadleafcommerce.catalog.domain.CategoryProduct;
import com.broadleafcommerce.catalog.domain.asset.ProductAsset;
import com.broadleafcommerce.catalog.domain.category.Category;
import com.broadleafcommerce.catalog.domain.category.CategoryRef;
import com.broadleafcommerce.catalog.domain.product.Product;
import com.broadleafcommerce.catalog.domain.type.InventoryCheckStrategy;
import com.broadleafcommerce.catalog.domain.type.InventoryType;
import com.broadleafcommerce.catalog.provider.RouteConstants;
import com.broadleafcommerce.catalog.service.CategoryProductService;
import com.broadleafcommerce.catalog.service.CategoryService;
import com.broadleafcommerce.catalog.service.asset.ProductAssetService;
import com.broadleafcommerce.catalog.service.product.ProductService;
import com.broadleafcommerce.common.dataimport.messaging.BatchCompletion;
import com.broadleafcommerce.common.dataimport.messaging.BatchCompletionRecord;
import com.broadleafcommerce.common.dataimport.messaging.BatchCompletionRecordStatus;
import com.broadleafcommerce.common.dataimport.messaging.BatchRecord;
import com.broadleafcommerce.common.dataimport.messaging.BatchRequest;
import com.broadleafcommerce.common.extension.TypeFactory;
import com.broadleafcommerce.common.extension.data.DataRouteByExample;
import com.broadleafcommerce.data.tracking.core.context.ContextInfo;
import com.broadleafcommerce.data.tracking.core.service.BulkPersistenceResponse;
import com.broadleafcommerce.data.tracking.core.service.Update;
import com.broadleafcommerce.data.tracking.core.type.OperationType;
import com.broadleafcommerce.money.util.MonetaryUtils;
import com.broadleafdemo.catalog.service.MyCategoryService;
import com.broadleafdemo.catalog.service.MyProductService;

import java.math.BigDecimal;
import java.security.SecureRandom;
import java.time.Instant;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Set;
import java.util.function.Function;
import java.util.stream.Collectors;

import io.azam.ulidj.ULID;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import lombok.RequiredArgsConstructor;

@DataRouteByExample(Product.class)
@RequiredArgsConstructor
public class CustomProductBatchHandler implements CustomBatchHandler {

    /**
     * Expected header fields expected on the CSV Flat File.
     */
    public static final String PART_NO = "Part Nbr";
    public static final String NAME = "Name";
    public static final String SHORT_DESC = "Short Description";
    public static final String LONG_DESC = "Long Description";
    public static final String MSRP = "SRP";
    public static final String IMAGE_URL = "Image URL";
    public static final String IS_STOCKING_ONLINE = "Is Stocking Online";
    public static final String MODEL_CODE = "Model Code";
    public static final String CLASS_CODE = "Class Code";

    private static final SecureRandom RANDOM = new SecureRandom();

    private final ProductService<Product> productService;
    private final ProductAssetService<ProductAsset> productAssetService;
    private final CategoryService<Category> categoryService;
    private final CategoryProductService<CategoryProduct> categoryProductService;
    private final CustomImportContextHelper contextHelper;
    private final TypeFactory typeFactory;

    @Override
    public String getImportType() {
        return "MY_CUSTOM_PRODUCT_IMPORT";
    }

    @Override
    public String getDataRouteKey() {
        return RouteConstants.Persistence.CATALOG_ROUTE_KEY;
    }

    @Override
    public BatchCompletion handle(BatchRequest batch) {
        ContextInfo context = contextHelper.getContext(batch);
        ContextInfo readContext = context.withOperationType(context, OperationType.READ);
        ContextInfo createContext = context.withOperationType(context, OperationType.CREATE);
        ContextInfo updateContext = context.withOperationType(context, OperationType.UPDATE);

        // Map of products with Category External ID as the key
        Map<String, List<Product>> productMap = new HashMap<>();
        Map<String, String> productCorrelationIdMap = new HashMap<>();
        Map<String, String> categoryProductCorrelationIdMap = new HashMap<>();
        Map<String, String> productAssetCorrelationIdMap = new HashMap<>();
        List<ProductAsset> productAssets = new ArrayList<>();

        Map<String, Product> existingProductMap =
                findExistingProductsByExternalId(batch, readContext);
        DiffsDetermined totalDiffs = new DiffsDetermined();

        for (BatchRecord record : batch.getRecords()) {
            Map<String, String> row = record.getRow();

            String partNumber = row.get(PART_NO);
            if (existingProductMap.containsKey(partNumber)) {
                Product product = existingProductMap.get(partNumber);

                totalDiffs.updateAll(determineDiffs(product, row));
                productCorrelationIdMap.put(product.getExternalId(), record.getCorrelationId());
                continue;
            }

            Product product = buildProduct(row);
            String categoryExternalId = getCategoryExternalId(row);
            productMap.computeIfAbsent(categoryExternalId, k -> new ArrayList<>())
                    .add(product);
            productCorrelationIdMap.put(product.getExternalId(), record.getCorrelationId());
            categoryProductCorrelationIdMap.put(product.getExternalId(), record.getCorrelationId());

            if (StringUtils.isNotBlank(row.get(IMAGE_URL))) {
                productAssets.add(buildProductAsset(row, product));
                productAssetCorrelationIdMap.put(product.getExternalId(),
                        record.getCorrelationId());
            }

        }

        BulkPersistenceResponse<Product> createProductResponse =
                createProducts(productMap, createContext);
        BulkPersistenceResponse<Product> updateProductResponse =
                updateProducts(totalDiffs.getUpdateProducts(), updateContext);
        BulkPersistenceResponse<CategoryProduct> categoryProductsResponse =
                persistCategoryProducts(productMap, createContext, readContext);
        BulkPersistenceResponse<ProductAsset> productAssetResponse =
                persistProductAssets(productAssets, createContext);

        return buildBatchCompletion(batch, createProductResponse,
                updateProductResponse,
                categoryProductsResponse,
                productAssetResponse,
                totalDiffs.getNoChangeRecords(),
                productCorrelationIdMap,
                categoryProductCorrelationIdMap,
                productAssetCorrelationIdMap);
    }

    /**
     * Builds a {@link BatchCompletion} object based on the outcomes of the custom import initiated
     * by the passed in {@link BatchRequest}
     *
     * @param batch
     * @param createProductResponse
     * @param updateProductResponse
     * @param categoryProductsResponse
     * @param productAssetResponse
     * @param noChangeRecords
     * @param productCorrelationIdMap
     * @param categoryProductCorrelationIdMap
     * @param productAssetCorrelationIdMap
     * @return
     */
    private BatchCompletion buildBatchCompletion(BatchRequest batch,
            BulkPersistenceResponse<Product> createProductResponse,
            BulkPersistenceResponse<Product> updateProductResponse,
            BulkPersistenceResponse<CategoryProduct> categoryProductsResponse,
            BulkPersistenceResponse<ProductAsset> productAssetResponse,
            Set<String> noChangeRecords,
            Map<String, String> productCorrelationIdMap,
            Map<String, String> categoryProductCorrelationIdMap,
            Map<String, String> productAssetCorrelationIdMap) {
        List<BatchCompletionRecord> completions = new ArrayList<>();
        completions.addAll(
                ImportUtil.getSuccessfulRecords(createProductResponse, productCorrelationIdMap,
                        Product::getExternalId));
        completions.addAll(
                ImportUtil.getSuccessfulRecords(updateProductResponse, productCorrelationIdMap,
                        Product::getExternalId));
        completions.addAll(
                ImportUtil.getSuccessfulRecords(categoryProductsResponse,
                        categoryProductCorrelationIdMap,
                        categoryProduct -> categoryProduct.getProduct().getExternalId()));
        completions.addAll(
                ImportUtil.getSuccessfulRecords(productAssetResponse,
                        productAssetCorrelationIdMap,
                        ProductAsset::getProductId));
        completions.addAll(
                ImportUtil.getFailedRecords(createProductResponse, productCorrelationIdMap,
                        Product::getExternalId));
        completions.addAll(
                ImportUtil.getFailedRecords(updateProductResponse, productCorrelationIdMap,
                        Product::getExternalId));
        completions
                .addAll(ImportUtil.getFailedRecords(categoryProductsResponse,
                        categoryProductCorrelationIdMap,
                        categoryProduct -> categoryProduct.getProduct().getExternalId()));
        completions.addAll(
                ImportUtil.getFailedRecords(productAssetResponse, productAssetCorrelationIdMap,
                        ProductAsset::getProductId));
        completions.addAll(getNoChangeRecords(noChangeRecords, productCorrelationIdMap));
        completions = combineRecords(completions);

        return new BatchCompletion(batch.getId(), batch.getImportId(), completions);
    }

    /**
     * CRUD Helper Methods
     */
    private BulkPersistenceResponse<Product> createProducts(Map<String, List<Product>> productMap,
            ContextInfo createContext) {
        List<Product> products = new ArrayList<>();
        productMap.values().forEach(products::addAll);

        return productService.createAllAllowingPartialSuccess(products, createContext);
    }

    /**
     * CRUD Helper Methods
     */
    private BulkPersistenceResponse<Product> updateProducts(Set<Product> products,
            ContextInfo createContext) {
        if (products.isEmpty()) {
            return new BulkPersistenceResponse<>();
        }
        Map<String, Product> idMap = products.stream()
                .collect(Collectors.toMap(Product::getId, Function.identity(),
                        (product1, product2) -> product1));
        List<Update<Product>> updateProducts = idMap.values().stream()
                .map(product -> new Update<>(product.getId(), product))
                .collect(Collectors.toList());

        return productService.updateAllAllowingPartialSuccess(updateProducts, createContext);
    }

    /**
     * CRUD Helper Methods
     */
    private BulkPersistenceResponse<CategoryProduct> persistCategoryProducts(
            Map<String, List<Product>> productMap,
            ContextInfo createContext,
            ContextInfo readContext) {
        Map<String, Category> categories = readCategories(productMap.keySet(), readContext);
        List<CategoryProduct> categoryProducts = buildCategoryProducts(categories, productMap);
        return categoryProductService.createAllAllowingPartialSuccess(categoryProducts,
                createContext);
    }

    /**
     * CRUD Helper Methods
     */
    private BulkPersistenceResponse<ProductAsset> persistProductAssets(
            List<ProductAsset> productAssets,
            ContextInfo createContext) {
        return productAssetService.createAllAllowingPartialSuccess(productAssets,
                createContext);
    }

    /**
     * Returns a map with the product external ID as the key and the product as the value.
     *
     * @param batch
     * @param readContext
     * @return
     */
    private Map<String, Product> findExistingProductsByExternalId(BatchRequest batch,
            ContextInfo readContext) {
        List<String> externalIds = batch.getRecords().stream()
                .map(record -> record.getRow().get(PART_NO))
                .collect(Collectors.toList());
        Map<String, Product> productMap = new HashMap<>();
        List<Product> products = ((MyProductService<Product>) productService)
                .readAllByExternalIds(externalIds, readContext);
        for (Product product : products) {
            productMap.put(product.getExternalId(), product);
        }
        return productMap;
    }

    /**
     * Returns a map with the category external ID as the key and the category as the value.
     *
     * @param categoryExternalIds A set of category external IDs.
     * @param contextInfo
     * @return A map of category external IDs.
     */
    private Map<String, Category> readCategories(Set<String> categoryExternalIds,
            ContextInfo contextInfo) {
        List<Category> categories =
                ((MyCategoryService<Category>) categoryService)
                        .readAllByExternalIds(categoryExternalIds, contextInfo);
        Map<String, Category> categoryMap = new HashMap<>();
        if (CollectionUtils.isEmpty(categories)) {
            return categoryMap;
        }
        for (Category category : categories) {
            categoryMap.put(category.getExternalId(), category);
        }
        return categoryMap;
    }

    /**
     * Builds the Category Product Relationship Assumes that the Category already exists in the
     * system.
     *
     * @param categoriesMap
     * @param productMap
     * @return
     */
    private List<CategoryProduct> buildCategoryProducts(Map<String, Category> categoriesMap,
            Map<String, List<Product>> productMap) {
        if (productMap.isEmpty()) {
            return Collections.emptyList();
        }
        List<CategoryProduct> categoryProducts = new ArrayList<>();
        for (String categoryExternalId : categoriesMap.keySet()) {
            Category category = categoriesMap.get(categoryExternalId);
            for (Product product : productMap.get(categoryExternalId)) {
                CategoryProduct categoryProduct = typeFactory.get(CategoryProduct.class);
                categoryProduct.setProduct(product);
                categoryProduct
                        .setCategory(typeFactory.get(CategoryRef.class).fromCategory(category));
                categoryProduct.setId(ULID.random(RANDOM));
                categoryProducts.add(categoryProduct);
            }
        }
        return categoryProducts;
    }

    /**
     * For this example, we'll assume a simple 2-level category: "Model" and "Class" The categories
     * are determined by a property called "Model" (parent category) and a second-level category
     * called "Class" (child category).
     *
     * These will be stored in Category#getExternalId as ${model}-{$class} (model dash class)
     *
     * @param row
     * @return
     */
    private String getCategoryExternalId(Map<String, String> row) {
        String model = row.get(MODEL_CODE);
        String classCode = row.get(CLASS_CODE);
        if (classCode == null) {
            return model;
        }
        return model + "-" + classCode;
    }


    /**
     * Sample builder method to convert a CSV row into a {@link Product}
     *
     * @param row
     * @return
     */
    private Product buildProduct(Map<String, String> row) {
        Product product = new Product();
        product.setId(ULID.random(RANDOM));
        product.setExternalId(row.get(PART_NO));
        product.setName(row.get(NAME));
        product.setSku(row.get(PART_NO));
        product.setDescription(getDescription(row));
        product.setMsrp(MonetaryUtils.toAmount(getBigDecimal(row, MSRP),
                MonetaryUtils.defaultCurrency().getCurrencyCode()));
        product.setDefaultPrice(
                MonetaryUtils.toAmount(getBigDecimal(row, MSRP),
                        MonetaryUtils.defaultCurrency().getCurrencyCode()));
        product.setActiveStartDate(Instant.now());
        product.setDiscountable(true);
        product.setIndividuallySold(true);
        product.setMetaTitle(row.get(SHORT_DESC));
        product.setMetaDescription(row.get(LONG_DESC));
        product.setOnline(true);
        product.setSearchable(true);
        product.setUri(ImportUtil.getUrlStr(row.get(PART_NO)));
        product.setInventoryType(InventoryType.PHYSICAL.name());
        product.setCurrency(MonetaryUtils.defaultCurrency());
        product.setAvailableOnline(getBoolean(row, IS_STOCKING_ONLINE));
        product.setInventoryCheckStrategy(InventoryCheckStrategy.NEVER.name());

        return product;
    }

    /**
     * Sample builder method to convert a CSV row into a {@link ProductAsset}
     *
     * @param row
     * @return
     */
    private ProductAsset buildProductAsset(Map<String, String> row, Product product) {
        ProductAsset productAsset = typeFactory.get(ProductAsset.class);
        productAsset.setId(row.get(PART_NO));
        productAsset.setAltText(row.get(PART_NO));
        productAsset.setPrimary(true);
        productAsset.setProductId(product.getId());
        productAsset.setProvider("UNSPECIFIED");
        productAsset.setTitle(row.get(PART_NO));
        productAsset.setType("IMAGE");
        productAsset.setUrl(row.get(IMAGE_URL));

        return productAsset;
    }

    /**
     * Row processing helper method
     */
    @Nullable
    private BigDecimal getBigDecimal(Map<String, String> row, String key) {
        String value = row.get(key);
        return value == null ? null : new BigDecimal(value);
    }

    /**
     * Row processing helper method
     */
    private String getDescription(Map<String, String> row) {
        String shortDesc =
                org.springframework.util.StringUtils.trimTrailingWhitespace(row.get(SHORT_DESC));
        String longDesc =
                org.springframework.util.StringUtils.trimTrailingWhitespace(row.get(LONG_DESC));
        if (StringUtils.isEmpty(longDesc)) {
            return shortDesc;
        }
        return shortDesc + " " + longDesc;
    }

    /**
     * Row processing helper method
     */
    private boolean getBoolean(Map<String, String> row, String key) {
        String val = row.get(key);
        return "Y".equals(val) || "Yes".equals(val) || "true".equalsIgnoreCase(val);
    }

    /**
     * Simple example to granularly determine row variation against an existing product
     *
     * @param existing
     * @param row
     * @return
     */
    private DiffsDetermined determineDiffs(Product existing, Map<String, String> row) {

        DiffsDetermined diffsDetermined = new DiffsDetermined();
        boolean diffExists = false;

        Product potentialUpdates = buildProduct(row);

        if (!StringUtils.equals(existing.getDescription(), potentialUpdates.getDescription())) {
            existing.setDescription(potentialUpdates.getDescription());
            diffExists = true;
        }
        if (!Objects.equals(existing.getMsrp(), potentialUpdates.getMsrp())) {
            existing.setMsrp(potentialUpdates.getMsrp());
            diffExists = true;
        }
        if (!Objects.equals(existing.getDefaultPrice(), potentialUpdates.getDefaultPrice())) {
            existing.setDefaultPrice(potentialUpdates.getDefaultPrice());
            diffExists = true;
        }
        if (!StringUtils.equals(existing.getMetaTitle(), potentialUpdates.getMetaTitle())) {
            existing.setMetaTitle(potentialUpdates.getMetaTitle());
            diffExists = true;
        }
        if (!StringUtils.equals(existing.getMetaDescription(),
                potentialUpdates.getMetaDescription())) {
            existing.setMetaDescription(potentialUpdates.getMetaDescription());
            diffExists = true;
        }
        if (existing.isAvailableOnline() != potentialUpdates.isAvailableOnline()) {
            existing.setAvailableOnline(potentialUpdates.isAvailableOnline());
            diffExists = true;
        }
        if (!StringUtils.equals(existing.getInventoryCheckStrategy(),
                potentialUpdates.getInventoryCheckStrategy())) {
            existing.setInventoryCheckStrategy(potentialUpdates.getInventoryCheckStrategy());
            diffExists = true;
        }
        if (diffExists) {
            diffsDetermined.getUpdateProducts()
                    .add(existing);
        } else {
            diffsDetermined.getNoChangeRecords()
                    .add(existing.getExternalId());
        }

        return diffsDetermined;
    }

    private Collection<? extends BatchCompletionRecord> getNoChangeRecords(
            Set<String> noChangeProducts,
            Map<String, String> productCorrelationIdMap) {
        return noChangeProducts.stream()
                .map(noChange -> new BatchCompletionRecord(
                        productCorrelationIdMap.get(noChange),
                        BatchCompletionRecordStatus.SUCCESS))
                .collect(Collectors.toSet());
    }

    @Data
    @NoArgsConstructor
    @AllArgsConstructor
    private class DiffsDetermined {
        Set<Product> updateProducts = new HashSet<>();
        Set<String> noChangeRecords = new HashSet<>();
        boolean missingRecords = false;
        String missingRecordError;

        public void updateAll(DiffsDetermined otherDiffs) {
            this.getUpdateProducts()
                    .addAll(otherDiffs.getUpdateProducts());
            this.getNoChangeRecords()
                    .addAll(otherDiffs.getNoChangeRecords());
        }
    }
}

Spring Configuration

Finally, you’ll want to register all your components with Spring. Below you will find an example configuration class:

package com.broadleafdemo.catalog.autoconfigure;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.lang.Nullable;

import com.broadleafcommerce.catalog.domain.CategoryProduct;
import com.broadleafcommerce.catalog.domain.asset.ProductAsset;
import com.broadleafcommerce.catalog.domain.category.Category;
import com.broadleafcommerce.catalog.domain.product.Product;
import com.broadleafcommerce.catalog.domain.product.Variant;
import com.broadleafcommerce.catalog.repository.category.CategoryRepository;
import com.broadleafcommerce.catalog.repository.product.ProductRepository;
import com.broadleafcommerce.catalog.service.CategoryProductService;
import com.broadleafcommerce.catalog.service.CategoryService;
import com.broadleafcommerce.catalog.service.asset.ProductAssetService;
import com.broadleafcommerce.catalog.service.product.ProductService;
import com.broadleafcommerce.catalog.service.product.VariantService;
import com.broadleafcommerce.catalog.service.rsql.RSQLEvaluationService;
import com.broadleafcommerce.common.extension.TypeFactory;
import com.broadleafcommerce.common.extension.cache.CacheStateManager;
import com.broadleafcommerce.data.tracking.core.Trackable;
import com.broadleafcommerce.data.tracking.core.filtering.fetch.FilterParser;
import com.broadleafcommerce.data.tracking.core.service.RsqlCrudEntityHelper;
import com.broadleafcommerce.data.tracking.core.web.ContextRequestHydrator;
import com.broadleafdemo.catalog.dataimport.CustomImportContextHelper;
import com.broadleafdemo.catalog.dataimport.CustomProductBatchHandler;
import com.broadleafdemo.catalog.service.MyCategoryService;
import com.broadleafdemo.catalog.service.MyProductService;

import cz.jirutka.rsql.parser.ast.Node;

@Configuration
public class MyCatalogAutoConfiguration {

    @Bean
    CustomImportContextHelper customImportContextHelper(
            ContextRequestHydrator contextRequestHydrator) {
        return new CustomImportContextHelper(contextRequestHydrator);
    }

    @Bean
    CustomProductBatchHandler customProductImportBatchHandler(
            ProductService<Product> productService,
            ProductAssetService<ProductAsset> productAssetService,
            CategoryService<Category> categoryService,
            CategoryProductService<CategoryProduct> categoryProductService,
            CustomImportContextHelper contextHelper,
            TypeFactory typeFactory) {
        return new CustomProductBatchHandler(
                productService,
                productAssetService,
                categoryService,
                categoryProductService,
                contextHelper,
                typeFactory);
    }

    @Bean
    @Primary
    CategoryService myCategoryService(CategoryRepository<Trackable> repository,
            RsqlCrudEntityHelper helper,
            ProductService<Product> productService,
            CategoryProductService<CategoryProduct> categoryProductService,
            RSQLEvaluationService rsqlEvaluationService,
            @Nullable CacheStateManager cacheStateManager,
            FilterParser<Node> parser) {
        return new MyCategoryService(repository,
                helper,
                productService,
                categoryProductService,
                rsqlEvaluationService,
                cacheStateManager,
                parser);
    }

    @Bean
    @Primary
    ProductService myProductService(ProductRepository<Trackable> repository,
            RsqlCrudEntityHelper helper,
            VariantService<Variant> variantService,
            CategoryProductService<CategoryProduct> categoryProductService,
            @Nullable CacheStateManager cacheStateManager,
            FilterParser<Node> parser,
            TypeFactory typeFactory) {
        return new MyProductService(repository,
                helper,
                variantService,
                categoryProductService,
                cacheStateManager,
                parser,
                typeFactory);
    }

}

Flex Package spring.factories Note

If you are going to include the Catalog Service jar inside a Flex Package composition, you will want to also make sure to define a spring.factories class allowing your Flex Package application to auto-register your auto-configuration class.

i.e. create a src/main/resources/META-INF/spring.factories file with the following contents:

org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
  com.broadleafdemo.catalog.autoconfigure.MyCatalogAutoConfiguration

Enable Custom Import in the Admin

In this next section, we’ll walk through the steps necessary to enable using the admin’s file importing interface with our custom importer handler.

You will want to create these extensions in the Import Service

Defining Custom Import Types

Let’s first create an enum that describes our CustomProductBatchHandler

package com.broadleafdemo.importservice.domain;

import lombok.Getter;

/**
 * Enum declaring a list of custom import types that can be converted into
 * {@link com.broadleafcommerce.dataimport.processor.specification.GlobalImportSpecification} and
 * registered as beans
 */
public enum CustomImportType {

    MY_CUSTOM_PRODUCT_IMPORT("MY_CUSTOM_PRODUCT_IMPORT", "My Custom Product Import", true, true);

    @Getter
    private String name;

    @Getter
    private String label;

    @Getter
    private boolean sandbox;

    @Getter
    private boolean catalog;

    CustomImportType(String name, String label, boolean isSandbox, boolean isCatalog) {
        this.name = name;
        this.label = label;
        this.sandbox = isSandbox;
        this.catalog = isCatalog;
    }

}

Next, we’ll create a GlobalSpecification that allows us to add it to the existing "Import Specs" that come with the system OOB.

package com.broadleafdemo.importservice.domain;

import org.springframework.lang.NonNull;

import com.broadleafcommerce.dataimport.processor.specification.DefaultSpecification;
import com.broadleafcommerce.dataimport.processor.specification.GlobalImportSpecification;

import lombok.Getter;

/**
 * Used as the default implementation of {@link GlobalImportSpecification} that adds the required
 * fields of catalogDiscriminated, sandboxDiscriminated, and label to {@link DefaultSpecification}
 */
@Getter
public class MyGlobalSpecification extends DefaultSpecification
        implements GlobalImportSpecification {

    private final boolean catalogDiscriminated;
    private final boolean sandboxDiscriminated;
    private final String label;

    public MyGlobalSpecification(String importType, boolean catalogDiscriminated,
            boolean sandboxDiscriminated, String label) {
        super(importType);
        this.catalogDiscriminated = catalogDiscriminated;
        this.sandboxDiscriminated = sandboxDiscriminated;
        this.label = label;
    }

    @Override
    public boolean canHandle(@NonNull @lombok.NonNull String importType) {
        return importType.equals(getImportType());
    }

}

Override the /specifications Endpoint

To include our custom specifications, we’ll want to override the default endpoints to also take into account our custom spec.

package com.broadleafdemo.importservice.web;

import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

import com.broadleafcommerce.common.extension.data.DataRouteByExample;
import com.broadleafcommerce.data.tracking.core.policy.Policy;
import com.broadleafcommerce.dataimport.domain.Import;
import com.broadleafcommerce.dataimport.domain.ImportSpecificationResponse;
import com.broadleafcommerce.dataimport.processor.specification.ImportSpecification;

import java.util.List;
import java.util.stream.Collectors;

import lombok.RequiredArgsConstructor;

/**
 * Override the import endpoint for specifications to return My Custom Specifications instead of the
 * defaults.
 */
@RestController
@RequestMapping("/imports")
@RequiredArgsConstructor
@DataRouteByExample(Import.class)
public class MyImportEndpoint {

    private final List<ImportSpecification> myImportSpecifications;

    @GetMapping("/specifications")
    @Policy(permissionRoots = "IMPORT")
    public List<ImportSpecificationResponse> getSpecifications() {
        return myImportSpecifications.stream()
                .map(spec -> {
                    return new ImportSpecificationResponse()
                            .setImportType(spec.getImportType())
                            .setCatalogDiscriminated(spec.isCatalogDiscriminated())
                            .setSandboxDiscriminated(spec.isSandboxDiscriminated())
                            .setRequiredScopes(spec.getRequiredScopes());
                })
                .collect(Collectors.toList());
    }

}

Create a Downloadable Example

The default import flows in the admin allow you to also specify an "example file" that’s tied with a particular Import specification. This is useful for business users that may want to download a template to be used as a starting guide.

To do this, we can just create an ExampleImportResolver

package com.broadleafdemo.importservice.web;

import org.springframework.core.annotation.Order;
import org.springframework.core.io.ClassPathResource;
import org.springframework.core.io.Resource;

import com.broadleafcommerce.dataimport.service.ExampleImportResolver;

import java.util.HashMap;
import java.util.Map;

@Order(-1000)
public class MyExampleImportResolver implements ExampleImportResolver {

    /**
     * Map containing example imports. The key is the import type, and the value the file the import
     * type should return. For example, for the import type "MY_CUSTOM_PRODUCT_IMPORT" the file to
     * be returned should be "custom-import-examples/custom_import.csv".
     */
    private Map<String, String> exampleImports = new HashMap<>();

    public MyExampleImportResolver() {
        exampleImports.put("MY_CUSTOM_PRODUCT_IMPORT", "custom-import-examples/custom_import.csv");
    }

    @Override
    public Resource resolveExample(String type, String fileType) {
        if (exampleImports.containsKey(type)) {
            return new ClassPathResource(exampleImports.get(type));
        }
        return new ClassPathResource("/does/not/exist");
    }
}

Finally, we’ll need to create a custom_import.csv file and put it in the /src/main/resources/custom-import-examples directory:

You can copy the contents of this into a file called custom_import.csv

Part Nbr,Name,Short Description,Long Description,Model Code,Class Code,SRP,Is Stocking Online,Image URL
"UNIQUE-ID-001","My Unique Product 1","Short Description 1","Long Description 1","CAT1","SUBCAT1","7.44","true","https://www.broadleafcommerce.com/cmsstatic/blc-nav-logo.svg"
"UNIQUE-ID-002","My Unique Product 2","Short Description 2","Long Description 2","CAT1","SUBCAT2","88.58","true","https://www.broadleafcommerce.com/cmsstatic/blc-nav-logo.svg"
"UNIQUE-ID-003","My Unique Product 3","Short Description 3","Long Description 3","CAT2","SUBCAT3","130.12","true","https://www.broadleafcommerce.com/cmsstatic/blc-nav-logo.svg"
"UNIQUE-ID-004","My Unique Product 4","Short Description 4","Long Description 4","CAT2","SUBCAT4","155.82","false","https://www.broadleafcommerce.com/cmsstatic/blc-nav-logo.svg"

Spring Configuration

Finally, let’s configure spring with all the components we’ve defined above:

package com.broadleafdemo.importservice.autoconfigure;

import org.apache.commons.collections.ListUtils;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Configuration;

import com.broadleafcommerce.dataimport.processor.specification.ImportSpecification;
import com.broadleafdemo.importservice.domain.CustomImportType;
import com.broadleafdemo.importservice.domain.MyGlobalSpecification;
import com.broadleafdemo.importservice.web.MyExampleImportResolver;
import com.broadleafdemo.importservice.web.MyImportEndpoint;

import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;

@Configuration
@ComponentScan(basePackageClasses = {MyImportEndpoint.class})
public class MyImportAutoConfiguration {

    @Bean
    public List<MyGlobalSpecification> myGlobalImportSpecifications() {
        return Arrays.stream(CustomImportType.values())
                .map(type -> new MyGlobalSpecification(type.getName(),
                        type.isCatalog(),
                        type.isSandbox(),
                        type.getLabel()))
                .collect(Collectors.toList());
    }

    @Bean
    public List<ImportSpecification> myImportSpecifications(
            List<? extends ImportSpecification> importSpecifications) {
        return ListUtils.union(importSpecifications, myGlobalImportSpecifications());
    }

    @Bean
    public MyGlobalSpecification myCustomProductImport() {
        return new MyGlobalSpecification(CustomImportType.MY_CUSTOM_PRODUCT_IMPORT.getName(),
                CustomImportType.MY_CUSTOM_PRODUCT_IMPORT.isCatalog(),
                CustomImportType.MY_CUSTOM_PRODUCT_IMPORT.isSandbox(),
                CustomImportType.MY_CUSTOM_PRODUCT_IMPORT.getLabel());
    }

    @Bean
    public MyExampleImportResolver myExampleImportResolver() {
        return new MyExampleImportResolver();
    }

}

Flex Package spring.factories Note

If you are going to include the Import Service jar inside a Flex Package composition, you will want to also make sure to define a spring.factories class allowing your Flex Package application to auto-register your auto-configuration class.

i.e. create a src/main/resources/META-INF/spring.factories file with the following contents:

org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
  com.broadleafdemo.importservice.autoconfigure.MyImportAutoConfiguration

Testing Custom Import

Let’s test out the new custom Product import in the admin.

  • First, log into the admin and go to Processes > Imports

  • Click on Create Import

  • You should see your MY_CUSTOM_PRODUCT_IMPORT added to the list of existing Product Import types.

MY_CUSTOM_PRODUCT_IMPORT

  • select MY_CUSTOM_PRODUCT_IMPORT, choose a Catalog and Sandbox you would like to import the products into.

  • You can download the sample CSV that you uploaded using the download example link at the bottom of the modal

MY_CUSTOM_PRODUCT_IMPORT configuration

  • Upload your CSV and initiate the Import process.

  • Once complete, you should be able to go to Catalog > Products to see your new products in the appropriate Sandbox

MY_CUSTOM_PRODUCT_IMPORT complete