Import Release Notes for 1.8.1-GA

The existing ProductSpecification and corresponding resource-tier handler (on the CatalogServices side) are deprecated and disabled by default in favor of the new CompleteProductImportSpecification and handler. To continue using the deprecated components and disable the new components, set broadleaf.import.product.specification.use-legacy to true in both the ImportServices properties and CatalogServices properties. Furthermore, any existing PRODUCT imports and batches should be allowed to fully complete processing with the old handler (and no more batch requests should be in the Kafka message queue) before the new components are enabled - this way the new product batch handler will not be forced to consume messages intended for the old handler.

All out-of-box specifications extending DefaultSpecification (except the deprecated ProductSpecification, which is unchanged) have had their initializeFieldConfigMap() methods removed. Field configuration has been moved to the populateHeaderFieldConfigsByRowType() method, which extensions should migrate to. The configuration itself should be the same as before, with the only difference being that it is now nested under a specific row type value. For specifications that only have a single row type, this should match the 'main record type' (which itself defaults to 'import type' if not set). Even for client specifications that do not currently extend the out-of-box specifications, it is strongly recommended to move away from all deprecated fields - while there are backwards compatibility measures in place to support the deprecated fields, the migrations are easy enough to recommend moving everything at once.

Import table now has a new specification_ref column - it’s nullable and existing rows can just have it as null. This supports the new specificationRef field in Import, discussed later in these notes. The liquibase changesets will need to be run to ensure the new column is present.

Mark operation in BatchItemCompletion as Nullable, as operation type is not necessarily provided in the input file or generated by the import service anymore. If clients have custom logic that references this field, they must ensure null-safety is checked.

Incorrect metadata message keys import.fields.specification and import.fields.options.specification.csv have been removed and replaced with import.fields.file-type and import.fields.options.file-type.csv, and also updated in StartImportAction and DefaultImportFileTypes

ImportManager/DefaultImportManager no longer directly injects/keeps a List<ImportSpecification> and instead relies on the new ImportSpecificationService for providing specifications. The findSpecificationOrDefault() method has been removed in favor of this new component.

ImportEndpoint no longer directly injects/keeps a List<GlobalImportSpecification> and instead relies on ImportSpecificationService.

Basic plumbing has been introduced to support fetching specifications with context discrimination. For security purposes, the context info used to fetch the specification needs to be pre-validated before it is used. This means the importing context info has to be built and validated in an earlier phase than before, which resulted in breaking changes to the initiateImport flow in DefaultImportManager. At a high level, initializeContextFields no longer instantiates importing catalog ID, and that initialization is deferred until after the specification is fetched.

PriceDataExampleImportResolver no longer delegates to DefaultExampleImportResolver

The getRowWithAdditionalDataAdded() method in CSVErrorsReportService has a changed signature that now accepts an ImportSpecification for the purpose of honoring dynamic mapping configuration

Unused RecordTypes static class has been removed from OfferCodeSpecification. Should not affect most clients, since the specification itself is only available since 1.8.0, which clients have not yet started consuming.

Introduce new ImportCompletedItemsGroup in metadata with separate successful and non-successful items grids, deprecating use of ImportDetailsGridExternal from ImportDetailsView and ImportDetailsFormView

Introduce a new optional specificationRef field to Import and ImportRequest, which is currently unused out-of-box, but provides an opportunity for extensions to implement behavior for requesting/finding specific specifications

ImportFieldConfig now supports a defaultValue, which will be set on the row by DefaultImportProcessor if the value for the field is null in the original input

Global variables in ImportFieldConfig are now explicitly marked as final

Update batching logic in DefaultImportProcessor such that dependents are now also counted against batch size limits instead of only counting main records. For example, let’s say the batch size limit is 2 and the input contains mainRecordA with 3 dependents and mainRecordB with 3 dependents. Previously, only main records were counted against batch size, so the batching logic would have only counted 2 records and thus included all of those items in a single batch. Now, since dependents count, mainRecordA with its dependents (4 total records) will reach the batch size limit and will be sent in a separate batch from mainRecordB with its dependents. To be clear, batching will still always include all dependents of a main record in the same batch and will not break them across multiple batches.

Introduce new resolveRecordTypeBeforePropertyMapping() method on ImportSpecification to describe how record type should be resolved from a row before any property mapping occurs, and update DefaultImportProcessor to use it

Introduce new shouldAutoGenerateOperationTypeForEachRecord() and shouldAutoGenerateResourceTierIdForEachRecord() methods on ImportSpecification to describe whether the import service should eagerly resolve/generate operation types and resource tier IDs for rows, and update DefaultImportProcessor to use them. This is primarily intended to support newer, simpler imports which do not rely on resource tier ID, correlation ID, and operation type concepts.

Introduce new shouldAllowUnmappedHeaders() method on ImportSpecification to describe whether unmappable headers should be passed through to the resource tier service or pruned, and update DefaultImportProcessor to use it

Introduce new dynamic field mapping methods to ImportSpecification and DefaultSpecification, as well as a DynamicHeaderFieldMapping interface that specifications can leverage to easily add dynamic mapping logic as an alternative to explicit, hard-coded field mappings via ImportFieldConfig

Introduce a default TranslationDynamicHeaderFieldMapping that can be used by specifications to map translations following a specific syntax pattern

Deprecate ImportSpecification 's 'global' getFieldConfigMap() and getRequiredHeaders() methods in favor of new row-type-specific getRequiredHeaders and getHeaderFieldConfigsByRowType, and update DefaultSpecification to make populating the new data easy via a new populateHeaderFieldConfigsByRowType() method. DefaultImportProcessor has been updated to preserve backwards compatibility with the deprecated methods while honoring the new ones.

Introduce new getName() method on ImportSpecification which is used by default as the specification ref

In DefaultSpecification, update the isRequiresRowTypeHeader computation and the deprecated getRequiredHeaders computation to happen in a thread-safe manner

Update CSVErrorsReportService to support the new dynamic mapping concepts, as well as honor the new methods and fields on ImportSpecification when generating errors report files

Introduce new CompleteProductImportSpecification representing the new product import specification, along with a new CompleteProductExampleImportResolver for its example file

Deprecate the old ProductSpecification

ImportManager now exposes a public buildImportingContextInfo() method for use by other components such as CSVErrorsReportService

Update DefaultImportValidator to honor new import specification methods and fields

Introduce new BooleanNormalizer, BooleanValidator, EnumeratedValueValidator, and MoneyAmountValidator components for use by specifications

The GET /imports/specifications endpoint (defined in ImportEndpoint) now accepts and resolves a ContextInfo argument. This is intended to support the possibility of client customizations filtering specifications by context. The caller of the endpoint will be expected to provide a context matching the 'importing context' they want to target (instead of whatever arbitrary context they may be browsing or initiating the import from). This will ensure the specification filtration occurs on the right context. Existing API callers should largely be unaffected with the default implementation, which does not truly use the context argument for any filtration logic.

Fix a bug in ImportDetailsGridExternal configuration where pagination was broken due to numbered vs narrowed misconfiguration

Fix an issue where CSV parsing logic in CSVFileReader and CSVErrorsReportService failed on Windows and classic Mac line separators

Introduce SkuInventoryImportSpecification to fix an issue where INVENTORY imports did not require inventory-specific scopes/permissions. With the latest changes, INVENTORY imports now require the INVENTORY scope and ALL_INVENTORY permissions.