Customer Master Data Incremental Query Sync: A 15-Minute Polling Strategy from ERP to Integration Platform

What This Strategy Solves

In one retail project, the same customer records were being maintained by hand in both the ERP and the marketing cloud. Three months later, the sales team tried to reconcile the two systems and found that more than a hundred codes did not match. The root cause was simple: there was no unified query-and-sync channel for master data. We used the Qeasy Data Integration Platform (Qeasy) to take over. From the ERP side, we pulled active customers into the platform in incremental windows based on modify_time. All downstream document flows then used the platform data as the single source of truth, and the problem was resolved.

Data Flow and Field Mapping

The data flow is unidirectional. The source ERP is the authoritative owner of customer master data, and the target is the Qeasy integration platform itself. This strategy is only responsible for pulling the data completely and on time, leaving it for downstream strategies to consume.

The source request uses a time window plus pagination. Key fields are mapped as follows:

The key fields returned by the source response are: number (customer code, business primary key), id (system internal primary key), and modify_time (modification time, used for watermark advancement). On the target side, this strategy uses a "write no-op" — it does not write directly to a target business system, but instead stores the data in the platform's transit layer, ready for downstream strategies to subscribe to as needed.

How to Configure It in Qeasy

When configuring this strategy in the Qeasy Data Integration Platform, the core is three things: API registration, variable binding, and empty-target handling.

API Registration: Choose the ERP category for the source platform, fill the API path with /jdy/v2/bd/customer, method GET, effect QUERY. For the target platform, choose the Qeasy integration platform itself, method POST, effect EXECUTE. The request body of this step is empty and only serves as a callback trigger for downstream processes.

Variable Binding: In the request parameters, bind modify_start_time to {{LAST_SYNC_TIME}}*1000, and modify_end_time to {{CURRENT_TIME}}*1000. There are two details that often cause trouble here: first, the timestamp unit must be milliseconds; second, the watermark should be advanced based on the modify_time of the latest record returned by the source API, not based on the platform's current time — otherwise you will get "duplicates pulled" or "rows missed".

Empty Target Handling: The number and id of the target API are both filled with 0, and idCheck is set to true. This is so that the platform generates an execution batch number for each batch of data, making it easy for downstream strategies to trace by batch. A common pattern among Qeasy customers is "centralized management of code mappings": place the mapping table between the source system number and the platform-internal UUID on the platform side, and downstream writes can directly look up this table, avoiding each strategy maintaining its own mapping.

Implementation Steps

There are three steps in the rollout, advancing steadily.

Step 1: Full Initialization. On the day of deployment, run a one-time full load, pulling all "in-use" customers from the ERP into the platform at once. In this step, set modify_start_time to a very early fixed timestamp (for example, 2020-01-01), and modify_end_time to the current time. Paginate through all pages until done. After the full load is complete, enable the incremental mode.

Step 2: Enable Incremental Polling. Configure the crontab as */15 8-22 * * *, that is, run every 15 minutes from 8 AM to 10 PM each day. High-frequency sync during daytime business hours, and stop at night. This both ensures business-perceived real-time performance and avoids the ERP nightly closing peak.

Step 3: Cover Night-time Gaps. Add one more nightly compensation run at 2:23 AM each day, with the crontab 23 2 * * *. The time window covers data from 10 PM the previous day to 2 AM the current day, to avoid missing any modifications on the ERP side at night. This is the "incremental plus full-coverage dual track" pattern often used by Qeasy customers: high-frequency incremental during the day, with a fallback full-coverage run at night.

Pitfall Review

Watermark Advancement Misalignment. A typical mistake is to advance the watermark using the platform's current time, which causes the same batch of data to be repeatedly pulled to the downstream and produces dirty data on the platform side. The reliable approach is to use the maximum modify_time returned by the source API as the new LAST_SYNC_TIME.

Pagination Boundary Misses. When page_size is set to 50, if the last page has fewer than 50 records, forgetting to handle the "exit when the next page is empty" logic in the loop will cause an infinite loop or missed pulls. We recommend using the built-in paginator in Qeasy, setting a maximum page-count upper bound as a safeguard.

Night-time Data Vacuum. Configuring only the daytime 15-minute polling means that any customer modifications on the ERP side at night will not be visible before the next business day's opening. Adding a dawn compensation task solves it.

Inconsistent Code Formats. The ERP-side customer code may be a mix of letters and numbers. If the platform side parses it as a pure number, precision will be lost. We recommend turning on "transmit as string" in the source API request body to avoid type conversion issues.

Empty Target Loses Batch Traceability. If the target API does not generate a batch number, tracing "when was this batch pulled" becomes very painful later. We generally enforce a standard that even empty targets must have a batch callback — this is the standard practice agreed upon on Qeasy.

When This Applies and When It Does Not

Applicable scenarios: the ERP is the master data source for customer records, and active customers need to be reliably synced into the integration platform, ready to be subscribed to by downstream documents, marketing, and reporting scenarios. Not applicable: scenarios requiring bidirectional sync (platform changes also need to be written back to the ERP), scenarios where the same customer record has write entries in multiple systems, and scenarios where a single customer record has more than 200 fields and requires deep customization.