Practical Tutorial for the Query Xiaoman Products Strategy: Pulling Master Data from CRM into the Middleware Layer
What This Strategy Solves
In a real engagement with a retail-and-manufacturing customer, the CRM held the authoritative product master data while the downstream ERP was the source for all subsequent business documents. Once product codes diverged between the two, orders, inventory and pricing all silently went wrong. The first thing we did on the Qeasy integration platform was to pull the product data from the CRM into a middleware layer in a stable, repeatable way. The strategy named Query Xiaoman Products plays exactly this role — it writes nothing to a target system, only fetches data through an incremental window.
Data Flow and Field Mapping
The data flow is clean: Xiaoman OKKI CRM → Qeasy Integration Platform (middleware) → waiting for the next write strategy to land in the downstream ERP. This strategy only covers the first half.
| Dimension | Source (Xiaoman OKKI CRM) | Target (Qeasy Middleware) |
|---|---|---|
| API path | /v1/product/list (GET) | Write no-op (POST) |
| Effect | QUERY | EXECUTE |
| Business key | product_no | 0 (placeholder, mapped by next strategy) |
| Idempotency check | idCheck disabled | idCheck enabled |
| Auto-fill response | autoFillResponse=true | — |
The request is a typical page-plus-time-window payload: start_index, count, start_time, end_time, removed. start_time and end_time are the scheduling core and are bound to {{LAST_SYNC_TIME|datetime}} and {{CURRENT_TIME|datetime}} so that each run only takes changes since the last successful sync.
How to Configure on Qeasy
On the Qeasy data integration platform, configure this strategy with the following points:
- Create the strategy using the Query template. Set the source platform to Xiaoman OKKI CRM and the target platform to the Qeasy platform itself.
- API metadata follows the source JSON:
/v1/product/list, GET,product_noas the business key,idCheckkept off. - Request parameters are maintained in the
requestarray: hard-codestart_index=1,count=20,removed=0; bindstart_timeandend_timeto the platform's built-in time variables. - Response handling turns on
autoFillResponseso the response is materialized as an observable data view, which is invaluable for troubleshooting. - Target side selects the Write no-op API, which lands the data in the middleware layer instead of writing directly to the ERP. This is a common pattern on customer sites: land first, distribute later. Centralized code-mapping management is kept in the downstream strategy rather than mixed into the first one.
- Scheduling is set per
crontab: source at 03:02, target at 02:23 — staggered to avoid overlap.
Implementation Steps
We recommend a three-phase rollout: incremental start point, full backfill, schedule frequency.
- Incremental start point: the first run should not rely on
LAST_SYNC_TIME. Manually set an explicit starting date (for example, "2024-01-01 00:00:00") to avoid scanning the entire table on day one. - Full backfill: add a one-shot manual trigger on Qeasy for initial historical data. After the full backfill is complete, switch back to incremental-window mode — this is the typical dual-track approach of incremental plus full.
- Schedule frequency: once a day in the early morning is enough. CRM product master data rarely changes more than a few times per day, and over-scheduling only burns API quota.
- Dependencies and sequence: in the orchestration, this strategy is sequence A; the downstream write-to-ERP strategy is sequence B with
depends_onA, forming a clear chain.
Pitfalls Recap
- Using
CURRENT_TIMEforstart_time: a classic mistake is to set bothstart_timeandend_timeto the current time, which returns an empty window every time. The safe approach isstart_time = last successful time,end_time = current time. - Pagination not fully traversed:
count=20is the default and will silently lose records once the product catalog grows. Configure a paging loop in Qeasy until the returned record count is smaller thancount. - Ignoring the
removedfield: on customer sites, we often see reconciliation disputes where a product is deleted in the CRM but still in use in the ERP. Always runremoved=0andremoved=1in separate windows and periodically sync deletion events. - Writing directly to the downstream ERP in this first strategy: this strategy is intentionally limited to landing in the middleware. Forcing an ERP write into the very first query strategy mixes code-mapping, field trimming and retry logic together and becomes unmaintainable.
- Scheduling collisions: source at 03:02 and target at 02:23 are deliberately staggered to give the landing enough time. When both are set to the same minute, the downstream strategy will pick up data that has not yet finished landing, producing mysterious mismatches.
When to Use and When Not to Use
Use when: master data lives in the CRM and needs to be stably pulled into a middleware layer by a time window, with a separate write strategy distributing to the ERP afterward. Do not use when: real-time sync is required (use event triggers instead of polling); the source itself is already a middleware and no second landing is needed; or the business is highly sensitive to deletion events and must use a subscription API rather than polling.