Qeasy Data Archiving & Cleanup in Practice: Batch Deletion of Data Older Than Two Months by Strategy

What This Strategy Solves

After running integrations between multiple business systems for a long time, the intermediate layer accumulates a large amount of historical documents. When we worked on a supply chain integration for a retail enterprise, we ran into exactly this: dozens of sync strategies between the WMS and ERP systems piled up log tables and intermediate database entries quickly over half a year, slowing down queries and increasing storage costs. Worse, when cross-strategy tracing was needed, the old data itself became a "distraction."

The purpose of this strategy is very clear: delete intermediate data older than two months in batch by strategy name (Strategy) dimension, keeping only the recent traceability window. It belongs to the "system maintenance" level cleanup action rather than a business sync action, and is suitable as a fallback governance strategy for the integration platform.

Data Flow and Field Mapping

The data flow of the cleanup action is different from the regular A→B sync. It is a loop of "cleanup trigger → platform internal cleanup action → platform write-back log."

One design detail worth noting: after the deletion action is done, we also configure a "write empty operation" strategy (crontab 23 2 * * *), whose role is to trigger one more empty write right before the archiving window as a "pre-sentinel" for the successful execution of the cleanup action. This is a typical fallback pattern—if the empty write fails, the cleanup task will be delayed according to the dependency relationship, avoiding any action on dirty data.

How to Configure on Qeasy

In the Qeasy Data Integration Platform, the core of this strategy's configuration is two parts: API metadata + scheduling plan.

1. Metadata modeling: Add WebAPI metadata, select DeleteStrategyData for the API, method POST, effect set to QUERY (the platform processes the deletion as a query semantic). Keep idCheck enabled, buildModel disabled, autoFillResponse enabled, and the response fields datetime and params will be automatically filled by the platform.
2. Request object configuration: Add several object typed fields (target_1, target_2, ...) in the request array, each object's value left empty, and pass in the strategy names via variables. Here it is recommended to centralize the strategy names into a single mapping table to manage, avoiding scattering across multiple request objects—this is the "centralized encoding mapping management" pattern we most often do for clients.
3. Target-side configuration: Configure a "write empty operation" WebAPI metadata on the target side as a dependency node, ensuring that the upstream is stable when the cleanup action happens.
4. Scheduling plan: The source strategy crontab is set to 15 3 * * *, and the target empty-write strategy crontab is set to 23 2 * * *, using the time difference of cron to form a natural sequential order.

Implementation Steps

We generally launch cleanup strategies of this type in three phases to avoid one-shot problems.

Phase 1: Small-scale gray release. First pick 1-2 non-critical strategies to trial-run, such as the lower-priority ToC sales return order. Observe the params returned by the platform and the execution time, confirm that the deletion scope matches expectations (only the intermediate layer is deleted, not the source/target systems).

Phase 2: Expand coverage. Bring core business strategies such as ToB sales outbound orders and ToB sales return orders into the cleanup scope. At this point, it is recommended to also turn on the "full dry-run" switch—do one dry-run first, the platform only returns the amount of data to be deleted without actually executing, check it and then release.

Phase 3: Normal operation. The crontab triggers at 3:15 AM every day, keeping a two-month window. Two months is a relatively stable compromise: it can cover most cross-month reconciliation scenarios, and keeps the intermediate database at a reasonable size. If the business side requires a shorter window, just adjust the platform's internal time threshold, no need to change the field mapping.

Lessons Learned

1. Don't hard-code "two months ago" as a business field. At first, we tried to add a cutoff_date field in the request body for the user to pass themselves, and then problems appeared such as clients passing 2025-13-01 and time zone confusion. The safe approach is to leave the time window to the platform to handle internally, exposing only the strategy name set as input parameters.
2. Typical mistake: deleting the wrong strategy. When multiple strategies coexist, once the naming of target_1, target_2 is confusing, the cleanup scope will cross over. It is recommended to add an is_required=true check in the request object, and add comments for each strategy name in the mapping table.
3. Don't forget to configure the empty write strategy. Without the 23 2 * * * empty write pre-action, the cleanup action may execute before the data stabilizes, creating the illusion of "deleting too early."
4. Audit logs should be stored separately. The datetime and params returned by the platform are the only credentials for post-event traceability. It is recommended to drop them into an independent archive log table rather than mixing them with business logs.
5. Concurrency limits in the public cloud environment. If too many strategies are cleaned at one time (in our actual test, more than 20 strategies may trigger throttling), it is recommended to add random jitter to the crontab (change 15 3 * * * to one with random seconds), avoiding request concentration at one second.

Applicable and Inapplicable Scenarios

Applicable: Long-term accumulation of intermediate-layer data, query performance degradation; the business side only cares about the traceability range of the recent 1-3 months; a large number of integration strategies (>10), requiring fine-grained cleanup at the strategy level.

Not applicable: Historical data cleanup of source/target systems themselves (this solution only cleans the intermediate layer); industries with regulatory requirements for long-term archiving (such as pharmaceuticals, finance), where the two-month window may not meet compliance; and lightweight integration scenarios with very few strategies where the cleanup benefit is not obvious.