Resume processing and CDC only modifications of table filters

[feal]

Hello

We have a Qlik CDC only task and we performed a modification on a filer transformation on a table.

We stopped that task , perform Import task with the desired modifications and perform (for mistake) resume processing of the same task.

The filer was loaded correctly (and unexpectedly !)

We remember in the past documentation that , if we have to perform a filter modification, we are aware that we will perform stop+import of the task (or perform modification by Qlik Enterprise Manager GUI and save the currently running task) and perform reload target

Can you explain us what happened ? in what case resume processing is acceptable for filter modifications scanarios ?

Lt us know

Regards

Alessandro Ferrario

[john_wang]

Hello @feal ,

Thanks for reaching out to Qlik Community!

We'd like to confirm the behavior for you, if you can share the source and target endpoint type then we may provide more accurate information.

Regards,

John.

[ShaunaB1]

Hi,

I read this and I got some ideas related to it. You can consider it as a suggestion if you want. Here it's: 

If you change a filter in a Qlik CDC task and just resume it, it might still work because the task picks up changes from where it stopped. It’s like pausing and playing a video. But if you need to start over or from a specific point, you should reload the target. Always check Qlik’s guidelines for the best steps.

I hope my suggestion will help you to solve this issue.

[feal]

Hi Joghn

i attacehd JSON definition (Export with endpoints,) ,  log after import/resume propcessing  and log generated during the stop before modifications

Let me know

Regards

AF

[john_wang]

Hello @feal ,

Thank you for sharing the details.

I conducted a quick test utilizing Oracle as the source and SQL Server as the target endpoint. I observed that the updated filter expression takes effect following the RESUME task operation. Here are the steps I followed:

1. Applied a filter with "ID > 10".
2. Stop the task.
3. Modified the filter from "ID > 10" to "ID > 100", save the changes.
4. RESUME the task, confirming that the new filter expression became active.

To summarize, altering the filter expression does not necessitate the export/import of task JSON files unless you specifically aim to adjust task settings without utilizing the Replicate or QEM console GUI.

Additionally, I noticed an issue with your task as indicated in the log file at line #9:

2024-04-09T11:54:34 [TASK_MANAGER ]I: Task 'R1_HP_PERSON_STRUCTURE_CHANGES' running CDC only in resume mode after recoverable error, retry #5

I suggest opening a support ticket and attaching these files. Our support team will assist you in resolving the abnormal stopping issue and addressing any concerns regarding filter behavior. Additional information may be requested through the support ticket.

By the way, I'm deleting the log files in the community as they may contain sensitive information.

Regards,

John.

[feal]

Hi John

thank you for your quick answer.

By the way ;. this time we perform stop+import+resume processing NOT by Qlik GUI but with REST API commands : can this difference may be relevent in tabel filter transformation behaviour ?

Let me know

regards

AF

[john_wang]

Hello @feal ,

I do not think so. If you can provide your script in support case then I'd like to double-confirm for you.

Regards,

John.

[feal]

Hi John

so you can confirm that this operation/way

- stop task

- import of a task

- resume processing of the task 

is always possible , also in he case  of task is CDC only  and with modifications of table  filters?

So reload target is not mandatory for this scenario ?

?

Accordingly to current documentation there is no caveat or different specification on task re-start   

Adding or removing filter ranges #Adding or removing filter ranges | Qlik Replicate Help

Let us know 

regards

AF

[Heinvandenheuvel · Accepted solution]

You always need to STOP followed by some sort of start to make tasks or endpoint changes active, the only exception being LOGGING LEVELS which take effect immediatly (Just as well! :-).

It doesn't matter whether the (re)start is done with Replicate UI, AEM, AEM-API or REPCTL.

For a CDC only task a reload is not needed - nor possible.

For a FULL + CDC task a reload is forced and required to make sure the target data matches the rules.

As soon as you IMPORT (or SAVE) any table with any change will be in 'queued' state and reloaded on task activation. Using the GUI - SAVE you'll get some relatively clear messages (see below). Using task json import there will be no message, but the table(s) will be in the queued pile and will reload.

The only way to stop a table reload in a FULL+CDC task is to restart with RUN ADVANCED - TABLES ALREADY LOADED + TIMESTAMP. This may well be justified if you happen to know that say a filter change is appropriate for the target - but there is no way Replicate could make that decision as it does not understand business rules. And for bigger changes, like adding a column, resuming with tables already loaded would clearly be wrong and not give you that new column and similar without further manual interventions.

Hein.

[feal]

OK Thanks and regards !!

AF