Full Load with CDC vs CDC

[user12345AA]

HI All,

I'm beginner and exploring the things.

My scenario : From Source I have to pull all data and load into target, then it has to pull the CDC data daily from Source to target based on the schedule.

I bit confused with the behaviour of Full load with Applied changes and Applied Changes alone.

Full Load and Applied Changes:

When I create a task with both Full load and Applied Changes and upon scheduling the task, each time it is identifying the records under Full load and loading into Target table. Suppose If I keep my Task as in running status, then the CDC records being captured. 

Applied Changes:

When I create a task with Applied changes and schedule to run, it is not identifying the CDC data, unless the task is in running.

Please suggest me , on how I should create the task in order to work for my scenario. 

[sureshkumar]

Hello @user12345AA 

Kindly refer below article

An Introduction to Qlik Replicate Tasks: Full Load... - Qlik Community - 2019167

Regards,

Suresh

[DesmondWOO]

Hi @user12345AA ,

For your scenario, you should choose "Full Load and Applied Changes".

"Applied Changes" alone does not perform data loading from the source table to target table. Instead of, Replicate reads changes from the transaction log (depends on the source endpoints) and applies changes to the target database. 

Regards,
Desmond

[john_wang]

Hello @user12345AA ,

Thanks for reaching out to Qlik Community!

In Qlik Replicate, there are 2 stage of a task: Full Load , and Change Processing. The 2 stages can be enabled both, or either one.

• Full Load (with Change Processing disabled)
  In Full Load stage, Qlik Replicate verifies and detects the target table (depending on the task "Target Table Preparation" setting), re-create, or truncate the target table before loading the data.
  In this stage, Qlik Replicate initialize the existing history data rows, and replicate them from source endpoint DB to target endpoint DB. Please take note in this stage Qlik Replicate retrieve data from source DB tables.
• CDC Change Processing
  In Change Processing stage, Qlik Replicate gets changes from source endpoint transaction log and applies the change to target endpoint DB. Please take note in this stage Qlik Replicate retrieve changing data from source transaction log (eg Redo Log in Oracle, TLOG in SQL Server , OPLOG in MongoDB etc).
• Full Load + Change Processing
  If both of them are enabled, whenever Qlik Replicate task is starting up, it will startup Change Processing thread first and cache all the changes during the Full Load stage; IN the meanwhile, the Full Load stage will initialize the history data rows. after the existing Full Load stage done, the CDC Change Processing will follow up smoothly. This is automatically switch oever.

Hope this helps.

John.

[user12345AA]

Thanks for the reply @ John_wang

Will the above stages works with same behaviour,even when it is scheduled because I have seen if my task is in running status then only the both stages  CDC and Fullload+CDC started processing CDC data. When I schedule the CDC stage not loading the data and Fullload+CDC going on with history data everytime.

[john_wang]

Hello @user12345AA ,

If both Full Load and Change Processing (see below sample) are set to ON:

Then after the task startup, the task will keep in running because CDC processing is a continuous progress - unless the task is stopped manually , or by scheduler.

So far for your question, there is no difference between the task startup/stop manually (eg via Qlik Replicate, or QEM console), or via scheduler.

Feel to let me know if you need any additional information.

Regards,

John.

[john_wang]

Hello @user12345AA ,

If both Full Load and Change Processing (see below sample) are set to ON:

Then after the task startup, the task will keep in running because CDC processing is a continuous progress - unless the task is stopped manually , or by scheduler.

So far for your question, there is no difference between the task startup/stop manually (eg via Qlik Replicate, or QEM console), or via scheduler.

Feel to let me know if you need any additional information.

Regards,

John.

[user12345AA]

When I schedule a task , after the completion it gets stopped, in that case you mean we cannot get the CDC records from source. Ideally the task should be in running status to get the CDC records. Is my understanding correct?