Increase performance workflow

[MartijnWanders]

Hi,

I want to understand how it is possible to increase the runtime of my workflow. My customer want to have data reloaded within minute from data warehouse to data mart.

I have one Data Warehouse task (full load) with a runtime of 1min15 and a data mart task (incremental) with runtime 35sec if I run then manually.

If I create a workflow then it the total time is 2min10. Somehow my data mart has a longer runtime in my workflow. I can not run in paralell and it must be in a workflow. I tried to split the data warehouse task after a duplication. But then I can not run paralell either.

We use Snowflake as warehouse tool. Within Snowflake you can tweek some features to increase warehouse size for faster calculating and process more concurrent queries with adding clusters.

But If I do this then I see nothing happening with runtime. I have the feeling that Qlik sent every querie in sequence so you don't have many concurrent queries to process (I think). In Snowflake I don't see queed queries either.

Does somebody struggled with the same challenge? Or is this the max what I'm asking from Compose. If that is the case  no problem. I'm just curious how this works.

Thank you very much!

Greetz Martijn W.

[Nanda_Ravindra · Accepted solution]

@MartijnWanders Workflow performance really depends on several factors, like the maximum number of database connections, JVM memory setting, and the hardware you are using for Compose. This being said, in your comment you mentioned you didn't see anything in Snowflake that queed queries. So I don't think tweaking in Snowflake will help you in this.

When you run a datamart, we first load the dims and then we load the fact table. If dims have no relation within them then the loading of dims goes in parallel. I would suggest you make sure you have the following default setting in your setup. If this doesn't help with the performance, then I would suggest you engage with our professional service team who can assist you with this. 

In Data warehouse setting:

Maximum number of database connections: 10

JVM Memory:   -Xms1024M -Xmx4096M

In the Databmart setting:

Load type set to Incremental loading

@MartijnWanders Workflow performance really depends on several factors, like the maximum number of database connections, JVM memory setting, and the hardware you are using for Compose. This being said, in your comment you mentioned you didn't see anything in Snowflake that queed queries. So I don't think tweaking in Snowflake will help you in this.

When you run a datamart, we first load the dims and then we load the fact table. If dims have no relation within them then the loading of dims goes in parallel. I would suggest you make sure you have the following default setting in your setup. If this doesn't help with the performance, then I would suggest you engage with our professional service team who can assist you with this. 

In the Data warehouse setting:

Maximum number of database connections: 10

JVM Memory:   -Xms1024M -Xmx4096M

Below is the link for determining the required number of database connections:

https://help.qlik.com/en-US/compose/May2022/Content/ComposeDWDL/Main/DW/General/determining_db_conne...

Hope this helps!

Thanks,

Nanda

[MartijnWanders]

Hi Nanda, thank you for your reply and I have the same settings as you mentioned. I will contact professional services.