Stress test QMS (QMC) - Issue

[Sahal]

Hi,

We have a new installed QlikView publisher environment with 3 NODES (for now: May 2021 SR2), that have some issue with.

We created 30 tasks and started them all (10 on each node) to stress test the QMS/QDS and what we noticed was that Queued and/or Started tasks got stuck in QMC (Status) and did not finish, fail or move on to next execution plan. To go around this issue we have to restart the QDS or "Send Work Order" from Setup->QDS->Advance sheet for the status tasks to be "finished" (All tasks are finished but just stuck...)

We try to play around with QVManagementService.exe.config and the QMSChunkSize/"MaxReceivedMessageSize" setting but don´t really know what limitation and/or if we should changes these settings when multiple task are running at the same time or not.

Might we need to create a bugg ticket with QlikTech on this, or is there some changes that can be made, to not get this fixed? 

[Chip_Matejowsky]

Hi @Sahal,

Based upon your description it sounds that there are communication issues between the QlikView Management Service (QMS) and the 3 QlikView Distribution Service (QDS) nodes.

Suggest that you review the Qlik whitepaper Scaling the QlikView Publisher that I provided in your previous forum post, as it details common issues and steps to improve QlikView Publisher performance; especially in clustered environments. The ApplicationData Folder section does a great job in describing the back end processes involved and the Best Practices section provides practical configuration actions you can take to improve QDS performance.  Please shout back if you have any questions on this whitepaper.

Best Regards

[Sahal]

Hi @Chip_Matejowsky 

Thank you for your feedback(s) they are really helpful. 

In this case, we have made different changes in the config files (even changes that are not present by default) for all QDS, server level changes,  separated the disks to hold different folders (App data, Source document etc..) but still no luck,  I would say we are trying to find needle in a haystack for this one.....

Our tests are using 30 tasks that are 3 MB in size (same qvw file just copied 30 times) and they run for 20 sec, no heavy applications because our test is to "stress test QMS ".. But like you said, Yes the issue is between the QMS and QDS or rather the workload/task results that for some reason when starting multiple tasks its unable to update the result and the tasks get stuck even if they are finished.

There is no error and/or any information in any logfiles becasue there is "nothing wrong" (well there is) only that the QMC gui wont update ...

Might this be a task for support? 

UPDATE* We just updated the version to May 2022 SR2 (12.70.20200.0) latest version - Still same issue!

[Chip_Matejowsky]

Hi @Sahal,

Is the QlikView Management Service running on it's own server, or do you have it installed along with a QlikView Distribution Service? For multinode Publisher environments, it is recommended to have QMS installed on it's own server. Which of the entries in the %ProgramFiles%\QlikView\Distribution\QVDistributionService.exe.config file did you edit or add? 

There could be some disk access latency causing the communication issues. Check the QDS Application Data folder (defined in QMC > System > Setup > Distribution Services > QDS@ > General tab) on each QDS node for Cluster_date.txt files. From the whitepaper:

If  the QlikView Distribution Service process has failed to reach the files in the Notification System for more than 10 minutes the log files, with the name "Cluster_date.txt," will be created in the "ApplicationData" subfolder for each QlikView Distribution Service node. This is an indication of disk access latency which can occur when there is a disk I/O request of more that 2-3 requests over time.

To alleviate the latency, an additional step you can take is to move the QDS Notification files to their own directory. This is especially necessary when dealing with a high load system. This will move the Task Execution notifications, the load balancing file, and the task execution history notification files. Refer to article How to move the QlikView Distribution Service (QDS) Notification files.

Best Regards

[Sahal]

Hi @Chip_Matejowsky 

•  QMS is installed on its own server.
• The file entries that we have edited is this (see image):
• No Cluster_date.txt files is created on any node (see image from one of the nodes)
• Most of the files have been moved to separate disks Notification files have been moved to its own disk on fileshare server (that is not heavily used at all)

I'm not sure if this is a bug or something we are doing is missing, however as i stated, the qvw files are small and when we run the stress test we run the tasks over and over again... I would believe that you guys (at Qliktech) have done the same but with more tasks, and i would be surprised if you have not encountered the same issue...

[Chip_Matejowsky]

Suggest that you add the following entry to QDS config.exe file (NOTE: only needs to be added to one of the config.exe files in the cluster, not all 3)
<!-- Purges deleted task and it's triggers and results. Once per day. Should only be enabled for one QDS node in a cluster. --> 

<add key="EnableAdvancedFilePurge" value="true"></add>

If set to “true,” the QlikView Distribution Service (QDS) process is more aggressive when cleaning out historical records from the Trigger files primarily. This reduces the size of the files, which allows for improved throughput in terms of faster opening, writing and closing of the files. This is more preeminent on tasks with frequent runs (many times per day, for example). It can be diagnosed by sorting the contents of the “TaskResults” subfolder in the “ApplicationData” folder by size. The TaskResult XML files should be small, less than 50 Kb. File sizes of hundreds of Kb or even a few Mb represent a challenge for the QDS process as a larger file takes longer to lock, open and read. 

Apart from all of these suggestions, your next best step would be to have a discussion with your Qlik account manager about an engagement with Qlik Professional Services.

Best Regards

[Sahal]

Hi,

That option we had a look at, but because our XML files in task results are less then 50KB we decided that it was unnecessary to do so, But are you saying  that even if the files are < 50 KB we should do it? 

We have created a incident on the support portal for this, because this needs to be looked on from QlikTechs side more. 

[Chip_Matejowsky]

Yes as it will help alleviate the processing burden on QDS.

Best Regards 

[Sahal · Accepted solution]

In

This was the one that we was looking for (don't know if all other changes made a difference but good to have in a big environment) 

QVManagementService.exe.config

<!-- Interval for retrieving all task statuses from Distribution Service to update the local cache (in minutes) -->
<add key="FullTaskStatusRefreshIntervalInMinutes" value="1" />

[Chip_Matejowsky]

Just a word of caution - moving from default value of 30 minutes to 1 minute may cause issues. You may want to set the value to 5 or 10.

Best Regards