Directory Service Connector is offline. for Reload and Distribution jobs during upgrade

[jbehan]

We are upgrading from QlikView 12.1 to QlikView 12.7 onto new servers, into a new domain.  Most all of our  Reloads work without issue, however none of our Reload and Distributes for our App-UI’s do.  We consistently receive the following error “QDSMain.Exceptions.DscOffLineException: Directory Service Connector is offline.”

In both Windows’ Services and the QMC, the QDS service is clearly running.  All services are using the a service account from the new domain.  The new service account has access to all of our file locations.  At this point, we are thinking that the issue is more related to Distribution Services than Directory Service Connector – but we’re open to suggestions.  Thank you.

[Chip_Matejowsky]

Hi @jbehan,

So this issue is occurring on only one Publisher reload/distribution task post upgrade to v12.70?

Can you provide the full task log, or at least portion with all of the errors?

For the task, what User Type is the QVW being distributed? Check to see if any of the users that the QVW is being distributed to are no longer in the user directory.

Best Regards

[Jill_Masinde]

@jbehan  It is also importanct to understand if this is a cluster environment.  Is the QDS and DSC running on different/same machine?  If running on different machine are they using the same service account?  

Verify that you can connect to DSC .  In QMC, go to System->Setup->Directory Service Connectors and click the DSC cluster entry. The Summary page copy the link to connect to the DSC instance. 

Log in into the QDS machine using the QDS service account, start a browser and enter the DSC links in the address bar.  Are you able to connect to each one, or do they return an error page?

[jbehan]

Hello Jill.  Yes, our QDS is on a stand-alone VM, and all other services are on another system.  When I login to the server with my service account, and enter the QDS URL, I get an XML page in return.  I get the XML page on both servers.

[jbehan]

Hello Chip

I kept attempt to post but I do not see my new content after refreshing the page.  My apologies for any duplicates. 

This is across all Reload and Distributes. Reloads are fine. The QDS is on a stand-alone VM, just as it is in production.  I confirmed that the users in the Distribution all existed in Active Directory, and in the new domain. We've worked with both users and user groups as distribution members. 

Here is a representative error.

(3/1/2023 1:08:50 PM) Information: DistributeTask[27e5930a-16df-4887-a177-8c11ea729581] ("Reload and Distribute of InBasket Messages Summary.qvw"): TryStart - Starting task. Triggered by 'TaskCompletedTrigger'. Id:91783e5a-c2be-474e-8dab-024a8057d4be. ExecID:60ac0325-d685-4598-8a89-5b9cfbf5c7d7

(3/1/2023 1:08:50 PM) Information: Entering Task Execution.

(3/1/2023 1:08:50 PM) Information: ClusterID=1

(3/1/2023 1:08:50 PM) Information: QDSID=5a2d371d-8dc4-ee01-4ca9-2a8b31e2bfc4

(3/1/2023 1:08:50 PM) Information: TaskID=27e5930a-16df-4887-a177-8c11ea729581

(3/1/2023 1:08:50 PM) Information: ExecID=60ac0325-d685-4598-8a89-5b9cfbf5c7d7

(3/1/2023 1:08:50 PM) Information: MaxRunTime=1.00:00:00

(3/1/2023 1:08:50 PM) Information: MachineName=S188-APQVDEV02

(3/1/2023 1:08:50 PM) Information: Task execution max attempts=1

(3/1/2023 1:08:50 PM) Information: Task execution current attempt=1

(3/1/2023 1:08:50 PM) Information: Task Dependencies are OK

(3/1/2023 1:08:50 PM) Information: The task "Reload and Distribute of InBasket Messages Summary.qvw" is starting

(3/1/2023 1:08:50 PM) Information: Document is marked to be Reloaded with fresh data. Initializing Reload for Distribution.

(3/1/2023 1:08:50 PM) Information: Opening "\\s188-apqvdev02\qlikview\PRD\App - UI\InBasket Messages Summary.qvw"

(3/1/2023 1:08:50 PM) Information: Allocating new QlikView Engine. Current usage count=0 of 9 (of type non-reader).

(3/1/2023 1:08:50 PM) Information: Max retries:5

(3/1/2023 1:08:50 PM) Information: Attempt:01

(3/1/2023 1:08:51 PM) Information: Opened the QlikView Engine successfully. ProcessID=395344

(3/1/2023 1:08:51 PM) Information: Allocated QlikView Engine successfully. Current usage count=1 of 9 (of type non-reader). Ticket number=1806.

(3/1/2023 1:08:51 PM) Information: Loading document "\\s188-apqvdev02\qlikview\PRD\App - UI\InBasket Messages Summary.qvw" (5220.79 Mb)

(3/1/2023 1:08:52 PM) Information: Loading. LoadTime=00:00:01.0005923

(3/1/2023 1:08:54 PM) Information: Loading. LoadTime=00:00:03.0024781

(3/1/2023 1:08:58 PM) Information: Loading. LoadTime=00:00:07.0044934

(3/1/2023 1:09:06 PM) Information: Loading. LoadTime=00:00:15.0068016

(3/1/2023 1:09:11 PM) Information: Physical FileSize=5220.79 Mb. Memory Allocation Delta for this file=20.88 Mb. Available Physical Memory Before Open=247688.93 Mb. Available Physical Memory After Open=249798.62 Mb. Total Physical Memory=262143.42 Mb.

(3/1/2023 1:09:11 PM) Information: Attempted to load the document without data.

(3/1/2023 1:09:11 PM) Information: The document was loaded successfully.

(3/1/2023 1:09:11 PM) Information: Document was opened successfully

(3/1/2023 1:09:11 PM) Information: Starting reload

(3/1/2023 1:09:11 PM) Information: The Source Document is being reloaded. DocumentPath=\\s188-apqvdev02\qlikview\PRD\App - UI\InBasket Messages Summary.qvw

(3/1/2023 1:09:12 PM) Information: Reloading.

(3/1/2023 1:09:13 PM) Information: Reloading..

(3/1/2023 1:09:14 PM) Information: Reloading...

(3/1/2023 1:11:34 PM) Information: Slow down logging. Log every <60> seconds.

(3/1/2023 1:11:34 PM) Information: Reloading

(3/1/2023 1:11:49 PM) Information: The Source Document reload complete. DocumentPath=\\s188-apqvdev02\qlikview\PRD\App - UI\InBasket Messages Summary.qvw

(3/1/2023 1:11:49 PM) Information: Memory Allocation Delta for this file=18585.95 Mb. Available Physical Memory Before Reload=249798.50 Mb. Available Physical Memory After Reload=231331.67 Mb.

(3/1/2023 1:11:49 PM) Information: The Source Document was reloaded successfully.

(3/1/2023 1:11:49 PM) Information: QlikView->Settings->Document Peferences->Generate Logfile in document is not set.

(3/1/2023 1:11:49 PM) Information: Reload finished successfully

(3/1/2023 1:11:49 PM) Information: Start saving document

(3/1/2023 1:13:05 PM) Information: Check in document: \\s188-apqvdev02\qlikview\PRD\App - UI\InBasket Messages Summary.qvw

(3/1/2023 1:13:05 PM) Information: Closing the document.

(3/1/2023 1:13:08 PM) Information: Closed the QlikView Engine successfully. ProcessID=395344

(3/1/2023 1:13:08 PM) Information: Start document distribution.

(3/1/2023 1:13:08 PM) Information: Opening source document

(3/1/2023 1:13:08 PM) Information: Opening document: "InBasket Messages Summary.qvw"

(3/1/2023 1:13:08 PM) Information: Allocating new QlikView Engine. Current usage count=0 of 9 (of type non-reader).

(3/1/2023 1:13:08 PM) Information: Max retries:5

(3/1/2023 1:13:08 PM) Information: Attempt:01

(3/1/2023 1:13:10 PM) Information: Opened the QlikView Engine successfully. ProcessID=36876

(3/1/2023 1:13:10 PM) Information: Allocated QlikView Engine successfully. Current usage count=1 of 9 (of type non-reader). Ticket number=1807.

(3/1/2023 1:13:10 PM) Information: Loading document "\\s188-apqvdev02\qlikview\PRD\App - UI\InBasket Messages Summary.qvw" (5274.53 Mb)

(3/1/2023 1:13:11 PM) Information: Loading. LoadTime=00:00:01.0003615

(3/1/2023 1:13:13 PM) Information: Loading. LoadTime=00:00:03.0003365

(3/1/2023 1:13:17 PM) Information: Loading. LoadTime=00:00:07.0020767

(3/1/2023 1:13:25 PM) Information: Loading. LoadTime=00:00:15.0067976

(3/1/2023 1:13:41 PM) Information: Loading. LoadTime=00:00:31.0162192

(3/1/2023 1:14:13 PM) Information: Loading. LoadTime=00:01:03.0324894

(3/1/2023 1:15:02 PM) Information: Physical FileSize=5274.53 Mb. Memory Allocation Delta for this file=16993.03 Mb. Available Physical Memory Before Open=248238.56 Mb. Available Physical Memory After Open=232890.18 Mb. Total Physical Memory=262143.42 Mb.

(3/1/2023 1:15:02 PM) Information: Attempted to load the document with data.

(3/1/2023 1:15:02 PM) Information: The document was loaded successfully.

(3/1/2023 1:15:05 PM) Information: Reduce skipped, nothing is selected.

(3/1/2023 1:15:05 PM) Information: Document was opened successfully

(3/1/2023 1:15:05 PM) Information: Determined 3 static recipients spread over 1 Distribution Resources.

(3/1/2023 1:15:05 PM) Information: Deleting temporary files.

(3/1/2023 1:15:05 PM) Information: Closing the document.

(3/1/2023 1:15:07 PM) Information: Closed the QlikView Engine successfully. ProcessID=36876

(3/1/2023 1:15:07 PM) Information: Document distribution done.

(3/1/2023 1:15:07 PM) Error: The task "Reload and Distribute of InBasket Messages Summary.qvw" failed. Exception: || QDSMain.Exceptions.TaskFailedException: Task execution failed with errors to follow. ---> QDSMain.Exceptions.DscOffLineException: Directory Service Connector is offline. || at QDSMain.DistributeTask.DistributionExecution(String fileName, Document qvbDocument, DistributionRequest distributionRequest, IExecutingTaskResult executingTaskResult, String repeatVariableName, String currentRepeatVariableValue) || at QDSMain.DistributeTask.Distribution(String fileName, DistributionRequest distributionRequest, IExecutingTaskResult executingTaskResult, String repeatVariableName, String currentRepeatVariableValue) || at QDSMain.DistributeTask.PerformExecute(IExecutingTaskResult executingTaskResult) || --- End of inner exception stack trace --- || at QDSMain.DistributeTask.PerformExecute(IExecutingTaskResult executingTaskResult) || at QDSMain.Task.AbstractTask.TaskExecution(CurrentExecutionArgs args)

(3/1/2023 1:15:07 PM) Information: Task Execute Duration=00:06:17.6801746

(3/1/2023 1:15:07 PM) Information: Sending Alert mail

(3/1/2023 1:15:08 PM) Warning: Failed to send mail to 6082132569@vtext.com

(3/1/2023 1:15:08 PM) Information: TaskResult.status=Finished

(3/1/2023 1:15:08 PM) Information: Notifying all triggers of new state: "FinishedWithErrors"...

(3/1/2023 1:15:08 PM) Information: Notifying all triggers of new state: "FinishedWithErrors" - Completed

[jbehan]

Hello @Chip_Matejowsky
I kept attempt to post a response to you, but I do not see my new content after refreshing the page. Attempting to reply via Jill's email notification. My apologies for any duplicates.
This is across all Reload and Distributes. Reloads are fine. The QDS is on a stand-alone VM, just as it is in production. I confirmed that the users in the Distribution all existed in Active Directory, and in the new domain. We've worked with both users and user groups as distribution members.
Here is a representative error.

(3/1/2023 1:08:50 PM) Information: DistributeTask[27e5930a-16df-4887-a177-8c11ea729581] ("Reload and Distribute of InBasket Messages Summary.qvw"): TryStart - Starting task. Triggered by 'TaskCompletedTrigger'. Id:91783e5a-c2be-474e-8dab-024a8057d4be. ExecID:60ac0325-d685-4598-8a89-5b9cfbf5c7d7
(3/1/2023 1:08:50 PM) Information: Entering Task Execution.
(3/1/2023 1:08:50 PM) Information: ClusterID=1
(3/1/2023 1:08:50 PM) Information: QDSID=5a2d371d-8dc4-ee01-4ca9-2a8b31e2bfc4
(3/1/2023 1:08:50 PM) Information: TaskID=27e5930a-16df-4887-a177-8c11ea729581
(3/1/2023 1:08:50 PM) Information: ExecID=60ac0325-d685-4598-8a89-5b9cfbf5c7d7
(3/1/2023 1:08:50 PM) Information: MaxRunTime=1.00:00:00
.....
(3/1/2023 1:15:02 PM) Information: The document was loaded successfully.
(3/1/2023 1:15:05 PM) Information: Reduce skipped, nothing is selected.
(3/1/2023 1:15:05 PM) Information: Document was opened successfully
(3/1/2023 1:15:05 PM) Information: Determined 3 static recipients spread over 1 Distribution Resources.
(3/1/2023 1:15:05 PM) Information: Deleting temporary files.
(3/1/2023 1:15:05 PM) Information: Closing the document.
(3/1/2023 1:15:07 PM) Information: Closed the QlikView Engine successfully. ProcessID=36876
(3/1/2023 1:15:07 PM) Information: Document distribution done.
(3/1/2023 1:15:07 PM) Error: The task "Reload and Distribute of InBasket Messages Summary.qvw" failed. Exception: || QDSMain.Exceptions.TaskFailedException: Task execution failed with errors to follow. ---> QDSMain.Exceptions.DscOffLineException: Directory Service Connector is offline. || at QDSMain.DistributeTask.DistributionExecution(String fileName, Document qvbDocument, DistributionRequest distributionRequest, IExecutingTaskResult executingTaskResult, String repeatVariableName, String currentRepeatVariableValue) || at QDSMain.DistributeTask.Distribution(String fileName, DistributionRequest distributionRequest, IExecutingTaskResult executingTaskResult, String repeatVariableName, String currentRepeatVariableValue) || at QDSMain.DistributeTask.PerformExecute(IExecutingTaskResult executingTaskResult) || --- End of inner exception stack trace --- || at QDSMain.DistributeTask.PerformExecute(IExecutingTaskResult executingTaskResult) || at QDSMain.Task.AbstractTask.TaskExecution(CurrentExecutionArgs args)
(3/1/2023 1:15:07 PM) Information: Task Execute Duration=00:06:17.6801746
(3/1/2023 1:15:07 PM) Information: Sending Alert mail
(3/1/2023 1:15:08 PM) Information: TaskResult.status=Finished
(3/1/2023 1:15:08 PM) Information: Notifying all triggers of new state: "FinishedWithErrors"...
(3/1/2023 1:15:08 PM) Information: Notifying all triggers of new state: "FinishedWithErrors" - Completed

[Chip_Matejowsky]

Hey @jbehan,

Potentially issue could be that TCP port 4730 is blocked from the Publisher server to the other server running the DSC. So suggest that you check to see if port is open. Note that it is perfectly fine to have a DSC installed on multiple servers in a QVS environment, so you may want to install the DSC service on the Publisher server as well as the QVS server.

Also, can you check the DSC and QMS logs for warnings/errors? I'd like to know if there are any notable entries there. 

Do you have a case open with Qlik Support? If not, suggest that you open one for direct assistance.

Best Regards

[jbehan · Accepted solution]

Thanks for your response, @Chip_Matejowsky.

We were able to open a telnet session from our Publication server to our QVS/QDS with the IP and 4730.  Also, we are were not noting any errors or warnings in our logs – other than those related to DSC. 

We did, however, resolve the issue.  As you suggested, we installed the DSC directly on the QDS node within our cluster.  Although the documentation recommends the DSC be installed on the same system as the Management Service in order to reduce chatter, it was sounded like it was still an acceptable approach.  As soon as we had it installed, the Reload and Distribute jobs worked.  We believe the failures were really related to distribution, as files were being reached, and reloads were occurring.

About an hour after the fix, we had our scheduling meeting with Qlik support.  Based on our troubleshooting steps and findings, the best guess was that the DSC portion of the installation must have been corrupted.  Perhaps it was somehow interrupted in some way due to the number of anti-virus/fire-wall related services in play.  In retrospect, I would recommend eliminating all AV/FW prior to installation, then made my file copies (this was a migration), and then applied the AV scanning exclusions right away.  It is possible that an installation repair of the DSC would have resolved this issue.  I would have preferred the DSC being on the same node as the Management Service as recommended.