LogAnalysis App: The Qlik Sense app for troubleshooting Qlik Sense Enterprise on Windows logs

Mario_Petre · Sep 9, 2022 4:09:13 AM

It is finally here: The first public iteration of the Log Analysis app. Built with love by Customer First and Support.

"With great power comes great responsibility."

Before you get started, a few notes from the author(s):

It is a work in progress. Since it is primarily used by Support Engineers and other technical staff, usability is not the first priority. Don't judge.
It is not a Monitoring app. It will scan through every single log file that matches the script criteria and this may be very intensive in a production scenario. The process may also take several hours, depending on how much historical data you load in. Make sure you have enough RAM 🙂
Not optimised, still very powerful. Feel free to make it faster for your usecase.
Do not trust chart labels; look at the math/expression if unsure. Most of the chart titles make sense, but some of them won't. This will improve in the future.
MOD IT! If it doesn't do something you need, build it, then tell us about it! We can add it in.
Send us your feedback/scenarios!

Chapters:

01:23 - Log Collector
02:28 - Qlik Sense Services
04:17 - How to load data into the app
05:42 - Troubleshooting poor response times
08:03 - Repository Service Log Level
08:35 - Transactions sheet
12:44 - Troubleshooting Engine crashes
14:00 - Engine Log Level
14:47 - QIX Performance sheets
17:50 - General Log Investigation
20:28 - Where to download the app
20:58 - Q&A: Can you see a log message timeline?
21:38 - Q&A: Is this app supported?
21:51 - Q&A: What apps are there for Cloud?
22:25 - Q&A: Are logs collected from all nodes?
22:45 - Q&A: Where is the latest version?
23:12 - Q&A: Are there NPrinting templates?
23:40 - Q&A: Where to download Qlik Sense Desktop?
24:20 - Q&A: Are log from Archived folder collected?
25:53 - Q&A: User app activity logging?
26:07 - Q&A: How to lower log file size?
26:42 - Q&A: How does the QRS communicate?
28:14 - Q&A: Can this identify a problem chart?
28:52 - Q&A: Will this app be in-product?
29:28 - Q&A: Do you have to use Desktop?

Environment

Qlik Sense Enterprise on Windows (all modern versions post-Nov 2019)

How to use the app:

Go to the QMC and download a LogCollector archive or grab one with the LogCollector tool
Unzip the archive in a location visible to your user profile
Download the attached QVF file
Import/open it in Qlik Sense
Go to "Data Load Editor" and edit the existing "Logs" folder connection, and point to the extracted Log Collector archive path
1. If you are using a Qlik Sense server, remember to change the Data Connection name back to default "Logs". Editing via Hub will add your username to the data connection when saved.
Go to the "Initialize" script section and configure:
1. Your desired date range or days to load
2. Whether you want the data stored in a QVD
3. Which Service logs to load (Repository, Engine, Proxy and Scheduler services are built-in right now, adding other Qlik Sense Enterprise services may cause data load errors).
LOAD the data!

My workflow:

I'm looking for a specific point in time where a problem was registered
I use the time-based bar charts to find problem areas, get a general sense of workload over time
I use the same time-based charts to narrow in on the problem timestamp
Use the different dimensions to zoom in and out of time periods, down to a per-call granularity
Log Details sheets to inspect activity between services and filter until the failure/error is captured
Create and customise new charts to reveal interesting data points
Bookmarks for everything!

Notable Sheets & requirements:

Anything "Thread"-related for analysing Repository Service API call performance, which touches all aspects of the user and governance experience
- Requirement: Repository Trace Performance logs in DEBUG level. Otherwise, some objects may be empty or broken.
Commands: great for visualizing Repository operations and trends between objects, users, and requests
Transactions: Repository Service API call performance analysis.
- Requirement: Repository Trace Performance logs in DEBUG level. Otherwise, some objects may be empty or broken.
Task Transactions: very powerful task scheduling analysis with time-based filters for exclusion.
Log Details sheets: excellent filtering and searching through massive amounts of logs.
Repo + Engine data: resource consumption and Thread charts for Repository and Engine services, great for correlating workloads.

*It is best used in an isolated environment or via Qlik Sense Desktop. It can be very RAM and CPU intensive.

The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.

Related Content

Optimizing Performance for Qlik Sense Enterprise - Qlik Community - 1858594

QFabian · ‎2023-01-09

@Mario_Petre , thanks for this, very clear and usefull

rockabs · ‎2023-07-20

It's been almost one year that I am using this application. I love it for our stakeholder that we able to analyze before Qlik help many cases. He released latest version which will give nicely about file lock system traceability.

The way @Mario_Petre always having in detail about his way of working about how that algorithm works that will give so confident like us to explain in smarter way to our entrepreneurs. Appreciate your work!

p_verkooijen · ‎2023-07-20

@Mario_Petre created a new Lib pointing to the logs and script failed

Unexpected token: 'bi_qlik', expected one of: 'biff', 'dif', 'fix', 'html', 'json', 'kml', 'ooxml', ...

Concatenate (working) LOAD 'Engine' as Source, 'AuditActivity' as SourceType, FileName() AS [Log Source], * from lib://ServerLogs (>>>>>>bi_qlik<<<<<<)/servername/Engine/Audit/BI_AuditActivity_Engine_2022-04-20T06.10.04Z.log (txt, utf8, embedded labels, delimiter is '\t', no quotes) where

Fixed it by enclosing the variable $(vFile) with [] on sheet "File loading"

* from $(vFile)
* from [$(vFile)]

p_verkooijen · ‎2023-07-20

@Mario_Petre for somewhat faster processing I changed the following (bold)

We have logfiles dating back to 2018 🙂 now only scanning the relevant year(s) of logfiles

Section: Initialize

//This will load the last x days of logs from the chosen location. Increase value or use specific date range filter above
Set NoOfDays=30;
Let DateFrom = Date(Now() - NoOfDays);
Let DateTo = Date(Now());
Let YearFrom = Year(Now() - NoOfDays);
Let YearTo = Year(Now());

Section: File loading

for vYearLoop = $(YearFrom) to $(YearTo)

for each vFile in FileList('$(dir)/*$(vYearLoop)*.$(Ext)')

next vFile;

next vYearLoop
set vYearLoop;

makunii · ‎2023-08-14

Hello @Mario_Petre

I downloaded the Log Analyzer app but didn't see the QIX Performance and QIX Performance 2 sheet.

Thank you

Regards,

Marco

sujeet_shirude · ‎2023-08-15

Hello @Mario_Petre;

@Troy_Raney ;

@Sonja_Bauernfeind

I ran into the same issue as @makunii I downloaded the Log Analyzer app but didn't see the QIX Performance and QIX Performance 2 sheet.

Any chance we can get the latest version of the app that has QIX Performance and QIX Performance 2 sheet?

Thanks in advance!

Kalkumar · ‎2024-02-10

@Mario_Petre cannot see QIX Performance and QIX Performance 2 sheet.

Sonja_Bauernfeind · ‎2024-02-13

Hello @Kalkumar

I've attached the version you are looking for. I've also replaced the previous one with an even more recent one now. Both include the QIX sheets.

All the best,
Sonja

LogAnalysis App: The Qlik Sense app for troubleshooting Qlik Sense Enterprise on Windows logs