Is it possible to separate in Qlikview 11 DATA vs. Document (metadata/definition)? I have a large document (fact tables in tens of MLNs). The documents has about 20 Tabs (similar number of scripts for data retrieval). It is populated with Data. QV.exe

[#1]

Hi Qlik gurus!

Is it possible to separate in Qlikview 11

DATA vs. Document (metadata/definition)?

I have a large document (fact tables in tens of MLNs).

The documents has about 20 Tabs

(similar number of scripts for data retrieval).

It is populated with Data.

QV.exe in RAM is more than 5 Gig.

Saving of the document takes several minutes.

Saving involves both the document (code+charts)

and data.

Data kept in QVD files:

stage1: SQL LOAD (from backend) => QVD (done once per day or once per week).

Stage2: QVD=> QVD (Analytical flags formulated),

Vertical filtering done by busn.vertical.

Stage3: QVD=> resident table (optimized, using prefiltered data).

Is it possible to specify (on a level of option):

"Save only code and document definition"

(while I am in a development mode)?

"Do not load the data".

The only option I have come to up now is

to divide data by a business vertical

and (while in a debug/develoment mode)

load ONLY a small subset of the data

to be used in development mode

(split by a business vertical).

Thank you in advance,

Alexei

[marcus_sommer · Accepted solution]

At first take a look if compression is enabled (document properties) and if then turn it off. The same with section access (at least while you develop other parts). Further it could be useful to develop locally or within a faster environment - I mean where you could save the file on a SSD instead within a company-network.

The next point which you could consider is to split the application into several applications and/or to reduce the number of records which will be loaded respectively you could reduce your loaded data to certain selections (menu/file/reduce).

- Marcus

[#3]

Thank you, Marcus:

Changing at the document level

Document Properties=> General => Save Format  => Compression

from High to None

has made saving of the document 3 times faster(!) typically from 45 secs into 15 secs.

>develop locally

Yes, local machine.

>could save the file on a SSD

Yes, SSD drive.

> instead within a company-network

Network is not used (except sql connection to back-end). All QVD and QVW files are local.

>The next point which you could consider is to split the application into several applications

Yes, that is a good point. I will do it after the development will be over.

>reduce the number of records which will be loaded respectively you could reduce your loaded data to certain selections >(menu/file/reduce).

Finally I found what I was searching for:

Menu => File => Reduce Data => Remove All Values.

Only the "skeleton" of the document is saved on file, i.e. all code and all GUI charts.

NO DATA.

The file can be distributed. It can be thus stripped of production data and populated with test data.

Or populated from QVD files from production data again.

Reduce Data=> Keep possible values

does what I was trying to achieve programmatically   (using ApplyMap in LOAD) -it filters out and removes the non related data. After that we can save the QVW file with a different name (i.e. "xxx_DEBUG_data_only.qvw", work on the code and structure, then populated this file with full data if  needed, than overwriting the original QVW file).

I also found what I was trying to achieve: full history of the log files:

Settings=> Document Properties=>  General=> "Generate Log File" (checked) with the "TimeStamp in Log File Name" checked.

After that we have the full audit trail of the load logs. Very important on production (I log all counts, all variable values into log file).

Yes, Qlikview is powerful.

Thank you again,

Alexei