Is an optimized LOAD with a RESIDENT afterwards faster?

[datanibbler]

Hi,

I have read some articles on optimized and unoptimized qvd_loading, but now I have a specific question:

=> I am just looking at an app that, for some reason, loads about 200mio records from a qvd in an unoptimized LOAD - that is apparently because there is one keyfield being constructed from two separate fields in that LOAD, otherwise it is a plain LOAD that could be optimized.

Is it worthwhile doing an optimized LOAD on this and then doing a RESIDENT LOAD where the keyfield can be constructed? Would that still be faster?

Thanks a lot!

Best regards,

DataNibbler

[Kushal_Chawda]

AFAIK, if load is not optimized then it is better to perform while loading it from QVD instead of doing it in resident load. Because resident load performs the calculation in memory, so if you huge data set then better to create the key while loading the data from QVD

[datanibbler]

Thanks kushal! I will test that.

[stabben23]

Hi Friedrich,

Why not do a preceding load where you create the key, I dont know if this will do Your load optimized, just an idea.

Resident loads on fact tables is what I try to avoid because it will be time consuming.

[stabben23]

Hmm, bad idea, dident work

[Peter_Cammaert]

No, because the initial Load passes each individual record straight to the Preceding load and thereby loses all gains from a quick QVD-into-memory block load.

Good try though

[Peter_Cammaert]

Just to make the set of suggestions a bit larger: how about an optimized load into memory, and - since they're just primary keys that you are creating - a simple LEFT JOIN of the resident table to itself?

I know that a technique like this has a horrendous impact on memory consumption (200M rows you say?) but I never actually checked the performance gain/loss...

[datanibbler]

More stuff to take into the sandbox ... Thanks a lot guys!

[Kushal_Chawda]

I think on very huge data set, preceding load does not work well that's what I read from the post of HIC

[datanibbler]

Well - loading the qvd without that keyfield_construct seems to have taken about 50% off the load_time - but then I have to do either a RESIDENT LOAD or that LEFT JOIN you proposed. Let's see.