Is it possible to load QVD-files using wildcard search?

[MEJ]

I wonder if it is possible to have a loadscript where I know the start of the filename, but not the ending.

For example:

Backlog_2019-05-20.qvd

Backlog_2019-05-21.qvd

Backlog_2019-05-22.qvd

Is it possible to say in the FROM statement Backlog_*.QVD or something like that? 

If I am reading from automatically created QVD files with "today's date", I don't know what dates they were stored beforehand. 

Hope the question is understandable. 

[m_woolf · Accepted solution]

Yes. Just like your example.

[MEJ]

Great! Now it worked. I had some other errors in the load script that is now fixed. At least I learned a new function.

However, I would still like to have everything stored into one file, not as 200 separate QVD files after a one year. Not sure what the load time difference is, loading from 1 large file containing 1 million records vs 200 QVD files containing 5000 records in each file. Probably it is a lot quicker to load from 1 file, as it otherwise needs to access 200 files, which probably takes longer.

[Brett_Bleess]

I would concur with your thoughts regarding single versus many, as with the many in this case the file operations alone will likely add a decent number of seconds to each round, whereas if you load optimized from the single file, I would expect things to be quite quick, just wanted to confirm you were thinking along the correct lines here.

Cheers,
Brett

[MEJ]

Hi Brett,

That was my initial worry, that it would take a long time to load from multiple files. So far it has generated 20 files or so (~400 kB per file), and it takes less than a second to load all. No problem whatsoever. The advantage of having multiple files, is that I can delete one of the files in case it contains invalid data, which happens once in a while. I would prefer to have a gap in the series, rather than having potential spikes in revenue/margins/quantity that would screw up the graphs. As I understand, is that it is rather difficult to correct incorrect data in a QVD file. In case the data load would take an excess amount of time due to too many files, I would load the multiple QVD files into one QVD file at some point.

Thanks

[Brett_Bleess · Accepted solution]

Yeah, that is a good point, and about the only protection with QVD files is to be sure you have known good backups you can fall back to if something does go wrong, so I would say your approach is the right way in this case given the times you are seeing etc. and your use case!  Best of luck with things.

Regards,
Brett