Import New Dates Only

[#1]

Hello,

I have a question regarding importing new dates only between 2 source files (File1, File2). 

I've already loaded one file (File1), now I received another file (File2), but would like to load New Dates only when I reload. 

File1: consists of date range 1/12012 to 4/4/2012

File2: consists of date range 4/1/2012 to 4/10/2012

Question: Is there a way to use ReportDate column within these files to import only 4/5-4/10 (new dates only)?

Basically incremental load File2, but only load new dates, so ignore 4/1-4/4.

Additonal Question: What if I want to reload with a specific date, such as since 4/1/2012?  Is there a way to specify in edit script load since this specific date?  Is there a command?  or variable is preferred?

Is there sample edit script command to be used for the two questions, I would really appreciate it.

Sincerely.

[Jason_Michaelides]

Hi,

Try this:

Data:

LOAD * FROM File1...;

LET vMaxDate = Peek('ReportDate');     EDIT: Assumes File 1 was in date order (ascending).

CONCATENATE (Data)

LOAD * FROM File2...

WHERE ReportDate > $(vMaxDate);

You can set vMaxDate to whatever date you want of course.

Hope this helps,

Jason

Message was edited by: Jason Michaelides

[#3]

Thanks.

Assuming that your code will reload both File1 and File2.

Question: Is there a way NOT have to reload File1?  The dashboard already shows all data for File1, and File1 consists of all historic data in 2011, so very large and time consuming.  Is there a command to just load File2?  Going forward, I would receive additiona much smallerl incremental files, so any files and dates that's already loaded would be preferred not have to be reloaded.

Of course, I will specific the variable vMaxDate, so no dups will be loaded from new files (in this case File2).

Appreciate your help again.

[Jason_Michaelides]

2 options.

1. Use a QVD file as this will load very quickly.

2. Use a partial reload to only load File2

Hope this helps,

Jason

[#5]

What does option 2 partial reload syntax look like?

Looks like option 2 will skip File1 and load new data only, which seems like the most efficient method.

BTW, how does the code ignore re-loading File1?  Or does it have to scan the file no matter what for new dates?

Your help is appreciated!

[Jason_Michaelides]

It's been a while since I've used Partial Reload.  Start with the F1 help and search on this forum - loads of examples of how to use it.

[Jason_Michaelides]

Just as an aside - if you do go down the partial reload route then of course File2 will be completely reloaded every time and it will be increasing in size every day.  I'd consider going down a classic incremental load route using QVDs as these will load faster than whatever your File1 and File2 sources are.