File as a source, directory wildcard

[allann]

We created a task that has a bunch of files without file extensions by using /directory/* in the Location of full load data file(s). Even though the documentation states a full file path is needed, we're able to pull all the data. We're trying to create a similar task but these files are *.csv. When I use * it won't work. 

When we used /*. *. it worked but it didn’t pick up any data. Any ideas for the wildcard in this situation?

[SwathiPulagam · Accepted solution]

Hi @allann ,

I tested with File as a source by giving just the directory\* keeping files with CSV extension and I see all the data loaded.

Please test one more time and share the task diagnostic package.

Thanks,

Swathi

[allann]

When I do that, I get "Failed to init unloading table.
Failed to write record id: 2, Number of values: 5 is not equal to number of columns: 32."

Just to confirm the direct path to 1 file worked, I did put /filename.csv and that pulled in 111 rows.

Attached log file.

[SwathiPulagam]

Hi @allann,

Try this option: Copy your CSV file to another folder and run the same mapping. let me know how it goes.

Thanks,

Swathi

[allann]

We had to figure out a naming convention and use the * within the filename(s).

[Heinvandenheuvel]

>> We had to figure out a naming convention and use the * within the filename(s).

for real? You use the "*" as a regular character in a filename? By choice? That's the silliest thing I've heard in years. Just Don't!

Hein.

[allann]

No I mean instead of trying to do /*.csv, the asterix had to be within the file path.

[Heinvandenheuvel · Accepted solution]

>> of trying to do /*.csv, the asterix had to be within the file path.

Ok, I supposed that's what the topic tittle also suggested.

I don't think that is going to work but if it did it, then the syntax would be more like <fixed_part_of_path>/*/*.csv

no?

For 'complex' source file locations you may want to look at generating/using REFERENCE files instead of direct file location. Reference file hold points to files.

Hein

[Megha_More]

Hi @SwathiPulagam ,

Can you please explain the steps for full load because there is an option to specify metadata (add field)? But when we give * that time, there might be changes in the metadata of files.

PFA for the same.

Regards,

Megha