Skip to main content
Announcements
Qlik Connect 2024! Seize endless possibilities! LEARN MORE
cancel
Showing results for 
Search instead for 
Did you mean: 
kuruvaram
Contributor
Contributor

How to get Talend to wait for a file to land in S3

I have a file that lands in AWS S3 several times a day. I am using Talend as my ETL tool to populate a warehouse in Snowflake and need it to watch for the file to trigger my job. I've tried tWaitForFile but can't seem to get it to connect to S3. Has anyone done this before?

Labels (1)
  • v8.x

1 Reply
jeoste
Creator
Creator

Waiting for a file to  land means you're in event mode, so real time/ESB mode. With a route and a cAWSS3 you might be able to listen to this folder.

If you don't have ESB, using ETL you have to schedule the job every 5 minutes for example, connect to s3, download file if present, if not stop the job. The next 5 minute the job will do the same process

If the file that lands on S3 is not scheduled there is no other way for you to wait for an event using ETL