Orchestrating multiple product workflows

[RonFusionHSLLC]

I'm sure we are not the only shop that has issues orchestrating daily processing schedules, complicated by replication, compose, qlikview and qliksense application runs.

Our OLTP system replicates,  custom processing introduces additional data and tables to prep for DW/DM loads.  Our DW loads have a mix of pre/single/post load ETL.  Our DM loads have pre/post loading.   We then have QlikView and QlikSense load/publishing tasks that need the DM's in a stable state.  Then there is Nprinting, it needs all this done to do it's thing.  On top of all that, we have custom jobs that extract and report raw data for various purposes.

All this relies on every system running on-time and without errors and without conflict.

Is there any comprehensive solution that could orchestrate all these products/tasks, having the ability to detect errors, stop processing, wait for completion...etc.  We use a number of workflows in compose, but that only gets us so far.

ron

[ashkya5]

I’m familiar with all of those. Not sure how random pieces of tech from an arbitrary cloud vendor really addresses my more generic question, though? Furthermore, isn’t ADF also designed for batch pipelines with a definite duration (e.g. AFAIR, activities time out in 7 days, not to mention costs accumulated during that time…) making it very similar to Airflow in my scenario regarding overall behavior, failure modes etc.