Hey there, I recently started a new job, and I have some questions for you.
Company: A retail chain with countless millions of sales rows. daily thousands of sales, Currently, data is currently retrieved and stored in the database using Qlik automations every morning.Qvd files (in the Qlik format) were then read by several dashboards.
Scenario:
Since I'm the only business intelligence (BI) specialist at the company, my duties entail retrieving data from the so-called PostgreSQL "data warehouse" (which, to me, is more of a database since it consists of just raw facts and dimensions tables that need to be joined), having that data transformed, and then using Qlik to produce various dashboards.
Right now, the business has made the decision to handle everything using Qlik, thus the data is extracted from the database using Qlik, transformed using a Qlik load script, and then used in various reports.
Because the company intends to grow significantly in the future, I'm aiming to modernize the stack, establish a more robust ELT/ETL procedure, and develop a more scalable strategy.
What would you advise using at each stage of the procedure? (Ideally using open-source)
Just so you know, this is an intermediate person (excellent knowledge of Python, SQL, and ELT/ETL).
Thank you
