ETL Processes Became 5 Times Faster as a Result of a Refined Data Warehouse: Twinero Case
Goal
Twinero’s DWH (data warehouse) was built with ETL (extract, transform, load) processes based on Pentaho IDE. In terms of the existing infrastructure, Windows-generated files (.xml files) were run under Unix. The approach was outdated and needed to be refined and modernized with Python. Besides this task, Valiotti Analytics also built a new analytical repository to enhance reporting.
Results
- Refined ETL processes with accurate outputs run five times faster than before
To optimize the analytical warehousing, we modernized the infrastructure.
We combined Python frameworks with DBT as an ETL tool and set up orchestration through Apache Airflow. As a result, SQL queries became faster and database tables were made five times more quickly.
As a result, all the reports were rewritten, and tables were normalized to align the data for further analytics.
- Enhanced reporting for valuable insights on the Client’s lifecycle and borrowing
We built new dashboards in Metabase and Tableau. In terms of the new data infrastructure, the Client was able to easily access and analyze historical data to get a better understanding of business specifics and trends. What is more, the customers were segmented based on payment categories, overdue period and scope, etc.
Now, Twinero can get better insights into its customers and do this in a more transparent way. For example, they can monitor the scope of users within a specific timeframe based on different categories and better analyze the overall situation.
Tips
- Perform reverse-engineering of all the created objects to find out how they work.
- Abandon legacy code for an up-to-date solution for enhanced flexibility, performance, and scalability.
- Data processing is best organized as separate manageable blocks. In this case, the engineering process is presented by a so-called pipeline or a consecutive transformation of raw data into manageable blocks. Such an approach allows spotting if there was a problem at each stage, as well as if the stage was implemented or not.
Learn How Data Insights Can Benefit Your Business
Wondering what value data insights can bring your business? Get in touch, and we'll answer your questions!
Other Case Studies
-
AI Sales
Advanced analytics boosted AI Sales with revenue tracking, AI vs. human performance insights, real-time alerts, and improved bot strategies for higher efficiency.
Read more -
-