ETL Processes Became 5 Times Faster as a Result of a Refined Data Warehouse: Twinero Case
Twinero’s DWH (data warehouse) was built with ETL (extract, transform, load) processes based on Pentaho IDE. In terms of the existing infrastructure, Windows-generated files (.xml files) were run under Unix. The approach was outdated and needed to be refined and modernized with Python. Besides this task, Valiotti Analytics also built a new analytical repository to enhance reporting.
- Refined ETL processes with accurate outputs run five times faster than before
To optimize the analytical warehousing, we modernized the infrastructure.
We combined Python frameworks with DBT as an ETL tool and set up orchestration through Apache Airflow. As a result, SQL queries became faster and database tables were made five times more quickly.
As a result, all the reports were rewritten, and tables were normalized to align the data for further analytics.
- Enhanced reporting for valuable insights on the Client’s lifecycle and borrowing
We built new dashboards in Metabase and Tableau. In terms of the new data infrastructure, the Client was able to easily access and analyze historical data to get a better understanding of business specifics and trends. What is more, the customers were segmented based on payment categories, overdue period and scope, etc.
Now, Twinero can get better insights into its customers and do this in a more transparent way. For example, they can monitor the scope of users within a specific timeframe based on different categories and better analyze the overall situation.
That's how Rostyslav Klopochuk, Twinero Country Manager, described our work together on this project:
Valiotti Analytics helped us build a new data pipeline from scratch, replacing legacy tools such as Pentaho IDE with modern ones. They suggested the implementation of a modern data warehouse architecture for analytics and tuned instances to work with software. They also reacted quickly and introduced a huge number of improvements in the data analysis structure of our company. Moreover, several useful analytic dashboards were built for us during the project. We appreciate their approach as they dove deeply into the crucial details and understood our business constraints to react to data failure immediately. In order to support the new data stack, the team helped us build a monitoring system for the data routine. We are very pleased to work with Valiotti Analytics.
- Perform reverse-engineering of all the created objects to find out how they work.
- Abandon legacy code for an up-to-date solution for enhanced flexibility, performance, and scalability.
- Data processing is best organized as separate manageable blocks. In this case, the engineering process is presented by a so-called pipeline or a consecutive transformation of raw data into manageable blocks. Such an approach allows spotting if there was a problem at each stage, as well as if the stage was implemented or not.
