ETL Processes Became 5 Times Faster as a Result of a Refined Data Warehouse: Twinero Case

Goal

Twinero’s DWH (data warehouse) was built with ETL (extract, transform, load) processes based on Pentaho IDE. In terms of the existing infrastructure, Windows-generated files (.xml files) were run under Unix. The approach was outdated and needed to be refined and modernized with Python. Besides this task, Valiotti Analytics also built a new analytical repository to enhance reporting.

Results

  1. Refined ETL processes with accurate outputs run five times faster than before

To optimize the analytical warehousing, we modernized the infrastructure.

Refined ETL

We combined Python frameworks with DBT as an ETL tool and set up orchestration through Apache Airflow. As a result, SQL queries became faster and database tables were made five times more quickly.

Apache Airflow

As a result, all the reports were rewritten, and tables were normalized to align the data for further analytics.

  1. Enhanced reporting for valuable insights on the Client’s lifecycle and borrowing

We built new dashboards in Metabase and Tableau. In terms of the new data infrastructure, the Client was able to easily access and analyze historical data to get a better understanding of business specifics and trends. What is more, the customers were segmented based on payment categories, overdue period and scope, etc.

Now, Twinero can get better insights into its customers and do this in a more transparent way. For example, they can monitor the scope of users within a specific timeframe based on different categories and better analyze the overall situation.

Tips

  • Perform reverse-engineering of all the created objects to find out how they work.
  • Abandon legacy code for an up-to-date solution for enhanced flexibility, performance, and scalability.
  • Data processing is best organized as separate manageable blocks. In this case, the engineering process is presented by a so-called pipeline or a consecutive transformation of raw data into manageable blocks. Such an approach allows spotting if there was a problem at each stage, as well as if the stage was implemented or not.

Learn How Data Insights Can Benefit Your Business

Wondering what value data insights can bring your business? Get in touch, and we'll answer your questions!

Contact Us

Other Case Studies

  • Buff

    Regained focus on business KPIs with dashboards transforming data into revenue.

    Read more
  • Scalista

    Refined an ETL Project for Flawless Performance and Seamless Data Workflows

    Read more
  • betPawa

    A Flexible and Scalable DWH system Re-Built from Scratch with Improved Data Processing Time and Quality

    Read more