ETL Processes Became 5 Times Faster as a Result of a Refined Data Warehouse: Twinero Case

Client

Twinero is a Spanish fintech company that offers short-term microloans to Spanish citizens. It focuses on rapid micro-lending with no paperwork.

Problem

The existing ETL processes were to be reviewed and modernized with Python. A new analytical repository was to be built.

Results

An optimized analytical warehouse with an improved reporting process and a reduced number of errors when creating reports.

Technologies

Our Tech Stack

Rostyslav Klopochuk

Country Manager, Twinero

"Valiotti Analytics helped us build a new data pipeline from scratch, replacing legacy tools such as Pentaho IDE with modern ones. They suggested the implementation of a modern data warehouse architecture for analytics and tuned instances to work with software. They also reacted quickly and introduced a huge number of improvements in the data analysis structure of our company. Moreover, several useful analytic dashboards were built for us during the project. We appreciate their approach as they dove deeply into the crucial details and understood our business constraints to react to data failure immediately. In order to support the new data stack, the team helped us build a monitoring system for the data routine. We are very pleased to work with Valiotti Analytics."

Goal

Twinero’s DWH (data warehouse) was built with ETL (extract, transform, load) processes based on Pentaho IDE. In terms of the existing infrastructure, Windows-generated files (.xml files) were run under Unix. The approach was outdated and needed to be refined and modernized with Python. Besides this task, Valiotti Analytics also built a new analytical repository to enhance reporting.

Results

  1. Refined ETL processes with accurate outputs run five times faster than before

To optimize the analytical warehousing, we modernized the infrastructure.

Refined ETL

We combined Python frameworks with DBT as an ETL tool and set up orchestration through Apache Airflow. As a result, SQL queries became faster and database tables were made five times more quickly.

Apache Airflow

As a result, all the reports were rewritten, and tables were normalized to align the data for further analytics.

  1. Enhanced reporting for valuable insights on the Client’s lifecycle and borrowing

We built new dashboards in Metabase and Tableau. In terms of the new data infrastructure, the Client was able to easily access and analyze historical data to get a better understanding of business specifics and trends. What is more, the customers were segmented based on payment categories, overdue period and scope, etc.

Now, Twinero can get better insights into its customers and do this in a more transparent way. For example, they can monitor the scope of users within a specific timeframe based on different categories and better analyze the overall situation.

Tips

  • Perform reverse-engineering of all the created objects to find out how they work.
  • Abandon legacy code for an up-to-date solution for enhanced flexibility, performance, and scalability.
  • Data processing is best organized as separate manageable blocks. In this case, the engineering process is presented by a so-called pipeline or a consecutive transformation of raw data into manageable blocks. Such an approach allows spotting if there was a problem at each stage, as well as if the stage was implemented or not.

Learn How Data Insights Can Benefit Your Business

Wondering what value data insights can bring your business? Get in touch, and we'll answer your questions!

Contact Us

Other Case Studies

  • Scalista

    Refined an ETL Project for Flawless Performance and Seamless Data Workflows

    Read more

  • betPawa

    A Flexible and Scalable DWH system Re-Built from Scratch with Improved Data Processing Time and Quality

    Read more

  • Mentorshow

    Comprehensive Reports Allow an EdTech Startup to Analyze User Behavior and Refine Its Product Strategy

    Read more