Refined an ETL Project for Flawless Performance and Seamless Data Workflows: Scalista Case

Client

Scalista is an Austrian online and mobile marketing agency that helps organizations grow in a very complex landscape. They have a proprietary project, Cloudista, for data harvesting.

Problem

They failed to connect Google BigQuery to Tableau to build reports. The project had to be modified due to the updates of used Python libraries. QA testing was required for the project’s flawless performance.

Results

The problem of the failed connection between BigQuery and Tableau was detected and eliminated. A new approach to building Tableau reports was developed. Project maintenance for flawless performance was completed. The Client’s users receive correct reports in a timely manner.

Technologies

Our Tech Stack

Goal

  • The Client’s project, Cloudista, which is used for data harvesting through cloud technologies, failed to perform correctly. The Python-based project is an ETL that exports marketing data from Scalista users’ accounts to GCP BigQuery. The data should be used to build Tableau reports, but the process kept failing.
  • Scalista’s specialist left the company, and they had no more resources to solve the task. The Client turned to Valiotti because of our experience in the tech stack (Python, Google BigQuery, Google Sheets) and process automatization.
  • During the project, two new tasks were set: to QA Cloudista’s performance and revise the ETL project.

Results

1. We designed a new approach to data transfer from BigQuery to Tableau via Google Sheets.

Firstly, we conducted research to find out the root of the problem—why it was impossible to connect BigQuery to Tableau. The reason was that the system used a custom request with a lot of calculated fields to enable the connection. It failed to deliver due to a limited request length.

To solve the problem, we developed a new approach: with the help of a custom request, the data from BigQuery was regularly exported to Google Sheets. Then, Google Sheets was easily connected to Tableau, which allowed for building analytical reports.

case study pipeline

Here is an example of a Google Sheet data mart:

Google Sheets Data Mart Example

2. The Cloudista performance was refined.

During the project, we discovered some ETL-related problems. Since the project is coded in Python, and the versions of the used libraries were updated, we needed to make the appropriate changes. After studying the code, project documentation, technologies, and the libraries’ changelogs, we edited the code for flawless performance.

What’s more, we compared the data from Facebook and Google Ads with the data sourced by the ETL project. Some modification of data processing was required.

Tips

  • Maintain a dialog between a data analyst and the Client. It’s best if the Client provides as many details as possible and formalizes the tasks.
  • When a vendor is dealing with an already-developed project, provide project documentation. It will help them understand the context much better.

Learn How Data Insights Can Benefit Your Business

Wondering what value data insights can bring your business? Get in touch, and we'll answer your questions!

Contact Us

Other Case Studies

  • betPawa

    A Flexible and Scalable DWH system Re-Built from Scratch with Improved Data Processing Time and Quality

    Read more

  • Mentorshow

    Comprehensive Reports Allow an EdTech Startup to Analyze User Behavior and Refine Its Product Strategy

    Read more

  • Skycoach

    Non-expert In-House Team Receives and Acts upon Professional Guidance to Establish Monitoring of Product Insights

    Read more