Refined an ETL Project for Flawless Performance and Seamless Data Workflows: Scalista Case

• The Client’s project, Cloudista, which is used for data harvesting through cloud technologies, failed to perform correctly. The Python-based project is an ETL that exports marketing data from Scalista users’ accounts to GCP BigQuery. The data should be used to build Tableau reports, but the process kept failing.
• Scalista’s specialist left the company, and they had no more resources to solve the task. The Client turned to Valiotti because of our experience in the tech stack (Python, Google BigQuery, Google Sheets) and process automatization.
• During the project, two new tasks were set: to QA Cloudista’s performance and revise the ETL project.

1. We designed a new approach to data transfer from BigQuery to Tableau via Google Sheets.

Firstly, we conducted research to find out the root of the problem—why it was impossible to connect BigQuery to Tableau. The reason was that the system used a custom request with a lot of calculated fields to enable the connection. It failed to deliver due to a limited request length.

To solve the problem, we developed a new approach: with the help of a custom request, the data from BigQuery was regularly exported to Google Sheets. Then, Google Sheets was easily connected to Tableau, which allowed for building analytical reports.

Here is an example of a Google Sheet data mart:

2. The Cloudista performance was refined.

During the project, we discovered some ETL-related problems. Since the project is coded in Python, and the versions of the used libraries were updated, we needed to make the appropriate changes. After studying the code, project documentation, technologies, and the libraries’ changelogs, we edited the code for flawless performance.

What’s more, we compared the data from Facebook and Google Ads with the data sourced by the ETL project. Some modification of data processing was required.