Refined an ETL Project for Flawless Performance and Seamless Data Workflows: Scalista Case
Goal
- The Client’s project, Cloudista, which is used for data harvesting through cloud technologies, failed to perform correctly. The Python-based project is an ETL that exports marketing data from Scalista users’ accounts to GCP BigQuery. The data should be used to build Tableau reports, but the process kept failing.
- Scalista’s specialist left the company, and they had no more resources to solve the task. The Client turned to Valiotti because of our experience in the tech stack (Python, Google BigQuery, Google Sheets) and process automatization.
- During the project, two new tasks were set: to QA Cloudista’s performance and revise the ETL project.
Results
1. We designed a new approach to data transfer from BigQuery to Tableau via Google Sheets.
Firstly, we conducted research to find out the root of the problem—why it was impossible to connect BigQuery to Tableau. The reason was that the system used a custom request with a lot of calculated fields to enable the connection. It failed to deliver due to a limited request length.
To solve the problem, we developed a new approach: with the help of a custom request, the data from BigQuery was regularly exported to Google Sheets. Then, Google Sheets was easily connected to Tableau, which allowed for building analytical reports.
Here is an example of a Google Sheet data mart:
2. The Cloudista performance was refined.
During the project, we discovered some ETL-related problems. Since the project is coded in Python, and the versions of the used libraries were updated, we needed to make the appropriate changes. After studying the code, project documentation, technologies, and the libraries’ changelogs, we edited the code for flawless performance.
What’s more, we compared the data from Facebook and Google Ads with the data sourced by the ETL project. Some modification of data processing was required.
Tips
- Maintain a dialog between a data analyst and the Client. It’s best if the Client provides as many details as possible and formalizes the tasks.
- When a vendor is dealing with an already-developed project, provide project documentation. It will help them understand the context much better.
Learn How Data Insights Can Benefit Your Business
Wondering what value data insights can bring your business? Get in touch, and we'll answer your questions!
Other Case Studies