Staying sane in the modern world of data engineering is a non-trivial mission. As data engineers we ask ourselves the same question several times a day: is everything okay with my data? If you know what I’m talking about, you also know that, let’s be honest, our primary concern is…

So you’ve got a ton of data that needs to be processed in real-time, huh? Don’t worry, in this tutorial I’ll show you how to stream data from Kafka compatible streaming platform to Amazon S3. But wait, there’s more! I’ll also cover how to create partitioned outputs, which allows you…

Learn how to create a data pipeline in GCP — I’ve recently worked on a project that required me to collect data from Google PubSub and load it into different BigQuery tables. I’ve faced many challenges during this process so I would like to share my experience building a complete data pipeline in Google Cloud Platform. Problem statement Let’s say we have…