Open in app

Sign In

Write

Sign In

Olga Braginskaya
Olga Braginskaya

32 Followers

Home

About

Published in

Dev Genius

·Pinned

Unleash Your Pipeline Creativity: Local Development with Argo Workflows and MinIO on Minikube

As a data engineer, you know the joy of wrangling massive datasets and navigating complex data pipelines. Argo Workflows, a popular workflow engine for Kubernetes, is your trusty companion in this data-driven journey, allowing you to define, run, and manage data pipelines as code. However, like any adventure, there are…

Data Engineering

12 min read

Unleash Your Pipeline Creativity: Local Development with Argo Workflows and MinIO on Minikube
Unleash Your Pipeline Creativity: Local Development with Argo Workflows and MinIO on Minikube
Data Engineering

12 min read


Published in

Dev Genius

·Pinned

How we mastered dbt: A true story

Staying sane in the modern world of data engineering is a non-trivial mission. As data engineers we ask ourselves the same question several times a day: is everything okay with my data? If you know what I’m talking about, you also know that, let’s be honest, our primary concern is…

Firebolt

13 min read

How we mastered dbt: A true story
How we mastered dbt: A true story
Firebolt

13 min read


Apr 9

From Kafka to Amazon S3: Partitioning Outputs

So you’ve got a ton of data that needs to be processed in real-time, huh? Don’t worry, in this tutorial I’ll show you how to stream data from Kafka compatible streaming platform to Amazon S3. But wait, there’s more! I’ll also cover how to create partitioned outputs, which allows you…

Kafka

16 min read

From Kafka to Amazon S3: Partitioning Outputs
From Kafka to Amazon S3: Partitioning Outputs
Kafka

16 min read


Published in

Towards Data Science

·Oct 4, 2021

PubSub to BigQuery: How to Build a Data Pipeline Using Dataflow, Apache Beam, and Java

Learn how to create a data pipeline in GCP — I’ve recently worked on a project that required me to collect data from Google PubSub and load it into different BigQuery tables. I’ve faced many challenges during this process so I would like to share my experience building a complete data pipeline in Google Cloud Platform. Problem statement Let’s say we have…

Data Pipeline

12 min read

PubSub to BigQuery: How to Build a Data Pipeline Using Dataflow, Apache Beam, and Java
PubSub to BigQuery: How to Build a Data Pipeline Using Dataflow, Apache Beam, and Java
Data Pipeline

12 min read

Olga Braginskaya

Olga Braginskaya

32 Followers

Data engineer. I believe in Python and cats.

Following
  • Adi Polak

    Adi Polak

  • Daniel Bourke

    Daniel Bourke

  • Nikita Schneider

    Nikita Schneider

  • Sigal Shaharabani

    Sigal Shaharabani

  • Alex Romanov

    Alex Romanov

See all (22)

Help

Status

Writers

Blog

Careers

Privacy

Terms

About

Text to speech

Teams