this post was submitted on 04 Mar 2024
6 points (87.5% liked)

Data Engineering

379 readers
2 users here now

A community for discussion about data engineering

Icon base by Delapouite under CC BY 3.0 with modifications to add a gradient

founded 1 year ago
MODERATORS
ericjmorey@programming.dev
6
Building a Data Pipeline from Scratch (medium.com)
submitted 8 months ago by nydas@lemmy.world to c/data_engineering@programming.dev
0 comments fedilink hide all child comments
 

I’ve written a series of Medium articles on creating a Data Pipeline from scratch, using Polars and DeltaTables. The first (linked) is an overview with link to the GitHub repository and each of the deeper dive articles. I then go into the next level of detail, walking through each component.

The articles are paywalled (it took time to build and document), but the link provided is the ‘family & friends’ link which bypasses the paywall for the Lemmy community.

I hope some of you may find this helpful.

no comments (yet)
sorted by: hot top controversial new old
there doesn't seem to be anything here