Data Engineering

1 readers

1 users here now

Discussion on Data Engineering topics. Data pipelines, tools and technologies, databases and DBMS, best practices:

Rules:

Limited to data engineering, no general CS/programming posts.
No technical questions. Example: how to fix this bug in my code.
No marketing
No resumes, jobs
No PII

founded 1 year ago

MODERATORS

lucas50a@lemm.ee

What is Data Lineage? (airbyte.com)

submitted 1 year ago by jackattackson@lemm.ee to c/dataengineering@lemm.ee

0 comments fedilink hide all child comments

I get questions like this a lot:

Where did this data come from?
How do I know I can trust the source?
What types of QA checks were applied to this data?

Data lineage is such a chronic issue in data engineering. This blog post from Airbyte gives a good overview & mentions some interesting products/projects that can maybe help out with data lineage.

Unfortunately, I have limited flexibility to purchase or install tools for this in my current role. Anyone rolled their own solution for this?

no comments (yet)

sorted by: hot top controversial new old

there doesn't seem to be anything here