Discover the Best Open-Source Data-engineering Projects Sorted by Stars

Apache Superset is a Data Visualization and Data Exploration Platform

Learn how to design, develop, deploy and iterate on production-grade ML applications.

📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.

Free Data Engineering course!

Workflow engine for Kubernetes

Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines

The Data Engineering Cookbook

Data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes.

Roadmap to becoming a data engineer in 2021

Always know what to expect from your data.

An orchestration platform for the development, production, and observation of data assets.

Fancy stream processing made operationally mundane

🧙 The modern replacement for Airflow. Build, run, and manage data pipelines for integrating and transforming data.

The open source high performance data integration platform built for developers.

Open Source Feature Flagging and A/B Testing Platform

Feature Store for Machine Learning

Kestra is an infinitely scalable orchestration and scheduling platform, creating, running, scheduling, and monitoring millions of complex pipelines.

SQL Translator is a tool for converting natural language queries into SQL code using artificial intelligence. This project is 100% free and open source.

lakeFS - Data version control for your data lake | Git for data

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).