Machine Learning Workflow/Pipeline Orchestration Tools
- 01. KALE (Kubeflow Automated pipeLines Engine) aims at simplifying the Data Science experience of deploying Kubeflow Pipelines workflows.
- 02. Flyte - the Workflow Automation Platform for Complex, Mission-Critical Data and Machine Learning Processes at Scale
- 03. MLRun, the scalable open source pipeline orchestration framework as a managed service in the Iguazio Data Science Platform.
- 04. ZenML is an extensible open-source MLOps framework to create reproducible Pipelines
- 05. Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes.
- 06. Kedro is an open-source Python framework for creating reproducible, maintainable and modular data science code.
- 07. Luigi is a Python package to build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures and command line integration
- 08. Metaflow enables data scientists to build, improve, and operate end-to-end workflows independently
- 09. Valohai is a MLOps platform that automates everything from data extraction to model deployment.
- 10. Dagster is a data orchestrator for machine learning, analytics, and ETL
- 11. Couler aims to provide a unified interface for constructing and managing workflows on different workflow engines
- 12. Iguazio ML Pipeline Orchestration Manage your workflow end-to-end using a user-friendly environment, featuring fully integrated workflow management, experiment tracking and AutoML tools
- 14. Polyaxon runs with all popular deep learning frameworks and machine learning libraries, enabling you to quickly push ideas to production.
- 15. Prefect Orchestration Engine makes it easy to build, test, and run dataflows right from your Python code.