Audience
Anyone who wants to to programmatically author, schedule and monitor workflows
About Apache Airflow
Airflow is a platform created by the community to programmatically author, schedule and monitor workflows. Airflow has a modular architecture and uses a message queue to orchestrate an arbitrary number of workers. Airflow is ready to scale to infinity. Airflow pipelines are defined in Python, allowing for dynamic pipeline generation. This allows for writing code that instantiates pipelines dynamically. Easily define your own operators and extend libraries to fit the level of abstraction that suits your environment. Airflow pipelines are lean and explicit. Parametrization is built into its core using the powerful Jinja templating engine. No more command-line or XML black-magic! Use standard Python features to create your workflows, including date time formats for scheduling and loops to dynamically generate tasks. This allows you to maintain full flexibility when building your workflows.
Other Popular Alternatives & Related Software
JAMS is an automation orchestration and job scheduling solution that works across applications, APIs, and scripting languages. Run, monitor, and manage critical IT processes—from simple batch jobs to cross-platform workflows—from a single pane of glass.
JAMS can automate jobs on any platform - Windows, Linux, UNIX, IBM i, zOS, and OpenVMS and includes native application integrations to run jobs specific to databases, BI tools, and ERP systems. Its extensive automation features enable you to run jobs on any schedule, as well as trigger off the completion of other events. JAMS centrally monitors the status of all jobs, provides notifications of failure (or success), and maintains a detailed audit trail and log of every execution.
Learn more
Union Cloud
Union.ai is an award-winning, Flyte-based data and ML orchestrator for scalable, reproducible ML pipelines. With Union.ai, you can write your code locally and easily deploy pipelines to remote Kubernetes clusters.
“Flyte’s scalability, data lineage, and caching capabilities enable us to train hundreds of models on petabytes of geospatial data, giving us an edge in our business.”
— Arno, CTO at Blackshark.ai
“With Flyte, we want to give the power back to biologists. We want to stand up something that they can play around with different parameters for their models because not every … parameter is fixed. We want to make sure we are giving them the power to run the analyses.”
— Krishna Yeramsetty, Principal Data Scientist at Infinome
“Flyte plays a vital role as a key component of Gojek's ML Platform by providing exactly that."
— Pradithya Aria Pura, Principal Engineer at Goj
Learn more
JS7 JobScheduler
JS7 JobScheduler is an Open Source workload automation system designed for performance, resilience and security. It provides unlimited performance for parallel execution of jobs and workflows. JS7 offers cross-platform job execution, managed file transfer, complex no-code job dependencies and a real REST API.
Platforms
- Cloud scheduling from Containers for Docker®, Kubernetes®, OpenShift® etc.
- True multi-platform scheduling on premises for Windows®, Linux®, AIX®, Solaris®, macOS® etc.
- Hybrid use for cloud and on premises
User Interface
- Modern, no-code GUI for inventory management, monitoring and control with web browsers
- Near real-time information brings immediate visibility of status changes and log output of jobs and workflows
- Multi-client capability, role based access management
High Availability
- Redundancy and Resilience based on asynchronous design and autonomous Agents
- Clustering for all JS7 products, automatic fail-over and manual switch-over
Learn more
Amazon CloudWatch
Amazon CloudWatch is a monitoring and observability service built for DevOps engineers, developers, site reliability engineers (SREs), and IT managers. CloudWatch provides you with data and actionable insights to monitor your applications, respond to system-wide performance changes, optimize resource utilization, and get a unified view of operational health. CloudWatch collects monitoring and operational data in the form of logs, metrics, and events, providing you with a unified view of AWS resources, applications, and services that run on AWS and on-premises servers. You can use CloudWatch to detect anomalous behavior in your environments, set alarms, visualize logs and metrics side by side, take automated actions, troubleshoot issues, and discover insights to keep your applications. CloudWatch alarms watch your metric values against thresholds that you specify or that it creates using ML models to detect anomalous behavior.
Learn more
Integrations
Company Information
The Apache Software Foundation
Founded: 1999
United States
airflow.apache.org
Videos and Screen Captures
Other Useful Business Software
The AI workplace management platform
By combining AI workflows, predictive intelligence, and automated insights, OfficeSpace gives leaders a complete view of how their spaces are used and how people work. Facilities, IT, HR, and Real Estate teams use OfficeSpace to optimize space utilization, enhance employee experience, and reduce portfolio costs with precision.
Product Details
Platforms Supported
Cloud
