You also need to check in the JSON into your VCS despite it being a generate artifact. Now you gotta make sure to run the script on changes. OK, so you are using some service that expects JSON to config. > If I need a for-loop, I'll write my own script to generate the JSON. I would love a framework that just orchestrates tasks (defined as a command + an image) according to a schedule, or based on the outcome of other tasks, and gives me a UI to view those outcomes and restart tasks, etc. Sometimes the operator author makes "helpful" (opinionated) changes that refactor or rewrite the native API. But with Airflow, you need to figure out what _Airflow_ operator and method wraps the Google SDK method you're trying to call. For example, to start a transaction on Spanner, Google has a Python SDK with methods to call their API. Operators often just add an API layer over top of existing ones. You almost need a dedicated Airflow expert to handle minor version upgrades or figure out why a task isn't running when you think it should. If I need a for-loop, I'll write my own script to generate the JSON. Configuration should be a way to change an application's behavior _without_ changing the code. I've used Airflow for a few years and here's what I don't like about it: Tasks that fail can be retried by Prefect automatically. I plumb the above up with a Prefect flow and it builds a DAG that runs the flow.Įverything that can be run in parallel does so, everything that has some other input waits for the input. I have another task that writes a list to a database.Īnother task puts errors to an error table, the machines that failed get filtered from the results and run into this task. I have a task that filters good results from bad. Note that the above don't depend on each other.
Airflow 2.0 task groups windows#
I get a list of windows machines and pass that to a task. I get a list of linux machines and pass that to a task to run. That task is an sqlalchemy query so I can pass the task a filter. I have a list of machines in a database so I create a task to get them.
Airflow 2.0 task groups software#
With prefect I created a task 'collect machine details for windows', another 'collect machine details for Linux', another 'collect software inventory'. I needed to collect disk size, ram, software inventory, some custom config, if present.
We looked into it ~1 year ago or so, I haven't heard a lot about it lately, I wonder if anyone has had success with it at scale. I stood up a prefect deployment for a hackathon and I found that it solved a ton of the issues with airflow (sane deployment options, not the insane file-based polling that airflow does). We'll stay at 2.0 until we eventually move off airflow altogether.
Airflow 2.0 task groups upgrade#
We've since tried multiple times to upgrade past the 2.0 release and hit issues every time, so we are just done with it. The scalability improvements in airflow 2 were a boon for our runtimes since before we would often have 5-15 minutes of overhead between task scheduling, but man it was a bear of an upgrade. Upgrades have been an absolute nightmare and so disruptive. Considering we have a pretty simplified DAG structure, I wish we had gone with a simpler, more robust/scalable solution (even if just rolling our own scheduler) for our specific needs. We've hit scaling issues at the k8s level, scheduling overhead in airflow, random race conditions deep in the airflow code, etc.
We have had SO many headaches operating airflow over the years, and each time we invest in fixing the issue I feel more and more entrenched. ingestion = ingest > validate > publish > scrub PII > publish) so we really don't need all the flexibility that airflow provides. Our DAGs are all config-driven which populate a few different templates (e.g. We weren't aware of a great alternative when we started. We've also been running airflow for the past 2-3 years at a similar scale (~5000 dags, 100k+ task executions daily) for our data platform.
0 kommentar(er)
