Alternative schedulers and their usage.


Airflow is probably the most used scheduler with many features and pretty demanding in terms of resources.

airflow overview of airflow scheduler


We use the following docker-compose.yaml to deploy airflow which includes the following containers


Each job is composed by a Direct Acyclic Graph which can be scheduled with a given frequency and starting time.

We use the crontab syntax to schedule the jobs:

min h day month day_name

The DAG can be called in python with the following operator which calls the function parseS3

with DAG(dag_id="s3_pull",start_date=datetime.datetime(2022,11,14),tags=["froms3"],schedule='30 3 * * *') as dag:
    def s3_pull():
    PythonOperator(task_id="s3_pull", python_callable=s3_pull)

Here some python examples