scheduler

Alternative schedulers and their usage.

airflow

Airflow is probably the most used scheduler with many features and pretty demanding in terms of resources.

airflow overview of airflow scheduler

deployment

We use the following docker-compose.yaml to deploy airflow which includes the following containers

• redis
• webserver
• scheduler
• workers
• init db
• flower

DAGs

Each job is composed by a Direct Acyclic Graph which can be scheduled with a given frequency and starting time.

We use the crontab syntax to schedule the jobs:

min h day month day_name

The DAG can be called in python with the following operator which calls the function parseS3

with DAG(dag_id="s3_pull",start_date=datetime.datetime(2022,11,14),tags=["froms3"],schedule='30 3 * * *') as dag:
    def s3_pull():
        parseS3(day=yesterday)
    PythonOperator(task_id="s3_pull", python_callable=s3_pull)

Here some python examples