cloud providers

The main cloud providers for the ’20 decade are aws, gcp and azure but digitalocean, linode and others are used too. Here is a good overview on aws.

filesystem

Filesystems are a great option for backups, cheap cold storage. Big files are usually stored in parquet

• aws s3
• gcp filestore
• azure files
• digitalocean space

query engine

• gcp big query
• hive

orchestration

• kubernetes
• microk8s
• openshift
• docker swarm

spark

• aws EMR
• azure deltashare
• databricks

storage

relational

• aws RDS
• on prem postgres

document

• mongo
• aws dynamoDB

instances

• aws Elastic Map Reduce
• aws ec2 instances
• gcp compute engine
• digitalocean droplet

ML ops

• aws sage maker
• ML flow
• Kubeflow
• netflix metaflow
• Kedro
• h2o autoML

warehouse, query

• aws athena: a serverless Analytics service to perform interactive queries over AWS S3
• aws redshift: fully managed, petabyte data warehouse service over the cloud
• gcp big query
• snowflake
• hadoop hive

load balancing

• celery
• asw elastic load balancer
• aws api gateway

data integration, ETL, batch processing

• aws glue
• aws lambda
• apache airflow
• gcp pubsub
• hadoop yarn
• snowflake (with storage)
• dbt

logs, tracking

• on prem elasticsearch
• on prem kibana, grafana
• aws cloudwatch
• prometheus

cloud monitoring

• datadog

cloud formation

• terraform

messagging

• apache kafka
• redpanda
• aws kinesis

business intelligence

• on prem metabase
• on prem superset
• azure powerBI, tableau