cloud providers
The main cloud providers or hyperscaler for the ’20 decade are aws,
gcp and azure and they differentiate for their scalability from fixed
resource booking data centers like digitalocean, linode and others are
used too. Here is a good overview on
aws.
App deployment on-prem vs
AWS
On prem: I launch an instance from my cloud
provider, I log into it with ssh, I install the libraries and launch the
application, install and configure nginx, create certificate with
certbot and rerun nginx. AWS:
- I create a VPC
- I create a internet gateway
- I add a subnet and add routes from the public
- I create an EC2 in that VPC
- I create an elastic IP to attach to the EC2
- I create security groups that allow at least ssh and http
- I log into the instance, install libraries, run the application,
launch it, install nginx
- I can’t use certbot so I need to activate cloudfront
- I create a load balancer
- I create a target group where my EC2 sits
- I create a cloudfront distribution
- request a certificate for your cloudfront domain
filesystem
Filesystems are a great option for backups, cheap cold storage. Big
files are usually stored in parquet
- aws s3
- gcp filestore
- azure files
- digitalocean space
query engine
orchestration
- kubernetes
- microk8s
- openshift
- docker swarm
spark
- aws EMR
- azure deltashare
- databricks
storage
relational
document
instances
- aws Elastic Map Reduce
- aws ec2 instances
- gcp compute engine
- digitalocean droplet
ML ops
- aws sage maker
- ML flow
- Kubeflow
- netflix metaflow
- Kedro
- h2o autoML
warehouse, query
- aws athena: a serverless Analytics service to perform
interactive queries over AWS S3
- aws redshift: fully managed, petabyte data warehouse
service over the cloud
- gcp big query
- snowflake
- hadoop hive
load balancing
- celery
- asw elastic load balancer
- aws api gateway
data integration, ETL,
batch processing
- aws glue
- aws lambda
- apache airflow
- gcp pubsub
- hadoop yarn
- snowflake (with storage)
- dbt
logs, tracking
- on prem elasticsearch
- on prem kibana, grafana
- aws cloudwatch
- prometheus
cloud monitoring
messagging
- apache kafka
- redpanda
- aws kinesis
business intelligence
- on prem metabase
- on prem superset
- azure powerBI, tableau