cloud providers
The main cloud providers or hyperscaler for the '20 decade are aws, gcp and azure and they differentiate for their scalability from fixed resource booking data centers like digitalocean, linode and others are used too.
Here is a good
overview on aws
.
App deployment on-prem vs AWS
On prem
: I launch an instance from my cloud provider, I log into it with ssh, I install the libraries and launch the application, install and configure nginx, create certificate with certbot and rerun nginx.
AWS
:
- I create a VPC
- I create a internet gateway
- I add a subnet and add routes from the public
- I create an EC2 in that VPC
- I create an elastic IP to attach to the EC2
- I create security groups that allow at least ssh and http
- I log into the instance, install libraries, run the application, launch it, install nginx
- I can't use certbot so I need to activate cloudfront
- I create a load balancer
- I create a target group where my EC2 sits
- I create a cloudfront distribution
- request a certificate for your cloudfront domain
filesystem
Filesystems are a great option for backups, cheap cold storage. Big files are usually stored in parquet
- aws s3
- gcp filestore
- azure files
- digitalocean space
query engine
- gcp big query
- hive
orchestration
- kubernetes
- microk8s
- openshift
- docker swarm
spark
- aws EMR
- azure deltashare
- databricks
storage
relational
- aws RDS
- on prem postgres
document
- mongo
- aws dynamoDB
instances
- aws Elastic Map Reduce
- aws ec2 instances
- gcp compute engine
- digitalocean droplet
ML ops
- aws sage maker
- ML flow
- Kubeflow
- netflix metaflow
- Kedro
- h2o autoML
warehouse, query
- aws athena: a serverless Analytics service to perform interactive queries over AWS S3
- aws redshift: fully managed, petabyte data warehouse service over the cloud
- gcp big query
- snowflake
- hadoop hive
load balancing
- celery
- asw elastic load balancer
- aws api gateway
data integration, ETL, batch processing
- aws glue
- aws lambda
- apache airflow
- gcp pubsub
- hadoop yarn
- snowflake (with storage)
- dbt
logs, tracking
- on prem elasticsearch
- on prem kibana, grafana
- aws cloudwatch
- prometheus
cloud monitoring
- datadog
cloud formation
- terraform
messagging
- apache kafka
- redpanda
- aws kinesis
business intelligence
- on prem metabase
- on prem superset
- azure powerBI, tableau