messaging
1/6

Messaging

Messaging distinguish between producers and consumers and arranges the messages into topics. To consume a topic the offset is used as a bookmark to efficiently navigate through the topic, consume only new messages or a particular subset.

middleware

Here is a middleware in python to publish and consume messages from a topic.

Endpoints are:

  • /push/<topic> produce a message to a topic
  • /consume/<topic> consume the whole topic
  • /latest/<topic> returns the messages in the topic not already consumed
  • /latest_time/<topic> returns the messages starting from a given time
  • /realtime/<topic> shows the kafka stream
  • /delete/<topic> deletes a topic
  • /topics/ shows the topics
  • /spec swagger documentation
  • / app user interface

Kafka

Kafka is deployed using a docker-compose file.

This includes zookeper which contains the volumes for storing the topic contents.

zookeeper:
image: confluentinc/cp-zookeeper:latest
hostname: zookeeper
container_name: zookeeper
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
volumes:
- zk-data:/var/lib/zookeeper/data # for Data
- zk-txn-logs:/var/lib/zookeeper/log # for transaction logs

kafka engine which requires a careful definition of the listeners

kafka:
image: confluentinc/cp-kafka:latest
hostname: kafka
container_name: kafka
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9093,OUTSIDE://host.docker.internal:9092
KAFKA_LISTENERS: INTERNAL://:9093,OUTSIDE://:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
KAFKA_RETENTION_MS: 604800000
KAFKA_RETENTION_BYTES: -1
extra_hosts:
- "host.docker.internal:172.0.0.1"

and eventually an UI

kafka-ui:
#image: confluentinc/cp-enterprise-control-center:latest
image: docker.redpanda.com/vectorized/console:latest
#hostname: control-center
depends_on:
- kafka
# ports:
# # - "9021:9021"
# - "127.0.0.1:9021:8080"
environment:
KAFKA_BROKERS: 'kafka:9093'
PORT: 9021

mosquitto

Mosquitto is designed for IoT where many devices are sending information (in a log fashion) to a central server. User applications can then consume topics and run analytics.

kinesis

pubsub