- Getting RabbitMQ
- sudo apt-get install rabbitmq-server
- /usr/sbin/rabbitmqctl status
- sudo pip install pika
- RabbitMQ terminology
- producer – publishes the message to an exchange
- channel – sub-socket inside open TCP socket to RabbitMQ (TCP sockets are re-used to avoid overhead connection costs)
- consumers – consume the message from the queue
- basic.consume – passively receive messages from the queue (push mechanism)
- basic.get – poll-based requests, essentially subscribe, basic.consume of 1, unsubscribe
- auto_ack is set true, fires off after receiving, or use basic.ack if auto_ack is false
- basic.reject rejects the message, requeue=true puts it back on the queue, use requeue=false for malformatted messages
- queue properties
- exclusive – can only be subscribed by 1 consumer, useful for testing
- auto-delete – delete as soon as last consumer unsubscribes
- durable – should be re-created after re-starting?
- exchanges – publishers send messages to exchanges, who then route them to one or more queues
- direct – routing key must match with the name of the queue
- $channel->basic_publish($message, $exchange=’’, $routing=’test-queue’)
- fanout – broadcast the message to every queue that’s bound to a particular exchange
- topic exchange – can use a wildcard key, * if you have more characters, # for everything
- Vhosts
- each vhost has its own exchanges, bindings and queues
- permissions are per vhost
- default: username=guest, password=guest
- rabbitmqctl list_vhosts
- rabbitmqctl add_vhost
- rabbitmqctl delete_vhost
- Persistence
- set exchange durable=true, survives the restart
- set queue durable=true, survives the restart
- set message persistent=true (delivery mode = 2)
- persistent message is flagged in persistence log after being delivered and ACKed
- if the server crashes before the garbage collector has run, it will be re-delivered
- persistence can cause 10x decrease, but SSD helps
- structure clusters into critical (persistent, SSD) and fast (no persistence)
- Queue management
- rabbitmqctl list_queues
- rabbitmqctl list_queues -p specific.vhost
- rabbitmqctl rotate_logs suffix_to_appent.to_old_logs
- For messages requiring confirmation, the publisher can set reply-to header of the AMPQ message, and then listen for a confirmation to arrive on a separate queue
- Inside a cluster, RabbitMQ does not automatically replicate the queues – for persistent queues this would imply being copied over the network, and being copied to disk, which could pose significant performance issues
- As a follow-up, if the queue was marked durable, and the node crashed, the node would have to be resurrected for that queue’s messages to be retrieved from persistent storage
- Exchanges in RabbitMQ are lightweight – they’re essentially routing tables storing the routing keys and queue names. Therefore replicating (and scaling out) exchanges is pretty easy
- What happens when you publish the message into the exchange, and the exchange node fails? You lose the message
- If you employ AMPQ transaction, it will block until you get a confirmation
- You could also be listening for publisher confirms and have some logic in publisher to deal with messages that were lost by the exchange
- Storing metadata on disk allows for easier restart, storing metadata in RAM allows for easier declaration of new exchanges, queues or bindings, as that operation will block until all of the nodes in the cluster have propagated it
- But what if you’re doing RPC with separate anonymous queues for ACKs? Super-noisy.
- RabbitMQ requires one node stores metadata on disk, everybody else can store metadata in RAM
- Declaring a mirrored queue requires hard-coding the specific node name into your application
- Cannot mirror on all nodes, since potentially that would degrade performance
- A newly declared mirror queue will copy the contents of the queue from the point of declaration – it doesn’t fully mirror the queue
- Mirrored queue is really another queue with a fanout exchange
- Using publisher confirms on mirrored queue will cause ACKs to be delivered only when ALL slaves received the message
- You can have a weird situation when the master host fails and ACK never arrives
- When the master fails, all consumers need to re-attach to the new master, the failover is not automatic
- This means the app has to understand and process consumer cancellation notices
- Shovel is RabbitMQ’s cross-data-center replication plugin, effectively subscribes to the master queue, and re-publishes the data to remote server
