Google Cloud PubSub
Google Cloud Pub/Sub is a scalable, durable, fully managed publish/subscribe messaging queue system that is real-time and guarantees delivery.
It brings the flexibility and reliability of enterprise message-oriented middleware to the cloud.
- Globally available service: messages can be published and consumed from any region.
- REST API for producers and consumers.
- At-least-once delivery.
- Exactly-once processing.
- No provissioning.
- Scale responsively and automatically.
- Multi-cloud and hybrid applications on open architecture.
- Integrated with other GCP services.
- Open API.
- Global by default.
- Compliance and security.l
It provides many-to-many, asynchronous messaging that decouples senders and receivers and facilitates secure and highly available communication among independently written applications.
- Message: the data that moves through the service.
- Topic: a named entity that represents a feed of messages.
- Subscription: a named entity that represents an interest in receiving messages on a particular topic. Subscription can be shared between several subscribers.
- Publisher (also called a producer): creates messages and sends (publishes) them to the messaging service on a specified topic.
- Subscriber (also called a consumer): receives messages on a specified subscription. It sends an acknowledge, after receiving the message from topic.
A scalable service should be able to handle increases in load without noticeable degradation of latency or availability:
- Number of topics
- Number of publishers
- Number of subscriptions
- Number of subscribers
- Number of messages
- Size of messages
- Rate of messages (throughput) published or consumed
- Size of backlog on any given subscription
- Failure due to load could happen when a sudden increase in traffic in the service (or in other software components running on the same hardware or in software dependencies) results in resource scarcity.
- Availability can also degrade due to human error, where one makes mistakes in building or deploying software or configurations.
- Faulty Tolerance: A system’s availability is measured on how well it deals with different types of issues, gracefully failing over in a way that is unnoticeable to end users.
For Cloud Pub/Sub, the two most important latency metrics are:
- The amount of time it takes to acknowledge a published message.
- The amount of time it takes to deliver a published message to a subscriber.
- Runs in multiple data centers
- It is horizontally scalable: number of topics, subscriptions and messages.
- Google PubSub is divided into two primary parts: control plane and data plane.
It handles moving messages between publishers and subscribers.
It maintains messages lifecycle.
Servers in Data Plane are called forwarders.
Client receives the confirmation after sending the message, when more of the 50% of the forwarders store the message.
It handles the assignment of publishers and subscribers to servers on the data plane.
Servers in Control Plane are called routers.
When clients connect to PubSub:
- Routers decide datacenter, based on latency, and balance the messages across forwarders, returning a sorted list to clients
- Clients make the final call about the forward to which they will connect.
- When publishers and subscribers are connected to their assigned forwarders, they do not need any information from the routers (as long as those forwarders remain accessible). Therefore, it is possible to upgrade the Control Plane without affecting any already connected clients.
A publisher application creates and sends messages to a topic.
Cloud Pub/Sub offers at-least-once message delivery and best-effort ordering to existing subscribers
The general flow for a publisher application is:
- Create a message containing your data.
- Send a request to the Cloud Pub/Sub Server to publish the message to the desired topic.
Custom Attributes: attributes can be text strings or byte strings
Balancing: messages can be batched based on request size (in bytes), number of messages, and time.
Retrying Requests: publishing failures are automatically retried, except for errors that do not warrant retries.
Concurrency Control: threading and support for concurrency is available but varies by language
To receive messages published to a topic, you must create a subscription to that topic.
Only messages published to the topic after the subscription is created are available to subscriber applications.
The subscription connects the topic to a subscriber application that receives and processes messages published to the topic.
A topic can have multiple subscriptions, but a given subscription belongs to a single topic.
- Cloud Pub/Sub delivers each published message at least once for every subscription.
- By default, a message that cannot be delivered within the maximum retention time of 7 days is deleted and is no longer accessible.
- A message published before a given subscription was created will usually not be delivered.
In pull delivery, your subscriber application initiates requests to the Cloud Pub/Sub server to retrieve messages.
- The subscribing application explicitly calls the pull method, which requests messages for delivery.
- The Cloud Pub/Sub server responds with the message (or an error if the queue is empty) , and an ackID.
- The subscriber explicitly calls the acknowledge method, using the returned ackID to acknowledge receipt
- Several topics to be consumed from same subscriber
- App Engine standard and Cloud functions subscribers.
- Environments where Google Cloud dependencies (credentials and client library) cannot be setup.
- The Cloud Pub/Sub server sends each message as an HTTPS request to the subscriber application at a pre-configured endpoint.
- The endpoint acknowledges the message by returning an HTTP success status code. A non-success response indicates that the message should be resent.
Cloud Pub/Sub dynamically adjusts the rate of push requests based on the rate at which it receives success responses.
It requires client libraries.
- High ratio of messages (much more than 1/second).
- Critical efficiency and throughput.
- When you have a public HTTPS endpoint with non self-signed certificate.
Lifecycle of a Subscription
By default, subscriptions expire after 31 days of inactivity
If Cloud Pub/Sub detects subscriber activity, the subscription deletion clock restarts.
Using subscription expiration policies, you can configure the inactivity duration or make the subscription persistent regardless of activity.
You can also delete a subscription manually.
Although you can create new a subscription with the same name as a deleted one, the new subscription has no relationship to the old one.
By default, Cloud Pub/Sub ensures that subscriptions retain unacknowledged messages for 7 days from the moment of publication.
The Cloud Pub/Sub API exports metrics via Google Cloud Monitoring.
Google Cloud Monitoring allows you to create monitoring dashboards and alerts or access the metrics programmatically.
Metrics and resource types
- Metrics List: see the usage metrics that Cloud Pub/Sub reports to Google Cloud Monitoring.
- Monitored Resource Types: see the details for the pubsub_topic , pubsub_subscription snf pubsub_snapshot monitored resource types.
You can use the APIs and services quotas dashboard to monitor the current utilization for a given topic or subscription.
- Balancing workloads in network clusters. For example, a large queue of tasks can be efficiently distributed among multiple workers, such as Google Compute Engine instances.
- Implementing asynchronous workflows. For example, an order processing application can place an order on a topic, from which it can be processed by one or more workers.
- Distributing event notifications. For example, a service that accepts user signups can send notifications whenever a new user registers, and downstream services can subscribe to receive notifications of the event.
- Refreshing distributed caches. For example, an application can publish invalidation events to update the IDs of objects that have changed.
- Logging to multiple systems. For example, a Google Compute Engine instance can write logs to the monitoring system, to a database for later querying, and so on.
- Data streaming from various processes or devices. For example, a residential sensor can stream data to backend servers hosted in the cloud.
- Reliability improvements. For example, a single-zone Compute Engine service can operate in additional zones by subscribing to a common topic, to recover from failures in a zone or region.
Quota and Limits
Each requests or response counts at minimum, 1KB, p.e. request with 100 messages of 50B each and 30 bytes topic name count as 6KB, rather than 5030B.
- Limits on quota over publisher throughput
- Subscriber throughput
- Push subscriber throughput
- Streaming pull subscriber throughput
- Number of open streaming pull connections
- Rate of streaming pull connections
- Administrator operations
- Maximum number of topics per project is 10K.
- Maximum number of subscriptions per project is 10K.
- Maximum number of attached subscriptions per topic is 10K.
- Maximum number of attached snapshots per topic is 50K.
- Retention: unacknowledge messages are stored maximum 7 days.
- Subscription are deleted after idle clients after 31 days.
- Publish requests are maximum 10MB and 1K messages.
- Message size, attributes per message, attribute key size, attribute value size.
- Push delivery outstanding messages to maximum 10MB or 1000 messages.
- Streaming pull streams to maximum 10MB per open stream.
Price per TB.
First 10GB are free. After 10TB, on increasing the data volume, the fee per TB goes down.
Command Line Interface
gcloud pubsub topic create <topic-name>
gcloud pubsub topic list
gcloud pubsub subscriptions create <subscription-name> --topic <topic> --ack-deadline=<seconds>
gcloud pubsub subscriptions list
gcloud pubsub topic publish <topic-name> --message <content>
gcloud pubsub subscriptions pull <subscription-name> --auto-ack --limit <number-of-messages>
gcloud pubsub subscriptions ack <subscription-name> --ack-ids <ids>