Google Cloud PubSub

Google Cloud Pub/Sub is a scalable, durable, fully managed publish/subscribe messaging queue system that is real-time and guarantees delivery.

It brings the flexibility and reliability of enterprise message-oriented middleware to the cloud.

Features

• Globally available service: messages can be published and consumed from any region.
• REST API for producers and consumers.
• At-least-once delivery.
• Exactly-once processing.
• No provissioning.
• Scale responsively and automatically.
• Multi-cloud and hybrid applications on open architecture.
• Integrated with other GCP services.
• Open API.
• Global by default.
• Compliance and security.l

Publish/Subscribe system

It provides many-to-many, asynchronous messaging that decouples senders and receivers and facilitates secure and highly available communication among independently written applications.

• Message: the data that moves through the service.
• Topic: a named entity that represents a feed of messages.
• Subscription: a named entity that represents an interest in receiving messages on a particular topic. Subscription can be shared between several subscribers.
• Publisher (also called a producer): creates messages and sends (publishes) them to the messaging service on a specified topic.
• Subscriber (also called a consumer): receives messages on a specified subscription. It sends an acknowledge, after receiving the message from topic.

Performance

Scalability

A scalable service should be able to handle increases in load without noticeable degradation of latency or availability:

• Number of topics
• Number of publishers
• Number of subscriptions
• Number of subscribers
• Number of messages
• Size of messages
• Rate of messages (throughput) published or consumed
• Size of backlog on any given subscription

Availability

• Failure due to load could happen when a sudden increase in traffic in the service (or in other software components running on the same hardware or in software dependencies) results in resource scarcity.
• Availability can also degrade due to human error, where one makes mistakes in building or deploying software or configurations.
• Faulty Tolerance: A system’s availability is measured on how well it deals with different types of issues, gracefully failing over in a way that is unnoticeable to end users.

Latency

For Cloud Pub/Sub, the two most important latency metrics are:

• The amount of time it takes to acknowledge a published message.
• The amount of time it takes to deliver a published message to a subscriber.

Architecture

• Runs in multiple data centers
• It is horizontally scalable: number of topics, subscriptions and messages.
• Google PubSub is divided into two primary parts: control plane and data plane.

Data Plane

It handles moving messages between publishers and subscribers.

It maintains messages lifecycle.

Servers in Data Plane are called forwarders.

Client receives the confirmation after sending the message, when more of the 50% of the forwarders store the message.

Control Plane

It handles the assignment of publishers and subscribers to servers on the data plane.

Servers in Control Plane are called routers.

When clients connect to PubSub:

1. Routers decide datacenter, based on latency, and balance the messages across forwarders, returning a sorted list to clients
2. Clients make the final call about the forward to which they will connect.

• When publishers and subscribers are connected to their assigned forwarders, they do not need any information from the routers (as long as those forwarders remain accessible). Therefore, it is possible to upgrade the Control Plane without affecting any already connected clients.

Publishing messages

A publisher application creates and sends messages to a topic.

Cloud Pub/Sub offers at-least-once message delivery and best-effort ordering to existing subscribers

The general flow for a publisher application is:

1. Create a message containing your data.
2. Send a request to the Cloud Pub/Sub Server to publish the message to the desired topic.

Custom Attributes: attributes can be text strings or byte strings

Balancing: messages can be batched based on request size (in bytes), number of messages, and time.

Retrying Requests: publishing failures are automatically retried, except for errors that do not warrant retries.

Concurrency Control: threading and support for concurrency is available but varies by language

Consuming messages

To receive messages published to a topic, you must create a subscription to that topic.

Only messages published to the topic after the subscription is created are available to subscriber applications.

The subscription connects the topic to a subscriber application that receives and processes messages published to the topic.

A topic can have multiple subscriptions, but a given subscription belongs to a single topic.

At-Least-Once delivery:

• Cloud Pub/Sub delivers each published message at least once for every subscription.
• By default, a message that cannot be delivered within the maximum retention time of 7 days is deleted and is no longer accessible.
• A message published before a given subscription was created will usually not be delivered.

Pull subscription

In pull delivery, your subscriber application initiates requests to the Cloud Pub/Sub server to retrieve messages.

1. The subscribing application explicitly calls the pull method, which requests messages for delivery.
2. The Cloud Pub/Sub server responds with the message (or an error if the queue is empty) , and an ackID.
3. The subscriber explicitly calls the acknowledge method, using the returned ackID to acknowledge receipt

Use Cases

• Several topics to be consumed from same subscriber
• App Engine standard and Cloud functions subscribers.
• Environments where Google Cloud dependencies (credentials and client library) cannot be setup.

Push subscription

1. The Cloud Pub/Sub server sends each message as an HTTPS request to the subscriber application at a pre-configured endpoint.
2. The endpoint acknowledges the message by returning an HTTP success status code. A non-success response indicates that the message should be resent.

Cloud Pub/Sub dynamically adjusts the rate of push requests based on the rate at which it receives success responses.

It requires client libraries.

Use Cases

• High ratio of messages (much more than 1/second).
• Critical efficiency and throughput.
• When you have a public HTTPS endpoint with non self-signed certificate.

Lifecycle of a Subscription

By default, subscriptions expire after 31 days of inactivity

If Cloud Pub/Sub detects subscriber activity, the subscription deletion clock restarts.

Using subscription expiration policies, you can configure the inactivity duration or make the subscription persistent regardless of activity.

You can also delete a subscription manually.

Although you can create new a subscription with the same name as a deleted one, the new subscription has no relationship to the old one.

By default, Cloud Pub/Sub ensures that subscriptions retain unacknowledged messages for 7 days from the moment of publication.

Monitoring

The Cloud Pub/Sub API exports metrics via Google Cloud Monitoring.

Google Cloud Monitoring allows you to create monitoring dashboards and alerts or access the metrics programmatically.

Metrics and resource types

• Metrics List: see the usage metrics that Cloud Pub/Sub reports to Google Cloud Monitoring.
• Monitored Resource Types: see the details for the pubsub_topic , pubsub_subscription snf pubsub_snapshot monitored resource types.

You can use the APIs and services quotas dashboard to monitor the current utilization for a given topic or subscription.

Use Cases

• Balancing workloads in network clusters. For example, a large queue of tasks can be efficiently distributed among multiple workers, such as Google Compute Engine instances.
• Implementing asynchronous workflows. For example, an order processing application can place an order on a topic, from which it can be processed by one or more workers.
• Distributing event notifications. For example, a service that accepts user signups can send notifications whenever a new user registers, and downstream services can subscribe to receive notifications of the event.
• Refreshing distributed caches. For example, an application can publish invalidation events to update the IDs of objects that have changed.
• Logging to multiple systems. For example, a Google Compute Engine instance can write logs to the monitoring system, to a database for later querying, and so on.
• Data streaming from various processes or devices. For example, a residential sensor can stream data to backend servers hosted in the cloud.
• Reliability improvements. For example, a single-zone Compute Engine service can operate in additional zones by subscribing to a common topic, to recover from failures in a zone or region.

Quota and Limits

Each requests or response counts at minimum, 1KB, p.e. request with 100 messages of 50B each and 30 bytes topic name count as 6KB, rather than 5030B.

• Limits on quota over publisher throughput
• Subscriber throughput
• Push subscriber throughput
• Streaming pull subscriber throughput
• Number of open streaming pull connections
• Rate of streaming pull connections
• Administrator operations
• Maximum number of topics per project is 10K.
• Maximum number of subscriptions per project is 10K.
• Maximum number of attached subscriptions per topic is 10K.
• Maximum number of attached snapshots per topic is 50K.
• Retention: unacknowledge messages are stored maximum 7 days.
• Subscription are deleted after idle clients after 31 days.
• Publish requests are maximum 10MB and 1K messages.
• Message size, attributes per message, attribute key size, attribute value size.
• Push delivery outstanding messages to maximum 10MB or 1000 messages.
• Streaming pull streams to maximum 10MB per open stream.

Pricing

Price per TB.

First 10GB are free. After 10TB, on increasing the data volume, the fee per TB goes down.

Command Line Interface

gcloud pubsub topic create <topic-name>

gcloud pubsub topic list

gcloud pubsub subscriptions create <subscription-name> --topic <topic> --ack-deadline=<seconds>

gcloud pubsub subscriptions list

gcloud pubsub topic publish <topic-name> --message <content>

gcloud pubsub subscriptions pull  <subscription-name> --auto-ack --limit <number-of-messages>

gcloud pubsub subscriptions ack  <subscription-name> --ack-ids <ids>