Introduction to GCP Pub/Sub and Event Consumption Models

GCP Pub/Sub powers a variety of use cases in modern architectures: 

• Event-Driven Architectures: Decouples services for seamless communication and independent scalability.
• Data Ingestion Pipelines: Handles massive volumes of data from diverse sources for processing.
• Real-Time Analytics: Streams events to enable instant insights and monitoring.
• Workflow Automation: Triggers actions and automates processes based on incoming events.

‍

Google Cloud Pub/Sub ability to handle high volumes of data, enable decoupling of services, and support for real-time processing make it an ideal data backbone for sharing event data across an enterprise.

GCP Pub/Sub Key Elements: Topics, Subscriptions, and Messages

At the core of GCP Pub/Sub functionality are three fundamental elements: topics, subscriptions, and messages. Together with the data and control planes, these form the foundation of how event data flows through the platform.

‍

1. Messages

Messages are the payloads of information which flow through GCP Pub Sub.
Each message contains:

• Data: The actual information being transmitted.
• Attributes: Key value pair metadata that adds context to the message, such as timestamps or identifiers.

Custom attributes can be set on a message which help identify the context or type of the message without having to process the data. These attributes allow for efficient filtering or delivery of messages to consumers who are only interested in specific types of events.

‍

2. Topics

A topic is a named resource which holds a data feed of messages until it is consumed by or delivered to a consuming application. It often acts as an entry point for event data onto GCP Pub/Sub when an application publishes it’s data onto the topic.

Key features of topics:

• Scalability: Topics can handle millions of messages per second and support thousands of subscriptions, ensuring wide distribution.
• Asynchronous Communication: Decouples publishers and subscribers, allowing them to operate independently.
• Durability: Messages are stored reliably until all subscribers acknowledge receipt.
• Flexibility: Supports both Push and Pull delivery models, adapting to various use cases.

‍

3. Subscriptions

Subscribers are the consumers of messages published to topics. To receive or consume messages from Pub/ Sub, a subscription is required. Subscriptions are configured on a topic for each subscriber and define:

1. Delivery: Who will get the messages and how they will be delivered
2. Filtering: Filter out messages based on the message attributes, so the subscriber only gets the subset of events in the topic that it is interested in

The messages in the topic are available to each subscriber until they have been successfully consumed by that subscriber. That is, if subscriber A has read a message, that message will still be available to subscriber B until it too has successfully consumed it.

Delivery Methods:

There are two main ways in which GCP Pub/ Sub makes an event available to a subscriber:

1. Pull Subscription: Subscribers request messages from the topic when they are ready to process them.
2. Push Subscription: Messages are automatically delivered to an HTTP endpoint specified by the subscriber.

The technicalities of these two delivery methods are covered in more detail below

‍

4. Control and Data Planes

GCP Pub/ Sub has 2 main components under the hood which tie together the publisher subscriber model and orchestrate the flow of messages on the platform:

1. Control plane – Controls the assignment of all publishers and subscribers
2. Data plane — Manages the moving of messages between publishers and subscribers.

Message Delivery Models in GCP Pub/Sub:

‍

In GCP Pub/Sub, message delivery method refers to how messages are delivered from topics to subscribers. Pub/Sub offers two primary models for this: Push and Pull, each with its own workflow, advantages, and limitations.

‍

The pull model also has a Streaming variant which provides enhanced performance for maximum throughput and minimum latency.

‍

Push Subscription Model

In the Push model, GCP Pub/Sub automatically delivers messages to an HTTP endpoint specified by the subscriber. When a message is published to a topic, Pub/Sub wraps the message in an HTTP POST request and sends it to the endpoint. The subscriber processes the message and responds with an HTTP acknowledgment to confirm delivery.

How it works:

1. The subscriber defines an HTTP endpoint during subscription setup.
2. Pub/Sub sends messages to the endpoint as soon as they are available.
3. The subscriber processes the message and responds with an HTTP success code (e.g., 200 OK).

Pros:

• Simpler setup: No client library is needed; events are delivered as HTTP requests.
• Decoupled architecture: The service receiving the events doesn’t need to know Pub/Sub’s details
• Real-time delivery: Messages are sent immediately, reducing latency.
• Scalable for intermittent traffic: Ideal for handling traffic spikes, as Cloud Run can dynamically scale based on HTTP requests.

Cons:

• Network dependency: Relies on HTTP availability; network issues can cause delays.
• Limited throughput in small regions: Push throughput is generally lower than Pull in certain regions.
• Always-on requirement: The subscriber endpoint must be consistently available, or messages may dead-letter if retries fail.

Use Cases:

• We want to integrate with an external system via a webhook or existing http request/ response-based systems such as an API which is already serving HTTP requests some text
  • Enables these systems to easily consume events from Pub/ Sub with minimal code changes and no added dependencies on GCP Client libraries.
• We want to implement a Fan In pattern to consume events from a single HTTP Request based Subscriber which acts as a Load Balancer and/or Message Router to other systems:some text
  • Enables events from multiple Pub/Sub Topics to be received at a single HTTP subscriber endpoint.
  • The receiving HTTP endpoint can act as a load balancer and re-direct messages to different backends.
  • This allows us to add new topics and push to the same subscriber endpoint which in turn routes the events to other services
• Trigger a process or workflow immediately in response to an event

Workflow for a push subscription, Push subscriptions  |  Pub/Sub Documentation  |  Google Cloud

‍

Pull Subscription Model

In the Pull model, subscribers explicitly request messages from a Pub/Sub topic at their own pace. This allows the subscriber to control when and how messages are processed.

How it works:

1. The subscriber sends a request to the Pub/Sub server to pull messages.
2. Pub/Sub responds with a batch of messages and acknowledgment IDs.
3. After processing, the subscriber sends an acknowledgment for each message to confirm delivery.

Pros:

• Full control: Subscribers decide when to retrieve messages and at what rate to process them.
• Higher throughput: Especially suited for high-load scenarios, as it avoids network overhead from HTTP requests.
• Resilience: If the subscriber is unavailable, messages remain in the topic and can be processed later.

Cons:

• More complexity: Requires integration with GCP client libraries and additional code for managing message acknowledgment.
• Coupling to Pub/Sub: Subscribers need to handle Pub/Sub-specific details, like managing acknowledgment IDs.
• Infrastructure requirements: For dynamic scaling against event load, solutions like GKE or Dataflow are required.

Workflow for a pull subscription, Pull subscriptions  |  Pub/Sub Documentation  |  Google Cloud

‍

Streaming Pull:

The Streaming Pull model is an extension of Pull subscriptions enabling high-throughput with low-latency. It uses a bidirectional streaming connection to continuously receive messages as they become available, minimizing delays.

‍

How it works:

1. The subscriber establishes a persistent connection with Pub/Sub.
2. Messages are streamed continuously over the connection.
3. Acknowledgments are sent back over the same connection.

Key benefits:

• Maximized throughput: Ideal for large-scale event processing.
• Reduced latency: Eliminates the need for repeated pull requests by keeping the connection open.
• Efficient for batch processing: Subscribers can process multiple messages in a single stream.
• It used by Pub/Sub client libraries wherever possible and is recommended for most pull subscription use cases to maximise throughput and reduce the overhead associated with repeated pull requests

‍

Choosing the Right Message Delivery Model

Google Cloud Pub/Sub serves as a powerful enabler for building event-driven microservices, offering the flexibility to choose between Push and Pull event consumption models based on your workload requirements.

While Push subscriptions simplify delivery and scaling for real-time applications, Pull subscriptions provide greater control and throughput for high-volume scenarios with the Streaming variant enabling efficient, low-latency processing in high-throughput systems.

Before finalising the choice of delivery model, it is important to also consider the scaling capabilities of the compute option being used when paired with the chosen delivery model.

In the next part of this series, we’ll explore containerization options on GCP, comparing platforms like Cloud Run and Google Kubernetes Engine (GKE) to help you build a scalable and resilient event-driven architecture.

Stay tuned to discover how to match your subscription model with the ideal platform to optimize your system's performance and scalability

‍