Cloud Pub/Sub : Global real-time messaging

What is pub/sub?

Pub/ Sub stands for Publisher/ Subscriber. It allows services to communicate asynchronously, with dormancies on the order of 100 milliseconds. It’s used for streaming data integration channels and analytics to ingest and distribute data. It’s inversely effective as messaging- acquainted middleware for service integration or as a line to parallelize tasks.

Pub/ Sub enables you to produce systems of event directors and consumers, called publishers and subscribers. Publishers communicate with subscribers asynchronously by broadcasting events, rather than by coetaneous remote procedure calls (RPCs). Publishers shoot events to the Pub/Sub service, without regard to how or when these events will be reused. It also delivers events to all services that need to reply to them. Compared to systems communicating through RPCs, where publishers must stay for subscribers to admit the data, similar asynchronous integration increases the inflexibility and robustness of the system overall.

How Pub/Sub works?

Cloud Pub/Sub Architecture

To understand the working of Pub/Sub, we first need to know about the architecture of cloud pub/sub which is shown in the image above.

As the name implies, this data streaming process is basically based on three parts: publisher, subject, and subscriber. This process is considered persistent, reliable, and flexible because it uses both publishers and subscribers. The Pub / Sub data streaming method primarily separates the sender from the recipient. This has found a secure way to process data between independently written applications. It has the ability to hold large amounts of data. This is made possible by the scalability of the service.

Data flow in Cloud Pub/Sub

The flow of data in the Pub / Sub service depends primarily on publisher, subject, and subscriber applications. A publisher application is an app engine that is any application that makes hypertext transfer protocol secure requests in a web service and publishes news about the topic. The subscribing application on the subscribing side subscribes to this topic. Subscribers include pull subscribers and push subscribers. Pull subscribers send hypertext transfer protocol secure requests to the Google API, and push subscribers accept data streamed over HTTPS.

Communication can be done in any manner :

1. Many subscribers for one publisher.
2. One publisher for many subscribers.

Types of Cloud Platforms providing pub/sub services

1. Google:

Google Cloud Pub / Sub provides messaging between applications. Cloud Pub / Sub is designed to provide reliable asynchronous messaging for many applications between applications. Publisher applications can post messages to “topics” and other applications can subscribe to the topic to receive messages.

2. IBM:

Publish / subscribe messaging allows you to separate an information provider from its consumers. Send and receive requests do not need to know anything about each other to send or receive information. Before a point-to-point IBM MQ application can send a message to another application, you need to know something about that application.

3. AWS:

Amazon Simple Notification Service (Amazon SNS) is a fully managed messaging service for application-to-application (A2A) and application-to-application (A2P) communication. The A2A pub / sub feature provides the topic of push-based many-to-many messaging with high throughput between distributed systems, microservices, and event-driven serverless applications. Amazon SNS topics allow publishing systems to fan out messages to numerous subscriber systems such as Amazon SQS queues, AWS Lambda functions, HTTPS endpoints, and Amazon Kinesis Data Firehose to process them in parallel. The A2P feature allows you to send messages to users on a large scale via SMS, mobile push, and email.

1. Amazon Simple Notification Service (Amazon SNS) is a fully managed messaging service for application-to-application (A2A) and application-to-application (A2P) communication. The A2A pub / sub feature provides the topic of push-based many-to-many messaging with high throughput between distributed systems, microservices, and event-driven serverless applications. Amazon SNS topics allow publishing systems to fan out messages to numerous subscriber systems such as Amazon SQS queues, AWS Lambda functions, HTTPS endpoints, and Amazon Kinesis Data Firehose to process them in parallel. The A2P feature allows you to send messages to users on a large scale via SMS, mobile push, and email.

Key Features

1. Stream analytics and connectors Native dataflow integration enables reliable, expressive, accurate, one-time processing and integration of event streams in Java, Python, and SQL.
2. Order bulk delivery The optional single-key layout simplifies stateful application logic without sacrificing horizontal scaling.
3. No partition is needed.
4. Cost-optimized recording with Pub / Sub Lite As a complement to Pub / Sub, Pub / Sub Lite is said to be the most cost-effective option for recording large numbers of events.
5. Pub / Sub Lite provides zone storage and gives you control over capacity management.

What is the lifecycle of a message in Pub/Sub?

An important part of using Pub / Sub effectively is to understand how messages are forwarded from publish to use. First, the editor sends a message to the subject. As soon as the Pub / Sub server receives the message, it adds fields such as the publish timestamp and message ID to the message. Pub / Sub sends this message to all subscriptions. From each subscription, the subscriber pulls the message through one of two pull calls (more on this later). Each subscription has a confirmation deadline that determines when the client can confirm this message before the server redistributes the message. If the customer holds the message beyond the agreed deadline, or if the customer submits a click request, the message will be redelivered.

What publisher settings are available, and what do they do?

Our Pub / Sub client library offers many configuration options for controlling message publishing, including batch functionality. Batch publishing helps reduce overhead by publishing multiple messages as part of a single public RPC. When you access publishing using the official client library, the message is not published immediately, but it is added to the batcher to be published with other messages. This behavior can be configured as follows: Maximum number of messages per batch Maximum stack size (in bytes) If the batch is not filled (delayed), how long to keep the batch of messages before sending. Increasing the number of messages per batch or the size of the batch reduces the number of public requests that need to be executed and improves throughput per public request. However, if you do not fill the stack fast enough, latency (the time between the client invoking the publication and the server confirming the publication) can be high. Conversely, reducing batch latency can improve latency, but if public requests cannot be moved fast enough (low throughput), public requests may be queued.

Benefits of Pub/Sub Messaging

• In today’s cloud architecture, applications are divided into smaller, independent building blocks that are easier to develop, deploy, and maintain. Pub / Sub messaging provides instant event notifications for these distributed applications.
• The publish and subscribe model allows you to create event-driven architectures and asynchronous parallel processing while improving performance, reliability, and scalability. Learn more about Pub / Sub messaging on AWS.

Other features of Pub/Sub Messaging

• Pub / Sub messaging makes it easy for developers to build sophisticated, complex-architecture applications in the cloud. With Pub / Sub, publishers and subscribers are separated and unaware that they exist.
• Subscribers are interested in a particular topic, and publishers send messages about the topic. The message is then immediately delivered or pushed to all subscribers to the topic.
• Using asynchronous pulling provides higher throughput in your application, by not requiring your application to block for new messages. Messages can be received in your application using a long running message listener, and acknowledged one message at a time.

The pros and cons of the Pub-Sub architecture pattern

Pub/Sub’s architectural patterns are widespread and for good reason. See how this pattern handles the performance of asynchronous messaging. Understanding common architectural patterns is essential for large-scale design of software architectures. Not only can you save time, but you can also ensure that your design is implemented. There is no need to reinvent the wheel when an architectural pattern that matches the developed architecture becomes available.

Pros:

• Pub/Sub activities are asynchronous (“Fire and Forget”). Therefore, there is little risk of performance degradation due to processes getting stuck with long-term data exchange interactions.
• Adding or removing subscribers to a topic is a configuration issue. No complicated programming is required. Therefore, the Pub/Sub system offers a high degree of scalability and flexibility.

Cons:

• Testing can be difficult. The interaction is asynchronous, so testing is not about making a request and analyzing the results. Rather, you should send the message to the system and then observe the behavior of the process under test to see when and how the message is being processed.
• Also, managing the test flow can be more difficult if the process under test requires many messages about the topic over time.
• Unexpected increases in message emissions can lead to network bottlenecks and can have unexpected consequences for consuming message brokers.
• Message formatting and message exchanges require well-defined policies. Otherwise, the message consumption will be garbled and error prone.
• Unexpected increases in message emissions can lead to network bottlenecks and can have unexpected consequences for consuming message brokers.
• Message formatting and message exchanges require well-defined policies. Otherwise, the message consumption will be garbled and error prone.