Web APIs and Streaming Data

Traditional WebAPI is inefficient during transactions that take long durations. Below are some popular alternatives to RESTFul API.

 

 

 

Webhooks

There is a client that registers with interested events and callback URL with the server. The client tells the server the client’s URL that the server POST updates to. When an event happens, the server POST to the client URL. Some things to consider when working with Webhooks.

  • Callback URL registered via subscription endpoint (on the client)
  • Ensures delivery through retries and reduces failiers
  • Apps running behind firewalls can send but receiving can be blocked
  • Webhooks are driven by events which means if a lot of events, it can be noisey
  • Webhooks is not yet a standard

 

WebSockets

Client sends a handshake (HTTP) to server, asks to do WebSocket. Server responds that it can and both agree to use the WebSocket API. This opens a bi-directional communication channel. WebSocket considerations:

  • Bidirectional communication is low latency (full-duplex via single TCP connection)
  • There is a significant reduced overhead for requests; important for realtime communications
  • Clients need to be responsible for establishing and maintaining connections (client must support Websocket)
  • Servers need to scale if many clients are connected.

 

GraphQL

Similar to Webhooks, the client tells the server what events it wants to subscribes to and how to get the event messages. The transport protocol is agnostc, however the most popular ones used is WebSockets (Apollo) and Server-Sent events (Sangria).

  • Like WebSockets, this is subscription that is read only (no bidirectional, only initial request is sent)
  • Subscriptions are client-driven, defined by the client app on what data to include in the events
  • GraphQL has own subscription language (GraphQL Schema Definition Language SDL)
  • Live Queries –

 

HTTP Streaming

Client submits request to server then the server continues to push data to the client. This can be done by sending data in chunks, which is popular for backend system communication. This option is what Twitter uses.

  • Can stream over simple HTTP and is supported by most browsers
  • This is not a form of bidirectional communication.

 

HTTP Live Streaming HLS

HLS was developed by Apple and the protocol used for their data streaming. For example, below is a general flow of video live streaming.

Camera –> Encoder –> Transcoder –> Server (stores and controls distribution) –> Edge / Caching (CDN) –> client (end user)

A packager will receive the data that is coming (eg RTSP) into outgoing formats and fragments acceptable by the different clients. This usually includes a manifest that explains the package being sent.

A transcoder converts one or more codecs/bitrates/resolutions to another. Like the packager it can repackage the data into different fragments. For example, it can send out a high, medium and low bitrate fragments simulataniously.

 

gRPC

Uses HTTP/2 and Protocol Buffers (created by Google). That means the connection uses .proto files which are serialized/deserialized during transport. The communication is bidriectional.

 

KAFKA vs XMPP

XMPP (Extensible Messaging and Presence Protocol) is a communication protocol for message-oriented middleware based on XML (Extensible Markup Language). It is used for sending and receiving instant messages, presence information, and other data between devices and servers.

Kafka is an open-source distributed event streaming platform used for handling real-time data feeds. It is designed to handle high volume, high throughput, and low latency data streams and is often used in big data and real-time streaming applications.

In summary, XMPP is a protocol for instant messaging and presence, while Kafka is a distributed event streaming platform for handling real-time data feeds. They are different technologies used for different purposes.

 

 

References

(7) Event driven API Strategies: from WebHooks to GraphQL Subscriptions – YouTube