Simplified data pipelines with Pulsar transformation functions

Click for: original source

They provide a low-code way to develop basic processing and routing of data using existing Pulsar features. Using functions in the cloud is a very efficient way of creating iterable workflows that can transform data, analyze source code, make platform configurations, and do many other useful jobs. As you develop a function you will quickly realize a need for a solid foundation of utilities and formatting. By Christophe Bornet.

This piece discusses:

  • About transformation functions
  • Function operations
  • Example configuration
  • Transformation function compute operation
  • Taking transformation functions further
  • Deploying the functions on Astra Streaming
  • Getting started with transformation functions

Similar to designing microservices, functions have boilerplate code and need standardized processes, and writing the boilerplate code can feel like valuable time spent on a seemingly mindless task. You bring value to a project by creating its core logic, not by creating JSON-parsing methods. You will code examples and links to further reading as well in this article. Good read!

[Read More]

Tags app-development data-science apache big-data