ServiceNow and Kafka: Stream Connect (2/3)

Three Ways to Connect ServiceNow and Kafka

In 2023, the Rynex team worked closely with one of our clients to validate different ways to connect ServiceNow and Kafka. We want to share that experience with the community, in the hope that it helps you in your own project and saves you time on research.

As part of our research, we examined three possible ways (proofs of concept) to connect ServiceNow to Kafka:

• Data Streaming with the Confluent Kafka Connector
• Data Streaming with ServiceNow Stream Connect (this article)
• Data Synchronization with an Integration Platform

Part two picks up where the Confluent connector left off - with the route that ServiceNow provides natively.

What Stream Connect Is

The Stream Connect plugin acts as a bridge that extracts relevant data from ServiceNow tables and publishes it to Kafka topics. It links your Kafka environment to your ServiceNow instance, letting you stream data between the instance and your external systems. Unlike the Confluent connector from part one, Stream Connect works with standard Kafka and does not require an enterprise license for the Confluent platform.

It does, however, require a license from ServiceNow. Only then do the Kafka-related functionalities of the Hermes Messaging Service become available. This applies even to non-production environments, because ServiceNow spins up additional services (servers) in its data center. The Hermes Messaging Service is a multi-tenant, multi-cluster data transport and queuing service built on Apache Kafka. It lets your instance produce and consume large volumes of Kafka events, and is the Now Platform capability available as part of Stream Connect for Apache Kafka, Log Export Service (LES), and Instance Data Replication (IDR).

To use the Hermes Messaging Service, activate the following plugins:

• com.glide.hub.flow_trigger.kafka
• com.glide.hub.etl_consumer.kafka
• com.glide.hub.kafka_consumer
• com.glide.hub.action_step.kafka

Important: if you want to try ServiceNow Stream Connect, first consult your contact at ServiceNow. A proof of concept cannot be carried out without the proper infrastructure in place on the ServiceNow side.

Key Components

• Publish and process Kafka events at scale: Publish events to your Kafka environment from your ServiceNow instance, and consume Kafka events from your external systems at high volume with low latency.
• Build flows that produce and consume Kafka events: Stream Connect is integrated with Flow Designer, providing a low-code way to publish and process Kafka messages.
• Reuse your transform configurations: Import data from your Kafka environment and process it using your existing Robust Transform Engine (RTE) or transform map configurations.
• Script your own consumers: Configure a consumer that uses your own scripts to process data from a Kafka topic.
• Monitor performance: Keep an eye on your consumers with detailed reporting of statistics and performance metrics.

Limitations

• Maximum message size: 2 MB
• Maximum message size per second: 2 MB (produced or consumed)
• Number of topics: 30 per instance
• Maximum number of partitions: 32 per topic
• Retention policy: 36 hours

Conclusion

In our specific business case we were unable to use this method, because we did not have a license for the Hermes Messaging Service. Even so, we explored key areas of the connector and its capabilities, which lets us draw a few conclusions.

Data streaming with ServiceNow Stream Connect lets you use multiple data sources and manipulate data during message transfer between ServiceNow and Apache Kafka. The final cost of implementation will vary from case to case, based on the ServiceNow platform's pricing policy. You should therefore weigh the cost of Stream Connect against the value it brings to your organization, and compare it with alternative solutions.