ServiceNow and Kafka: The Confluent Connector (1/3)

Three Ways to Connect ServiceNow and Kafka

In 2023, the Rynex team worked closely with one of our clients to validate different ways to connect ServiceNow and Kafka. We want to share that experience with the community, in the hope that it helps you in your own project and saves you time on research.

As part of our research, we examined three possible ways (proofs of concept) to connect ServiceNow to Kafka:

• Data Streaming with the Confluent Kafka Connector (this article)
• Data Streaming with ServiceNow Stream Connect
• Data Synchronization with an Integration Platform

This is the first article in the series, and today we focus on data streaming with the Confluent Kafka Connector.

How the Connector Works

The Confluent Kafka Connect ServiceNow Source connector is available in the Confluent Connector Hub and can be installed into your Confluent Kafka environment with a few clicks. It moves data and updates from a ServiceNow table to Apache Kafka in real time.

The connector consumes data from a ServiceNow table to add and update records in a Kafka topic. The matching Sink connector can be used to transfer data from Kafka back to ServiceNow.

Key Components

• Supports one task: The ServiceNow Source connector runs only one task - one table is handled by one task.
• Delivery guarantee: Under normal operation, the connector delivers each ServiceNow update to Kafka exactly once. Duplicates are still possible in some situations, though: during temporary network degradation, after non-graceful shutdowns, or when connectors and tasks are rescheduled.
• Automatic retries: The connector may hit network failures when connecting to the ServiceNow endpoint. In that case it automatically retries to poll the data. The property retry.max.times controls how many retries are attempted, and an exponential back-off is added to each retry interval.
• Supports HTTPS proxy: The connector can connect to ServiceNow through an HTTPS proxy server.
• Elasticity: Two parameters enforce the throughput limit: batch.max.rows and poll.interval.s. By default the connector uses 10,000 records and a 20-second interval. If a large number of updates occur within the interval, it paginates records according to the configurable batch size. Since ServiceNow provides precision down to one second, the lowest possible poll.interval.s setting is one second.
• Real-time and historical lookup: The connector supports a specific starting point in history. It can retrieve all historical records in one run and then catch up with real-time records.

Limitations

The ServiceNow Source connector is very powerful. It is important to note, however, that it does not support the following table types:

• System Audit (sys_audit)
• Audit Relationship Change (sys_audit_relation)
• History (sys_history_line)

A separate task is required for each table. If many tables are involved, this approach may not be the best solution.

Setting Up the Environment

• Deploy the ServiceNow instance with a configured API user that has the required permissions for the defined tables.
• Configure and deploy the Kafka cluster along with the required user groups, users, and topics.
• Activate and configure the ServiceNow Source connector. Note: the connector may not be part of your license (check this), and it can incur additional costs based on runtime and data volume.

Testing and Validation

• Perform extensive testing to ensure the accuracy and consistency of data synchronization.
• Validate data integrity and verify that errors are handled appropriately.

Connecting the Confluent Connector

Confluent Kafka offers Connector Plugins to transfer data from Kafka to ServiceNow and from ServiceNow to Kafka. In this walk-through we export data from ServiceNow to Kafka using the ServiceNow Source connector, which consumes data from a ServiceNow table to add and update records in a Kafka topic.

• Select the ServiceNow Source connector plugin and choose the topic for exporting data from ServiceNow.
• Define granular access: restrict the connector to the desired environment and topics only.
• Enter the authentication details for the ServiceNow instance.
• Define the output format and adjust the configuration if required.
• Set the starting time in UTC (yyyy-MM-dd): the point from which all updates and creations are exported from ServiceNow. By default the creation time is used; set an earlier time to import historical data from the beginning.
• Specify the size of the connector: sizing depends on the number of topics and the volume of exported data, and can be scaled later for additional throughput.
• Name the connector and start it.
• Check the status of the Connect plugin.

Conclusion

This is a strong approach for a business that already uses Confluent Kafka. Compared with the methods in the following articles, it stands out for its relatively low cost and low implementation complexity.

Its main limitation: it does not let you manipulate the information while it is being transferred between Confluent Kafka and the ServiceNow platform.