Skip to content

README.md

Latest commit

 

History

History
159 lines (115 loc) · 4.54 KB

README.md

File metadata and controls

159 lines (115 loc) · 4.54 KB

Apache Pulsar Connectors

This repository contains the IO connectors for Apache Pulsar.

Connectors are packaged as NAR files and can be deployed into any Pulsar installation by placing them in the connectors directory or mounting them into the apachepulsar/pulsar Docker image.

Available Connectors

Sources

Connector Description
Canal MySQL binlog via Alibaba Canal
Debezium (MySQL, PostgreSQL, MongoDB, MSSQL, Oracle) CDC via Debezium
DynamoDB Amazon DynamoDB Streams
File Local filesystem
Kafka Apache Kafka
Kinesis Amazon Kinesis Data Streams
MongoDB MongoDB change streams
NSQ NSQ messaging
RabbitMQ RabbitMQ / AMQP

Sinks

Connector Description
Aerospike Aerospike database
Alluxio Alluxio distributed storage
Azure Data Explorer Azure Data Explorer (Kusto)
Cassandra Apache Cassandra
Elasticsearch / OpenSearch Elasticsearch and OpenSearch
HBase Apache HBase
HDFS3 Hadoop HDFS
HTTP HTTP endpoint
InfluxDB InfluxDB time-series database
JDBC (PostgreSQL, MariaDB, ClickHouse, SQLite, OpenMLDB) JDBC databases
Kafka Apache Kafka
Kinesis Amazon Kinesis Data Streams
MongoDB MongoDB
Redis Redis
Solr Apache Solr

Adaptor

Connector Description
Kafka Connect Adaptor Run Kafka Connect connectors on Pulsar

Prerequisites

Note: This project includes a Gradle Wrapper so no separate Gradle installation is needed. Use ./gradlew on Linux/macOS and gradlew.bat on Windows.

Building

Compile and assemble all modules:

./gradlew assemble

NAR files are produced under each connector's build/libs/ directory.

Build a specific connector:

./gradlew :elastic-search:assemble

Build the distribution package containing all connector NARs:

./gradlew :distribution:pulsar-io-distribution:assemble

Check source code license headers:

./gradlew rat spotlessCheck

Auto-fix license headers:

./gradlew spotlessApply

Running Tests

# All unit tests
./gradlew test

# Specific connector
./gradlew :elastic-search:test

# Specific test class
./gradlew :elastic-search:test --tests "ElasticSearchSinkTests"

Using Connectors

With Docker

Mount connector NARs into the Pulsar container:

docker run -v /path/to/connectors:/pulsar/connectors apachepulsar/pulsar

Manual Installation

Copy NAR files to the connectors/ directory of your Pulsar installation:

cp elastic-search/build/libs/pulsar-io-elastic-search-*.nar $PULSAR_HOME/connectors/

Versioning

This repository follows its own release cadence, independent from Apache Pulsar releases. All connectors are released together as a single release. The initial release version matches the Pulsar version at the time of the split.

Each release specifies which Pulsar versions it is compatible with. The pulsar-io-core API has been stable for years, so connectors are generally compatible across Pulsar 4.x releases.

Contributing

Contributions are welcome! Please see CONTRIBUTING.md in the main Pulsar repository for guidelines.

License

Licensed under the Apache License, Version 2.0. See LICENSE for details.
⚡