Is Using KoP (Kafka-On-Pulsar) a Good Idea?

Pulsar Summit San Francisco 2022 – San Francisco 🇺🇸

Aug. 2022

Ricardo Ferreira

Slides

skip-backskip-back  /  skip-forwardskip-forward
download-clouddownload-cloud PDF

Abstract

Building microservices around Apache Kafka is your job, and life is great. One day, you hear community members talking about some neat Apache Pulsar features, and get you intrigued. I mean, we all love Kafka, but you can’t avoid wondering if migrate one of your projects to Pulsar is a good idea. Then it happens. You find Pulsar supports Kafka clients natively via a protocol handler called KoP: Kafka-On-Pulsar. This gets you pumped. Is that it? Can I go ahead and simply point my microservices to Pulsar and be a hero with this migration? But you must be responsible; and history says you shouldn’t believe migrations like this are refactoring free. Reason you may get interested in this session. We will revisit the architecture behind protocol handlers to understand what it means having one enabled on Pulsar. Plus, we will discuss the internals of KoP. Finally, we will use a show-and-tell approach to detail the effort to migrate a microservice written for Kafka to Pulsar, and whether the code need to change for this.

Video

Previous page In the land of the sizing, the one-partition Kafka topic is king
Next page Do It Yourself: Programmable Metrics using OpenTelemetry

© 2018 - 2026 Ricardo Ferreira

Search is powered by Pagefind. Just hit CTRL+K or CMD+K to start searching.

Powered by Hugo with Dream and Devrel themes.

Open Source

I contribute to LangChain4j, an idiomatic open source Java library for building LLM-powered applications on the JVM. Recent work includes adding native vector search embedding stores so developers can build RAG, recommendation engines, and AI memory systems.

I also ported RedisVL to Go, an open source, AI-native client that brings vector search, semantic caching, LLM memory, semantic routing, rerankers, and an MCP server to the Redis ecosystem for Golang developers.

Speaking

I’ve been speaking at conferences since 2008 and doing it full time as part of my work since 2018. Some of the events I’ve spoken at include AWS re:Invent, Microsoft Ignite, Google Cloud Next, KubeCon, Oracle OpenWorld, QCon, Strange Loop, Kafka Summit, Pulsar Summit, JavaOne, DevNexus, JFokus, JNation, and All Things Open.

Recordings of my talks live on my YouTube channel, and the code I write for talks, demos, and tutorials is on my GitHub.

Who am I?

I work at the intersection of distributed systems, AI, and data infrastructure, turning complex technology into things developers can understand, adopt, and build with.

Lately, that means hands-on AI engineering: building vector search, semantic caching, agent memory, and RAG into the data layer; contributing to LangChain4j and RedisVL for Golang; and figuring out how to make AI agents secure enough to ship.

The AI-native work isn’t a pivot. It draws on the same systems-design foundation I’ve built for 20+ years, moving data fast, at scale, close to compute, watching where systems break; now applied to vectors and agents. I have worked on event streaming with Apache Kafka and Flink at AWS and Confluent; observability at Elastic; as well as RDBMS, NoSQL, and Big Data at Oracle. That foundation is exactly what separates AI demos that work on stage from AI systems that survive production.

Social Links