Protobuf

1 article

Agents AI Alexa Amazon Q AMQP Apache Apache Kafka Apache Pulsar Architecture AWS AWS Kinesis Bastion Host Bedrock BRIO Certification Cloud CodeWhisperer Confluent Craft Data Structures Databases Deserializer Dock Docker Elastic ElastiCache Elasticsearch Extreme Go Golang Grafana Hadoop Heartbeat Infrastructure-as-a-Code Java JMS Kafka LangChain LangGraph Laptop Learning Lenovo LinkedIn Linux LLM Logitech Logstash MCP Messaging Metricbeat Multiple Columns MX Anywhere Observability OpenTelemetry Performance Programming Projects Prometheus Protobuf RAG Redis Search Security Serialization Serializer Sorted Set Streaming Terraform ThinkPad Tools Topic Treat Vector Databases VSCode Workaround X1

Data Sharing between Java & Go using Kafka and Protobuf

Thursday, Jul 9, 2020

One of the most incredible things in Apache Kafka is that record values are shapeless, meaning that developers can write any set of bytes they want, and it works just fine. This can be pretty powerful if they share data among different programming languages.

However, this characteristic is sometimes inadequate. It creates coupling problems as producers and consumers need to negotiate a common format that they can rely on to write and read data. Therefore, it is essential to use a neutral format – likely something accepted within the industry, such as Avro, Protobuf, and JSON Schemas. This post will focus on the problems raised by using Protobuf to share data between Java and Go.

Ricardo Ferreira
3 minute read

© 2018 - 2026 Ricardo Ferreira

Search is powered by Pagefind. Just hit CTRL+K or CMD+K to start searching.

Powered by Hugo with Dream and Devrel themes.

Open Source

I contribute to LangChain4j, an idiomatic open source Java library for building LLM-powered applications on the JVM. Recent work includes adding native vector search embedding stores so developers can build RAG, recommendation engines, and AI memory systems.

Public Speaking

I’ve been speaking at conferences since 2008 and doing it full time as part of my work with DevRel since 2018. My talks go deep on the systems I build with: distributed systems and event streaming, AI engineering and vector search, and the data infrastructure that has to hold up when the demo ends and production begins. Some of the events I’ve spoken at include AWS re:Invent, Microsoft Ignite, Google Cloud Next, KubeCon, Oracle OpenWorld, QCon, Strange Loop, Kafka Summit, Pulsar Summit, JavaOne, DevNexus, JFokus, JNation, and All Things Open.

Ricardo Ferreira presenting on the main stage at AI DevWorld
On the main stage at AI DevWorld

You can find my upcoming and past talks on my speaking calendar. Recordings also live on my YouTube channel, and the code I write for talks, demos, and workshops is on my GitHub.

Consulting and Professional Services

If you’d like to hire me as a consultant for your projects, speak at your event, or lead a hands-on workshop for your team, contact me at riferrei@riferrei.com. I can understand the scope of your request and provide a free estimate.

Who am I?

I work at the intersection of AI, data infrastructure, and distributed systems. In practice, that means hands-on AI engineering: building vector search, semantic caching, agent memory, and RAG into the data layer, and figuring out how to ship AI agents securely. I also contribute to the LangChain4j open-source project.

The AI-native work isn’t a pivot, though. It draws on the same systems-design foundation I’ve built for 20+ years: designing data systems for scale, processing fast data, watching where systems break; now applied to vectors and agents. I have worked on RDBMS and Big Data at Oracle; event streaming with Apache Kafka and Apache Flink at Confluent; observability at Elastic; AI and developer tooling at AWS; and NoSQL and vector stores at Redis.

This foundation is exactly what separates AI demos that work on stage from AI systems that survive production.

Social Links