If you know me well, you must know that I am a big fan of movies. It is not uncommon to see me in situations where if I need to explain something rather complex, I end up using movies as metaphors to understand what I am trying to explain. In this post, I will explain what Apache Kafka has in common with the character Rocky Balboa.
Rocky Balboa is a fictional character created and played by Sylvester Stallone, who tells the history of a boxer that faces numerous challenges throughout his life and career. But in every single movie of the series (eight produced until this date), it is impossible not to fall in love with the character. This is the first and most important thing that Kafka has in common with this character.
Kafka is a technology that creates a strong emotional bond with people. It is hard to explain why this happens, but the reality is that if you ask anyone from the community why Kafka is so great — they will have a hard time explaining. It is an invisible love that starts small, but as you keep using the technology, it grows to something beyond comprehension, just like what happens with Rocky.
When you first start watching his movies, you can’t avoid but see yourself in character. You can see yourself facing his challenges; you can see yourself feeling the same emotions that he does. And that is something amazing because somehow the actor pulled off something really hard in the world of acting: emotional bond. If you ever watch a movie where you suddenly start to cheer for the character — whether it is portraying a good or a villain — that means that there is an emotional bond. Then you can’t avoid but run along with the character.
Creating this emotional bond requires having a unique personality. People need to believe in their hearts that there is something really different with that character, and therefore — they can open themselves to the loving part. The same happened with Kafka. Kafka has a unique personality that anyone in the world now understands, knowing that Kafka is:
- Something that they can trust;
- Something that simply works;
- Something that is very fast;
- Something that is scalable;
- Something that is battle-proven;
Along with these characteristics, Kafka also has a very charismatic personality that helps people falling in love with it, like what happens with Rocky. Rocky is humble because he is never looking to overcome anybody, hurt anybody or be more than anybody. You can see in the first movie that even when he is told to hurt someone (that scene where the mob boss asks him to collect a debt), he is bounded to forgive the debt and let the person go. Because in his heart, he is a good person.
So it is with Kafka. You don’t see anywhere content about Kafka being forcibly published to beat some other technology or prove itself. Kafka is this amazing technology that people choose to use because they like it and because it works. And the community that is behind Kafka also shares the same traits: they are humble, have a good heart, and, more importantly, are not bullies.
Rocky is also known for being able to overcome difficult situations with perseverance, obstinacy, and serenity. Throughout the entire movie series, you can see Rocky sometimes almost falling for some challenging situation. Still, he ultimately finds a way to pull himself up and come out as a winner. That is another trait that people like about Rocky because it mimics how we are with our lives. We don’t need to be always on edge, but we find ways to overcome our limitations and win when we are properly pushed.
Needless to say, so it is with Kafka. There has never been a single situation where you couldn’t use Kafka. This is not the same to say that Kafka doesn’t have limitations. This has to do with the fact that Kafka’s core is strong, and when presented with the most challenging of the situations, Kafka always finds a way to overcome the challenge with something that is part of its core. Kafka is adaptable, extensible, and a really bulletproof technology. Just like Rocky, Kafka might not always seem to have what you’re looking for; but if you look deeper enough (e.g., properly push it), Kafka always has some acceptable alternative.
Rocky is also notoriously known for constantly having enemies. It is funny, but you secretly keep this excitement to the scene where his new enemy will show up while watching his movies. If we pick a pattern behind every Rocky movie, either he will suffer a lot (both emotionally and physically) or have a new enemy. It is almost as if everybody is desperately trying to beat him down and/or take away what he has.
Kafka is also known for constantly having competitors. It is a technology that has been around for more than a decade now (if we count the time before it was released, GA). From time to time, there is always some other technology trying to beat Kafka down and/or take away what it has… which in the universe of software its adoption and market share.
The best way to discuss Kafka’s competitors is making analogies with Rocky’s enemies because, curiously — they also share similarities with the technologies that Kafka competes against. Let’s get it started.
Apollo Greed / JMS
At the beginning of the first Rocky movie, we are introduced to his character, a small-time boxer struggling in life, essentially working during the day as a loan collector and making small paid fights during the night for a living. Including his own trainer, everybody calls him “just a bum” because he is not actively pursuing anything in his life. Everything changes when he is given a chance to fight a serious contender, nothing more than the current champion Apollo Creed.
Right off the bat, we can see some fascinating similarities between Rocky and Kafka. As many people know, Kafka was not born Kafka but instead — it was the result of several internal projects on LinkedIn to come up with a perfect solution to their “data problem” or to put more professionally: to come up with a universal backbone that would allow any data to be stored, processed, and shared efficiently.
Apollo is this full of himself, a charming, provocative fighter that is so confident that no one is actually capable of beat him that he starts looking for a challenge to prove that he still has what it takes. Apollo shares many similarities with JMS. Not that JMS was the first messaging technology globally, but it was by far the most successful one. Mainly because it chooses to rely on standards, and because of that — it allowed itself to be used in many different workloads.
However, just like Apollo, JMS failed to understand that being the standard successful messaging technology for a very long time doesn’t mean being in that position forever. While trying to be the best, it forgot to evolve and improve itself as an option continuously, as well as it forgot that the world constantly changes and what was considered a large workload in the past (e.g., 10K messages per second) in today’s world is what Kafka eats for breakfast. That goes in line with recurring comments from Apollo about Rocky’s punches being the hardest that he ever had seen.
Another interesting fact about the first Rocky movie is the romance that Rocky starts with Adrian — a painfully shy woman that Rocky likes right off the start because she reminds many of him. Adrian was in this movie and throughout the entire series — the mountain that allows Rocky to withstand his most difficult challenges. As a result of this romance, Rocky eventually marries her during the second movie. I think the analogy here is clear to understand — but in case it isn’t — Adrian is to Rocky what Zookeeper is to Kafka. For many years Kafka relied on Zookeeper to do many of its core operations.
Back to Apollo, I think it is fair to say that just like him, JMS also had many enterprises backing it up, which contributed to its continuous success. These enterprises were his managers and sporting goods companies interested in profiting from his victories in Apollo’s case. In JMS, the vendors decided to back JMS up because it would allow them to sell their products since positioning them based on standards easily is more appealing than proprietary. The reason why there were, at a certain point in time, several JMS-based products like TIBCO, Fiorano, ActiveMQ, SonicMQ, webMethods, etc.
It is fair to say that JMS significantly improved the market share of these products because all of them could start promising interoperability between them, given that they were now based on the JMS spec.
While Apollo was worrying about profit and his next exhibition, Rocky started to train hard with Mickey, who saw a chance to transform Rocky from the “just a bum” to someone who could really make the difference. Mickey always believed in Rocky’s potential, and it would drive him mad at the idea of Rocky going on wasting his potential with small fights.
Something similar happened with Kafka. While seeing the potential of Kafka beyond LinkedIn, its creators (Jay Kreps, Neha Narkhede, and Jun Rao) decided then to make the technology open. In October of 2012, the technology became GA. It was born then the technology that would change the lives of many developers, and just like Rocky — it became something that developers all over the globe learned to love.
When it came the night where Apollo would fight Rocky, something happened right at the end of the first round: Rocky knocked down Apollo. Something that never has happened in Apollo’s career and therefore something that he always had been proud of. At this point, Apollo started to worry about Rocky, although his ego never fully faded.
Kafka knocking JMS down is something that any developer that has done any test to compare the two would also see. Kafka was designed with specific purposes in mind. Those purposes were handling large amounts of data and keeping up with any workload using known scaling out techniques from distributed systems. JMS was not built for that, and it is not a surprise to see Kafka beating JMS.
“It’s not about how hard you hit. It’s about how hard you can get hit and keep moving forward. How much you can take and keep moving forward.” — Rocky Balboa
This quote, extracted from one of the movies series, became a worldwide line for anyone trying to start a motivational speech. But it also best describes why Kafka performs so well under the most challenging workloads: it is a technology that knows how to get hit and keep moving forward.
Clubber Lang / AMQP
Rocky III is by far my favorite movie of the series. Maybe because of the Eyes of the Tiger soundtrack, but it is simply my favorite one. It shows Rocky as this successful boxer that after defeating Apollo, became the current champion.
Earlier in the movie, Rocky is shown fighting against Thunderlips (played by Hulk Hogan), a wrestling champion that doesn’t necessarily respect boxing rules. Right after the fight starts, Rocky realizes that fighting using his own style will not help him in defeating Thunderlips, and then Rocky starts to fight the very same style fought by him.
I like to think that Thunderlips is somehow equivalent to Hadoop. It is certainly a great technology to solve certain types of problems. Still, just like Rocky adapting himself to fight Thunderlips — Kafka also is equally capable of changing its style to solve different types of problems. It is not any surprise by now that Kafka can keep up with Hadoop and easily defeat it, just like Rocky was able to defeat Thunderlips. It turns out that Rocky is not as heavyweight as Thunderlips is but instead — he is faster and adaptable. Just like Kafka.
Meanwhile, in the movie, we can see this fighter called Clubber Lang (played by Mr. T) that grows a very intimate desire to be what Rocky is and to have what Rocky has. That is when we start to see all these scenes where Clubber is training really hard, making sure to study all Rocky’s moves, and more importantly — sustaining a particular training that would grant him powerful punches that would drain Rocky’s energy quickly since he knows that Rocky is known for his endurance.
When I say that AMQP is somehow equivalent to Clubber Lang, I mean this in this most respectful way because he is great, and so is AMQP. Though it is a standard AMQP managed to support multiple programming languages, that alone is more than enough to respect the technology. AMQP brokers are also known for being able to handle large amounts of data.
Curiously, we can see right at the beginning of the movie the first fight between Clubber and Rocky. Clubber effortlessly defeats Rocky because he applied what he had decided to be the key to defeat Rocky: powerful punches that would drain Rocky’s energy quickly.
Rocky’s defeat from Clubber could be tied to several factors that the movie tries to portrait, such as the fact that his trainer Mickey had a heart attack minutes before his fight or because Rocky was not training really hard as he was supposed to. But the reality is that Clubber won the fight because he trained appropriately for it and brought a strategy to the ring.
The same can be said about AMQP. If you look closely at any performance study that claims that AMQP is faster than Kafka, you will see that there are many knobs to tweak to accomplish that. The most notorious one is disabling persistence. AMQP was not designed to persist data, just like any other messaging technology. Persistence in AMQP is just a way to address fault tolerance, where the brokers could provide business continuity even in a broker failure. But if you leave this feature turned on continuously — it will slow down the broker and its capability of being fast. This is not to say that AMQP is bad, but being fast due to an adaptation of a knob doesn’t really equate to saying that the technology is fast by default.
The remaining of the movie shows Rocky training along with Apollo Creed for a rematch against Clubber Lang. Probably one of the most famous quotes from this movie came from Apollo when he says to Rocky that he had lost “his edge” and to defeat Clubber; he would need to regain his “eye of the tiger.”
After Rocky really puts his mind into the training, we can see Rocky really absorbing Apollo’s fighting style, which is best known for his speediness, dancing style of movement, and his unique way through several punches in sequence.
Just like Rocky, Kafka keeps itself humble and never stops improving. A quick look into Kafka’s KIP (Kafka Improvement Proposal) program can quickly show how many cool features have been added to Kafka throughout the years, where many of them might even have been influenced by how some other technologies work.
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
While having KIP’s is good, Kafka remains very active in the community, which might be even more important than just keeping adding features. It is amazing how the people keep cheering for Kafka throughout the years. Kafka’s personality is unique.
Rocky put himself through a decent amount of pain during the final fight in getting Clubber Lang crucifying him with his powerful punches. Meanwhile, Rocky remains in the position of simply absorbing Clubber punches and sometimes even taunting him for more. It turns out that Rocky’s strategy is to turn Clubber’s own strategy against him: while giving his powerful punches, Clubber started to get tired and therefore; opening up a window for Rocky to get him good.
That goes back to the point where AMQP — as good as it might be — was not designed to withstand workloads that require data to be continuously persisted and processed, which is the basis for streaming. Just like Clubber Lang, AMQP gets tired if persistence is enabled; therefore, it might work well for quick/small bursts, but just like Clubber Lang, it doesn’t hold his breath for much longer. On the other hand, Kafka has persistence by default, so it can do what it does without persistence being the reason to slow it down.
Ivan Drago / Amazon Kinesis
Rocky IV is the fourth installment of the series that portrays Rocky making friends with Apollo Creed. After Apollo helped Rocky to defeat Clubber in the previous movie, we can see how close friends they became even though he once was one of Rocky’s enemies.
Similarly, we can see an equal friendship being built between Kafka and JMS. More and more, there are use cases where developers writing code for JMS-based apps would like to use Kafka seamlessly. A good example is support for JMS that Confluent introduced to its Kafka-based platform called Confluent Platform.
But the movie starts to unroll its plot by introducing Ivan Drago — a new fighter from Russia that emerged from the ground up to be the perfect fighting machine capable of defeating anyone. Ivan proposes an exhibition fight to Rocky after arriving in the U.S with his wife and trainers.
I think Ivan Drago shares many similarities with Amazon Kinesis, a service built from the group up to be this unquestionable thing that handles any scale because it has been built as a native service on top of AWS. And just like Drago, Kinesis counts with a phenomenal infrastructure to ensure its success. Also, just like Drago that is shown doing demos of his punches with a machine measuring its efficiency, many demos show how amazing Kinesis is with some exciting numbers.
One of the saddest scenes of the movie is when Drago defeats Apollo but ends up killing him with an unstoppable brute attack. It turns out that Apollo convinced Rocky to give up the challenge of fighting against Drago so he could do it instead. But regardless of who would fight Drago — both Rocky and Apollo were expecting a simple exhibition match, and they never thought that the Russian contender would be willing to bring the fight to a deadly level.
The death of Apollo in the hands of Drago also has significance in the technology world, notably when developers started to use Amazon Kinesis for every application that would require messaging. JMS, which used to be a long-term candidate for this type of use case, suddenly started to disappear because developers thought that using Kinesis would be the right way to approach messaging.
It is not that I don’t think that Kinesis is a good service, but blindly believing that Kinesis is great just because it has been built on top of a great infrastructure such as AWS is what I think is nonsense. Similarly, the Russians from the movie would claim that Drago is unbeatable because of his unique training system designed by the Soviet superiority. While clearly, we know that this inference doesn’t hold — there will always be people that would fall for this fallacy.
Enraged by the guilt of thinking that it was his fault for letting Apollo die, Rocky decides to challenge Drago for a fight while vacating the title. Rocky was decided to beat Drago to avenge his friend and as a way to live in peace with himself. An interesting fact about this movie plot is that Rocky decides to challenge Drago in the Soviet Union where Rocky would know that conceptually — Drago would be on his natural habit and wouldn’t. This is one of those plots that make you think that Rocky is stupid but, in reality, isn’t much for the contrary. He wants to use this opportunity to reinvent himself in a territory that he was not built for.
I think this is also true for Kafka. Though many people thought Kafka would not work properly in the cloud, Confluent proved otherwise by providing the first fully managed service for Apache Kafka, known as Confluent Cloud. It turns out that not only Kafka works great in the cloud, but it also works great across different cloud providers, notably AWS, GCP, and Azure. This is not to say that building a service to run Kafka in the cloud is easy. But with the proper engineering, it became possible.
This is essentially what we see in the movie regarding Rocky’s training in the Soviet Union. Facing a solid winter, he trains hard with some very unorthodox methods that are arguably meant for people twice his size.
When the time comes for the final fight between Rocky and Drago, there are many moving parts to be noticed besides the fact that the two of them are fighting. The most notable one is the crowd that obviously is cheering for Drago. This reminds me a lot of those developers that automatically cheer for any cloud-based technology for the simple fact that the cloud is awesome. While that is not untrue, the reality is that betting against any technology that is not cloud-native is a huge mistake.
When Kinesis started to gain some traction some years ago, many people would say that Kafka is dead because Kafka was born to be on-premises. It turns out that Kafka works well in the cloud and sometimes even better than Kinesis. That is when we started to see more and more people cheering for Kafka in the cloud, just like what happens in the movie when the crowd begins to root for Rocky, given his ability to get hit and keep moving forward.
Unsurprisingly, Rocky eventually beats Drago and thus the fight. Rocky learns throughout the fight that Drago is not this unbeatable machine he claims to be but instead — that Drago is just a man. Controversially, Drago claims that Rocky “is not human, he is like a piece of iron,” meaning to say that Rocky is hard to break despite all Drago’s efforts in applying what he has against Rocky.
I truly believe that Kafka is also a piece of iron. It is tough to break Kafka even in the most demanding scenarios. Likewise, Kinesis is not that unbeatable technology that never breaks. While it is admittedly an outstanding service from AWS, the reality is that it can be broken, just like what happened with Drago in the movie. I can’t avoid but think that this was the reason for AWS to come up with its own managed service for Apache Kafka that competes against Kinesis in many ways.
Tommy Gun / Apache Pulsar
Rocky V is the movie from the series that people claimed to be the worst, but I liked it. It gave back the perspective of Rocky being again this regular person since one of the twists of this movie is that he loses all his fortune because of an unscrupulous accountant.
The movie is all about this young new boxer called Tommy Gun (which is played by Tommy Morrison) that wants to be what Rocky once was. Anecdotally, he desires the part of Rocky that had money, glory, and fame, rather than the part of Rocky that people fall, which is his likable personality.
Rocky takes Tommy under his wings and starts to train him so Tommy could get what he craves, which is to be as good as Rocky. This is interesting because it shows Rocky evolving from a fighter to a trainer with lots of experience to share, making Rocky remember his former trainer Mickey that devoted his last days entirely to him.
Apache Pulsar reminds me a lot of Tommy Gun. It craves to be what Kafka is, and just like Tommy learning from Rocky to ensure being the best — Pulsar learned everything it knows from Kafka. Pulsar is known for copying everything that Kafka does but tries to do it better.
Although Pulsar lacks its own personality, it is a great technology. It does a few things that Kafka currently doesn’t, such as having an infinite storage strategy that doesn’t rely on the cluster nodes but instead it relies on Apache BookKeeper. While this approach arguably increases the operational overhead of the cluster to be maintained, it provides an economical way for the cluster to add more storage capacity without necessarily having to add more computing power. Kafka has a KIP under discussion to address this as well.
As Tommy trains with Rocky, he keeps getting better, knocking down his opponents continuously, proving that he actually learned what Rocky had to teach. The movie evolves very fast on this matter, trying to empathize with Rocky’s strong connection with his trainee since it gives Rocky a chance to be what Mickey once was to him. Kafka is a disruptive technology that brought distributed systems very clever solutions to hard problems. In a sense, it is a technology that will be the “trainer” of other technologies.
Up to that point of the movie, we don’t actually see Tommy as a Rocky enemy, given the strong connection built between them. Still, everything changes when a boxing promoter named George Washington Duke (played by Richard Gant) sees in Tommy a way to make lots of money. Duke starts to shower Tommy with luxuries and promises that he will be the champion that he wants to be. But obviously, Tommy can’t see more than meets the eye because he is too young and doesn’t have Rocky’s experience.
I think that also applies to Pulsar and Kafka’s relationship. Both Apache projects are equally good, though obviously, Kafka has way more experience, hence why it is considered more battle-proven than Pulsar. But they are not enemies, or at least they shouldn’t be. However, companies are trying to profit from Pulsar, and they know that Pulsar will never be number one while Kafka is around. The reason why in the last 12 months, there has been a lot of paid content trying to get a fight between Kafka and Pulsar with claims that Pulsar is way better than Kafka.
Back to the movie, there is a point where Tommy decides to leave Rocky and embracing Duke as his manager. The movie then shows Tommy winning the current champion Union Cane in the first round almost effortless — like if Cane was not in the same level as Tommy and leaving everybody thinking that he might not be as good as Rocky because he never fought real strong adversaries as Rocky did.
Tommy is then pressed by the media that repeats that he will never be the real champion unless he fights a worthy opponent such as Rocky; they drive the point home when one reporter announces, “…A Rocky Balboa he’ll never be!”. They insist that Cane was nothing but a “paper champion” since Cane did not win the title from Rocky in the first place. Lastly, people start calling Tommy “Baby Rocky,” “Rocky’s Robot,” and “Rocky’s Shadow.” All of that pushes Tommy to go out and challenge Rocky for a fight.
It is incredible how this is exactly what is going on between Kafka and Pulsar. Pulsar seems to insistently try to beat Kafka to prove itself and show to the world that it is not a second-rate technology. This is fine because it is normal for a technology that creates a trend like Kafka did; others will always try to do better. Take virtualization and containerization, for example.
Tommy starts a fight with Rocky that quickly turns into chaos because the fight takes place in the streets of Philadelphia instead of the ring. Punches are thrown, kicks are delivered, and the fight progresses to multiple rounds where Rocky and Tommy are unwilling to give up. Until the moment where Rocky is almost beat down but starts to remember the wars that he had fought, the even more challenging enemies that he had defeated, and ultimately — the words of Mickey saying, “get up, you son of a bitch, ’cause Mickey loves ya.”
This is the part of the movie where he regains his strength and embraces the fight he was trying to avoid as much as possible. Rocky then goes savagely onto Tommy that at a certain point in time, is unable to defend Rocky’s punches and attacks. Rocky finally defeats his former trainee that at this point, he can’t avoid but accept that Rocky is… well, Rocky is Rocky!
I really don’t know if Pulsar would ever beat Kafka, but I am inclined to believe that, at least for now, this is not possible. Pulsar has some interesting features, but Kafka is Kafka. Kafka has more experience, it is humble enough to keep evolving, it can be super fast when it needs to be, and its ecosystem is very mature. All these things, I think Pulsar will get as well given time. Just not now, IMO.
What happened with Adrian/Zookeeper?
After Rocky V, there were three more movies in the series, Rocky Balboa, Creed, and Creed II respectively. All three of them are great movies, where Rocky get to be present again, but it’s hard to avoid not noticing that one significant character had gone: Adrian.
It turns out that Adrian passed away after being diagnosed with ovarian cancer between the events of Rocky V and Rocky Balboa after being diagnosed with ovarian cancer, leaving Rocky widower. That is dramatically portrayed in the sixths installment of the series with Rocky visiting her grave. Adrian was more than a wife to Rocky — she was his mountain. There were many occasions where Rocky was about to crumble before Adrian have him rescued, and she was always pivotal in putting him on his feet again.
In the Kafka world, we have Zookeeper, which plays an important role in keeping Kafka on its feet and serving its clients. In the past, clients would literally connect to Zookeeper to fetch information about the cluster, which corroborates the fact that Zookeeper played an important role in Kafka’s world. But then the KIP-500 was approved, which means that Zookeeper will be removed from Kafka in replacement for a self-managed metadata quorum. That means that Zookeeper will be dead for Kafka, just like Adrian for Rocky.
But just because Adrian is gone doesn’t mean that she will be ever forgotten. Rocky keeps the best of her in his heart, and everything he learned from her will be kept forever. The same goes for Zookeeper. Kafka now can do autonomously everything that it once needed Zookeeper for, and Kafka will keep the best of Zookeeper on his heart, which technically is its clustering protocol.
Apache Kafka is a Fighter!
I am sure that we can all agree that Kafka is a fighter, just like Rocky. It keeps being pushed, being challenged, being tested, being put to prove, and just like Rocky, it always finds a way to win. It is never easy, but Kafka wins.
Yo Adrian, we did it!
Thanks for reading this blog, though, and I hope you had some fun while reading it. To get aware of more content like this, follow me on Twitter: @riferrei.