Databricks Zerobus: Event Streams + The Lakehouse
I had not been thinking much about Kafka lately, but depending on who you ask, Kafka is either sitting comfortably at the top of the streaming world or beginning a slow decline into abstraction. The truth is probably somewhere in the middle. I’ve seen a number of newish streaming tools, many Rust-based, come in like a splash and leave nothing but a few bubbles.
Kafka is entrenched in the streaming world, and this has created a problem for the Lake House architecture. The integration isn’t “as easy as pie,” … far from it.
We live in an era where complexity is increasingly viewed as a liability. Businesses are less willing to hire teams of platform engineers whose primary responsibility is keeping complicated infrastructure alive, patched, tuned, upgraded, and operating correctly. Everyone wants the benefits of streaming systems. Fewer people want the operational burden that comes with them.
- That reality creates opportunity.
Every few years, a vendor looks at an established technology and says, “Surely there has to be an easier way.” Sometimes they are right. Sometimes they are not. Either way, the cycle repeats itself. Databricks, for one, has always been at the bleeding edge of turning products and tech on their heads. Like em’ or not, that’s just the way they roll.
And now, they’ve done it with streaming.
We have seen it happen in data processing. Spark spent years dominating the conversation before tools like DuckDB, Polars, and Daft began to gain traction by focusing on simplicity and developer experience. They did not kill Spark. They simply started eating around the edges.
The same thing appears to be happening in the streaming world.
Then I heard a few words that immediately got my attention.
- Databricks.
- Streaming.
- gRPC.
- Delta Lake.
That combination was enough to make me curious. 
Databricks recently introduced Zerobus, a new approach to streaming data directly into a lakehouse, and on paper, it sounds almost suspiciously simple. Naturally, that made me want to try it for myself. The goal of this article is simple. I want to explore what Zerobus is, why it exists, how it compares to traditional streaming technologies like Kafka, and what the actual experience looks like when you sit down and try to use it.
Because the truth about technology rarely lives in the marketing materials. It usually lives somewhere between the documentation and the first error message.
Why Zerobus Exists
Before we talk about Zerobus specifically, we need to talk about the state of streaming.
Streaming and near-real-time data processing have become mainstream. Organizations increasingly want fresh data, operational analytics, event-driven architectures, and AI systems that react immediately to changing information. At the same time, companies are under pressure to reduce infrastructure costs and simplify operations.
- That creates an interesting tension.
Kafka is not going away. Anyone claiming otherwise is probably selling something. Kafka has become deeply embedded in thousands of organizations and powers some of the largest data systems in existence. But that does not mean there is no room for alternatives.
The fact that Databricks invested significant engineering effort into building Zerobus tells us something important. Streaming data into a lakehouse remains harder than many organizations would like it to be.
Today, most modern data architectures revolve around the lakehouse. Whether you are using Delta Lake, Apache Iceberg, or some combination of open table formats, the lakehouse has become the central storage layer for analytics, machine learning, and increasingly operational workloads.
The challenge is that getting streaming data into that lakehouse often requires a fairly substantial collection of moving parts.
- Kafka clusters.
- Streaming engines.
- Connectors.
- Monitoring.
- Scaling.
- Maintenance.
- Security.
- Networking.
And everything else that inevitably appears once a system reaches production. Zerobus attempts to remove much of that complexity.
What Is Zerobus?
At a high level, Zerobus provides a serverless mechanism for pushing streaming data directly into a Delta table.
Instead of building and maintaining a traditional streaming platform, applications send data directly to Databricks through APIs built around technologies such as gRPC and Apache Arrow Flight. The design philosophy appears to be straightforward. Push data, land it in Delta, avoid the infrastructure.
For organizations that already consider the lakehouse their primary data platform, that is an appealing proposition. Rather than introducing additional systems to manage and operate, Zerobus aims to make the lakehouse itself the destination for streaming workloads.
Whether that approach works at scale remains a larger discussion, but conceptually, it is easy to understand why customers might find it attractive.
- Less infrastructure.
- Fewer moving pieces.
- A more direct path from the event producer to the Delta table.
Time To Break Something
Rather than spend the entire article discussing architectural diagrams and product positioning, I wanted to actually use Zerobus. The setup process is surprisingly straightforward.
- You need a Databricks workspace.
- You need a target Delta table.
- You need authentication.
- You need an SDK.
And then you can start sending data. The Zerobus endpoint itself is derived from your Databricks workspace information. Once authentication is configured using a service principal and OAuth credentials, you have everything you need to begin writing code.
For the destination table, I used the Divvy bike trips dataset because it is public, familiar, and easy to work with.
The interesting part comes when you start writing the client.
Since Zerobus supports Apache Arrow, I naturally chose Arrow as the transport format. Arrow continues to show up everywhere in modern data systems, and seeing native support here was encouraging.
After defining the Arrow schema and building the client code, the overall workflow looked something like this:
- Create a stream.
- Read source data.
- Convert data into Arrow batches.
- Push those batches to Zerobus.
The Reality Of Trying It
As anyone who has spent time in data engineering already knows, software demonstrations rarely work the first time.
Mine certainly did not. The first surprise came when attempting to run the code from a Databricks Serverless environment. I was unable to reach the Zerobus endpoint from Serverless compute, and I could not find any documentation explicitly calling out this limitation.
Switching to an all-purpose cluster immediately got me farther.
That was a good sign. The next error involved schema compatibility between my Arrow data and the destination table.
- Also, a good sign.
At least now the SDK was communicating with the endpoint. The Zerobus client successfully established an Arrow Flight connection, which meant the plumbing was working. After a few rounds of experimentation, some schema adjustments, and a little Arrow casting magic, everything finally clicked.
Data started flowing. Approximately 140,000 records were successfully streamed into the Delta table. The moment of truth arrived when I queried the table and confirmed that the records had landed exactly where they were supposed to be.
There is something deeply satisfying about seeing a system finally work after spending an hour fighting with authentication, schemas, permissions, and mysterious error messages.
What About The Files?
One thing immediately came to mind after the initial success. Files. Lots and lots of files.
Streaming systems naturally create smaller batches of data, and lakehouses are notoriously sensitive to file management. Anyone who has spent time working with Delta Lake or Iceberg knows that file sizes matter. After my test run, the Delta table contained multiple parquet files generated from the ingestion process.
Nothing surprising there.
Running OPTIMIZE consolidated those files into a much healthier state, significantly reducing the number of underlying parquet files.
For a toy example, this is not a concern. For production workloads involving hundreds of terabytes or petabytes of data, it becomes a very important consideration. Any organization adopting direct-to-lakehouse streaming needs to think carefully about compaction strategies, optimization schedules, clustering approaches, and long-term storage maintenance.
The complexity may have moved, but it has not completely disappeared.
Final Thoughts
After spending some time experimenting with Zerobus, I think a few things are becoming increasingly clear. First, streaming continues to grow in popularity. Second, the lakehouse has become the preferred architecture for a large portion of the modern data industry.
Third, streaming remains one of the most operationally complex areas of data engineering. Those three realities create a market opportunity for products that reduce friction and lower barriers to entry.
- That is exactly where Zerobus seems to fit.
The experience was surprisingly straightforward once the initial setup hurdles were cleared. There was no Kafka cluster to deploy. No connectors to manage. No streaming infrastructure to tune. The SDK experience was clean, and integration with Delta Lake felt natural.
Now, does that mean Kafka is doomed?
Of course not.
Kafka will continue powering enormous data platforms for years to come. But for organizations whose primary goal is simply to get event data into a lakehouse quickly and efficiently, Zerobus offers an interesting alternative. The most impressive part may be how aggressively it simplifies the architecture. What previously required multiple systems, multiple teams, and significant operational expertise can now be accomplished with a relatively small amount of code.
That is not something to ignore.
It is still early, and there are plenty of questions around scale, file management, performance characteristics, and operational best practices, but after spending some time with it, I can confidently say that Zerobus deserves serious attention. And the fact that it speaks Apache Arrow certainly does not hurt.
If you are already streaming data into a lakehouse, I would be interested in hearing about your setup and whether you see products like Zerobus as the future or simply another tool in the toolbox.
