, , ,

Databricks Zerobus Streaming Ingestion for Delta Lake House

Yeah, so … I’ve heard rumbling and mumblings about, here and there. But I had yet to try it out for myself. I trust nothing I can’t put my hands on. Something about being raised in the cornfields of the Midwest, always be skeptical of anything that seems like Black Magic.

Zerobus in Black Magic.

Look, let me break it down for ya’ll real easy like.

  • Streaming data ingestion has long been hard, complex, and expensive.
  • Everyone (SaaS) wants to make streaming more approachable and easy.
  • Integration: Streaming data into Lake House architecture has not been straightforward.
  • Databricks solved and removed the complexity of Streaming + Lake House.

I don’t care if you are a Databricks fan or not; anyone who is serious has to admit that they make incredible products, have brought Spark to the masses, have broken down all AI barriers, and are generally able to make breakthroughs in tech. Zerobus is that breakthrough for streaming.

To top it off, when I tried it out, I was able to use Apache Arrow RecordBatches to stream data into a Delta Lake table.

Databricks provides several SDKs for Zerobus.

I used the Python SDK above, as you saw in the opening code.

What’s the big deal?

Well, did you see how much code it took to create a Zerobus streaming client that writes directly to a Delta Lake table? That’s crazy. Sure, it was just for fun and playing, but there isn’t a whole lot to it. The Zerobus architecture from Dataricks removes all the complexity of Kafka, Flink, Spark Streaming, and the like.

  • Databricks knows that streaming to the Lakehouse is an enterprise requirement, and many orgs will balk at running custom, bespoke setups to push streaming data into the Lakehouse.

This is the age we have been living in, the abstraction age. Zerobus has accomplished what many folk have tried to do. We’ve had many tools coming along, like Flink and Arroyo that promised to simplify stream processing, and they did. But Databricks, like always, went the extra step: not only did Zerobus make streaming data easy, but the first-class integration into the Lake House was the missing ingredient.

The fact that I could use Apache Arrow + Zerobus made my toes tingle.

Here’s to the future my friends.