Skip to content

DataNotesData engineering

Why Data Engineering is Booming in 2025

Baraa Khatib Salkini5 min read

Subscribe to DataNotes

Join 213,000+ curious readers.

Subscribe takes you to Substack to confirm, and you can unsubscribe anytime (privacy policy).

Subscribe to DataNotes

Subscribe takes you to Substack to confirm, and you can unsubscribe anytime (privacy policy).

In this issue5 min left

Hey friends!👋

Oh yeah… it’s finally our moment. For years data engineers were the ones working quietly in the background. Nobody really noticed us.

But not anymore. Now everyone wants AI, and guess what they discovered? Without data engineers, it all falls apart.

To understand why let me tell this little story

A few weeks ago, I ran a little experiment. I took one AI model and fed it two different versions of the same dataset.

  • First, I gave it the raw dump. Hundreds of tables, no clear structure, messy formats, duplicates everywhere, missing values, and no metadata to explain what anything meant. Total chaos.
  • Then I gave it a second version of the same data. This time it was properly modeled, cleaned, and described. Instead of hundreds of random tables, I reduced it to a handful of well-structured, integrated ones that actually made sense together.

The difference? Night and day.

With the raw dump, the AI spit out answers that looked “okay” at first glance… but dig deeper and they were wrong, inconsistent, and sometimes completely hallucinated.

With the clean, structured dataset, the AI suddenly became sharp. Its answers were clear, consistent, and actually useful for real decision-making.

That little test reminded me of a simple truth I’ve seen my whole career:

👉 AI is only as smart as the data you give it.

And that, my friend, is exactly why Data Engineering is booming right now.

But here is the thing. This did not happen overnight. The role of the data engineer itself had to evolve to reach this point. I have been working with data since 2010, and I have lived through each stage of this evolution.

Let me take you through that journey.

How We Evolved as Data Engineers

Before 2015: The ETL Era

If you rewind 10 or even 20 years, nobody called us data engineers. The title back then was ETL Developer.

Our job was pretty straightforward on paper: move data from source systems into a data warehouse, usually overnight in big batch jobs. We built models for BI tools using platforms like SSIS, Talend, Informatica, or DataStage. Everything was locked into one tool.

We were the hidden heroes in the background. Analysts and BI developers gave us requirements, and we delivered pipelines. That was the cycle. Nothing fancy.

2015 to 2020: The Big Data and Cloud Shift

Then came the cloud, and suddenly the ground shifted under our feet. Companies started moving away from heavy on-prem ETL tools and jumped into cloud and big data platforms like Hadoop, Spark, AWS, and Azure.

That’s when the job title started changing. ETL Developer became Data Engineer, because the role had grown beyond just extract, transform, load.

We also got a new customer inside companies: the data scientist. They didn’t want pre-modeled BI tables. They needed raw, large-scale data for training machine learning models.

To support them, we built data lakes. Huge storage areas where you could dump everything: logs, files, raw tables. At first it felt revolutionary. But without structure and governance, most of these lakes quickly turned into data swamps. The data was there, but messy, inconsistent, and almost impossible to use effectively.

2020 to 2023: The Lakehouse and Data Mesh Era

After that wave, companies started realizing that just collecting data wasn’t enough. They needed an actual data strategy.

This is when the idea of the data lakehouse took off. It brought together the trusted structure of a data warehouse with the flexibility and scale of a data lake.

At the same time, people started talking about the data mesh. The idea was simple but powerful: treat data like a product. Document it, own it, and share it across teams.

For us as data engineers, the role expanded again. We weren’t just moving data anymore. We were now building data products: clean, documented, reliable datasets that analysts, data scientists, and even AI systems could use directly.

2023 to Today: The AI Era

Now we’ve entered the age of AI. It’s the hottest topic everywhere. But here’s the reality:Without data engineers, AI does not work.

Companies are finally waking up to the fact that data is one of their most valuable assets. The quality of their AI depends entirely on the quality of their data.

Our role today is not just to feed dashboards or help analysts. Our job is to build the AI-ready data layer. Clean, structured, governed, and productized.

And here’s the truth: if a company jumps straight into AI tools without a strong data strategy, they will fail. But if they start with strategy, bring in strong data engineering, and build clean products, then everything clicks together. AI, BI, analytics, and decision-making all work like a charm.

The Big Picture

That is the journey. From ETL developers running nightly jobs, to cloud and big data engineers, to builders of data products, and now the foundation of the AI era.

At the end of the day, we as data engineers are the ones implementing the company’s data strategy and building the data layer that everything else depends on.

And once that layer is in place, you can do almost anything with data:

  • Build BI dashboards and reports
  • Run advanced analytics and forecasting
  • Power machine learning and generative AI
  • Expose data through APIs so other apps and services can use it
  • Even support operational systems that rely on real-time, trusted data

If the data layer is weak, none of this works. If the data layer is strong, everything from BI to AI to live operations works like a charm.

That is the real impact of data engineering today.

And honestly, it feels good to finally see the world notice what we have been building all along.

Thanks for reading ❤️

Baraa