Skip to content

72% of data teams prioritize AI coding. Just 24% prioritize AI for pipeline quality

Mykhailo PoliarushUpdated 5 min read

In 2026, data platform engineering is defined by a gap between speed and trust. AI is now part of daily work in data teams, but investment in checking what it produces has not kept up: in the dbt Labs 2026 State of Analytics Engineering survey (dbt Labs is a data tooling vendor), 72% of respondents prioritize AI-assisted coding, while only 24% prioritize AI-assisted pipeline management such as testing, observability and quality controls. This article collects verified data on where data platforms stand, what holds data teams back, and what good practice looks like.

What does a modern data platform look like in 2026?

The lakehouse, which keeps data in open files on low-cost object storage and adds warehouse-style tables and governance on top, is where most large companies say they are heading. In Dremio's State of the Data Lakehouse in the AI Era (January 2025, 563 IT decision-makers), 55% of organizations ran most of their analytics on a lakehouse, and 67% planned to within three years. In Dremio's 2026 edition, a survey of 101 data leaders, 87% expect the lakehouse to be their primary data architecture by 2027. Dremio sells a lakehouse platform, so read these as one vendor's samples.

Practitioner data on one building block is more cautious. Apache Iceberg is one of several open table formats used to build a lakehouse, so its numbers are not a lakehouse adoption rate, but in the dbt Labs 2026 survey of 363 data practitioners and leaders, only 9% use Iceberg in production and 68% have no current plans for it. The most cited barriers in the same survey were knowledge gaps (27%) and unclear use cases (27%).

Apache Iceberg adoption among data teams, 2026

In production
9%
In proof of concept
12%
Planning adoption
6%
No current plans
68%

Source: dbt Labs (vendor), 2026 State of Analytics Engineering (n=363, Dec 2025 to Feb 2026)

The two pictures fit together. Leaders plan to consolidate, while the teams that build pipelines adopt new formats one use case at a time. Neither sample is the whole market: Dremio surveys buyers of data infrastructure, and dbt Labs surveys practitioners, many of them close to its own community.

How is AI changing data engineering work?

Quickly. In the dbt Labs 2025 State of Analytics Engineering (459 respondents, surveyed in late 2024), 80% of respondents used AI in their daily workflow, up from 30% a year earlier. By the 2026 survey, 72% prioritize AI-assisted coding in their development process.

What has not changed is where the hours go. In the 2025 report, 57% of respondents said they spend most of their workday maintaining or organizing data sets, nearly the same as the year before, and the 2026 report still finds that a majority spend most of their time this way. AI makes new models and pipelines quicker to write. It does not yet remove the maintenance that comes after.

What are the main challenges data teams face?

Data quality. Poor data quality was the most reported challenge in the dbt Labs 2025 survey, cited by over 56% of respondents, and it was again the most frequently reported obstacle in 2026.

Wrong answers reaching people. In the dbt Labs 2026 survey, 71% are concerned about hallucinated or incorrect data reaching stakeholders. When an AI assistant or agent answers questions from company data, a broken pipeline no longer just breaks a dashboard; it produces a confident, wrong answer.

Ownership and governance. Ambiguous data ownership is a challenge for 41% of dbt Labs respondents, unchanged year over year. In Dremio's 2026 survey (vendor data, 101 respondents), 70% said siloed data and weak governance are the main obstacles to getting value from AI, and 40% pointed to poor data quality and missing semantic definitions.

Cost. In the dbt Labs 2026 survey, 57% report higher warehouse and compute spend, while only 36% report higher team budgets. Spending on platforms is rising faster than the budgets of the teams that run them.

Some problems are easing. In the same survey, the share citing integration of data from various sources as a top challenge fell from 35% to 27%, and lack of stakeholder trust as a primary challenge fell from 33% to 24%, while the share calling trust in data an important priority rose from 66% to 83%.

What changed for data teams from 2025 to 2026

What changed for data teams from 2025 to 2026
CriterionShare of respondents20252026
Say increasing trust in data is important66%83%
Say shipping data products faster is important50%71%
Say cost reduction is important48%53%
Cite integrating data from various sources as a top challenge35%27%
Cite lack of trust in data from stakeholders as a primary challenge33%24%
Cite data literacy among stakeholders as a barrier39%36%

Source: dbt Labs (vendor), 2026 State of Analytics Engineering (n=363, Dec 2025 to Feb 2026)

Why does AI-ready data decide whether AI projects ship?

Because a model can only be as right as the data it reads. Gartner predicts that through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data. That is a forecast, not a measurement, but it matches what data teams report about their own priorities.

Budgets have started to follow. In the dbt Labs 2025 survey, 30% of respondents reported budget increases for their data teams, against 9% a year earlier, and 40% reported headcount increases, against 14%. AI tooling was the area where the largest share of respondents (45%) planned to increase investment, followed by data quality and observability (38%), according to the same report.

AI-ready data is not a product to buy. It means data that is tested, documented, owned and described with shared business definitions, so that a person and an AI system read a metric the same way.

What does good data platform engineering look like?

The teams that close the gap between speed and trust tend to do the same few things:

  • Test data like code. Schema checks, freshness checks and business rules run in CI on every change, and pipelines fail loudly instead of passing bad rows downstream.
  • Name an owner for every data set. Each table that feeds a report, a model or an agent has a team accountable for it and an agreed contract with its consumers.
  • Define metrics once. A semantic layer or shared metric definitions stop "revenue" from meaning three different things across dashboards and AI tools.
  • Watch cost as a metric. Track compute spend per pipeline and per consumer, and remove duplicate copies before buying more capacity.
  • Adopt open formats for a reason. Move to a lakehouse or an open table format where it solves a real problem, such as several engines reading the same data, not because a roadmap says so.
  • Review AI-written pipelines. Let AI draft transformations and documentation, and keep engineers reviewing the logic and the tests before anything reaches production.

A practical first step is an audit of the ten data sets your business and your AI use cases depend on most: who owns them, how they are tested, and what they cost to run.

How can SDCLabs help with your data platform?

SDCLabs builds and runs data platforms the way we build software: AI agents draft specs, pipeline code and tests, senior engineers review and decide what ships, and automated tests check every change. We are the founders of the open-source testing frameworks Codeception and CodeceptJS, so testing is where we start, not where we finish.

If your data team ships faster than it can verify, see our data platform engineering services or tell us about your platform. For the AI side of the same problem, read about our AI-ready data engine and our report on the state of AI engineering in 2026.

Sources checked on 2026-11-17.

Frequently asked questions

What is the biggest challenge for data teams in 2026?

Poor data quality. It was the most reported challenge in the dbt Labs 2025 survey, cited by over 56% of respondents, and again the most frequent obstacle in 2026.

Are companies moving to data lakehouses?

Leaders say yes: in Dremio's January 2025 survey, 55% ran most analytics on a lakehouse and 67% planned to within three years. Practitioner data on one building block is more cautious: only 9% of dbt Labs 2026 respondents use Apache Iceberg, one open table format, in production.

Why does AI-ready data matter?

AI answers are only as good as the data behind them. Gartner predicts that through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data.

Where should a data team start improving its platform?

Audit the data sets your business and AI use cases depend on most: name an owner for each, test them in CI on every change, define shared metrics once, and track what each pipeline costs.

Related posts

Get an estimate for your product.