The Real Value of Data Analysts in the AI Era

I often see claims that data analytics and data science are on their way out, soon to be replaced by AI. Some people love demonstrating how they can hand AI a CSV file and have it produce an entire analysis and report, as if that were all the job involved.

But anyone who has worked in analytics in a complex business environment, especially at a large internet company, knows there is much more to it. Where does the input come from? Why choose those dimensions and metrics? What business problem is the analysis supposed to solve? And how will the recommendations actually be implemented?

The work these demos skip is precisely what tests an analyst's ability.

Where data analysts really add value

First, define the analytical framework and quantify its dimensions and metrics

Analytics projects at school often start with a dataset: inspect the available fields, explore them, and try to let conclusions emerge naturally. In practice, however, analysis requires deductive reasoning guided by business hypotheses, rather than slicing the data first and searching for explanations afterward. That means talking with business stakeholders to understand the context, existing strategies, and the problem they actually need to solve.

Take a content app where the business asks: “How can we improve retention?”

Without a clear hypothesis, it is easy to start breaking down retention by gender, age, and region. If nothing stands out, try acquisition channels, entry points, or new versus existing users. If that still fails, combine several dimensions. You can end up doing a lot of analysis without knowing what to change.

After discussing the problem with stakeholders, you can instead build a framework around how users consume content:

Content distribution: Does the current distribution strategy meet users' needs? Do recommendations match their content preferences? If not, consider adjusting the distribution strategy.
Content supply: Are users leaving because the content they want is unavailable? If so, consider acquiring that content.
Feature usability: Are users running into obstacles, such as being unable to find something they left unfinished? If so, consider improving the relevant features.

These hypotheses need to be tested against data, which may disprove them. But a framework gives the analysis direction and connects it to potential business actions. Thorough communication also saves analysts a great deal of aimless querying and exploration.

Once the framework is clear, it needs to be quantified by choosing the right populations, metrics, and time periods:

Population: Who are we analyzing, and who are we comparing them with? Select groups based on the hypothesis, such as new versus existing users or users with different content preferences, and establish a meaningful benchmark. An app's next-day retention of 60% this month tells us little on its own. Even if last month's figure was 40%, we still need to check whether acquisition sources and measurement definitions are comparable.
Metrics: How do we measure a business concept? How do we tell whether users “like this type of content”? Like and save rates are possible measures, alongside reading completion and time spent. But using exposed users versus readers as the denominator produces different meanings. The definition needs to match the question.
Time period: How long should we observe, and which period should we compare against? For a large content app with frequent usage, a full week can be a starting point for examining routine consumption, covering both weekdays and weekends while accounting for holidays and campaigns. Long-term retention requires a longer observation window, and comparison periods should have similar business conditions where possible.

These choices reveal an analyst's business experience and quantitative judgment. It also pays to think one step further: if users who consume a certain type of content have higher retention, does that content improve retention, or are already-active users simply more likely to consume it?

Second, navigate the joins and querying logic of real business data

The CSV in a demo or the tables in a Kaggle project come with fields, definitions, and data granularity already organized. But in a real business, user profiles, event logs, content information, and experiment assignments are often spread across different tables that require complex joins before analysis can begin.

Choosing the wrong field or join condition can affect the final result. For example, joining a user table to an event table may produce multiple rows per user. Without handling that correctly, the query can count users more than once. The SQL runs, and the numbers look plausible, but the logic is wrong.

Even an initial assessment of the data requires decisions about how to define the population, when measurement should begin, and how to connect group assignments, exposures, and subsequent behavior. This can involve several tables, sometimes more than a dozen. The challenge includes verifying that every step retrieves the data you actually intended.

A warehouse may also contain many historical intermediate tables. Two tables may both refer to “active users,” while one counts app launches and the other requires meaningful content consumption. Similar-looking fields can mean very different things. Understanding those differences requires documentation, business context, and historical knowledge.

Giving AI all the raw tables and asking it to write SQL does not automatically solve these problems. AI can speed up coding, but analysts still need to review definitions, joins, and results. You need to understand the data yourself to judge whether it has been retrieved correctly.

Third, work across teams to turn findings into measurable business impact

Finishing a report and offering a few recommendations is usually an intermediate step.

Returning to the earlier example, suppose the analysis suggests improving the “Continue reading” entry point because users cannot find where they left off. Someone still needs to discuss the solution with product, coordinate engineering schedules, fill gaps in event tracking, and design an experiment to evaluate the change.

Teams have competing priorities, and engineering resources are limited. Explaining the evidence, securing resources, resolving disagreements, and getting the change shipped all test an analyst's broader capabilities.

Once the change is live, a rise in retention alone does not prove it worked. An A/B test, or causal inference methods where appropriate, is needed to determine whether the change caused the improvement and to assess its benefits and effects on other metrics.

The full value of analytics spans this entire process, from identifying a problem to validating the business outcome.

How should analysts use AI at work?

The fact that this work requires experience and judgment does not mean analysts should keep doing everything the old way. I see two levels at which to start.

The most immediate use is improving everyday efficiency

Once table structures and metric definitions are clear, AI can draft SQL, help debug code, generate data processing and visualization scripts, and organize reports. These tasks involve plenty of repetitive work that AI can assist with, while analysts review the key logic and results.

Going further, bring AI into parts of the team's business workflows

An agent can participate in a more complete stretch of the analytical process.

If a company has its own agent, it can use MCP or CLI integrations to connect to internal platforms and retrieve requirements documents, chat context, warehouse data, and experiment results. For example:

Gather business context: Read requirements documents and relevant conversations to summarize the analytical objective, existing strategies, and open questions.
Retrieve analytical data: Find the relevant warehouse tables, generate and execute queries using agreed definitions, and obtain experiment configurations and results from the experimentation platform.
Run the analytical workflow: Apply methods documented by the team to validate data, break down metrics, and conduct an initial analysis, then produce a draft with data sources and items that still need verification.

Making this workflow reliable requires documenting the team's business knowledge and analytical methods.

One part is turning mature analytical workflows into standard operating procedures, then packaging them as skills an agent can call. Examples include investigating unexpected metric changes and segmenting users.

The other part is documenting the tables, fields, and metric definitions commonly used in that business area: each table's granularity, join keys, exclusion rules, how retention and meaningful consumption are calculated, and which historical tables should no longer be used. Validated SQL queries can be saved as reference examples as well.

This gives the agent a clear basis for querying and analysis, reducing the need to guess field meanings and the risk of choosing the wrong tables or mixing definitions. Errors found during review can feed back into the documentation and skills, making subsequent use more reliable.

I believe AI will replace some repetitive querying, table preparation, and report compilation, while amplifying the capabilities of experienced analysts. Analysts can actively turn their experience into knowledge and workflows AI can use, freeing more time for understanding the business, testing hypotheses, and putting strategies into practice.

But the fundamentals still matter. You need to understand how to do the analysis yourself to guide AI, check its work, and take responsibility for the final conclusions.