How to hire a data engineer

Key takeaways

• Scope against the pipeline problem: greenfield build, legacy migration, warehouse optimization, or AI data infrastructure. Each is a different engineering problem.
• Data engineers are evaluated best on how they think about pipeline failures and recovery, not on which orchestration framework they know.
• Three data engineer subtypes: pipeline builder (ETL/ELT at scale), warehouse engineer (query optimization and data modeling), and ML data engineer (feature engineering and model data pipelines for AI systems).
• First 30 days: a pipeline audit with failure modes documented, a first new pipeline or pipeline improvement shipped by week two.
• Most common failure: hiring a warehouse engineer to build a new pipeline, or hiring a pipeline builder to optimize an existing warehouse.

Why this question matters

Data engineering is one of the engineering disciplines with the widest gap between the title and the actual work. Two people with "data engineer" on their resume may have built completely different systems: one optimizing SQL query performance in BigQuery, one designing real-time Kafka consumers for event-driven pipelines. Getting the specific scope clear before the search starts is what produces a useful shortlist.

The decision frame: Pipeline problem first, tools second

Three questions before writing the JD.

What is the data pipeline problem? Is data not flowing at all (build problem), flowing but breaking often (reliability problem), flowing but too slowly for the analytical use cases (performance problem), or flowing correctly for existing use cases but not set up to support AI feature development (ML data infrastructure problem)?

What's the scale? Orders of magnitude matter: a pipeline handling thousands of events per day and one handling millions per hour require different engineering approaches, even if both use the same tool names in the JD.

What does "reliable" mean in this context? How often are data pipeline failures acceptable? Is a one-hour outage in the analytics pipeline a P0 incident or a minor inconvenience? The reliability requirement shapes the engineering approach and the profile of engineer who's built for it.

Scoping the role

Data engineer engagements fall into three subtypes.

Pipeline builder (ETL/ELT). The engineer builds the data pipelines that move data from source systems (product databases, third-party APIs, event streams) to the analytical layer (warehouse, data lake, or feature store). The work involves source connectors, transformation logic, scheduling and orchestration, and error handling. This is the broadest data engineering profile, the default when someone says "data engineer."

Warehouse engineer. The engineer focuses on the analytical layer: data modeling, query optimization, schema design, and the data products that analysts and stakeholders consume. They're more SQL-intensive and closer to the analytics consumer than to the raw data source. If the pipeline is working but reports are slow and the data model is a mess, this is the profile you need.

ML data engineer. The engineer builds the data infrastructure that AI and ML systems need: feature stores, training data pipelines, model evaluation datasets, and the monitoring that ensures the data feeding AI systems is clean. This profile requires understanding both data engineering and the requirements of ML training and inference, a rarer combination that commands a premium.

Evaluating a senior data engineer

The wrong evaluation is a SQL performance quiz in isolation. Knowing how to write an efficient window function is useful; knowing when a performance problem is a query problem versus a data model problem versus an infrastructure problem is what matters at the senior level.

Pipeline failure walkthrough. Ask the candidate to describe a production data pipeline they owned that had a significant failure. Not a bug they caught in development, something that went wrong in production, lost data or delivered incorrect data, and required diagnosis and remediation. Ask: how did they detect it, how did they diagnose it, what did they fix, and what did they change in the design to prevent it from happening again? The post-mortem instinct is the most predictive signal in data engineering evaluation.

Data model design problem. Give the candidate a business scenario, a product with users, events, and transactions, and ask them to design a data model for analytical queries. Watch for: do they ask about the queries before designing the model? Do they think about slowly changing dimensions? Do they account for the update patterns of the source data? The design reveals data modeling judgment that's hard to fake.

Orchestration trade-off conversation. Ask about a decision they've made about orchestration or scheduling, Airflow, Prefect, dbt, or another tool. Not which one they've used, but what trade-offs they considered when choosing it. Do they understand the operational overhead of different tools? Do they reason about the failure modes of different scheduling approaches? Senior data engineers have opinions on orchestration that they can defend.

The first 30 days

Week one: pipeline audit. The data engineer should have read access to the source systems, the orchestration setup, and the warehouse from day one. Week one is spent understanding what's there: what pipelines exist, what their failure rate is, what the data freshness SLAs are, and where the highest-risk points in the pipeline graph are. End of week one: a written summary of the pipeline landscape and the top three risks.

Week two: first improvement shipped. A pipeline fix, a new connector, a monitoring alert added to an unmonitored pipeline. Something observable and measurable. The goal is to prove the deployment path from development to production.

Week three: first new pipeline or schema change. A new data source connected, a schema migration completed, or a new analytical model added. Something with defined scope and a test that confirms the data is correct before it reaches downstream consumers.

Week four: data quality review. Sit with the data engineer and review data quality metrics across the key pipelines. What's the current error rate? What's the data freshness across the critical paths? What's the one pipeline most likely to fail next? A 30-day data quality review surfaces the systemic issues that the pipeline-by-pipeline view misses.

Common failure patterns

Two patterns account for most data engineer mis-hires.

The scope was "help us get our data in order" without specifying the order of the problem. One team hired a warehouse engineer when they needed a pipeline builder: the data wasn't flowing at all, but the hire spent the first month optimizing SQL queries. Specify the problem, not the job title.

The data engineer was hired before the source systems were stable. A data engineer building pipelines from a product database that changes schema every week is constantly building against a moving target. If the product engineering team is in active refactoring of the systems the data engineer will consume, delay the data engineering hire until the schemas stabilize, or scope the hire explicitly around schema management.

What to do next

Write the pipeline problem in two sentences before writing the JD. The problem is not "we need a data engineer", it's "our analytics pipeline fails twice a week and analysts can't trust the numbers in their dashboards." That problem statement produces a shortlist of engineers with reliability-focused instincts. The JD follows from the problem.