How to evaluate an AI-native engineer

“AI engineer” can describe a product engineer integrating a model API, an applied machine learning engineer improving task performance, or a platform engineer operating training and inference systems. Hiring becomes noisy when the same title is used for different work and every candidate is tested with the same generic questions.

Define the role before the interview

Start with the outcome the person will own in the next six to twelve months. The tools may change during that period. The product responsibility usually changes more slowly.

Some senior engineers span these boundaries. That does not remove the need to prioritize. A startup hiring its first AI product engineer should favor someone who can own the product path end to end. A research-heavy company may accept less frontend breadth in exchange for deep experimentation and model knowledge. A company already shipping several AI features may need platform ownership more than another prototype builder.

The five evaluation dimensions

Score evidence across the complete product lifecycle. A candidate does not need equal depth in every area, but the required strengths should match the role you defined.

1. Product and problem judgment

Can the candidate turn “add AI” into a specific user decision or task? Look for clarifying questions about users, current workflow, acceptable error, human review, and what success would change. Strong candidates narrow the problem before selecting a model.

Ask: “Which part of this workflow should remain deterministic?” A thoughtful answer distinguishes language or perception tasks from business rules, permissions, calculations, and irreversible actions.

2. System design beyond the prompt

Ask the candidate to trace input, context, retrieval, model selection, structured output, validation, tools, stored state, and user feedback. They should explain trust boundaries and recognize that authorization belongs in application code.

Good answers include fallback behavior, timeouts, retries, idempotency, provider limits, data retention, and what should happen when the model produces malformed or uncertain output.

3. Evaluation and experimentation

The candidate should be able to define representative examples and acceptance criteria before endlessly tuning prompts. Depending on the task, that can combine deterministic checks, retrieval metrics, rubric-based review, calibrated model graders, human evaluation, and production signals.

Ask how they would detect a regression after changing a prompt, model, tool schema, or document pipeline. “We will try it manually” may be reasonable for an early spike; it is not a complete production strategy.

4. Security, privacy, and control

Look for practical boundaries: least-privilege tools, server-side permissions, secret management, isolation between users or tenants, validation of model output, limits on external content, and explicit approval for consequential actions.

A candidate does not need to quote a security framework. They should treat the model as a probabilistic component inside an application, rather than the final authority for access or action.

5. Production and cost ownership

Strong engineers discuss latency budgets, caching, context size, provider failure, tracing, deployment, rollback, data migrations, support, and cost per successful task. They connect technical metrics to user experience and business value.

Ask what they would monitor during the first week after launch and what thresholds would trigger rollback, a model switch, or human review.

A practical interview loop

A compact loop can produce better evidence than a long sequence of trivia interviews. Give every interviewer a clear dimension and a shared scorecard.

1. Role calibration
   Hiring manager and technical lead agree on the six-month outcome, required depth, acceptable gaps, and the evidence each stage should collect.
2. Project walkthrough
   The candidate explains one system they personally influenced: the user problem, their decisions, alternatives, failures, measurements, and what they would change now.
3. Product and system design
   Use a realistic Botmer-style workflow rather than a puzzle. Let the candidate ask questions and evolve the architecture as new constraints appear.
4. Small work sample
   Assess a bounded artifact such as an implementation plan, evaluation design, code review, or narrow prototype. Time-box it and avoid unpaid production work.
5. Operating review
   Introduce a failure: rising latency, prompt injection, cross-tenant retrieval, provider outage, or quality regression. Ask how they diagnose, contain, recover, and prevent recurrence.

Use project evidence carefully

Separate personal contribution from team outcome. Ask which decision the candidate owned, what they implemented, how it was reviewed, what evidence changed their mind, and which limitation remained. A clear account of a failed approach can be stronger evidence than a polished demo with vague ownership.

Do not require candidates to reveal confidential prompts, customer data, private code, or employer metrics. They should be able to explain decision patterns and tradeoffs without exposing protected material.

Use a realistic work sample

Choose a task close to the role but small enough to discuss within the interview. For an AI product engineer, a useful prompt is:

Ask the candidate to cover:

• the user journey and where a human remains in control;
• document ingestion, retrieval, permissions, freshness, and citation behavior;
• the model contract, structured output, validation, and fallback;
• an initial evaluation set and the production feedback loop;
• prompt injection and cross-account data risks;
• latency, cost, observability, release, and rollback;
• what to build in four weeks and what to defer.

There is no single correct architecture. Score whether the candidate makes assumptions visible, connects design choices to the task, identifies the highest-consequence failures, and can reduce scope without losing the learning objective.

Allow responsible AI tools

If the role expects engineers to work with coding assistants, allow them in the work sample and observe the workflow. Ask the candidate to explain generated code, verify external assumptions, run tests, review security boundaries, and identify what they would not delegate. Tool use is part of the evidence; silent acceptance of generated output is also evidence.

Score evidence consistently

Use anchored scores instead of “strong yes” based on conversational confidence. Each interviewer should record evidence before the debrief.

Common false positives

• Tool vocabulary without ownership: listing vector databases, agent frameworks, or models without connecting them to a requirement.
• Demo speed without verification: shipping quickly while ignoring tests, permissions, data boundaries, and failure recovery.
• Research depth for the wrong role: impressive model knowledge when the immediate need is reliable product integration and delivery.
• Conventional backend strength without AI evaluation: solid services and infrastructure but no method for measuring probabilistic behavior.
• Over-engineering: proposing multi-agent orchestration, custom training, or complex infrastructure before establishing the simplest useful workflow.

Make the tradeoff explicit

No candidate covers everything. Record the gaps you are choosing and how the team will cover them. A strong product engineer may need support on model training. A deep ML engineer may need a product partner and platform support. A platform specialist may not be the person to discover the first user workflow.

Reference points

This scorecard draws on established guidance for AI evaluation and risk management. The interview structure and hiring synthesis are Botmer’s own.

• NIST AI Risk Management Framework — considering validity, reliability, safety, security, privacy, transparency, and accountability through the AI lifecycle
• NIST AI RMF Playbook — suggested actions organized around Govern, Map, Measure, and Manage
• OWASP Top 10 for LLM Applications 2025 — practical security risks to probe during design and operating interviews
• OpenAI evaluation guidance — task-specific evaluation criteria, datasets, and graders for comparing behavior