AI and LLM engineering jobs: what hiring teams want

When a startup posts an AI or LLM engineering role, it is almost never asking you to train a foundation model. It is asking you to make a model someone else trained behave reliably inside a product: retrieve the right context, measure whether the output is correct, and keep latency and cost in check while real users hammer it. The job is mostly software engineering with one specialized layer on top, and the teams hiring fastest can tell within twenty minutes whether you have built that layer or only read about it.

This post is the concrete version. For the broader question of how AI is changing engineering work overall, read AI and software engineering jobs. Here we go narrow: the four skills that actually show up in interviews, who is hiring for them, what they pay, and how to position a resume so the screen does not bounce you.

What does an 'AI engineer' actually do?

There are three job shapes hiding under one title, and the comp and skill bar differ across them. Read a job post carefully before you assume which one it is.

The first row is where the volume is. A founder who just closed a round wants a shipping engineer who can wire a model into the product this quarter, not a PhD. That is good news: the entry skills are learnable on a laptop, and you can show them with a side project. If you also want the difference between this and a classic ML role, ML engineer vs software engineer draws that line in detail.

What are the four skills hiring teams actually test?

Across application-engineer loops, the same four areas come up again and again. None of them require deep math. All of them require having built something real.

1. RAG that does not hallucinate

Retrieval-augmented generation is table stakes: pull relevant documents, put them in the prompt, ground the answer in them. The naive version (embed everything, top-k cosine search, stuff into context) is a fifteen-minute tutorial. The interview tests the next layer. How do you chunk a 90-page PDF so retrieval works? What do you do when the right answer is split across three chunks? How do you handle a query the corpus cannot answer instead of letting the model invent something? Teams want to hear chunking strategy, reranking, and a refusal path, not just 'I used a vector database.'

2. Evals: how you know it works

This is the single biggest signal of seniority in AI work, and the most common gap. Anyone can get a demo to work once. The hard part is knowing whether version two is better than version one when the output is free text. Strong candidates talk about building an eval set (50 to 200 graded examples), defining what 'correct' means for their task, and using a model as a judge with a clear rubric where exact-match will not work. If you have ever caught a regression because your eval score dropped from 0.86 to 0.79 before it hit users, lead with that story.

The fastest way to sound senior in an AI interview is to describe how you measure quality, not how you prompt.

3. Production LLM systems: latency, cost, failure

A model call is a network call to a non-deterministic, sometimes-slow, sometimes-down dependency that bills per token. Production-minded engineers handle that like any other unreliable upstream: timeouts, retries with backoff, fallback to a cheaper model, streaming so the user sees tokens early, and caching where the input repeats. Worked example: a support-answer endpoint that calls a large model on every request might cost $0.04 per call. Route the 60 percent of easy queries to a small model at $0.003 and you cut blended cost by roughly half while keeping quality on the hard 40 percent (illustrative, not advice). Founders notice engineers who think in dollars per request.

4. Fine-tuning: knowing when not to

Fine-tuning is the most over-asked-about and least-often-needed skill. The senior answer to 'would you fine-tune this?' is usually 'not yet.' Better retrieval, a clearer prompt, and a few examples in context solve most quality problems faster and cheaper than collecting a training set. Fine-tuning earns its keep for a narrow style or format you cannot get with prompting, or to make a small cheap model match a big one on one specific task. Know the decision tree (prompt, then retrieve, then few-shot, then fine-tune) and you will outscore people who reach for a GPU first.

Who is hiring, and what does it pay?

The demand is concentrated but broad. AI startups hiring for their own product, of course, but also fintech, developer tools, healthcare, and legal companies bolting LLM features onto existing products. You can watch which ones just funded a hiring push on the recent raises page. For the sector view, which startup sectors are hiring engineers and the AI startup engineering job market break it down further.

Base salaries for AI application engineers track senior software pay in the same city, sometimes with a modest premium when the skill is scarce. They are not a separate pay scale. If a recruiter implies AI is automatically a 30 percent bump, treat it as a negotiation tactic and check it against the market in senior software engineer salary, SF and NYC 2026. Equity often matters more than the AI premium anyway; how much equity by stage covers that side.

How do you position a resume for these roles?

Screeners for AI roles are looking for evidence you have built, not consumed. The cleanest signal is a deployed project with numbers attached. Reframe what you already have.

• Name the system, not the buzzword. 'Built a RAG pipeline over 12,000 support docs, reranked top-20 to top-5, cut hallucinated answers from 18 percent to 4 percent on a 150-example eval set' beats 'experience with LLMs and vector databases.'
• Lead with evals. If you measured quality, say the metric and the delta. This is the rarest thing on the average AI resume.
• Show cost and latency awareness. 'Reduced p95 from 9s to 2.1s by streaming and routing easy queries to a smaller model' tells a founder you think in production.
• Do not fake depth. If you have never fine-tuned, do not list it. The screen will find out, and the application-engineer track does not need it.
• Keep it ASCII and tight. General resume hygiene still applies: see resume tips for startup engineers.

You can pressure-test the wording with our free resume scorer: it flags vague claims and rewards concrete, measured ones, which is exactly the bar an AI screen sets. Then drop your resume in the pool so we can match you against companies hiring for this work right now.

What if I am a strong engineer but new to LLMs?

Most teams hiring for the application track would rather take a strong generalist who can learn the LLM layer than a prompt hobbyist with shaky engineering fundamentals. The model-specific skills above are a thin layer, and a good engineer absorbs them in weeks. Build one honest end-to-end project: ingest a corpus, retrieve, generate, and grade the output with an eval set you wrote yourself. That single project covers three of the four competencies and gives you something concrete to talk about. If you are weighing this move against staying in classic backend or platform work, backend, frontend, fullstack career paths and AI and software engineering jobs are the right companions.