Most GenAI courses teach you how to prompt a model. The Databricks Generative AI Engineering pathway focuses on the engineering side: how to deploy AI responsibly, control costs, evaluate performance, and connect models to existing data systems.
This interview with Enver Ablaiev is a walkthrough of the first course in that pathway: the concepts that matter, the decisions they inform, and what the whole thing looks like from a data engineering perspective.
Specialises in data platforms, ETL pipelines, and scalable cloud-based architectures using Python, Apache Spark, Databricks, Apache Airflow, and SQL/NoSQL databases.
What were you expecting when you started the Databricks GenAI Engineering pathway?
Enver Ablaiev: I expected the course to be mostly about AI models: which ones are the newest, which to use, and what has just launched. The first lesson reframed that completely. New models arrive constantly, so if your strategy is "use the best model," it's out of date almost immediately. What sets you apart is your data: your records, your domain expertise, your operational history. The model is easy to swap out. Your data is the hard part to copy.
After eight years of building data platforms, that was good to hear. The hard work of getting clean, governed, well-modelled data in one place is the part I already know how to do. GenAI sits on top of that; it doesn't replace it.
How does the course explain what an LLM is, under the hood?
Enver: It keeps it usefully simple. At its core, a large language model is a prediction engine — it looks at the text so far and works out the most likely next token, then repeats the process. The whole trick is a chain of "What word probably comes next?" The course captures this in what it calls the Cognitive Engine — massive data feeds the model, the model processes it, and the outcome is human-like output.
One thing that surprises people is that it's non-deterministic. Ask the same question twice, and you can get two different answers, because the model samples from probabilities rather than looking up a fixed fact. Once I understood that, hallucinations stopped feeling mysterious. A model optimising for "plausible next word" can produce something plausible and wrong, with full confidence, because it has no sense of truth, only patterns.
What are the key technical concepts a practitioner needs to understand about how LLMs work?
Enver: Three things drive a model's cost, speed, and capacity. First, tokenization — models don't read words, they read tokens, chunks of text turned into numbers. Tokens aren't standardized across models, so the same sentence can cost differently depending on which model you use. One detail worth knowing: reasoning models work through problems internally, and those hidden reasoning steps still count as output tokens even though you never see them.
Not all LLMs tokenise in the same way</strong
| Term | Description |
|---|---|
| Input tokens | The number of tokens in the prompt |
| Output tokens | The number of tokens the LLM generates in response |
| Reasoning models | The act of "reasoning" is included in the overall number of output tokens |
Second, the context window — the model's working memory. Everything it can pay attention to at once: your prompt, any retrieved documents, its own response. When it fills up, the oldest content drops out, which is exactly when a model starts "forgetting" your instructions or filling gaps with invented details.
Third, parameters are the internal weights that represent the model's reasoning capacity. More parameters generally mean more nuance, but the cost grows quickly.
How do you get useful output from an LLM in a business context?
Enver: Think of an LLM as the most well-read intern alive — they've read every book, article, and study, never sleep, never forget a fact. But they take instructions completely literally and know nothing about your business. Ask them to "write a sales email", and you'll get something technically correct and entirely generic. Without guidance, it's just bad work.
Two things fix that. The first is prompt engineering — being specific and well-structured. The second, and the one I think matters more, is grounding: instead of letting the model answer from general knowledge, you give it specific, verified context from your own data. Grounding turns a general-purpose model into one that knows your company, and it's your best defence against the two biggest problems in enterprise AI: hallucinations and stale knowledge.
Is every problem a generative AI problem?
Enver: No, and the course is direct about this. Before building anything, run a quick decision check:
- Can output quality be measured clearly?
- Does the task require deterministic accuracy?
- Does the task involve language understanding or generation?
- Does the task require synthesis or open-ended output?
- Does the task depend on proprietary enterprise knowledge?
You only reach the last question — does it depend on your own proprietary knowledge? — after the task has survived the four before it. If the answer there is yes, that's a genuine fit for GenAI plus RAG.
How should teams think about choosing which model to use?
Enver: Don't use the most powerful model, use the most efficient one that still meets your needs. Models come in tiers — small, medium, large, and frontier — and the job isn't to climb to the top. It's to stop at the smallest one that clears the bar.
The cost difference is stark: a frontier model on the same workload can run roughly $3,800 a month versus about $150 for a small model, a 25x swing just from picking the wrong tier.
If your data can't leave your own environment, the most capable API model simply isn't an option that constraint alone can decide the choice. Beyond that, the factors to weigh are task complexity, quality requirements, and the cost-versus-latency trade-off.
What is RAG, and why does it matter?
Enver: Retrieval-Augmented Generation solves the core problem that a model doesn't know your data. When a question comes in, the system first searches your internal documents for the most relevant snippets, adds them to the prompt, and only then asks the model to answer. The model is working from your verified documents and can cite them.
What I liked most about RAG is that it updates instantly. Change a policy, re-index it, and the system reflects it immediately — no expensive retraining. That's why it fits customer support, security and compliance, and internal knowledge search so well.
When does it make sense to move from a simple GenAI call to an agent?
Enver: Plain generative AI is linear — prompt in, answer out. An agent adds reasoning and multi-step workflows. Instead of just answering, it can break a goal into smaller tasks, check its own progress, and adjust as it learns new information. Generative AI is the brain; an agent is the worker that uses that brain to get something done.
| Component | What it does |
|---|---|
| LLM | The "brain" that controls core logic and sequencing; other LLMs handle sub-tasks. |
| Tools | External resources the agent uses via tool calling. |
| Planning | Breaks complex goals into manageable tasks. |
| Memory | Short-term: session tracking; long-term: historical state, knowledge, preferences. |
The practical question is knowing when not to use one. If your workflow is a stable "if this, then that" — translate a document, fill a template — a plain, non-agentic call is cheaper, faster, and easier to debug. Save AI agents for open-ended, multi-step problems where the path changes as you go, and you want to automate the reasoning, not just the execution.
How do you evaluate whether a GenAI system is actually performing well?
Enver: Evaluation is what lets you confidently switch from an expensive model to a cheaper one once they perform the same. Three approaches each make different trade-offs.
- Humans in the loop are the most reliable for nuance and trust, but they don't scale, and reviewers often disagree.
- LLM-as-judge scales well — you can grade thousands of responses at low cost, and the judge explains its reasoning — but judges tend to favour longer answers, favour whatever is listed first, and prefer outputs from their own model family.
- Benchmarks give you direct comparisons and work well as regression tests, but they can be gamed and are often cleaner than your real production data.
The course's advice, which I thought was sound: maintain your own living evaluation set built from real production data and edge cases. Don't rely on public leaderboards.
How does all of this connect to Databricks specifically?
Enver: The core governance principle is straightforward: bring artificial intelligence to your data, not the other way around. Everything stays within your security perimeter, governed by the Unity Catalogue, rather than being sent to external APIs.
What struck me was how familiar the production pattern looks once you map it out. A RAG app on Databricks follows a clean sequence: govern your data in Unity Catalogue, build a Vector Search index that auto-syncs when data changes, retrieve context and add it to the prompt, generate a grounded answer, register the entire chain in MLflow, and deploy it to Model Serving. Look closely, and it's a data pipeline — ingestion, transformation, a sync job, a serving layer, and lineage tracking. The vocabulary is new; the shape is one I already know.
Related insights
The breadth of knowledge and understanding that ELEKS has within its walls allows us to leverage that expertise to make superior deliverables for our customers. When you work with ELEKS, you are working with the top 1% of the aptitude and engineering excellence of the whole country.
Right from the start, we really liked ELEKS’ commitment and engagement. They came to us with their best people to try to understand our context, our business idea, and developed the first prototype with us. They were very professional and very customer oriented. I think, without ELEKS it probably would not have been possible to have such a successful product in such a short period of time.
ELEKS has been involved in the development of a number of our consumer-facing websites and mobile applications that allow our customers to easily track their shipments, get the information they need as well as stay in touch with us. We’ve appreciated the level of ELEKS’ expertise, responsiveness and attention to details.
