Evaluating Paid vs Local LLMs for QA Test Design: A Practical Comparison

How to choose your local LLM?

If you want to run large language models on your own computer without sending data to the cloud, there are two main options to consider: Ollama and LM Studio. Both let you use LLMs locally, which is helpful if your company blocks cloud services like ChatGPT or when budgets are tight. However, each tool is designed for a different kind of user.

Ollama

Ollama is a local LLM runtime mainly aimed at developers.

It runs models in the background and exposes a REST API that works much like the OpenAI API, just fully local. This makes it a good fit for automation, integrations, agents, MCP servers, and CI/CD pipelines. It also handles concurrent requests better, which matters if multiple tools or scripts need to use the model at the same time.

A simple way to think about Ollama is: “Docker for LLMs.” It also provides a graphical user interface, but it is very simple and limited in functionality

LM Studio

LM Studio is a desktop application with a graphical interface.

It focuses on ease of use. You can browse available models, download them, and start chatting right away in a chat-style UI. It’s great for trying out prompts, experimenting with different models, and doing demos. LM Studio also exposes a REST API, but automation and backend usage are not its main strengths. This makes LM Studio more suitable for individual or private use rather than large-scale automation.

A good mental model here is: “ChatGPT, but running locally.”

Which one to pick?

• Ollama is the better choice if you're a developer who needs automation, integrations, or programmatic access.
• LM Studio is best if you want to explore models, learn how they behave, or tune prompts interactively.

Both tools run models locally, keep your data private, and support popular open-source LLMs. The difference is not in what models they run, but in how you plan to use them.

AI for software testing: approach and results of ELEKS’ experiment

For the experiments, we used the same prompt to generate 10 test cases and a separate prompt to generate 10 checklists. Below you will find a detailed user story, the exact prompts used, and sample outputs for each run.

To measure time-related metrics, we used a custom script that calls both the OpenAI API and the Ollama API running locally on a virtual machine. This allowed us to compare response times under the same conditions.

Results of running prompts (test design) on different models

The table below summarises the captured metrics. We measured execution time using a generated script (TypeScript + OpenAI API, Ollama API). To calculate the quality and cost score, we reviewed the outputs, focusing on the quality of checks and how constraints were handled. Additionally, we uploaded the AI-powered test cases and checklists to the GPT-5.2 model for a more comprehensive review. The final assessment combines our analysis with the latest GPT model’s evaluation.

* Some tests are too generic, steps are underspecified, and there are logic duplications in the checklist 
**  It was fast, but the quality of the output was worse than expected, as constraints were skipped 

Translating user stories into testable requirements: online store purchase

Let’s start with the user story. Imagine that as a registered customer, you want to purchase a product from the online store, so that you can receive the selected item at your address.

In this business context, the online store allows customers to browse products, add them to their cart, and complete purchases with different payment options. This user story covers the end-to-end checkout flow for a single product.

Key assumptions

Before defining test coverage, we establish several baseline conditions:

• User is registered and logged in
• Product is available in stock
• Delivery service is available for the user’s address
• The payment gateway is operational

Acceptance criteria (AC)

We structured our acceptance criteria using the Given-When-Then format for clarity. This structure enables comprehensive test coverage by making each criterion independently testable.

AC1: Given the user is logged in, when the user adds a product to the cart, then the product is displayed in the cart with the correct price and quantity.

AC2: Given the cart contains a product, when the user proceeds to checkout, the checkout page is displayed with the order summary.

AC3: Given that the checkout page is open, when the user enters a valid delivery address, the address is accepted and saved.

AC4: Given that a valid delivery address is provided, when the user selects a payment method and confirms payment, the payment is processed successfully.

AC5: Given that the payment is successful, when the order is completed, then an order confirmation page is displayed with the order number.

AC6: Given that the order is completed, the user receives an order confirmation email.

Scope boundaries

To maintain focused test coverage, we excluded:

• Guest checkout
• Multiple product purchase
• Order cancellation and refunds
• Promotions and discount codes

Testability notes (for test automation)

• Each acceptance criterion can be mapped to at least one test automation scenario
• Checkout flow is suitable for API, UI, and end-to-end automation
• Stable test data required for products, users, and payment methods
• Mocks or stubs are recommended for the payment gateway in lower environments

Potential test types

• UI test automation (Add to cart, checkout, confirmation)
• API test automation (Cart, Order, Payment endpoints)
• Integration testing (Payment and email services)
• Regression testing
• Smoke testing

Prompts

Prompt 10TCs

“Role: you are an experienced QA/Test Engineer. Given the following requirement(s), generate comprehensive 10 test cases that cover positive, negative, boundary, and edge cases. For each test case, include: Test Case ID, Description, Preconditions, Steps, Expected Result, and Actual Result (to be filled in during execution). Follow Acceptance Criteria. The number of test steps depends on how detailed the requirements are; if the requirements are not clear, steps should be high level. Requirements:”

Prompt 10 checklists

“Role: you are an experienced QA/Test Engineer. Generate 10 Test cases, including positive and negative for the given requirement at the end. The number of test cases should depend on the complexity of a requirement. Do not add description; provide just numbered test cases. Each test should start from 'Verify'. Requirements:”

Examples of generated test cases and checklists

Explore the potential of AI agents

Explore intelligent automation solutions

Challenges with local LLMs

ChatGPT typically prioritises explicit numeric constraints. For example, when instructed to “create 10 test cases,” it reliably generates exactly ten.

Local LLMs, however, often prioritise semantic reasoning over hard constraints. For instance, if the prompt says, “generate 10 test cases,” but also includes “the number of test cases should depend on the complexity of the requirement,” a local LLM may treat the numeric constraint as a soft guideline and produce 8–12 test cases instead. If I remove the second constraint: “the number of test cases should depend on the complexity of the requirement”, it will still generate more than 2 test cases. This illustrates how local LLMs behave in contrast to online paid services.

Additionally, even when explicitly instructed not to use a table format, some models (e.g., DeepSeek) still ignore this constraint and return the output in a table.

Model-based results assessment

In short, the paid ChatGPT/OpenAI API models (GPT-4.1, GPT-4o, and GPT-5.2) delivered the best overall results, which is expected. These models are trained on hundreds of billions (500–1000B+) of parameters, compared to the 7B-parameter local LLMs used in this evaluation.

For clarity, it makes sense to evaluate paid and free models separately.

Among the paid models, GPT-4.1 and GPT-5.2 demonstrated the best quality-to-cost ratio, with GPT-5.2 being the fastest in terms of response time while maintaining high output quality.

When analysing artifacts generated by free/local models, Qwen 2.5 7B and Mistral 7B produced the highest-quality outputs. Although they are slower than LLaMA 3.2, the overall artifact quality is noticeably better, especially for structured QA outputs such as test cases and checklists.

Comparison responses: ChatGPT vs local LLM

Which local model to choose for QA and activities like test design?

Each mode is characterised by a very important metric: the number of parameters. 

LLM parameters are the internal numeric weights of a large language model that store what the model has learned from data.

In simple terms, parameters are the “memory” of an LLM.

Each parameter is a number that influences how the model:

• understands words and context
• connects concepts
• predicts the next token

So, for these experiments, we’ve chosen 7B because this is a small LLM; it is a sweet spot, it can be easily deployed on machines with no GPU or limited resources (16-32 GB memory is enough), it also provides good reasoning.

Large Models (13B+) have better reasoning, but they become slow on CPU, and require high RAM (20-40GB).

If you have access to powerful machines with GPU support, using local LLMs is absolutely a viable and often preferable option. With sufficient hardware, local models can deliver strong results while preserving data privacy and avoiding cloud dependency.

However, for everyday and relatively simple tasks, such as generating test cases, checklists, or basic test documentation, small LLMs up to 8B parameters can already do a very good job and significantly speed up QA-related work.

If you have the option to use online services such as ChatGPT/OpenAI or Gemini (or similar platforms), they are generally recommended. These models are trained on vast amounts of data and typically provide higher precision, better instruction adherence, and more consistent outputs, especially for structured artifacts.

When a local LLM is the only option, it is advisable, together with IT support, to allocate the most powerful machine available. Based on the available hardware, you should then decide:

• which local LLM runtime to use (e.g., Ollama or LM Studio), and
• which specific model best fits your priorities.

At the time of this experiment, Qwen 2.5 7B and Mistral 7B delivered good-quality results, but they required almost twice the execution time compared to faster models such as LLaMA 3.2.

This ultimately becomes a trade-off decision. If speed is the top priority, faster models like LLaMA are a better choice. If output quality is more important, slightly slower models may be justified.

Conclusions

The choice between paid and local LLMs ultimately depends on your specific constraints and priorities. The key is understanding the trade-offs: faster models sacrifice some quality, while higher-quality local models require more processing time.

It is also important to note that models evolve rapidly. Before making a final decision, always review the latest benchmarks and evaluations for the models you are considering. Public benchmark platforms can provide up-to-date insights into quality, speed, and resource consumption, helping you make a more informed choice.

AI changes how software testing is done by looking at business risks, user impact, and past failures to highlight the most important features and workflows. This helps teams focus their testing where it matters most, boosting both efficiency and coverage. By using historical data and business priorities, AI can point out which parts need the most attention. Still, many complex AI models make decisions without clear explanations, which makes debugging and trusting their results hard. That’s why human oversight is crucial to keep the testing process reliable and high-quality.