Large Language Models (LLMs) have changed the way we build applications. Today, we can build chatbots, intelligent workflows, AI assistants, and even AI agents.
There are many frameworks available to build LLM powered applications like Spring AI, Langchain. However, before jumping into a framework, it is important to understand what happens underneath.
In this article, we will explore the main concepts behind LLM applications by directly interacting with OpenAI APIs.
Calling an LLM
Let's start with a simple request.
First, export your OpenAI API key.
export OPENAI_API_KEY="your-api-key"
Now, send a request to the OpenAI Responses API.
curl https://api.openai.com/v1/responses \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "gpt-4.1",
"input": "What is Java?"
}'
At the simplest level, we send an input to the LLM and receive an output.
Understanding Tokens and Next Token Prediction
LLMs work with tokens. A token can be a word, part of a word, or another small piece of text.
Imagine we provide the following text:
Java is a programming
The model may predict possible next tokens:
language 80%
platform 10%
tool 5%
...
The model selects a token and repeats the same process.
This process continues until the model completes the response.
Controlling the Response with Inference Parameters
We can influence how the model generates a response using inference parameters.
Temperature
Temperature controls the randomness of token selection.
Let's ask the model to create a slogan with a low temperature.
curl https://api.openai.com/v1/responses \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "gpt-4.1",
"temperature": 0.2,
"input": "Create a slogan for a chess application."
}'
A lower temperature generally produces more focused and predictable responses.
Now, increase the temperature.
curl https://api.openai.com/v1/responses \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "gpt-4.1",
"temperature": 1.0,
"input": "Create a slogan for a chess application."
}'
The response may now be more varied or creative.
In simple terms: Temperature controls how adventurous the model is when selecting the next token.
Top P
top_p controls the probability range considered when selecting the next token.
Imagine the model predicts:
Java 50%
Python 30%
Go 10%
Rust 5%
Other 5%
With top_p set to 0.8, the model focuses on tokens covering roughly the top 80% of the probability mass.
curl https://api.openai.com/v1/responses \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "gpt-4.1",
"top_p": 0.8,
"input": "Suggest a programming language for building a web API."
}'
Temperature controls randomness, while top_p limits the probability range considered during selection.
Usually, we do not need to aggressively change both.
Maximum Output Tokens
We can also limit the maximum number of tokens generated by the model.
curl https://api.openai.com/v1/responses \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "gpt-4.1",
"max_output_tokens": 100,
"input": "Explain microservices in detail."
}'
It is useful to understand the difference between input and output tokens.
Input tokens → Instructions + messages + context
Output tokens → Generated response
Tokens are important because models have context limits, and API usage is generally measured using tokens.
LLM Responses Are Not Guaranteed Facts
There is another important point about LLMs. LLM output is probabilistic.
The model generates a likely response based on the provided input and its learned patterns. This means a response may sound confident while still being incorrect.
This is commonly referred to as hallucination.
For example:
User: What is the status of order ORD-1001?
LLM: Your order has been delivered.
The response sounds perfectly valid. However, the model has no access to our order database.
This is why we should not treat an LLM as a database or a source of guaranteed facts. Later, we will see how external knowledge and tools help us provide real information to the model.
Getting Structured Responses
So far, the model has returned natural language responses. This is fine for a chatbot. However, applications often need structured data. Imagine we are building a movie application. We may need the model to return:
{
"title": "Interstellar",
"genre": "Science Fiction"
}
Simply asking the model to return JSON does not define a reliable application contract. OpenAI supports Structured Outputs, where we provide the expected JSON Schema.
curl https://api.openai.com/v1/responses \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "gpt-4.1",
"input": "Recommend a science fiction movie.",
"text": {
"format": {
"type": "json_schema",
"name": "movie",
"schema": {
"type": "object",
"properties": {
"title": {
"type": "string"
},
"genre": {
"type": "string"
}
},
"required": ["title", "genre"],
"additionalProperties": false
},
"strict": true
}
}
}'
The model now generates a response that follows the defined structure. Structured output is useful when the response needs to be processed by our application, displayed in a UI, or mapped to an application object.
Messages and Roles
Real applications need more than a simple input. We need to tell the model how it should behave and what the user wants. Messages can have different roles.
systtem/ developer - The system, developer roles contains application level instructions.
user - The user role represents the user's request.
assistant - The assistant role represents previous model responses when we provide conversation history.
Let's create a Java assistant.
curl https://api.openai.com/v1/responses \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "gpt-4.1",
"input": [
{
"role": "developer",
"content": "You are a Java expert. Explain concepts using simple examples."
},
{
"role": "user",
"content": "What is dependency injection?"
}
]
}'
Now, let's change the developer instruction.
curl https://api.openai.com/v1/responses \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "gpt-4.1",
"input": [
{
"role": "developer",
"content": "Explain technical concepts to a five-year-old child."
},
{
"role": "user",
"content": "What is dependency injection?"
}
]
}'
The user question is exactly the same. However, the model responds differently because the developer instruction changes its behaviour.
This separation is important when building LLM applications.
Chat Conversations
Consider the following conversation:
User: What is Spring Boot?
Assistant: Spring Boot is a framework for building Java applications.
User: Explain it more simply.
How does the model know what "it" means?
Let's send the first request.
curl https://api.openai.com/v1/responses \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "gpt-4.1",
"input": "What is Spring Boot?"
}'
The response contains an ID similar to:
resp_0654f03ef0ebbd98006a55cc8a9ffc819490468f200df33bde
We can continue from that response using previous_response_id.
curl https://api.openai.com/v1/responses \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "gpt-4.1",
"previous_response_id": "resp_0654f03ef0ebbd98006a55cc8a9ffc819490468f200df33bde",
"input": "Explain it more simply."
}'
The previous conversation context helps the model understand what "it" refers to.
Conversation Memory
A common misunderstanding is that the LLM automatically remembers our previous API requests. It does not. The model needs previous conversation context to understand earlier messages.
One option is to chain responses using previous_response_id as explained earlier.
Our application can also maintain the conversation history itself and provide the messages again.
curl https://api.openai.com/v1/responses \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "gpt-4.1",
"input": [
{
"role": "user",
"content": "My favorite programming language is Java."
},
{
"role": "assistant",
"content": "Got it. Your favorite programming language is Java."
},
{
"role": "user",
"content": "What is my favorite programming language?"
}
]
}'
In this approach, our application may store messages in a database and provide the required history with each request.
OpenAI also provides conversation objects for maintaining conversation state across multiple responses.
The important point is; Conversation memory is conversation context available to the model. It is not permanent knowledge stored inside the LLM.
Application instructions may also need to be provided again depending on how we manage conversation state.
Context Windows
Can we keep sending the entire conversation forever? No. Models have a context window.
The context may contain:
Developer instructions
+
Conversation history
+
Retrieved knowledge
+
Tool information
+
Current user request
All of this consumes tokens. As a conversation grows, applications may need to keep only recent messages, summarize older conversations, or select only relevant information.
This is why conversation memory usually needs a memory strategy instead of storing and sending everything forever.
The LLM Does Not Know Our Data
Now, imagine a customer asks:
What is the status of order ORD-1001?
Let's ask the model directly.
curl https://api.openai.com/v1/responses \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "gpt-4.1",
"input": "What is the current status of order ORD-1001?"
}'
The model does not have access to our order database. Similarly, it does not automatically know about:
Company policies
Internal documents
Customer accounts
Current order status
Application database records
We need to provide external information. Two common techniques are:
Documents / Knowledge → RAG
Live Application Data → Tools
Let's understand the difference.
External Knowledge with RAG
Imagine our company has an employee handbook.
The user asks:
How many annual leave days do employees receive?
The answer exists in the employee handbook.
Retrieval Augmented Generation, or RAG, retrieves relevant information and provides it to the model as context.
First, we split our documents into smaller chunks. Imagine one chunk contains:
Employees receive 21 days of annual leave per year.
We can create an embedding for this text.
curl https://api.openai.com/v1/embeddings \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "text-embedding-3-small",
"input": "Employees receive 21 days of annual leave per year."
}'
An embedding represents the meaning of the text as a vector of numbers.
Conceptually:
"Employees receive 21 days of annual leave per year."
↓
[0.018, -0.024, 0.091, ...]
We store the vector together with the original text in a vector store. When the user asks a question, we create another embedding.
curl https://api.openai.com/v1/embeddings \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "text-embedding-3-small",
"input": "How many annual leave days do employees receive?"
}'
Our application performs a similarity search between the question vector and document vectors.
The relevant chunk may be:
Employees receive 21 days of annual leave per year.
Now, we provide the retrieved context to the model.
curl https://api.openai.com/v1/responses \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "gpt-4.1",
"input": [
{
"role": "developer",
"content": "Answer the question using only the provided context."
},
{
"role": "user",
"content": "Context: Employees receive 21 days of annual leave per year.\n\nQuestion: How many annual leave days do employees receive?"
}
]
}'
The complete RAG flow looks like this:
The important point is:
RAG does not teach the model new knowledge. It retrieves relevant information and provides it as context.
Accessing Live Data with Tools
Let's return to our order question.
What is the status of order ORD-1001?
Order status is live application data. RAG is usually not the right solution. Instead, we can provide a tool.
Let's define a get_order_status tool.
curl https://api.openai.com/v1/responses \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "gpt-4.1",
"input": "What is the status of order ORD-1001?",
"tools": [
{
"type": "function",
"name": "get_order_status",
"description": "Returns the current status of an order.",
"parameters": {
"type": "object",
"properties": {
"order_id": {
"type": "string",
"description": "The order ID."
}
},
"required": ["order_id"],
"additionalProperties": false
},
"strict": true
}
]
}'
The model may return a function call similar to:
{
"type": "function_call",
"name": "get_order_status",
"arguments": "{\"order_id\":\"ORD-1001\"}",
"call_id": "call_abc123"
}
Notice what happened. The model did not execute our database query. It requested:
get_order_status(
order_id = "ORD-1001"
)
Our application executes the actual operation.
For example:
SELECT status
FROM orders
WHERE order_id = 'ORD-1001';
Assume our application receives:
{
"status": "IN_TRANSIT"
}
Now, we send the tool result back to the model.
curl https://api.openai.com/v1/responses \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "gpt-4.1",
"previous_response_id": "resp_0f18bff12dd43a47006a55cdf1679481959d8221e37c23aeb4",
"input": [
{
"type": "function_call_output",
"call_id": "call_GntNb1CYnOMWqvthZINOXcJt",
"output": "{\"status\":\"IN_TRANSIT\"}"
}
]
}'
The model can now generate a natural language response:
Your order ORD-1001 is currently in transit.
The complete tool-calling process is:
There is one very important detail:
The LLM does not execute our Java method, database query, or REST API.
The model decides which tool should be called and provides the arguments. Our application executes the actual operation. Tools are not limited to reading data. They can also represent actions such as:
create_order
cancel_order
send_email
book_delivery
create_support_ticket
This is how an LLM can interact with the outside world.
Combining Memory, RAG, and Tools
A real AI application may combine all these concepts.
Consider this conversation:
User:
What is our refund policy?
Assistant:
You can request a refund within 30 days.
User:
What about order ORD-1001?
Assistant:
ORD-1001 is currently in transit.
User:
Can I refund it?
The final question looks simple:
Can I refund it?
However, answering it may require several capabilities.
Conversation Memory - Understand "it" means ORD-1001
RAG - Retrieve the refund policy
Tool - Retrieve the current order status
Eventually, the model needs the relevant information.
curl https://api.openai.com/v1/responses \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "gpt-4.1",
"input": [
{
"role": "developer",
"content": "Answer customer questions using the provided policy and order information."
},
{
"role": "user",
"content": "Refund policy: Orders can be refunded within 30 days. Orders currently in transit must be delivered before a refund request can be created.\n\nOrder ORD-1001 status: IN_TRANSIT.\n\nThe customer is asking whether order ORD-1001 can be refunded now."
}
]
}'
The architecture starts to look like this:
Are We Building an Agent?
Consider what we have built so far:
LLM
+
Instructions
+
Conversation Memory
+
External Knowledge
+
Tools
The model can understand a request, use available knowledge, decide when a tool is required, and continue a conversation.
Our application is now starting to show agent like behaviour.
More advanced agents may perform multiple steps, use several tools, evaluate results, and continue until a task is completed. However, the fundamental concepts remain the same.
Why Do We Need AI Frameworks?
Without a framework, our application must manage:
API requests
Messages
Instructions
Structured outputs
Conversation history
Context windows
Embeddings
Vector stores
Retrieval
Tool definitions
Tool execution
Tool results
Retries
Observability
We can certainly implement all of this ourselves. However, as the application grows, the amount of infrastructure code also grows. This is where AI frameworks such as Spring AI, LangChain become useful. They provide abstractions around the same concepts we explored in this article.
Frameworks do not introduce magic into LLM applications. Underneath, we still have:
Models
Messages
Memory
Retrieval
Structured Outputs
Tools
Understanding these fundamentals makes it much easier to understand what an AI framework is actually doing for us.
Summary
In this article, we explored the fundamental building blocks of an LLM application using OpenAI APIs directly. When we combine instructions, memory, external knowledge, and tools, we begin to build intelligent, agent like applications. Now that we understand what happens underneath, we are ready to explore how AI frameworks simplify these concepts.
Comments
Post a Comment