Prompt engineering: The obvious things

• Be explicit
• Provide relevant context
• Specify the desired output (format, e.g., JSON, audience, e.g. PhD level, granularity: e.g., summary)
• Give constraints and boundaries (e.g., “Use only the supplied text,” “If information is missing, say so,” or “Don’t invent citations.” )

Zero-Shot Prompting

• Zero shot prompting provides no examples and relies entirely on the pre-existing knowledge in the LLM

One/Few-Shot Prompting

• Zero shot prompting provides no examples and relies entirely on the pre-existing knowledge in the LLM

Chain-of-thought (CoT) prompting

• Chain of thought prompts encourage the LLMs to autocomplete in a way that will simulate step by step reasoning¹

Tree of thoughts

• The actual ToT used a small controller that repeatedly called an LLM API

Conclusion: Prompt engineering

• test prompts against representative inputs
• examine failures, modify it, and test again
• try different styles of prompting to see what works the best for a certain goal

Retrieval augmented generation (RAG)

• RAG was primarily designed to address three potential issues of LLMs
• LLMs are trained on massive amounts of broad, general data.
  • The focus of LLMs on huge data corpora may lead to inferior performance in specialized tasks
  • This restricts their ability to incorporate new data after training
    • The cost and effort involved in training an LLM means they cannot be updated frequently, and may therefore struggle with very recent data/information
  • LLMs may generate convincing yet inaccurate responses (“hallucination”)
• We will see later on that LLMs may also struggle with named entity recognition (NER) of ontology terms. RAG-like approaches can be helpful here as well

RAG: Basic idea

• RAG can enable an LLM to provide precise answers beyond its initial training data
• Look up information in external data source

RAG: Basic idea

RAG: Example

• The basic algorithm of RAg is heuristic and can be most easily explained by an example.
• HOMEWORK We will construct a somewhat artificial example that can be run on a laptop and ask you to extend the example.
• Moby Dick, by Herman Melville, is one of the classics of American Literature.
• The story gets started on the island of Nantucket. A classic soup dish from the time that is still popular is called Clam Chowder

RAG: Example

• We will use RAG to show how to answer the question

• If we ask this question of a small local LLM (gemma3:12b), which is available via olamma or on hugging face), we do not get a satisfactory answer
• Imagine we have a database of books about Nantucket. Each book has a certain number of chapters
• We suspect that the answer can be found in on or other of the chapters.
• RAG uses vector similarity to retrieve chapters that display semantic similarity with a query
• RAG then presents the chapters to the LLM and asks the LLM to craft a response based on the information in the chapter

RAG: Moby Dick

• We download the text of Moby Dick from Project Gutenberg and split the text into the 135 chapters

This text is a comb...
CHAPTER 1. Loomings.: Call me Ishmael. Some years ago—never mind how long precisel...
CHAPTER 2. The Carpet-Bag.: I stuffed a shirt or two into my old carpet-bag, tucked it u...
CHAPTER 3. The Spouter-Inn.: Entering that gable-ended Spouter-Inn, you found yourself in...
CHAPTER 4. The Counterpane.: Upon waking next morning about daylight, I found Queequeg’s ...
CHAPTER 5. Breakfast.: I quickly followed suit, and descending into the bar-room ac...
CHAPTER 6. The Street.: If I had been astonished at first catching a glimpse of so o...
CHAPTER 7. The Chapel.: In this same New Bedford there stands a Whaleman’s Chapel, a...
CHAPTER 8. The Pulpit.: I had not been seated very long ere a man of a certain vener...
CHAPTER 9. The Sermon.: Father Mapple rose, and in a mild voice of unassuming author...
(...)

• See homework for the corresponding Python code

Langchain: Document

• LangChain is a Python library that accelerates LLM application development
  • modular abstractions for chaining prompts, memory, and external tools together.
  • foundation for building agents, retrieval-augmented generation (RAG) systems, and complex workflows.
• We encourage students to explore LangChain in the homework.
• Document: LangChain’s standard container for a piece of text plus its associated metadata

from langchain_core.documents import Document

chapter_docs = [
    Document(page_content=body, metadata={"chapter": header})
    for header, body in real_chapters
]

Langchain: Vector DB

• Chroma: an open-source vector database that runs locally
• Initializing Chroma embeds the chapters using the nomic-embed-text model
• vector_store.add_documents(chapter_docs): Creates one vector for each of the documents
• Chroma stores the triple — the original text, its vector, and its metadata (the chapter tag) — all together in moby_dick_chapters.
• vector_store (line 12) now contains a searchable index of all chapters

from langchain_chroma import Chroma
from langchain_ollama import OllamaEmbeddings

# Get Embeddings Model
embeddings = OllamaEmbeddings(model="nomic-embed-text")

# Initialize ChromaDB as Vector Store
vector_store = Chroma(
    collection_name="moby_dick_chapters",
    embedding_function=embeddings
)
vector_store.add_documents(chapter_docs) 

Naive query

llm = ChatOllama(model="gemma3:12b", temperature=0)
chowder_question = "What is a good place to eat chowder in early nineteenth century Nantucket?"
no_rag_response = llm.invoke(chowder_question)
print(no_rag_response.content)

• The response is amusing (and hallucinated)

LangChain: RAG

from langchain_ollama import ChatOllama
from langchain_core.prompts import PromptTemplate

retriever = vector_store.as_retriever()

prompt_template = """Use the context provided to answer 
the user's question below. If you do not know the answer 
based on the context provided, tell the user that you do 
not know the answer to their question based on the context
provided and that you are sorry.

context: {context}

question: {query}

answer: """

# Create Prompt Instance from template
custom_rag_prompt = PromptTemplate.from_template(prompt_template)

• retriever: wraps vector_store so it can be used to fetch semantically similar objects
• custom_rag_prompt is a PromptTemplate object — a structured representation of a prompt with named variable slots

LangChain: Querying the Vector DB

results = vector_store.similarity_search(
    'Inns where one can eat clam chowder',
    k=2
)

for res in results:
    print(f"* {res.page_content[:200]} [{res.metadata}]\n\n")

Here we retrieve the chapters with the most similar vectors, e.g.,

• We pass texts to the LLM and ask the LLM to analyze the texts and answer

LangChain: RAG - create the chain of commands

from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

rag_chain = (
    {"context": retriever | format_docs, "query": RunnablePassthrough()}
    | custom_rag_prompt
    | llm
    | StrOutputParser()
)

• rag_chain is written in LangChain Expression Language (LCEL) — chains built with the | operator
• Works like Unix pipes, where each stage’s output becomes the next stage’s input.
• retriever | format_docs — the raw input (i.e., the chowder question) is fed into retriever, producing a list of Documents (similar to previous slide)
  • The Document obejcts are piped into format_docs, producing one joined string. This becomes the value at key context.
• custom_rag_prompt: formats result of the query into a prompt text according to the template
• llm: Sends the prompt to our LLM (gemma)
• StrOutputParser(): formats output of the LLM

LangChain: RAG - run the query

• The following code invokes the chain of commands from the previous slide using our chowder question.

rag_chain.invoke(
  chowder_question
)

• Now we receive a much more useful answer

RAG-HPO: Motivation

• LLMs are resource-intensive and slow in their reasoning compared to other machine learning methods

• LLMs are prone to hallucinations, in which the model generates incorrect information and confidently asserts it as fact

• LLMs can be fine-tuned, but this is computationally expensive and requires substantial expertise¹

• RAG on the other hand is more cost-effective and adaptable and is relatively easy to keep up to date, which is important for a resource such as HPO that is updated many times a year

RAG-HPO

• Conceptually, RAG-HPO has three stages
  • 1. Phase 1: Send a clinical text to an LLM and ask the LLM to identify clinical concepts. These are not yet HPO terms, but many represent concepts for which there is an HPO term. The result of phase 1 is a list of clinical phrases
  • 2. Phase 2: For each phrase returned by stage 1, perform a query in a vector database of HPO terms to identify \(k\) candidates¹ The result of phase 2 is a list of candidates for each phrase
  • 3. Phase 3: For each of the phrases, ask the LLM to judge which of the candidates is the best match for the phrase. This match becomes the HPO annotation for the phrase²

RAG-HPO workflow

• To set the stage for RAG-HPO, some amount of coding is needed that we will not cover in detail in this lecture.¹
• In the lecture, we will provide a bird’s eye view of the preparation and focus on the final query stages
• Main stages:
  • Ingest the HPO file (hp.json)
  • Extract the main components of each HPO term (label, definition, synonyms, organ category, ancestry)
  • Create a vector database with one vector for each HPO term
  • Initialize a large language model (locally, for this example)
  • Encode the query using the same embedding model
  • Have the LLM extract phrases from the input text that it thinks are clinical descriptions
  • For each clinical description
    • Find the \(k\) most similar HPO vectors using cosine similarity
    • Ask the LLM to decide if the clinical description can be represented by one of the \(k\) HPO terms
    • If yes, return the HPO term together with metadata including the HPO id

RAG-HPO: Structuring HPO data prior to vectorization

------------------------------------------------------------
HP ID:       HP:0004227
Info:        brachytelophalangism v
Direction:   0
Organ system:Limbs
Lineage:     All (HP:0000001) -> Phenotypic abnormality (HP:0000118) -> \
    Abnormality of limbs (HP:0040064) -> Abnormality of the upper limb (HP:0002817) -> \
    Abnormality of the hand (HP:0001155) -> Abnormal finger morphology (HP:0001167) -> \
    Aplasia/Hypoplasia of fingers (HP:0006265) -> Aplasia/Hypoplasia of the 5th finger (HP:0006262) -> \
    Short 5th finger (HP:0009237) -> Short distal phalanx of the 5th finger (HP:0004227)
Definition:  Hypoplastic/small distal phalanx of the fifth finger.
------------------------------------------------------------

• Information is extracted from each HPO term as shown above. This is the sentence that is used for vectorization
• See hpo vectorization (Jupyter notebook) for implementation details.

RAG-HPO: Set up

from langchain_ollama import ChatOllama
llm_client = ChatOllama(model="gemma3:12b", temperature=0)
emd_model = get_embedding_model()
docs, emb_matrix = load_vector_db(meta_path="hpo_meta.json", vec_path="hpo_embedded.npz")
index = create_faiss_index(emb_matrix, metric='cosine')

• We initialize a gemma model as our LLM. This is a relatively small (less than 8GB) but surprisingly powerful LLM that can be run on a decent laptop with Ollama
• We load the embedding vectors and metadata as created in hpo vectorization (Jupyter notebook)
• We index the vectors using the FAISS library for quicker searching¹

RAG-HPO: Build the prompt

from langchain_core.messages import SystemMessage, HumanMessage

messages = [
    SystemMessage(content=system_message_I),
    HumanMessage(content=clinical_note),
]

response = llm_client.invoke(messages)

• This langchain code initializes a prompt message from a System and a Human component.
• The system message tells the LLM what role to play (see next slide)
• In this case, the user message is clinical description that we are asking the LLM to parse (see below)

RAG-HPO Prompt (System message)

RAG-HPO Prompt (Human Message)

• The following is a clinical description of an individual who has presented for diagnostic evaluation

RAG-HPO: First response

• The first step in RAG-HPO is to get the LLM to find pieces of text that could potentially represent phenotypic features.
• The point of RAG-HPO is that we can use these candidates to scan the HPO ontology for potential matches using a vector database
• We will subsequently ask the LLM in a second step which of the HPO terms returned by the vector database are actually good matches.

Here is an excerpt of the LLM response:

{
  "phenotypes": [
    {
      "phrase": "MCAS symptoms",
      "category": "Suspected"
    },
    {
      "phrase": "flushing",
      "category": "Abnormal"
    },
    {
      "phrase": "rashes",
      "category": "Abnormal"
    },
    {
      "phrase": "itching",
      "category": "Abnormal"
    },
    {
      "phrase": "disabled owing to orthostatic symptoms",
      "category": "Other"
    },
    (...)
}

RAG-HPO

• For each of the phrase entries from the list in the previous slide, RAG-HPO then asks for the best \(k\) matches from the vector database

query_vec = embed_query(phrase, embeddings_model, metric=metric)
dists, idxs = index.search(query_vec, top_k)

• dists is a list of the top_k most similar vectors,¹ sorted by similarity
• The list looks something like this²
  • hp_id: HP:0031284, phrase: ‘flushing’, ’similarity: 0.9999998807907104
  • hp_id: HP:0001033, phrase:‘facial flushing after alcohol intake’, similarity: 0.6326053142547607
  • hp_id: ‘HP:0033349’, phrase: ‘seizure flurries’, similarity: 0.5522459149360657
  • hp_id: ‘HP:0032762’, phrase: ‘focal autonomic seizure with pallor/flushing’, similarity: 0.5467579364776611
  • hp_id: ‘HP:4000054’, phrase: ‘exanthem’, ‘definition’: None, ‘organ_system’: ‘Integument’, similarity: 0.5450245141983032
  • (…)
• These are only candidates – we still need to ask the LLM which list item is actually the correct match

Understanding the vector DB search

def _collect_metadata_best(
    phrase: str,
    query_vec: np.ndarray,
    index: faiss.Index,
    docs: List[Dict[str, Any]],
    top_k: int = 500,
    similarity_threshold: float = 0.35,
    min_unique: int = 15,
    max_unique: int = 20
) -> List[Dict[str, Any]]:
    clean_tokens = set(re.findall(r'\w+', phrase.lower()))
    dists, idxs = index.search(query_vec, top_k)
    sims, indices = dists[0], idxs[0]

    seen_hp = set()
    results = []

    for sim, idx in sorted(zip(sims, indices), key=lambda x: x[0], reverse=True):
        if len(results) >= max_unique:
            break
        doc = docs[idx]
        hp = doc.get('hp_id')
        if not hp or hp in seen_hp:
            continue

        info = doc.get('info', '') or ''
        token_overlap = bool(clean_tokens & set(re.findall(r'\w+', info.lower())))

        #see next slide

• line 11 Tokenizes the input phrase into a lowercase set of word-tokens — e.g. “facial rash” → {“facial”, “rash”}

Understanding the vector DB search (2)

def _collect_metadata_best(
    phrase: str,
    query_vec: np.ndarray,
    index: faiss.Index,
    docs: List[Dict[str, Any]],
    top_k: int = 500,
    similarity_threshold: float = 0.35,
    min_unique: int = 15,
    max_unique: int = 20
) -> List[Dict[str, Any]]:
    clean_tokens = set(re.findall(r'\w+', phrase.lower()))
    dists, idxs = index.search(query_vec, top_k)
    sims, indices = dists[0], idxs[0]

    seen_hp = set()
    results = []

    for sim, idx in sorted(zip(sims, indices), key=lambda x: x[0], reverse=True):
        if len(results) >= max_unique:
            break
        doc = docs[idx]
        hp = doc.get('hp_id')
        if not hp or hp in seen_hp:
            continue

        info = doc.get('info', '') or ''
        token_overlap = bool(clean_tokens & set(re.findall(r'\w+', info.lower())))

        #see next slide

• line 12: Runs the actual FAISS similarity search: for query_vec (the phrase’s embedding), find the top_k (default 500) nearest neighbors in the index. index.search returns 2D arrays because FAISS supports batched queries (searching many query vectors at once)
• line 13: so dists[0] (for the first, and in this case, only query) unwraps the single row of results.
• sims is an array of 500 similarity scores (cosine similarities)
• indices is the corresponding array of 500 row-indices into docs.

Vector db search

• The vector DB search indenties the \(k\) most similar entries in the database (in this example, \(k=2\))
• These entries are returned for use in the LLM prompt (client code can also maintain a database of metadata related to each vector in order to have more information for crafting the prompt)

Understanding the vector DB search (3)

def _collect_metadata_best(
    phrase: str,
    query_vec: np.ndarray,
    index: faiss.Index,
    docs: List[Dict[str, Any]],
    top_k: int = 500,
    similarity_threshold: float = 0.35,
    min_unique: int = 15,
    max_unique: int = 20
) -> List[Dict[str, Any]]:
    clean_tokens = set(re.findall(r'\w+', phrase.lower()))
    dists, idxs = index.search(query_vec, top_k)
    sims, indices = dists[0], idxs[0]

    seen_hp = set()
    results = []

    for sim, idx in sorted(zip(sims, indices), key=lambda x: x[0], reverse=True):
        if len(results) >= max_unique:
            break
        doc = docs[idx]
        hp = doc.get('hp_id')
        if not hp or hp in seen_hp:
            continue

        info = doc.get('info', '') or ''
        token_overlap = bool(clean_tokens & set(re.findall(r'\w+', info.lower())))

        #see next slide

• line 18: Pairs up each similarity score with its doc index, and sorts descending by similarity.
• line 21: docs[idx]gives us the metadata at index idx. We skip if it is malformed or if we have already seen this entry
• line 26: info holds the cleaned text of a single HPO label or synonym, e.g., “flushing”, “arachnodactyly”
• line 27: token_overlap is True iff info shares any word with the input phrase’s tokens.

Understanding the vector DB search (4)

def _collect_metadata_best(
    phrase: str,
    query_vec: np.ndarray,
    index: faiss.Index,
    docs: List[Dict[str, Any]],
    top_k: int = 500,
    similarity_threshold: float = 0.35,
    min_unique: int = 15,
    max_unique: int = 20
) -> List[Dict[str, Any]]:
        #see previous slide
        if token_overlap or sim >= similarity_threshold or len(results) < min_unique:
            seen_hp.add(hp)
            results.append({
                'hp_id': hp,
                'phrase': info,
                'definition': doc.get('definition'),
                'organ_system': doc.get('organ_system'),
                'similarity': float(sim)
            })
    return results

• line 12: A candidate is accepted if any of three things is true:
  • it shares a word with the query phrase, or
  • its cosine similarity is ≥ 0.35, or
  • min_unique accepted results has not been achieved yet (a fallback that accepts anything, regardless of quality, purely to guarantee a minimum count)

Understanding the vector DB search (5)

def _collect_metadata_best(
    phrase: str,
    query_vec: np.ndarray,
    index: faiss.Index,
    docs: List[Dict[str, Any]],
    top_k: int = 500,
    similarity_threshold: float = 0.35,
    min_unique: int = 15,
    max_unique: int = 20
) -> List[Dict[str, Any]]:
        #see previous slide
        if token_overlap or sim >= similarity_threshold or len(results) < min_unique:
            seen_hp.add(hp)
            results.append({
                'hp_id': hp,
                'phrase': info,
                'definition': doc.get('definition'),
                'organ_system': doc.get('organ_system'),
                'similarity': float(sim)
            })
    return results

• line 21: Following the for loop, we return a list of candidates (one such list for each phrase in the original text)

RAG-HPO: Stage 3: Finding the best hit among the retrieved candidates

• RAG-HPO then uses a second prompt to ask the LLM to rate the candidates

{
  "phrase": "​…​",
  "category": "​…​",
  "original_sentence": "​…​",
  "candidates": [ {"term": "​…​", "id": "HP:…​"}, … ]
}

{"hpo_id": "HP:0001234"}

{"hpo_id": null"}

RAG-HPO: Stage 3: Finding the best hit among the retrieved candidates

• The corresponding user prompt incorporates the results of the vector search

{"phrase": "flushing", "category": "Abnormal", 
"original_sentence": "A 43-year-old Caucasian female experienced MCAS symptoms at age 18: 
    specific foods and odours caused flushing, rashes, itching, wheezing, dizziness and nausea", 
"candidates": [
    {"term": "flushing", "id": "HP:0031284"}, 
    {"term": "facial flushing after alcohol intake", "id": "HP:0001033"}, 
    {"term": "seizure flurries", "id": "HP:0033349"}, 
    {"term": "focal autonomic seizure with pallor/flushing", "id": "HP:0032762"}, 
    {"term": "exanthem", "id": "HP:4000054"}, 
    {"term": "blushing", "id": "HP:0001041"}, 
    {"term": "focal aware autonomic seizure with pallor/flushing", "id": "HP:0032761"}, 
    {"term": "hemoptysis", "id": "HP:0002105"}, 
    {"term": "fasciculation", "id": "HP:0002380"},
    {"term": "severe influenza", "id": "HP:0034249"}, 
    (...)
]}

• For this example, the LLM response was correct

RAG-HPO

• The authors showed excellent performance of their tool on a benchmark dataset
• The investigated different LLMs. The best performing was LLaMA-3 70B

SPIRES: Motivation

• Knowledge Bases and ontologies (KBs) encode domain knowledge in a structure that is amenable to precise querying and reasoning.
• However, the vast majority of human knowledge is communicated via natural language
• Large Language Models (LLMs) have shown great promise in interpreting biomedical texts, but often fail to identify ontology term identifiers and relations that are need to extract computable knowledge for KBs
• SPIRES aims to use LLMs to build KBs using NLP at the time of KB construction
• This can assist manual knowledge curation and acquisition while validating facts prior to insertion into KBs such as HPO and many others.

SPIRES: Main steps

• Literature triage: aids selection of relevant texts to curate
• Named Entity Recognition (NER): identify textual spans mentioning relevant things or concepts such as genes or ingredients
• grounding maps these spans to persistent identifiers in databases or ontologies (e.g., the HPO term label Arachnodactly is mapped to its identifier HP:0001166)
• Relation Extraction (RE) connects named entities via predicates such as ‘causes’ into simple triple statements.

Knowledge Bases: Challenges

Most KBs are built upon detailed knowledge schemas which prove challenging to populate.

• Schemas describe the forms in which data should be structured within a domain.
• For example, a food recipe KB may break a recipe down into a sequence of dependent steps
  • each step is a complex knowledge structure involving an action, utensils, and quantified inputs and outputs.
• Inputs and outputs might be a tuple of a food type plus a state (e.g. cooked)

SPIRES: Models and Schemas

SPIRES: Goals

• an automated approach for population of custom schemas and ontology models.
• The objective of SPIRES is to generate an instance (i.e. an object) from a text, where that instance has a collection of attribute-value associations.
• Each value is either a primitive (e.g. string, number, or identifier) or another inlined instance
• SPIRES can fill out schemas with linked data while bypassing a need for training examples
• SPIRES also makes use of a flexible grounding approach that can leverage over a thousand ontologies in the OntoPortal Alliance
• This grounding method offers far more consistent mapping to unique identifiers than hallucination-prone LLM querying alone.

SPIRES: Schemas

• Class: a collection of attribute constraints, which control the attribute-value pairs that can be associated with each instance.
• Range of an attribute: specifies the allowed value or values.
  • 1. a primitive type such as a string or number
  • 2. a class
  • 3. an enumeration of permissible value tokens (e.g. an enumeration of days of the week may include ‘Monday’, ‘Tuesday’, and so on).
• cardinality: Specification of the minimum and maximum number of values each instance can take.
• In addition, each schema element can have arbitrary metadata associated with it.

SPIRES algorithm

SPIRES software

• SPIRES allows users to write a YAML template with the target schema
• SPIRES translates this template into a prompt such as the following

SPIRES: Structured LLM responses

• THe above prompt ideally (and usually) will lead to an answer like this:

• it is now relatively easy to parse out the response to get structured data

SPIRES: grounding

Example

• If we request the GO term for integrase activity we expect the response to include GO:0008907, i.e., the ID of this term
• The id is prerequisite for further computational analysis
  • SPIRES returned the correct identifiers for 98 when using GPT-3.5-turbo and 97 with GPT-4-turbo.
  • Without SPIRES, GPT-3.5-turbo returned just 3 correct identifiers.
• That is, without grounding, LLMs tend to hallucinate and return no or the wrong ontology id.
  • This would lead to wrong results in downstream analysis (unless these ids are corrected!)

Original SPIRES vs t-SPIRES

• The original SPIRES has a very rich functionality that makes it easy to extract information as a structured knowledge graph with correct ontology terms using LLMs
• The full approach goes way beyond the scope of this course
• We will therefore preset Toy SPIRES (t-SPIRES), a simplified version of SPIRES that:
  • Creates a prompt that causes LLMs to return results according to a schema
  • Uses a schema that relates Medical Action Ontology (MAxO) terms to Human Phenotype Ontology (HPO) terms with one of two relations
  • Performs grounding on the JSON returned by the LLM to ensure the MAxO and HPO identifiers are correct.
• We are using different and simplified versions of the code and ideas presented in SPIRE, and recommend reading the original paper and using the original code for any serious work!

Results (treats)

• We provided 10 examples of MAxO terms that represent treatments of specific phenotypic abnormalities (HPO terms)
• The labels of the HPO terms are largely correct, but the LLM did not provide a correct MAxO label (or synonym)
• In a full application, we would use an HPO-RAG approach to identify the proper MAxO label and ground the term
• SPIRES performs an analogous step but does not use RAG to retrieve the most likely labels

• The LLM provide correct answers, but the texts only corresponding with the actual MAxO labels for 2 out of 10 texts
• Only two of ten LLM responses were exact matches to MAxO term labels or synonyms (✔️). With two exceptions the other proposed labels were close and very probably could be assigned correctly using RAG
• This shows the need for effective grounding approaches!

Results (diagnoses)

• The LLM provide correct answers for 8 out of 10 texts related to treatment
• The assignment of the relation type was correct in 100% of cases (20/20)

Biomedical knowledge graphs

Summary

• In SUM, RAG-HPO and SPIRES are tools to assign ontology codes and provide structure to biomedical data as parsed from the biomedical literature
• This process accelerates biocuration, makes data more easily amenable to use in software that expects ontology-encoded input data, and contributes to making data FAIR.

Sources

• Background: Huang, Y. et al. (2026) A Survey on Retrieval-Augmented Text Generation for Large Language Models. ACM Comput. Surv 12:1-38
• SPIRES: Caufield JH, et al. (2024) Structured Prompt Interrogation and Recursive Extraction of Semantics (SPIRES): a method for populating knowledge bases using zero-shot learning. Bioinformatics 40:btae104
• RAG-HPO: Garcia BT, et al. (2025) Improving automated deep phenotyping through large language models using retrieval-augmented generation. Genome Med 17:91
• Bhandari P (2024) A Survey on Prompting Techniques in LLMs arXiv 2312.03740
• Yao S et al (2023) Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv 2305.10601