In 1995, Resnik introduced a method for evaluating the semantic similarity between two concepts in an ontology with is_a relations.1
Core Idea: Associate probabilities with the concepts of an ontology.
Definitions:
Let \(\mathcal{C}=\left\{c_1,c_2,\ldots,c_n\right\}\) be the set of concepts in the ontology (permits multiple inheritance).
Define a function \(p:\mathcal{C}\longrightarrow \left[ 0,1\right]\).
\(p(c_i)\) is the probability of encountering an instance of class \(c_i \in \mathcal{C}\).
Ontology Probabilities & Information Content
Example: WordNet Excerpt
Hierarchy: Every pebble is a stone, and every stone is a natural object.
The Root: Every concept is a descendent of entity.
\(p(\text{entity}) = 1\)
Information Content: \(\log 1 = 0\)
Knowing a random term is a descendent of entity provides no added information. Conversely, the probability of a specific term like pebble is much lower, meaning its Information Content is correspondingly higher.
Information Content (IC)
Logic of Inheritance
Recall that: \(x\)instance_of\(A\) AND \(A\)is_a\(B\)\(\implies\)\(x\)instance_of\(B\).
Monotonicity: \(p\) is a monotonically increasing function as we move up the hierarchy.
If \(c_i\)is_a\(c_j\), then \(p(c_i) \leq p(c_j)\).
Root Probability: If the ontology has a unique root node \(r\), then \(p(r)=1\).
Resnik’s Definition
The Information Content (IC) of a concept is the negative log likelihood of its probability:
\[\text{IC}(t) = -\log p(t)\]
Intuition Behind IC
As the probability of a concept increases, its information content decreases. This mirrors information theory:
Common vs. Rare: We gain less new information from an observation of a common event than from a rare one.
The Root Term: Since the root subsumes all concepts:
\(p(\text{root}) = 1\)
\(\text{IC}(\text{root}) = -\log(1) = 0\)
Summary
The more specific a term is (lower probability), the higher its Information Content. The more general it is (higher probability), the lower its Information Content.
Basic Properties of Information Content (IC)
Mathematical Foundation
Information Content relies heavily on the properties of logarithms. These identities are essential for calculating IC across different branches of an ontology.
The following logarithmic identities1 are fundamental to the behavior of \(IC(t) = -\log p(t)\):
\[
\begin{aligned}
\log 1 &= 0 \\
\log xy &= \log x + \log y \\
\log x^r &= r \log x \\
\log \frac{1}{x} &= - \log x \\
\log_a x &= \frac{\log x}{\log a}
\end{aligned}
\]
Entropy
Game plan
This lecture covers a review of artificial neural networks and backpropagation as a prerequisite for understanding transformers and LLMs.
Let us assume we are analyzing a random variable \(X\), which can take on one of a finite number of specific values \(\left\{x_1,x_2,\ldots,x_n\right\}\) with a probability \(\left\{p_1,p_2,\ldots,p_n\right\}\), where the probability of the outcome \(x_i\) is \(p_i\) with \(0\leq p_i\leq 1\) and \(\sum_{i=1}^n p_i=1\).
We will use the notation \(p(X=x)\) or simply \(p(x)\) to denote the probability that the random variable \(X\) takes on some specific value \(x\), where \(x\in
\left\{x_1,x_2,\ldots,x_n\right\}\).
We will refer to each of the specific values that \(X\) can take on as outcomes.
Shannon defined the information content of an outcome \(x\) as
\[
h(x) = \log_2 \dfrac{1}{p(x)}
\tag{1}\]
Entropy (2)
Note that because of the properties of logarithms, \(\log_2 \frac{1}{p(x)} = -\log_2 p(x)\), which is the definition of information content. The entropy of the random variable \(X\), written as \(H(X)\), was defined by Shannon as the average information content of all of the possible outcomes of \(X\).1
Entropy and information content are measured in units called bits (not to be confused with the definition of a bit as a zero or one in computer science).
The entropy of a random variable \(X\) can be interpreted as the uncertainty of the random variable. For instance, if \(X\) has only one outcome (i.e., \(p(x')=1\) for some \(x'\)), then the entropy of \(X\) is zero.
On the other hand the entropy of \(X\) is maximized if there is maximal uncertainty about the outcome – this is the case if no outcome is more likely than the others, or stated differently, if \(X\) follows a uniform distribution.
Entropy of a random variable \(X=\left\{\mathtt{0, 1}\right\}\) representing Bernoulli process such as a coin toss for different values of \(p(X=1)\).
Maximum entropy is achieved if \(p(\mathtt{0}) = p(\mathtt{1}) = 0.5\), i.e., for a uniform distribution.
Proof
We want to show: among all probability distributions on \(n\) outcomes, entropy
\[
H(p) = -\sum_{i=1}^n p_i \log p_i
\]
is maximized by the uniform distribution \(p_i = 1/n\).
Constraint:\(\sum_{i=1}^n p_i = 1\), with \(p_i \geq 0\).
The method of Lagrange multipliers allows us to find maximums/minimums subject to a constraint
The Lagrangian
Introduce a multiplier \(\lambda\) for the normalization constraint:
the right-hand side doesn’t depend on \(i\), so every \(p_i\) must be equal; denoting the the common value \(c\): \[
\sum_{i=1}^n p_i = \sum_{i=1}^n c = nc = 1 \quad \Longrightarrow \quad p_i = \frac{1}{n}\quad \text{for all } i
\]
Since \(p_i > 0\), every diagonal entry is negative, and off-diagonal entries vanish.
\(H(p)\) is therefore strictly concave — the unique critical point is a global maximum, not a saddle or minimum.
\[
H(p) \leq H\left(\tfrac{1}{n}, \dots, \tfrac{1}{n}\right) = \log n
\]
for every probability distribution \(p\) on \(n\) outcomes, with equality iff\(p\) is uniform.
Interpretation: maximal uncertainty about the outcome corresponds exactly to maximal entropy — any deviation from uniformity introduces predictability and strictly lowers \(H\).
Entropy (5)
The function \(IC(x) = -\log p(x)\) is a natural one for measuring information content, for several reasons.
If there is no uncertainty, i.e., if \(p(x=1)\), then \(IC(x)= \log 1 = 0\).
On the other hand, the information content can be arbitrarily large as the probability of an outcome goes to arbitrary small quantities: \(\lim_{x\rightarrow 0}\log x = \infty\).
Say we have two random variables, \(X\) and \(Y\), that are independent from one another. It is reasonable to assume that the information content of the outcomes \(x\) and \(y\) must be additive. This is a natural consequence of the logarithm, because
\[
IC(x,y) = -\log_2 (xy) = -\log_2 x - \log_2 y = IC(x) + IC(y)
\]
Shannon’s Entropy: The Uniqueness Theorem
Claude Shannon’s definition of entropy is not arbitrary; it is the only function that satisfies three fundamental postulates.
Theorem: Uniqueness of Entropy
Let \(P = \{p_1, p_2, \dots, p_n\}\) be the probabilities of outcomes \(\{x_1, x_2, \dots, x_n\}\), where \(p_i \geq 0\) and \(\sum p_i = 1\).
A continuous function \(H(P)\) satisfies the following:
Maximality: \(H\) is maximized when \(p_i = \frac{1}{n}\) for all \(i\) (uncertainty is highest when all outcomes are equally likely).
Additivity: \(H(AB) = H(A) + H_A(B)\). If \(A\) and \(B\) are independent, \(H(AB) = H(A) + H(B)\).
Expandability: \(H(p_1, \dots, p_n) = H(p_1, \dots, p_n, 0)\) (adding an impossible event does not change entropy).
Then \(H\) must take the form:\[H(p_1, p_2, \dots, p_n) = \lambda \sum_{i=1}^n - p_i \log p_i\] where \(\lambda\) is a positive constant (usually \(\lambda=1\)).
Information Content in Gene Ontology (GO)
In the setting of GO, the probability of a term \(t\) is the probability that a randomly chosen gene product (usually: protein) is annotated to that term.
The Root Term: Assuming all genes are annotated to the root (the most general term):
\(p(\text{root}) = 1\)
\(\text{IC}(\text{root}) = -\log_2(1) = 0\)
General vs. Specific:
Terms used for many genes have low IC.
Terms used for very few genes have high IC.
Note
This reflects “surprise”: discovering a gene is annotated to the root is expected, while discovering a rare, specific annotation provides significant information.
IC: Numerical Examples
Assume we have a set of 256 annotated genes:
Genes Annotated to Term
Probability \(p(t)\)
IC calculation (\(-\log_2 p(t)\))
IC Value
256 (All)
\(256/256 = 1\)
\(-\log_2(1)\)
0
4
\(4/256\)
\(-\log_2(4/256) = 8 - 2\)
6
2
\(2/256\)
\(-\log_2(2/256) = 8 - 1\)
7
1 (Unique)
\(1/256\)
\(-\log_2(1/256)\)
8
Semantic Similarity (\(t_1 \leftrightarrow t_2\))
Resnik defined similarity between two terms based on the Information Content of their Most Specific Ancestor (MSA).
\[\text{IC}(t) = -\log p(t)\]
Resnik Similarity
The similarity is the IC of the most specific ancestor \(a\) in the set of common ancestors \(\mathcal{A}(t_1, t_2)\):
Ontologies for Medicine and Translational Research
Core Purpose
Ontologies have two major (interrelated) use cases
Interoperability between databases
Foundation for algorithms and computational tools
Ontologies: Interoperability
Human genetics is becoming a data-driven science
Global Alliance for Genomics and Health (GA4GH) and other groups are defining standards to exchange and integrate information between databases across the globe
Phenotypic descriptions that are very evocative for humans1:
myopathic electromyography
still walking 25 years after onset \end{itemize}
The following descriptions mean the same thing to you: generalized amyotrophy, generalized muscle atrophy, muscular atrophy, generalized, (etc)
Many publications have little2 information about the actual phenotypic features seen in patients with particular diseases and variants
Databases cannot talk to one another about phenotypes
Ontologies: Interoperability
An ontology provides a standardised “Term” for each entity in its domain that can be used for data exchange.
Field
Description
ID
HP:0000256
Name
Macrocephaly
Definition
Occipitofrontal (head) circumference greater than 97th centile compared to appropriate, age matched, sex-matched normal standards. Alternatively, an apparently increased size of the cranium.
Synonyms
Large head, big head, large cranium, large calvaria, Increased head circumference, …
Ontologies are used for myriad applications in biomedical research and translational applications. The greatest common denominator is: Compute over human knowledge!
Applications
Functional profiling in high-throughput experiments
Network biology
Model, reason and manage complex data systems (e.g., IBM Watson)
Natural Language Processing
et cetera
Haendel, Chute, Robinson (2018) Classification, Ontology, and Precision Medicine. N Eng J Med379:1452-1462
The Human Phenotype Ontology
Robinson Am J Hum Genet 2008
Köhler Am J Hum Genet 2009
Robinson Clin Genet. 2010
Schulz BMC Bioinformatics. 2011
Köhler Hum Mutat. 2012
Bauer Bioinformatics 2012
Köhler Nucleic Acids Res 2014
Köhler J Med Genet 2014
Zemojtel Sci Transl Med. 2014
Robinson PLoS Genet. 2014
Robinson Genome Res. 2014
Groza Am J Hum Genet 2015
Groza Database (Oxford). 2015
Smedley Nat Protoc 2015
Smedley Genome Med. 2015
Smedley Am J Hum Genet. 2016
Köhler Nucleic Acids Res 2017
Vasilevsky Nat Genet 2018
Köhler Nucleic Acids Res 2019
Köhler Curr Protoc Hum Genet. 2019
Carmody Orphanet J Rare Dis. 2020
Köhler Nucleic Acids Res 2021
Dhombres Am J Med Genet C 2022
Gargano Nucleic Acids Res 2024
>19,00 terms
244,440 annotations for 8,223 monogenic or chromosomal diseases
The Phenomizer: The first algorithm/software tool for supporting differential diagnostics with semantic similarity analysis
Sebastian Köhler et al. (2009) Clinical Diagnostics with Semantic Similarity Searches in Ontologies. Am J Hum Genet, 85:457–64.
Online version supported 2009-2022…no longer available.
The query
Query terms: The HPO is frequently used to create a set of terms that accurately describe the observed clinical abnormalities of an individualbeing evaluated for suspected rare genetic disease.
Physicians choose terms to represent the clinical anomalies seen in a patient1
Many tools are available. Some tools enable users to choose HPO terms by navigating the HPO hierarchy or by using autocomplete functions.
The query
Various text mining (named entity recognition) tools are available
Disease model: The HPO is frequently used to create a set of terms that accurately describe the observed clinical abnormalities of an individualbeing evaluated for suspected rare genetic disease.
ThHe HPO project has ca. 8500 disease models
Algorithms such as Phenomizer compare the query with each model to determine a ranked list of models that best match the query
Algorithms differ in how they calculate the strength of the match
Information content of Upslanted palpebral fissure is 3.46, that of Downslanted palpebral fissure is 3.01. To calculate the most informative common ancestor, we go to their common parent Slanting of the palpebral fissureHP:0200006, which has an IC of 2.59
Short stature is the direct parent of Proportionate short stature, and so the MICA is the IC of Short stature
Noonan syndrome 1 receives a higher score than Neurodevelopmental disorder 1, and so NS1 would be ranked before NDD1
The same process is performed for all 8500 candidate diseases to generate the final ranking (prioritization)
Example
The table in the previous slide demonstrates how to calculate the phenomizer score \[
\mathrm{sim}(Q\rightarrow D) = \mathrm{avg} \sum_{s\in Q}\max_{t\in D}\mathrm{sim}(s,t)
\] - \(\mathrm{sim}(s,t)\) is calculated as the IC of the maximally informative common ancestor (MICA), see Equation 3.
From raw score to p-value:
Raw similarity scores aren’t directly comparable across diseases:
A disease with many annotated terms has more chances for a lucky match
A small, specific query profile behaves differently than a large, vague one
To make the Phgenomizer score comparable across 8500 diseases, we want to know: How much does a score actually support a disease, given the query’s and the disease’s HPO term count and specificity?
Empirical p-value via simulation
For each candidate disease \(D\):
Generate \(N\)random query profiles of the same size as the observed query (terms sampled from the HPO term/annotation universe)
Compute the fuzzy similarity score between each random query and \(D\), exactly as for the real query
This builds a null distribution of scores specific to \(D\)’s phenotype profile and the query size
Empirical p-value:
\[
p = \frac{\#\{\text{random scores} \geq \text{observed score}\}}{N}
\]
Diseases are then ranked by \(p\)-value rather than raw score, making rankings comparable across diseases with very different profile sizes
This was the basis of the original Phenomizer algorithm
Empirical p-value via simulation
Feature vector (FV) method, ontological similarity search (OSS), and p value (OSS-PV)
Similarly, the probability of the absence of the disease (D–) given the test result is \[
\begin{align}
P(D^{-}\mid x=\mathrm{abn}) &= 1 - P(D^{+}\mid x=\mathrm{abn}) \\
&= \frac{P(x=\mathrm{abn}\mid D^{-})\cdot P(D^{-})}{P(x=\mathrm{abn})} \\
\end{align}
\tag{6}\]
Dividing Equation 5 by Equation 6 and replacing P(D–) with 1 – P(D+) gives: \[
\frac{P(D^{+}\mid x=\mathrm{abn}) }{1-P(D^{+}\mid x=\mathrm{abn}) } =
\frac{P(x=\mathrm{abn}\mid D^{+})}{P(x=\mathrm{abn}\mid D^{-})}
\times
\frac{P(D^{+})}{1 - P(D^{+})}
\]
This is \[
\text{Post-test odds of }D^{+} =
\frac{P(x=\mathrm{abn}\mid D^{+})}{P(x=\mathrm{abn}\mid D^{-})}
\times
\text{Pre-test odds of }D^{+}
\]
This is the equation used in the Bayesian approach to interpret test results.
The factor \(\tfrac{P(x=\mathrm{abn}\mid D^{+})}{P(x=\mathrm{abn}\mid D^{-})}\) is termed the likelihood ratio (LR)
When the test result equals to \(r\), we write \(LR(r)\).
Generally speaking, the likelihood ratio indicates how many times more (or less) likely a certain condition for a test result is expected to be observed in diseased, compared with non-diseased, people
The definition of the LR can be extended to multiple tests. Suppose \(X=(x_1,x_2,\cdots,x_n)\) is an array of \(n\) test results. Under the assumption that the tests are independent, \(\rm{LR}(X)\) is defined as:
Calculating \(P(h_i|\mathcal{D})\), the probability of an individual with disease \(D\) to have HPO term \(h_i\), is easy
If the HPO annotations for \(D\) include the HPO term \(h_i\), then \(P(h_i|\mathcal{D})\) is equal to the frequency of that term in the HPO annotations for the disease
The above line is one of 285603 lines in the phenotype.hpoa file
It specifies the frequency of Micrognathia HP:0000347 in individuals with Auriculocondylar syndrome 2
It is less obvious how to specify \(P(h_i|\neg \mathcal{D})\) - the frquency of \(h_i\) in individuals without disease \(D\).
In principle, we could collect data on the general population, omitting individuals with \(D\), and estimate the population prevalence of \(h_i\)
This is both impossible and also not a good model of the diagnostic situation in medical genetics (individuals who present for diagnosis are not an unbiased selection from the general population)
\(P(h_i|\neg \mathcal{D})\)
LIRICAL therefore makes the assumption that every individual in the population has some Mendelian disease.
Then we can estimate \(P(h_i|\neg \mathcal{D})\) as follows (with a total of \(N\) Mendelian diseases) \[
P(h_i\mid D_j) = \frac{1}{N-1}\sum_{k\neq j} P(h_i\mid D_k) \approx \frac{1}{N}\sum_{k} f_{i}^{D_k}
\]
Here, \(f_{i}^{D_k}\) is the frequency of HPO term \(i\) in individuals with disease \(k\)
This is the key assumption (“trick”) that allows us to apply the likelihood ratio framework to HPO-based differential diagnostic support
There are many details about how to calculate \(f_{i}^{D_k}\) in different situations and how to true-path rule is applied
The next slide gives an impression of the algorithms.1
LIRICAL also introduces an approach to estimate likelihood ratios for genotypes that we will not cover here.
LIRICAL: Calculating the likelihood ratio
Calculating the likelihood ratios for phenotypes
HPO annotations and frequency data: used to calculate \(\dfrac{P(x|\mathcal{D})}{P(x|\neg \mathcal{D})}\), with heuristics for non-exact matches.
Disease model annotated with four HPO terms (dark blue), annotated are “inherited” by ancestors (light blue)
Exact match to query term: \(P(h_{16}|\mathcal{D})=f_{16}^{\mathcal{D}}\)
Query matches descendent of annotated term: \(P(h_{14}|\mathcal{D})=\frac{f_{7}^{\mathcal{D}}}{2}=0.5\)
Query matches ancestor of annotated term: \(P(h_{8}|\mathcal{D})=f_{16}^{\mathcal{D}}\)
LIRICAL: Posterior probabilities improve interpretation of rankings
LIRICAL provides estimates of posttest probabilities
LIRICAL: Performance on 384 published case reports
LIRICAL: Detailed disease output page
Robinson PN et al., Interpretable Clinical Genomics with a Likelihood Ratio Paradigm Am J Hum Genet 2020;107:403-417
free for academic use
