distributional hypothesis

Distributional Hypothesis

The Distributional Hypothesis is that words that occur in the same contexts tend to have similar meanings (Harris, 1954). The underlying idea that "a word is characterized by the company it keeps" was popularized by Firth (1957), and it is implicit in Weaver's (1955) discussion of word sense disambiguation (originally written as a memorandum, in 1949). The Distributional Hypothesis is the basis for Statistical Semantics . Although the Distributional Hypothesis originated in Linguistics, it is now receiving attention in Cognitive Science (McDonald and Ramscar, 2001). The origin and theoretical basis of the Distributional Hypothesis is discussed by Sahlgren (2008).

• Statistical Semantics

External links

• Zellig S. Harris
• Warren Weaver
• Firth, J.R. (1957). A synopsis of linguistic theory 1930-1955. In Studies in Linguistic Analysis , pp. 1-32. Oxford: Philological Society. Reprinted in F.R. Palmer (ed.), Selected Papers of J.R. Firth 1952-1959 , London: Longman (1968).
• Harris, Z. (1954). Distributional structure. Word , 10(23): 146-162.
• McDonald, S., and Ramscar, M. (2001). Testing the distributional hypothesis: The influence of context on judgements of semantic similarity . In Proceedings of the 23rd Annual Conference of the Cognitive Science Society , pages 611-616.
• Weaver, W. (1955). Translation. In W.N. Locke and D.A. Booth (eds.), Machine Translation of Languages , Cambridge, MA: MIT Press.
• Sahlgren, Magnus (2008) The Distributional Hypothesis . Rivista di Linguistica (Italian Journal of Linguistics), 20 (1). pp. 33-53.

Navigation menu

The Distributional Hypothesis: semantic models in theory and practice

Posted on Sat 03 March 2018 in IT

This was the final project for the Data Semantics course at university - A report on distributional semantics and Latent Semantic Analysis. Here is the nicely-formatted pdf version  (with references).

What is the Distributional Hypothesis

When it comes to Distributional Semantics and the Distributional Hypothesis, the slogan is often "You shall know a word by the company it keeps" (J.R. Firth).

The idea of the Distributional Hypothesis is that the distribution of words in a text holds a relationship with their corresponding meanings. More specifically, the more semantically similar two words are, the more they will tend to show up in similar contexts and with similar distributions. Stating the idea the other way round may be helpful: given two morphemes with different semantical meaning, their distribution is likely to be different.

For example, fire and dog are two words unrelated in their meaning, and in fact they are not often used in the same sentence. On the other hand, the words dog and cat are sometimes seen together, so they may share some aspect of meaning.

Mimicking the way children learn, Distributional Semantics relies on huge text corpora, the parsing of which would allow to gather enough information about words distribution to make some inference. These corpora are treated with statistical analysis techniques and linear algebra methods to extract information. This is similar to the way humans learn to use words: by seeing how they are used (i.e. coming across several examples in which a specific word is used).

The fundamental difference between human learning and the learning a distributional semantic algorithm could achieve is mostly related to the fact that humans have a concrete, practical experience to rely on. This allows them not only to learn the usage  of a word, but to eventually understand  its meaning. However, the way word meaning is inferred is still an open research problem in psychology and cognitive science.

The weak and strong hypothesis

Before going any further, it is important to keep in mind that there are two versions of the distributional hypothesis, as pointed out by A. Lenci.

In the weak version of the hypothesis, the distribution of words is believed to be a way of inferring their semantic behaviour. Let us stress the term behaviour  in the previous sentence. The weak hypothesis does not state at any rate that the distribution of words will, in any way, tell something about the meaning of words. In fact, we may only hope (and this would already be a success!) to unveil the ways a word is used .

The idea here is that the semantic meaning of words has a specific effect on their distribution. By observing the distribution, we may then hope to infer something about the meaning. The distribution is thus seen as a latent variable  of the semantic meaning: observing the former may give hints about the latter.

The strong form of the hypothesis, on the other hand, states that words distribution have a causal relationship  with the way meaning is derived from the text. In fact, this theory entails that the statistical distribution of a word causes the semantic representation humans have of the corresponding idea.

In other words, while the weak version of the distributional hypothesis states that there is a relationship such that

but does not say anything about the inverse, the strong version believes that to hold as well.

Perhaps not surprisingly, the weak version is accepted by far more researchers of the (computational) linguistics fields than the strong one. It is not difficult to believe that meaning has a strong influence on the distribution of words, but it is not so easy to believe that distributions shape meaning by themselves.

For example, consider the following sentence:

"I hiked 2000 metres uphill in the mountains to have a swim in the lake".

Knowing the meaning of the word sea , humans (hopefully) know that the word lake  cannot just be replaced with the word sea  in the sentence above, even if there is the word swim , which is often seen together with sea . However, if we believed that distribution shapes meaning, then words with supposedly very similar distributions such as lake and sea  could be swapped one with the other. This is not the case, although this is only clear if the meaning of the context is understood.

As we will see, the relationship between distribution and meaning with respect to the distributional hypothesis is a difficult one, and we will discuss it in more depth later on.

How to deploy the idea

In order to investigate lexical meaning, we first need to set up a way to measure semantic similarity between words. When are two words similar? What does it even mean for two words to be similar ? These are important questions, as word similarity is believed to affect the way terms are processed in mental lexicon. We may define synonymy as being related to word interchangeability, but this is not a computationally implementable definition yet.

Thus, a way to make explicit the functional dependance between word distribution and semantic is needed.

Distributional semantics models rely on statistical analysis and linear algebra tools to implement its theory. We will now discuss in more depth Latent Semantic Analysis and hint at further developments.

Latent Semantic Analysis (LSA)

Figure 1 - Dot products between words (Evert)

Latent Semantic Analysis (LSA) is arguably the mathematical tool of distributional semantics. In its basic form, it allows to parse several texts and analyze similarities between them. It is an unsupervised learning method (i.e. it requires no human-preprocessed data). Although it is not the only possible method, others are roughly based on the same mathematics that LSA employs, so exploring the latter is enough to understand other approaches.

The idea of LSA is to regard words as points in a so-called distributional space . The distributional space is simply a vector space (most often \(\mathbb{R}^n\) ), where vectors represent words. The vector contains an average of all possible different words usages deduced from the input text corpora.

However simple this construction may seem, it has a very nice property: that distances between points correspond to semantic distances between words. In particular, measuring angles between vectors (which is as simple as calculating dot products) is a proxy for measuring the similarity of the corresponding words (see Figure 1).

An example of the result of text corpora processing through Latent Semantic Analysis is shown in Figure 2.

As an aside, notice that the name of this technique comes from the idea that semantic may be thought of as a latent variable responsible for the observed linguistic distributions.

Figure 2 - Latent Semantic Analysis-based grouping of documents basing on their keywords (Wikipedia)

How it is done

We briefly go through how Latent Semantic Analysis works from a technical point of view.

Figure 3 - Matrix example of LSA.

Consider \(n\) documents, which contain \(m\) different words all together. LSA starts by building a matrix of size \(m \times n\) :

Each row \(t_i^T\) represents a word, while each column \(d_j\) represents a document. The matrix entry \(x_{i,j}\) holds the occurrence count (or frequency) of the word \(i\) in the document \(j\) .

It is then possible to calculate similarity both between terms and between documents. In fact, the dot product \(t_i^T t_p\) computes the similarity between the word vectors \(i\) and \(p\) , whereas \(d_j d_k^T\) measures similarity between documents \(j\) and \(k\) .

It is easy to generate matrices of co-occurrence both for words and for documents, through the computation respectively of \(X^TX\) and of \(XX^T\) . In particular, \(X^TX\) will contain information about the similarity of words, basing on the provided corpora; \(XX^T\) will contain similar information about the documents.

However, the matrix generated by LSA is likely to be huge in size, and thus computationally difficult to handle. Furthermore, a good part of it may well consist of useless data: noise, synonyms and other information that may be stripped away without any concrete loss.

For these reasons, Singular Value Decomposition is applied to the matrix to extract the most meaningful pieces of information from it. In this way, similarity measures will still be computable, though with less computation effort.

However, the drawback is that the new dimensions are very unlikely to relate to any comprehensible concept: they will just be linear combinations of rows and columns.

To make this concern clearer, consider the following example.

{(car), (bottle), (flower)} \(\rightarrow\) {(1.3452 \* car + 0.2828 \* bottle), (flower)}

Here, two words are merged in one new joint dimension, while the third is retained as it is. However, although the interpretation of car and bottle is clear, it is not clear at all how the joint dimension should be thought of.

Critiques and more sophisticated approaches

As we have seen, LSA is able to generate co-occurrence patterns from the input it is provided. However, the idea that meaning may consist of abstractions from purely linguistic co-occurrence patterns has raised many critiques. As already pointed out, the biggest one is that co-occurrence information may only allow to infer the usage of a word, but not its meaning.

Moreover, one concern is understanding whether the current limits and weak points of the distributional semantics approaches are due to the implementations and current models. On the other hand, it may be possible that they are inherent to the distributional hypothesis itself.

When using Latent Semantic Analysis, further rules may be enforced. In fact, although text corpora may be used as they are in their raw form ( naive approaches), syntactically-savy models are possible as well. In these latter models, the distribution is analysed by keeping in mind the syntactic configurations words should obey to. In this way it should be possible to get closer to the ways terms are used in reality.

It is worth pointing out that LSA-based methods are often the starting point of distributional semantic investigations, but more complex methods have been devised on top of it.

One example of further technique consists on focusing on the notion of property of a concept, to try to capture the essence of its meaning. In this model, animal would be identified as property for dog , and wheels as property for car .

Another possible approach is that of semantic association. In this fashion, swim would find a relationship with water , as the former calls to mind the latter.

Critiques to the distributional approach

As we have seen, the distributional hypothesis in its weak form is usually accepted by field experts, but there is strong disagreement over the strong formulation.

One fundamental critique to the distributional hypothesis is that, as far as it could go, distributional semantic models may not go beyond learning word usage. In fact, learning words meaning is totally out of their scope. This is well conveyed by the Chinese Room thought experiment, first proposed by Searle in 1980:

Imagine to be locked in a room and to receive batches of Chinese characters. Crucially, you do not know Chinese and for you these characters are just meaningless symbols. You also receive "rules" (in English) on how to combine these characters, and how to respond to Chinese characters with other Chinese characters. Suppose that, after a certain amount of training, you have been able to learn to combine Chinese characters and to reply to Chinese messages in such a way that your answers are de facto indistinguishable from those given by Chinese native speakers. The key point is that even in this case it would still be true that you do not understand Chinese, even though you may be said to "speak it".

Another point of difficulty is that of composivity. By learning the meaning of the parts (i.e. the single words that make up a sentence), is it possible to learn the meaning of the whole sentence?

As we have seen, technical implementations of the theory (Latent Semantic Analysis) usually rely on basic linear algebra and vector spaces. In these models, each word is conceived as a point. However, the sum of the points representing the words of a sentence is an absolutely inadequate definition for the meaning of the sentence. This is because vector summation is commutative, whereas word summation should not.

In fact, consider the following examples:

1. The dog ate a cake. 2. The cake ate a dog. 3. The a cake dog ate.

If word-vector summation would be enough to define sentence meaning, then the three sentences would be believed to all have the same meaning, although this is clearly not the case. Remarkably, the third sentence does not even have any real meaning at all, although it is believed to be equal to the other two. The point here is that while swapping vectors in a sum is fine, swapping words in a sentence obviously is not.

More sophisticated models that account for non-commutativity in words usage have been developed, but this still looks like an open problem.

One further question, although weakly related to that of non-commutativity, is that of asymmetric relationships. For example, sentences like:

A bus has wheels.

allow to learn that there is a strict relationship between buses  and wheels , but it is not clear how to learn the fact that it is an asymmetric link. Indeed, it is difficult to find a bus without wheels, but it is not difficult at all to find wheels without a bus.

Moreover, experts argue that meaning is most often influenced by factors that cannot be learnt just by the textual representation of content. In fact, the speaker's intentions, the speaker's own way of expression and the context in which the content fits into all shape the meaning of the words it contains. The statistical methods employed by distributional semantics are not able to capture these factors, as they lay beyond its scope.

Also, keep in mind that up to now we have only considered well-formed language instantions, i.e. sentences in which the meaning exactly corresponds to the words used to express it. However, consider the following sentence:

I borrow books and sometimes they do not come back to me.

A human skilled enough with the English language easily understands that the speaker mistaked the word borrow  with the word lend , but would a computer be able to do this?

Ultimately, making inferences given a group of facts is the final goal of learning the meaning of words. However, it is unclear how this could be achieved without a real understanding of semantic meaning. In fact, one of the main critiques to distributional semantics is precisely this: that it cannot provide meaning for words.

Finally, it should be noted that the distributional semantics debate is just an instance of a broader debate. The two contrasting theories in this regard are the Abstract Cognition Hypothesis (ACH) and the Embodied Cognition Hypothesis (ECH).

Standing to Abstract Cognition Hypothesis, concepts and meanings are represented in human cognition by formal symbols. This stance supports the Distributional Hypothesis. On the other hand, the Embodied Cognition Hypothesis believes that meanings are stored in the same perceptual system from which its instance is experienced. This means that knowing the meaning of the word spider  implies being able to run some sort of internal simulator that re-enacts out concrete experience with spiders. This theory, of course, is in sharp contrast with the distributional hypothesis and its tools, since requires a deeper understanding of words, that goes beyond their usage.

Final remarks

As we have discussed at length, even if distributional-based semantic models could tell us something interesting about the language and words usage, it would reveal absolutely nothing about meanings. In fact, one could build graphs, buckets and at any rate classify words in fashionable and complex ways, but it seems pretty unlikely that the meaning of a word could be extracted (just) from those kind of analysis. This is the  vital problem of distributional semantics.

However, even though philosophical debates are far from settled, distributional semantics models have allowed good results in real-world applications, and so are used notwithstanding the critiques. Indeed, using vocabulary tests to compare the performance of Latent Semantic Analysis based models with the learning achieved through reading by school children, it was found that they scored similarly.

After all, it is important to realize what our goals are. If we would like a machine to understand ideas, concepts and words as a human would do, then there is probably no technique that would allow this in the current state of things. However, if we only require a machine to process and respond to linguistic queries, then distributional models can be used with a good success rate.

To make an analogy with another field, let us notice that we do not know how (artificial) neural networks work, although we use them and rely on them in more and more contexts. Crucially, we can not state that they are able to think  or understand  the work they do. For example, when searching for tieleman orchestra , Google suggests thielemann orchestra  instead. Do Google's algorithms know who Thielemann is, or that he is a director? Do they know what a director even is?

All of this is to say: then, why should distributional models not be used, if they have proved to be good enough to infer words usage, which is what matters in practice? Our lives lay on top of automated systems that work, although they do not understand their work. Distributional semantic models may well become one of these tools.

The difficulty of devising computational models that would extract meanings from terms and contexts is mostly related to the fact that we ourselves do not yet understand how we do it. It is unlikely, of course, that we will ever be able to teach a machine to do it without understanding it ourselves first.

At any rate, is seems unlikely (at least as of now) to ever be able to teach stupid machinery to learn meanings and concepts.

The fact that we are able to understand  the word circle , for example, has undoubtedly to do with our past experience and interaction with the real word: we have seen round things, balls and other different geometric shapes, and have an intuitive idea  of what a circle is. This allows us not only to use the word, but to have a real understanding of, which stands at a higher level. In fact, one may argue that the understanding that an 8-year-old could have of circle  would be at a superior level than any distributional model!

In other words, it is just difficult to believe that a simple algorithm/technique may be able to compete with the human conceptualization of the world. After all, just observing the statistical occurrence of the word idea  may make it look like it has a lot to do with lamps  and lights .

As Wittgenstein said, "the meaning of a word lies in its use", and looking at usage may be enough to replicate usage patterns. However, grasping and understanding that meaning looks like a totally different matter!

[References]

[1] Lenci Alessandro 2008. Distributional semantics in linguistic and cognitive research. Rivista di Linguistica 20.1, pp. 1-31

[2] Evert Stefan – Distributional Semantic Models (last accessed: 2018/02/26) http://esslli2016.unibz.it/wp-content/uploads/2015/10/dsm_tutorial_part1.slides.pdf

[3] Wikipedia – Latent Semantic Analysis (last accessed: 2018/02/26) https://en.wikipedia.org/wiki/Latent_semantic_analysis

[4] Landauer Thomas K., Susan T. Dumais 1997. A solution to Plato’s problem: the latent semantic analysis theory of acquisition, induction and representation of knowledge. Psychological Review CIV/2. 211-240

[5] Searle John 1980. Minds, brains and programs. Behavioural and Brain Sciences III. 417-424

[6] Wittgenstein Ludwig 1953. Philosophical Investigations. Oxford: Blackwell

[7] Firth John R. 1957. Papers in Linguistics. London, Oxford University Press

Distributed Representations

• First Online: 11 June 2019

Cite this chapter

• Uday Kamath 4 ,
• John Liu 5 &
• James Whitaker 4  

8938 Accesses

In this chapter, we introduce the notion of word embeddings that serve as core representations of text in deep learning approaches. We start with the distributional hypothesis and explain how it can be leveraged to form semantic representations of words. We discuss the common distributional semantic models including word2vec and GloVe and their variants. We address the shortcomings of embedding models and their extension to document and concept representation. Finally, we discuss several applications to natural language processing tasks and present a case study focused on language modeling.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Faisal Alshargi et al. “Concept2vec: Metrics for Evaluating Quality of Embeddings for Ontological Concepts.” In: CoRR abs/1803.04488 (2018).

Google Scholar  

Waleed Ammar et al. “Massively Multilingual Word Embeddings.” In: CoRR abs/1602.01925 (2016).

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. “Neural machine translation by jointly learning to align and translate”. In: CoRR abs/1409.0473 (2014).

Amir Bakarov. “A Survey of Word Embeddings Evaluation Methods”. In: CoRR abs/1801.09536 (2018).

Yoshua Bengio et al. “A neural probabilistic language model”. In: JMLR (2003), pp. 1137–1155.

Piotr Bojanowski et al. “Enriching Word Vectors with Subword Information”. In: CoRR abs/1607.04606 (2016).

Antoine Bordes et al. “Translating Embeddings for Modeling Multirelational Data.” In: NIPS . 2013, pp. 2787–2795.

José Camacho-Collados and Mohammad Taher Pilehvar. “From Word to Sense Embeddings: A Survey on Vector Representations of Meaning”. In: CoRR abs/1805.04032 (2018).

Ting Chen et al. “Entity Embedding-Based Anomaly Detection for Heterogeneous Categorical Events.” In: IJCAI . IJCAI/AAAI Press, 2016, pp. 1396–1403.

Ronan Collobert and Jason Weston. “A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning”. In: Proceedings of the 25th International Conference on Machine Learning . ACM, 2008, pp. 160–167.

Marta R. Costa-Jussà and José A. R. Fonollosa. “Character-based Neural Machine Translation.” In: CoRR abs/1603.00810 (2016).

Jocelyn Coulmance et al. “Trans-gram, Fast Cross-lingual Word embeddings”. In: CoRR abs/1601.02502 (2016).

Jacob Devlin et al. “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.” In: CoRR abs/1810.04805 (2018).

Paramveer S. Dhillon, Dean Foster, and Lyle Ungar. “Multiview learning of word embeddings via cca”. In: In Proc. of NIPS . 2011.

Bhuwan Dhingra et al. “Embedding Text in Hyperbolic Spaces”. In: Proceedings of the Twelfth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-12) . Association for Computational Linguistics, 2018, pp. 59–69.

Manaal Faruqui et al. Retrofitting Word Vectors to Semantic Lexicons . 2014.

Edouard Grave et al. “Learning Word Vectors for 157 Languages”. In: CoRR abs/1802.06893 (2018).

Jiatao Gu et al. Incorporating Copying Mechanism in Sequence-to-Sequence Learning . 2016.

Jeremy Howard and Sebastian Ruder. “Universal Language Model Fine-tuning for Text Classification”. In: Association for Computational Linguistics, 2018.

Armand Joulin et al. “Bag of Tricks for Efficient Text Classification”. In: CoRR abs/1607.01759 (2016).

Ramakrishnan Kannan et al. “Outlier Detection for Text Data: An Extended Version.” In: CoRR abs/1701.01325 (2017).

Yoon Kim et al. “Character-Aware Neural Language Models”. In: AAAI . 2016.

Anoop Kunchukuttan and Pushpak Bhattacharyya. “Learning variable length units for SMT between related languages via Byte Pair Encoding.” In: CoRR abs/1610.06510 (2016).

Maximilian Lam. “Word2Bits - Quantized Word Vectors”. In: CoRR abs/1803.05651 (2018).

Quoc V. Le and Tomas Mikolov. “Distributed Representations of Sentences and Documents”. In: CoRR abs/1405.4053 (2014).

Wang Ling et al. “Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation.” In: CoRR abs/1508.02096 (2015).

Minh-Thang Luong and Christopher D. Manning. “Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models.” In: CoRR abs/1604.00788 (2016).

Tomas Mikolov et al. “Distributed Representations of Words and Phrases and their Compositionality”. In: Advances in Neural Information Processing Systems 26 . 2013, pp. 3111–3119.

Andriy Mnih and Geoffrey E Hinton. “A scalable hierarchical distributed language model”. In: Advances in neural information processing systems . 2009, pp. 1081–1088.

Arvind Neelakantan et al. “Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space.” In: EMNLP . ACL, 2014, pp. 1059–1069.

Maximillian Nickel and Douwe Kiela. “Poincaré Embeddings for Learning Hierarchical Representations”. In: Advances in Neural Information Processing Systems 30 . Curran Associates, Inc., 2017, pp. 6338–6347.

Masataka Ono, Makoto Miwa, and Yutaka Sasaki. “Word Embedding based Antonym Detection using Thesauri and Distributional Information.” In: HLT-NAACL . 2015, pp. 984–989.

Jeffrey Pennington, Richard Socher, and Christopher D. Manning. “GloVe: Global Vectors for Word Representation”. In: Empirical Methods in Natural Language Processing (EMNLP). 2014 , pp. 1532–1543.

Sebastian Ruder, Ivan Vulic, and Anders Sogaard. A Survey Of Cross-lingual Word Embedding Models . 2017.

Tianze Shi and Zhiyuan Liu. “Linking GloVe with word2vec.” In: CoRR abs/1411.5595 (2014).

Andrew Trask, Phil Michalak, and John Liu. “sense2vec - A Fast and Accurate Method for Word Sense Disambiguation In Neural Word Embeddings.” In: CoRR abs/1511.06388 (2015).

Ashish Vaswani et al. “Attention is all you need”. In: Advances in Neural Information Processing Systems . 2017, pp. 5998–6008.

Luke Vilnis and Andrew McCallum. “Word Representations via Gaussian Embedding.” In: CoRR abs/1412.6623 (2014).

Download references

Author information

Authors and affiliations.

Digital Reasoning Systems Inc., McLean, VA, USA

Uday Kamath & James Whitaker

Intelluron Corporation, Nashville, TN, USA

You can also search for this author in PubMed   Google Scholar

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Kamath, U., Liu, J., Whitaker, J. (2019). Distributed Representations. In: Deep Learning for NLP and Speech Recognition . Springer, Cham. https://doi.org/10.1007/978-3-030-14596-5_5

Download citation

DOI : https://doi.org/10.1007/978-3-030-14596-5_5

Published : 11 June 2019

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-14595-8

Online ISBN : 978-3-030-14596-5

eBook Packages : Computer Science Computer Science (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Advertisement

• Previous Article
• Next Article

1 Introduction

2 related work, 3 theoretical framework, 4 formal definition and properties, 5 function analysis and generalization, 6 experiments, 7 conclusions, appendix a. proofs, acknowledgments, information theory–based compositional distributional semantics.

Action Editor: Kevin Duh

• Cite Icon Cite
• Open the PDF for in another window
• Permissions
• Article contents
• Figures & tables
• Supplementary Data
• Peer Review
• Search Site

Enrique Amigó , Alejandro Ariza-Casabona , Victor Fresno , M. Antònia Martí; Information Theory–based Compositional Distributional Semantics. Computational Linguistics 2022; 48 (4): 907–948. doi: https://doi.org/10.1162/coli_a_00454

Download citation file:

• Ris (Zotero)
• Reference Manager

In the context of text representation, Compositional Distributional Semantics models aim to fuse the Distributional Hypothesis and the Principle of Compositionality. Text embedding is based on co-ocurrence distributions and the representations are in turn combined by compositional functions taking into account the text structure. However, the theoretical basis of compositional functions is still an open issue. In this article we define and study the notion of Information Theory–based Compositional Distributional Semantics (ICDS): (i) We first establish formal properties for embedding, composition, and similarity functions based on Shannon’s Information Theory; (ii) we analyze the existing approaches under this prism, checking whether or not they comply with the established desirable properties; (iii) we propose two parameterizable composition and similarity functions that generalize traditional approaches while fulfilling the formal properties; and finally (iv) we perform an empirical study on several textual similarity datasets that include sentences with a high and low lexical overlap, and on the similarity between words and their description. Our theoretical analysis and empirical results show that fulfilling formal properties affects positively the accuracy of text representation models in terms of correspondence (isometry) between the embedding and meaning spaces.

The representation of text meaning is a fundamental issue in Natural Language Processing (NLP). It involves encoding natural language in a way that can be handled by information management systems. It is one of the main bottlenecks in Textual Information Access, Text Mining, Dialogue Systems, and so forth. The conflict between compositionality and contextuality (Frege’s principles) exhibits a tension between the meaning representation paradigms, namely, symbolic and distributional (Maruyama 2019 ).

The Principle of Compositionality states that the meaning of a whole is a function of the meaning of its parts and the syntactic way in which they are combined together. This principle is the main foundation of the symbolic representation paradigm, which attempts to relate language to propositional logic via extended semantic references (word referential meaning) and grammars that capture the language structures. For instance, in the symbolic paradigm, “table” and “sitting” are associated with their references (icons in Figure 1a ) and the utterance “sitting at a table” is the result of being combined by means of a syntactic structure.

Representation paradigms in computational linguistics.

On the other hand, The Principle of Contextuality states that the meaning of words and utterances 1 is determined by their context. This principle supports the distributional representation paradigm, in which the meaning of linguistic items is inferred from their context of use. Rather than symbols, words or utterances are represented as points in a continuous vector space (embeddings; i.e., vectors in Figure 1b ). For instance, the words “table” and “sitting,” and the utterance “sitting at a table” are projected in the representation space according to the textual context in which they usually appear. In contrast to the symbolic paradigm, the distributional paradigm interprets the meaning space as a continuous, that is, there exists a graded scale of meaning representations between “sitting,”“table,” and “sitting at a table.”

Over the last decade, distributional NLP approaches have undoubtedly been the predominant basis of NLP applications thanks to their predictive power over a sufficient amount of textual data. In particular, Neural Language Models, such as Generative Pre-trained Transformer (GPT) (Radford et al. 2019 ) and Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al. 2019 ), pre-trained over a huge amount of parameters and text collections, have demonstrated high performance on specific tasks with limited supervised data. However, in terms of meaning representation, there is considerable consensus in the literature that one of the main limitations of the distributional paradigm is its lack of systematicity (Johnson 2004 ; Talmor et al. 2020 ; Pimentel et al. 2020 ; Goodwin, Sinha, and O’Donnell 2020 ; Hupkes et al. 2020 ; Bender and Koller 2020 ). Robert Cumming defined language systematicity as follows (Cummins 1996 , page 591): A system is said to exhibit systematicity if, whenever it can process a sentence, it can process systematic variants, where systematic variation is understood in terms of permuting constituents or (more strongly) substituting constituents of the same grammatical category . Briefly, while in the symbolic paradigm the productive rules defining grammars admit infinite combinations and permutations, the distributional paradigm is somewhat conditioned by the word sequences observed in the training corpus.

Because distributional and compositional representation approaches present complementary properties (contextuality vs. systematicity), Maruyama ( 2019 ) raised the need for a Kantian Synthesis . The hypothesis is that the meaning of linguistic forms is given by a reflow between their context and components. In this line, a number of authors have proposed adding a compositional layer over distributional representations. These models are referred to as Compositional Distributional Semantic models (Mitchell and Lapata 2010 ; Arora, Liang, and Ma 2017 ; Clark and Pulman 2007 ; Coecke, Sadrzadeh, and Clark 2010 ). As shown in Figure 1c , in the distributional compositional paradigm, the utterance “sitting at a table” is not inferred from the context in which it appears, but by a composition function applied on the distributional representations of “sitting” and “table.” In other words, context of words determines their meaning. At the same time, the meaning of the utterance is given by the words that compose it and compositional operators. This allows us to exploit the compositionality systematicity on utterances whose length does not allow us to infer usage statistics.

The main goal of this article is the formal study of embedding, composition, and similarity functions in the context of Compositional Distributional representation models, allowing for the analysis of the problem independently of the task in which the representation is applied. For this purpose, we formalize the notion of Information Theory–based Compositional Distributional Semantics , the contribution of which to Distributional Compositional models consists of establishing formal constraints for embedding, composition, and similarity functions ( Figure 1d ). These constraints are based on Shannon’s concept of Information Content (IC), establishing a link between distributional semantics, compositional language structures, and Information Theory.

We apply the following methodology. We first establish formal properties for embedding, composition, and similarity functions based on Shannon’s Information Theory. Then we analyze the existing approaches under this prism, checking whether or not they comply with the established desirable properties. After this, we propose two parameterizable composition and similarity functions that generalize traditional approaches while fulfilling the formal properties. Finally, we perform an empirical study on several textual similarity datasets that include sentences with a high and low lexical overlap, and on the similarity between words and their description. Our theoretical analysis and empirical results show that fulfilling formal properties affects positively the accuracy of text representation models in terms of correspondence between the embedding and meaning spaces.

The article is structured as follows. Section 2 analyzes the main existing semantic approaches from a historical perspective and reviews Compositional Distributional Semantics models in more detail. Sections 3 and 4 describe the proposed theoretical framework for Information Theory–based Compositional Distributional Semantics (ICDS). Section 5 describes the proposed generalized similarity and composition functions, their properties, and their connection with existing functions. Section 6 displays our experiments and results. Finally, we draw our conclusions in Section 7 .

2.1 Text Representation Paradigms

To get an overview of the different text representation paradigms, we will consider the following set of properties.

Whenever the representation is able process a sentence, it can process systematic variants, where systematic variation is understood in terms of permuting constituents or (more strongly) substituting constituents of the same grammatical category.

The meaning representation is sensitive to the contexts in which the expression appears. There is a consensus among psycholinguists that the semantics of the basic units of language (words, collocations) is determined by their use (Wittgenstein 1953 ). Put in computational linguistic terms, this is known as the Distributional Hypothesis (Harris 1954 ; Firth 1957 ).

There is a multidimensional continuous space where the meaning representations are mapped. The distributional approach offers us a way to capture the full continuum of meaning (Erk 2009 ; Gladkova and Drozd 2016 ). Moreover, this space should be isometric with respect to the meaning affinity of expressions, that is, it should reflect semantic similarities within the space of embeddings. The continuity property offers important advantages, such as the ability to generalize by proximity in space and to incorporate dimensionality reduction mechanisms, and so on (Landauer 1997 ).

The representation system includes a function that measures the amount of information contained in the represented utterance. In general, the amount of information is measured in terms of specificity or likelihood, in concordance with Shannon’s notion of Information Content ( ⁠ I ( x ) = − log ( P ( x ) ⁠ ). This links the representation system with the notion of language model, and is a key aspect in representation models (Zhai 2008 ) and textual similarity measures (Lin 1998 ).

In this section, we will show that, since the beginning of NLP, these properties have been captured or lost as new paradigms have been established. The loss of properties in each paradigm was compensated for by the use of different techniques. Table 1 summarizes this historic evolution.

Semantic representation approaches and their properties. The table includes techniques used to mitigate drawbacks.

The first semantic models were based on logicist approaches assuming the Principle of Compositionality (e.g., via Preference Semantics [Wilks 1968 ]). These models establish a univocal relationship between a symbol (or a set of symbols) in context and its meaning (Boleda and Erk 2015 ). The space of senses is given by ontological resources such as WordNet (Fellbaum 1998 ). The symbolic paradigm is inherently systematic, since it connects with propositional logic. The other aspects are only partially captured. First, the compositional paradigm is fundamentally based on rules, not on the distributional features of language ( Usage Context ); however, the context of use is partially captured by techniques such as Lexical Sense Disambiguation (Navigli 2009 ) that maps words to senses according to their context, and Probabilistic Grammars (Sekine and Grishman 1995 ) trained on text corpora. Although the semantic space in the symbolic paradigm is discrete, the lack of Continuity is partially mitigated by means of similarity metrics between nodes within an ontology, for example, via Conceptual Density (Agirre and Rigau 1996 ) in WordNet. The lack of Information Measurability at the lexical level within these models has also been addressed via the deepness of nodes (specificity) in the hierarchical ontology (Seco, Veale, and Hayes 2004 ).

From the 1980s onward, the Statistical Paradigm, whose representation system was fundamentally based on the Vector Space Model (VSM) (Salton and Lesk 1965 ), gained strength. Under this paradigm, texts are represented as a bag of independent words, disregarding word order and grammar. Consequently, Systematicity is not captured, since the structures of the language are ignored. Some approaches incorporate a certain level of Systematicity by means of objects more complex than simple vectors, such as tensors (Baroni and Lenci 2010 ; Padó and Lapata 2007 ; Turney 2007 ). The natural language property that the VSM incorporates is Continuity on which similarity measures are applied, such as cosine, dot product, or Euclidean distance. Because the original VSM assumes word independence, it does not capture Usage context . This lack is tackled via dimensionality reduction approaches based on the Distributional Hypothesis, such as Latent Semantic Indexing (Deerwester et al. 1990 ) or Latent Dirichlet Allocation (Blei et al. 2003 ). 2 Regarding Information Measurability , the Term Frequency–Inverse Document Frequency (TF-IDF) term weighting function and its variants (such as projection functions that are within vector space) approaches the probability of the represented utterance and have a direct connection with the notion of Information Content in Shannon’s theory (Robertson 2004 ).

The next generation of meaning representation consisted of the Count-based Language Models. In these models, texts are represented as word sequences and their probability distribution (Andreas, Vlachos, and Clark 2013 ). Information Measurability is captured via perplexity , a concept directly based on Shannon’s notion of Information Content. Language Models are mainly usage-oriented, since perplexity is based on the frequency of words in given contexts. Texts are represented as a whole (holistic) in terms of the word sequence and its probability. Consequently, the lack of Systematicity is a limitation of Language Models because the probability of a word sequence cannot be inferred from the probability of its parts (Bengio et al. 2003 ). Just like in the VSM, a way of mitigating this lack consists in assuming statistical independence and applying the product of probabilities; this is the case of the n -gram model (Brown et al. 1993 ). Regarding semantic Continuity , Language Models represent texts as token sequences rather than a point in a continuous space. In addition, they do not provide a direct notion of similarity between word sequences, but between different Language Models (e.g., Kullback-Leibler divergence) or sequences vs. Language Models (perplexity).

Lastly, the most successful approaches over the last few years are supported by Neural Language Models, which are based on neural networks pre-trained over a huge text corpus on the basis of Usage Context (Devlin et al. 2019 ; Brown et al. 2020 ). Neural Language Models combine the properties of VSMs and Language Models. On the one hand, text embedding is given by the activation state of neural network inner layers. Therefore, as in the VSM, the representation space is a continuum, allowing generalization. Just like traditional Language Models, the network is trained to predict words, estimating a probabilistic distribution of word sequences given their context. In consequence, Neural Language Models capture both Usage Context and Information Measurability . Their limitation is still Systematicity . Finally, the Compositional Distributional paradigm covers all properties and is discussed in detail in the next section.

In Table 1 , a summary of the semantic models and the linguistic features of language that they address is shown. In view of this table, we can conclude that Continuity , Information Measurability , Usage Context , and Systematicity are complementary desirable properties that have been addressed from different paradigms, although covering all of them simultaneously remains a challenge for the Natural Language Processing community.

2.2 Distributional Semantic Representation Models: From Word Embeddings to Transformers

In 2008, Collobert and Weston ( 2008 ) demonstrated that word embeddings generated from a suitably large dataset carry syntactic and semantic meaning and improve the performance on subsequent NLP tasks. However, the hype regarding neural encoders really started in 2013 with the distributional semantic model known as Skip-gram with Negative Sampling (SGNS, Word2Vec package) (Mikolov et al. 2013 ). Some authors proposed the extension of this approach to represent longer linguistic units, such as sentences (Kiros et al. 2015 ) or documents (Le and Mikolov 2014 ; Kenter, Borisov, and de Rijke 2016 ). Static Neural Models (also referred to as Non-Contextual Neural Models) such as SGNS or GloVe (Pennington, Socher, and Manning 2014 ) optimize the correspondence between the scalar product of embeddings and their distributional similarity (Mutual Information) (Levy and Goldberg 2014 ; Le and Mikolov 2014 ; Arora et al. 2016 ). Assuming the distributional hypothesis (similar words appear in similar contexts), this ensures a certain isometry between the embedding space and meanings.

These embedding approaches are static. That is, the representation of each word is fixed regardless of the particular context in which it appears. The second generation of encoders are sequential models that are sensitive to the word sequence order. The Long Short Term Memory (LSTM) model (Hochreiter and Schmidhuber 1997 ) and its variants introduced a memory cell that is able to preserve a state over long periods of time. Several authors presented unsupervised, pre-trained, bidirectional LSTM–based encoders, such as CoVe (McCann et al. 2017 ) and ELMo (Peters et al. 2018 ), which can improve the performance on a wide variety of common NLP tasks. Next came graph-based approaches, which use a fully connected graph to model the relation of every two words within the input text. A successful implementation of this idea is the Transformer (Vaswani et al. 2017 ). In the Transformer, the neural network is pre-trained over a large text corpus under different self-supervised tasks such as Masked Language Modeling (MLM), Sequence to Sequence (Seq2Seq), Permuted Language Modeling (PLM), Denoising Autoencoder (DAE), Constrastive Learning (CTL), Replaced Token Detection (TRD), Next Sentence Prediction (NSP), and Sentence Order Prediction (SOP). Combining some of these tasks has also been proposed, such as BERT (Devlin et al. 2019 ), Transformer-XL, ALBERT, XLNet (Yang et al. 2019 ), ELECTRA (Clark et al. 2020 ), BART (Lewis et al. 2019 ), or GPT (Brown et al. 2020 ). Transformer-based Neural Language Models are able to solve tasks quite accurately from a limited set of training samples (via fine-tuning). Moreover, as Language Models, they have great predictive power over word strings (Radford et al. 2019 ; Brown et al. 2020 ).

However, it has been shown in the literature that contextual models do not always maintain the isometry with respect to the semantic similarity of words utterances. They concentrate the representation of words in hypercones of multidimensional space. This phenomenon has been called the representation degradation problem (Ethayarajh 2019 ; Gao et al. 2019 ; Li et al. 2020 ; Wu et al. 2020 ; Cai et al. 2021 ). The result is that contextual models are very predictive as Language Models, but not very effective in terms of text representation within a semantic space. In other words, although the neural network is able to predict words from the previous sequence and classification labels, the embedding space in which the texts are represented is not coherent with their meanings. According to different authors, such as Gao et al. ( 2019 ) and Demeter, Kimmel, and Downey ( 2020 ), this is due to an effect of infrequent words in the soft max optimization function. We will see that this limitation has an effect on our experiments. In addition, previous experiments show that the effect of the representation degradation problem is stronger in higher net layers where the word representations are more contextualized (Ethayarajh 2019 ). However, solving this problem is not the focus of the article and we leave its analysis for future work.

Some approaches, such as Sentence-BERT (Reimers and Gurevych 2019 ) and the Universal Sentence Encoder (Cer et al. 2018 ), address this problem by training the network over sentence pairs as a similarity classification task. However, the effectiveness of these models may decline when the compared texts have different characteristics from the text units on which they have been trained. In particular, Raffel et al. ( 2020 ) found that supervised transfer learning from multiple tasks does not outperform unsupervised pre-training. Yogatama et al. ( 2019 ) conducted an extensive empirical investigation to evaluate state-of-the-art Natural Language Understanding models through a series of experiments that assess the task-independence of the knowledge being acquired by the learning process. They concluded that the performance is sensitive to the election of the supervised training task. We confirmed this phenomenon in our own experiments.

Recent experiments suggest through probes that Neural Language Models do not capture the systematic nature of language (Talmor et al. 2020 ; Pimentel et al. 2020 ; Goodwin, Sinha, and O’Donnell 2020 ; Hupkes et al. 2020 ; Bender and Koller 2020 ). When probing, a researcher chooses a linguistic task and trains a supervised model to predict annotations in that linguistic task from the network’s learned representations. One of the most meticulous experiments was performed by Hupkes et al. ( 2020 ). This team developed five behavioral tests in order to analyze whether neural networks are able to generalize compositional aspects. The authors found that, for the majority of these tests, Recurrent, Convolution-based, and Transformer models fail. In summary, although tremendously powerful, Neural Language Models alone cannot represent previously unseen textual information by composition.

2.3 Compositional Distributional Semantic Models

In the Compositional Distributional approach, texts longer than one word are represented by means of a composition function that combines distributional representations of linguistic units. A large body of literature has shown that the sum or global average of word embeddings is very effective, often outperforming more sophisticated methods (Mitchell and Lapata 2010 ; Boleda 2020 ; Lenci 2018 ; Blacoe and Lapata 2012 ; Perone, Silveira, and Paula 2018 ; Baroni and Lenci 2010 ; Rimell et al. 2016 ; Czarnowska, Emerson, and Copestake 2019 ; Wieting and Gimpel 2018 ; Ethayarajh 2018 ). An intrinsic limitation of the additive approaches is that word order is not considered, since these composition functions are associative and, in addition, some experiments suggest that their effectiveness degrades with sentence length (Polajnar, Rimell, and Clark 2014 ). The tensor product has also been studied as a composition function (Clark and Pulman 2007 ); its main disadvantage is that the space complexity grows exponentially as more constituents are composed together.

As mentioned in the previous section, the Principle of Compositionality states that the composite meaning depends on the meaning of constituents and their syntactic relationships. In order to capture the information of linguistic structures, some researchers have trained an additive composition function on the basis of expression equivalences (Zanzotto et al. 2010 ), sentence similarity and paraphrasing tasks (Mitchell and Lapata 2008 ; Wieting et al. 2015 ), or sentiment labels in movie reviews (Socher et al. 2012 ). Their main limitations are that the composite structures are biased by the nature of the training corpora and are difficult to scale on more complex linguistic structures.

In another line of work, some authors proposed adding a symbolic layer on top of the distributional word representation. More specifically, Coecke, Sadrzadeh, and Clark ( 2010 ) proposed a composition function relying on the algebra of pre-groups and checked empirically the preservation of the vector dot product as an approach to similarity, although this property was not formally derived. Its main drawback is scalability (Zhang 2014 ). It has been applied to relational words (Grefenstette and Sadrzadeh 2011 ); simple phrases (Kartsaklis, Sadrzadeh, and Pulman 2012 ); and pronouns, prepositions, and adjective phrases (Zhang 2014 ). Smolensky et al. ( 2016 ) presented an initial work on mapping inference in predicate logic based on tensor product representations. This approach is limited by the need for a previous mapping between phrases or sentences and logical propositions.

Finally, other authors have grounded their compositional approach on Information Theory notions. Arora, Liang, and Ma ( 2017 ) proved that applying IDF (word specificity 3 ) weighted sum of vector plus Singular Value Decomposition achieves competitive results regarding some sequential models such as LSTM. Zhelezniak, Savkov, and Hammerla ( 2020 ) represented sentences as a sequence of random vectors and experimented with their Mutual Information approach as a sentence similarity measure. A common limitation of both approaches is that they do not consider the syntactic structure of the sentence.

In the context of the state of the art, we can situate our work as follows. We propose a framework for Compositional Distributional representation and study the formal desirable properties of embedding, composition, and similarity functions. Our work focuses on the problem of representation itself, taking as an objective the correspondence between distributional representations and meanings.

In this section, we describe the proposed theoretical framework. First, we establish a geometric interpretation of Distributional Semantics and its connection with Information Theory. Next, we formally define the notion of Information Theory–based Compositional Distributional Semantics and its formal properties.

3.1 Geometrical Interpretation of Distributional Semantics

In addition, Gao et al. ( 2019 ) proved that under some assumptions, the optimal embeddings of infrequent tokens in Transformer Language Models can be extremely far away from the origin. Li et al. ( 2020 , page 9121) observed empirically that “high-frequency words are all close to the origin, while low-frequency words are far away from the origin” in Transformer language models.

On the other hand, this phenomenon makes sense from the point of view of text representation. The absence of textual information should be a singular point in the representation space (origin of coordinates). This point should be equidistant to any text with a fixed amount of information. In other words, the set of possible texts with a fixed amount of information should form a sphere around the empty information point. In short, it makes sense that the representations are distributed around the origin of coordinates at a distance proportional to the amount of information they contain.

Figure 2 illustrates this idea. The longer a word sequence is, the lower its probability, increasing its specificity and, therefore, its IC. Consequently, as we remove words from an utterance, its embedding approaches the origin of coordinates (vector norm tends to zero). At the end, the probability of the empty word set is maximal, and therefore, its IC is minimal.

Geometric interpretation of text embedding based on Information Theory.

Another consequence of this geometrical interpretation of Distributional Semantic representation is that the meaning in the pragmatic context of any utterance must be represented as an infinite norm vector (human thinking in Figure 2 ). Assuming that the meaning in the pragmatic context of any utterance is extremely specific (time, place, actors, physic context, open-ended world knowledge, etc.), then we can assert that its probability in the semantic space tends to zero and therefore its IC according to Shannon’s Theory is infinite. The representation is therefore theoretically an infinite vector and its proximity depends exclusively on its angular distance.

Because it is a continuous representation space, the embedding distances should approximate the semantic similarity of their utterances. However, we can consider two different notions of semantic similarity. On the one hand, pragmatic semantic similarity refers to the meaning proximity of two utterances taking into account the context in which they are used. In our geometric space, since the pragmatic meaning corresponds with infinite norm vectors, this would translate to the angular distance between infinite norm vectors. In fact, cosine similarity is the standard measure in distributional representations.

On the other hand, literal similarity takes into account the amount of information provided by the utterance. For example, although “hello” may be equivalent to “hello from Barcelona” in a certain context of use, there is a difference in terms of literal similarity, since “hello” is under-specified regarding “hello from Barcelona.” That is, the space of possible meanings in the context of “hello” is broader and therefore its literal IC is lower. In empirical terms, literal utterance similarity represents the expected similarity given an annotator who is shown the two utterances without any information about their pragmatic context.

We can summarize the geometrical interpretation of embedding models as follows: The direction of an embedding represents the pragmatic meaning, and the vector norm of embedding represents how much information the literal utterance provides about its meaning in the pragmatic context . Returning to the example in Figure 2 , the meaning of “Barcelona” depends on its pragmatic context. As we add words ( “from Barcelona,”“hello, from Barcelona” ), the meaning in the pragmatic context of what the sender means to express becomes more precisely defined. That is, as we add context information to the expression when adding words, the embedding norm will become progressively longer and its semantic orientation more precise. Only in infinity do we arrive at the meaning in the pragmatic context. In this paper, we identify the properties that embedding, composition, and similarity functions should fulfill based on this geometric interpretation.

3.2 Information Theory–based Compositional Distributional Semantics

We formalize the notion of Information Theory–based Compositional Distributional Semantics as a tuple of three functions, namely: embedding , composition , and similarity . This framework synthesizes the distributional and compositional paradigms; the hypothesis underlying the notion of ICDS is that there are minimal linguistic units whose semantics are determined by their use and whose amount of information is determined by their specificity. On the other hand, the systematicity of language can be captured by compositional mechanisms while preserving the amount of information of the composite utterance .

Figure 3 illustrates the ICDS framework. The starting point in this example is a set of three words: “yellow,”“star,” and “galaxy.” The semantics of these atoms are inferred on the basis of their use or distributional properties, and not by composition. In our framework, we assume that there exists an embedding function π : S →ℝ n , which returns a vector representation for each basic linguistic unit x in the space S of basic linguistic units. Note that these semantic atoms are words but they could be multiword terms such as Named Entities or compound terms in general. For instance, it is not possible to infer the semantics of “John Smith” as a semantic composition of “John” and “Smith” and the semantics must be inferred from its use as a distinct linguistic unit. The same is true for some common terms; for example, the concept of “black hole” is not the semantic composition of the concepts of “hole” and “black.” In our framework, we will refer to these atoms as basic linguistic units . The framework also assumes that there exists a composition function ⊙ that returns a new embedding given two representations and a similarity function δ that estimates the embedding meaning proximity. The particularity of ICDS is that both the composition and the similarity functions must be consistent with the embedding Information Content (vector norms).

Embedding (π), Information Content (IC, vector norm), Composition (⊙), and Similarity (δ) functions in an Information Theory–based Compositional Distributional Semantics system.

In Section 2 we discussed four aspects that a semantic representation model should cover as desirable properties. The embedding function π captures Usage Context and Information Measurability . The composition function ⊙ addresses Systematicity and, whenever it satisfies certain properties, it maintains the Information Measurability coherence of composite representations. Finally, the semantic similarity function δ defines the continuous space of semantic representations.

The key point is that this formal framework captures the duality between Compositional and Distributional Semantics. The semantics of the basic linguistic units modeled by means of the embedding function π is determined by the textual context in which they appear, but, in turn, the semantics of complex structures is determined by the composition function ⊙ and the way in which words are combined.

In this article, we define the notion of Information Theory–based Compositional Distributional Semantics as follows:

Definition 1 (ICDS) An Information Theory–based Compositional Distributional Semantics representation is a tuple (π,δ,⊙) with an embedding, semantic similarity, and composition function. S being the space of basic linguistic units: π : S → R n , δ : R n × R n → R , ⊙ : R n × R n → R n

In the following sections we formalize a set of formal properties for each of the components of the above definition of ICDS. We cannot be sure that there are no other possible properties that can be added in future work. However, based on our review of related work, what we can be sure is that this set of nine properties is sufficient to capture the particularities and differences among existing embedding functions.

4.1 Embedding Function Properties

The first two properties in Definition 1 affect the embedding function. In the following, we will denote the atomic linguistic units (words, chunks) as x , y , or z .

Property 1 ( Information Measurability ) x being a linguistic unit, the vector norm of its embedding approximates the Information Content of x . ∥ π ( x ) ∥ ≃ I C ( x ) = − log ( P ( x ) )

Information Measurability formalizes what has already been observed in the experiments developed throughout the literature. That is, the correspondence between embedding vector norm and word specificity or frequency (see Section 3.1 ). It states that the embedding of the word sequence allows for estimating the probability of observing the corresponding word sequence, which is projected within representation space by π on the vector terminal point defined by the embedding vector. This property establishes an explicit link between text embedding and Information Theory.

The consequence is that stopwords and other frequent tokens (with low expected Information Content) are represented near the origin while infrequent words (with high expected Information Content) are projected further away from the origin of coordinates. In general, both Static and Contextual Neural Distributional Models comply with this property (Arora et al. 2016 ; Levy and Goldberg 2014 ; Gao et al. 2019 ; Li et al. 2020 ).

Property 2 ( Angular Isometry ). There must exist isometry between the angular position of embeddings and the expected similarity of utterances according to humans. Being |π( x )| = |π( y )|: cos ( π ( x ) , π ( y ) ) ∝ E ( SIM ( x , y ) )

where E ( SIM ( x , y ) ) represents the expected similarity according to humans, which corresponds with the pragmatic context of the utterances. We assume the hypothetical case of a representative set of humans who are presented with the utterances, and who are asked about their semantic similarity.

Angular Isometry states the correspondence between the embedding direction and the (pragmatic) meaning. This property formalizes the idea of desirable semantic isometry, that is, it establishes a correspondence between the representation of texts and their semantic similarity given a fixed degree of specificity (Information Content). Given two embeddings with similar length (equally informative according to the analysis in Section 3.1 ), the expected similarity between represented utterances according to human annotators should be correlated with the angle these vectors maintain.

4.2 Composition Function Properties

The next three properties concern the composition function ⊙. Unfortunately, it is not possible for the composition function to predict the probability (i.e., IC) of a compound expression (“yellow star”) given the probability of its components (P( “yellow” ) and P( “star” )) and its syntactic relationship. This is in fact an inherent limitation of Language Models in terms of compositionality (see Section 2.1 ). However, we can state constraints for the IC of the composite expression. These boundaries define three formal constraints illustrated in Figure 4 . The black and gray dots represent the component and compound representations, respectively.

Illustration of composition function properties.

Property 3 ( Composition Neutral Element ) Null information components (zero norm) do not affect the composition. ∥ v → 2 ∥ = 0 ⟹ ∥ v → 1 ⊙ v → 2 ∥ = ∥ v → 1 ∥

In principle, adding empty information should not affect the composition. It follows from Composition Neutral Element that when a embedding with (nearly) empty information is composed with another given embedding, the resulting composite embedding will be very close to the given embedding. For instance, “green” is semantically similar to “the green,” assuming that “the” is not informative ( Figure 4a ).

Property 4 ( Composition Norm Lowerbound ) The vector norm of the composite embedding is higher than or equal to the norm of each component; that is, the composition never reduces the Information Content. ∥ v → 1 ⊙ v → 2 ∥ ≥ ∥ v → 1 ∥ v → 1 ⊙ v → 2 ∥ ≥ ∥ v → 2 ∥

Composition Norm Lowerbound states that adding textual information should not reduce the final amount of information. That is, the IC (i.e., vector norm, see Section 3.1 ) of an utterance embedding increases when adding words. For instance, “green grass” is more informative than “green” and “grass” ( Figure 4b ).

Property 5 ( Composition Norm Monotonicity ) The norm of the composite vector is monotonic with respect to the angle between the compound vectors: ∥ v → 1 ∥ = ∥ v → 2 ∥ = ∥ v → 3 ∥ cos ( v → 1 , v → 2 ) > cos ( v → 1 , v → 3 ) ⟹ ∥ v → 1 ⊙ v → 2 ∥ < ∥ v → 1 ⊙ v → 3 ∥

Composition Norm Monotonicity states that the more semantically dissimilar the components are (greater angular distance), the more information the composite embedding contains. This property responds to the intuition that combining two semantically distant utterances should produce specific and infrequent utterances, that is, with a high information content associated with it. For instance, “green” and “colour” are semantically close and their representation vectors should therefore point to nearby areas in the infinite ( Figure 4c ). That is to say that “green” and “colour” have an angular distance that is less than that of two semantically unrelated words, such as “green” and “thought.” The compound term “green thought” is therefore more specific, less likely, and more informative than “green colour” ; and its vector norm should therefore be larger. In other words, under fixed single vector norms (equal IC), the semantic orientation similarity of components decreases the Information Content of the composition.

In addition to these three properties, there is another aspect to consider in ICDS. Compositionality requires sensitivity to the linguistic structure.

Property 6 ( Sensitivity to Structure ) Given the three embeddings v → 1 ⁠ , v → 2 ⁠ , v → 3 with an equal norm and angularly equidistant, their composition is not associative: ∥ v → 1 ∥ = ∥ v → 2 ∥ = ∥ v → 3 ∥ > 0 cos ( v → 1 , v → 2 ) = cos ( v → 1 , v → 3 ) = cos ( v → 2 , v → 3 ) > 0 ⟹ ( v → 1 ⊙ v → 2 ) ⊙ v → 3 ≠ v → 1 ⊙ ( v → 2 ⊙ v → 3 )

Let us explain the motivation for Sensitivity to Structure . The Principle of Compositionality assumes that linguistic units are progressively grouped into more complex semantic units, according to the structure of the language ( Figure 4d ). That is, the composition function must be applied on the representations according to how the words are considered to be structured. For example, assuming constituents, the composition function on the sentence “This house is very big” is applied following the brackets: “((This house) is (very big))” .

Unfortunately, we cannot define formal geometric properties that ensure that the distributional model captures the semantics of composition. However, a basic requirement is that the final representation is at least sensitive to the way in which the language is structured. That is, different ways of grouping words in the composition function should produce different representations. More formally, the vector composition function should not be associative.

4.3 Similarity Function Properties

The next three properties concern the similarity function δ (see Figure 5 ).

Illustration of the preconditions of the properties Angular Distance Similarity Monotonicity , Orthogonal Embedding Similarity Monotonicity , and Equidistant Embedding Similarity Monotonicity .

Property 7 ( Angular Distance Similarity Monotonicity ) Under equal vector norm (equal IC), similarity is a monotonic decreasing function with regard to the angular distance (and semantic orientation proximity). cos ( v → 1 , v → 2 ) > cos ( v → 1 , v → 3 ) ∥ v → 1 ∥ = ∥ v → 2 ∥ = ∥ v → 3 ∥ > 0 ⟹ δ ( v → 1 , v → 2 ) > δ ( v → 1 , v → 3 )

Angular Distance Similarity Monotonicity expresses the monotonicity of similarity regarding the semantic orientation of text representation vectors. Note that Angular Isometry (defined above) states the correspondence between the semantic similarity of text units and their embedding angle. Angular Distance Similarity Monotonicity ensures that the similarity function δ is coherent with this principle. More formally, similarity between two representation vectors increases if the cosine of the angle formed by the vectors increases, that is, if the angle is reduced ( Figure 5a ).

Property 8 ( Orthogonal Embedding Similarity Monotonicity ) Given orthogonal embeddings, the larger their norm is (their specificity), the less similar will be: cos ( v → 1 , v → 2 ) = cos ( v → 3 , v → 4 ) = 0 ∥ v → 1 ∥ < ∥ v → 2 ∥ , ∥ v → 3 ∥ < ∥ v → 4 ∥ ⟹ δ ( v → 1 , v → 2 ) > δ ( v → 3 , v → 4 )

Orthogonal Embedding Similarity Monotonicity expresses the decreasing monotonicity of similarity regarding the IC (i.e., vector norm) of components. That is, given two semantically unrelated utterances, the more they contain information, the more they are semantically dissimilar. Let us consider two semantically distant words such as “green” and “thought,” and two semantically distant utterances such as “green colour” and “retrospective thought” ( Figure 5b ). The more information we add to the semantically distant utterances, the lower their similarity should be.

Property 9 ( Equidistant Embedding Similarity Monotonicity ) Given two embedding pairs ( ⁠ v → 1 , v → 1 ′ ⁠ ) and ( ⁠ v → 2 , v → 2 ′ ⁠ ), and with ϵ → being a vector that represents their equidistance, then: v → 1 ′ = v → 1 + ϵ → , v → 2 ′ = v → 2 + ϵ → ∥ v → 2 ∥ > ∥ v → 1 ∥ > > ∥ ϵ → ∥ ⟹ δ ( v → 2 , v → 2 ′ ) > δ ( v → 1 , v → 1 ′ )

Equidistant Embedding Similarity Monotonicity captures the idea that given two utterances that are close to each other, their specificity gives them similarity. Figure 5c reflects this idea. The utterances “green” and “green grass” are not necessarily similar, given that the term “green” could refer to any kind of object. However, their similarity could be higher in a more specific context, for instance, “the top of the mountain is green” vs. “the top of the mountain is green grass.” Under our geometrical interpretation, the further away the related embeddings are from the origin (more specificity and content), the more they are semantically similar. More formally, under equal Euclidean distance, semantic similarity grows with the embedding vector norm.

In this section we analyze existing embedding, composition, and similarity functions (the tuple that defines an ICDS) on the basis of the formal properties stated in our formal framework. In addition, we define novel parameterizable functions that generalize existing approaches for composing. We also introduce the linguistic structures that will be considered in our experiments.

5.1 Embedding and Information Content Functions

According to previous studies, both static (SGNS, GloVe) and contextual (BERT, GPT) embedding models keep the correspondence between the embedding vector norm and the specificity of the represented linguistic unit (see Section 3.1 ). Therefore, the vector norm can be used as Information Content function, satisfying Information Measurability .

Note that under the distributional hypothesis ( “similar contexts imply similar meaning” ), PMI should approximate the semantic similarity between utterances. That is, a high PMI between two utterances implies a high co-occurrence ( P ( x , y )) with respect to their likelihood in isolation ( P ( x ) and P ( y )). Consequently, two infrequent terms that co-occur will have a high PMI, while a frequent term (e.g., a determiner) will have a low PMI with any other term.

The main drawback of static embedding functions is that the occurrence’s local context is not considered. In contrast, Contextual Neural Language Models (e.g., BERT, GPT) take into account the textual context of language units when embedding them (see Section 2 ). However, due to the representation degeneration problem (see Section 2.2 ), contextual models suffer from limitations in terms of Angular Isometry ( Property 2 ).

5.2 Composition Functions

In this section we study the properties of basic composition functions such as summation or averaging. The global average, namely, the direct average of all embedding vectors that make up an utterance, and sum have been found to be strong baselines for compositional tasks (Boleda 2020 ; Lenci 2018 ; Blacoe and Lapata 2012 ; Perone, Silveira, and Paula 2018 ; Baroni and Lenci 2010 ; Rimell et al. 2016 ; Czarnowska, Emerson, and Copestake 2019 ; Wieting and Gimpel 2018 ; Ethayarajh 2018 ).

In addition, in order to satisfy the formal properties, we propose a generalized composition function based on Information Theory that can be instantiated in different measures according to the chosen parameters.

Definition 2 (Generalized Composition Function) The generalized composition function F λ,μ :ℝ n ×ℝ n →ℝ n is defined as: F λ , μ ( v → 1 , v → 2 ) = v → 1 + v → 2 ∥ v → 1 + v → 2 ∥ · λ ( ∥ v → 1 ∥ 2 + ∥ v → 2 ∥ 2 ) − μ 〈 v → 1 , v → 2 〉

The first component on the right-hand side of the expression is the unit vector of their sum and determines the vector direction. The second component determines the vector norm (magnitude), which depends on the norm of single vectors and their inner product. This function generalizes the most common composition functions in the literature. In particular, it leads to the sum of vectors ( F Sum ) when λ = 1 and μ = −2, and to the pairwise average vector 4 ( F Avg ) when λ = 1 4 and μ = − 1 2 ⁠ .

F Sum and global average do not comply with Composition Lowerbound ( Property 4 , see Table 2 ). For instance, two vectors pointing to opposite directions can cancel each other. Furthermore, they do not comply with Composition Norm Monotonicity ( Property 5 ). In fact, the more distant in orientation the original vectors are, the shorter the length of the composite vector. In addition, the sum and global average are the only non-associative functions in this set. Therefore, they also fail to satisfy Sensitivity to Structure ( Property 6 ). The four remaining functions are sensitive to the text structure. Global average does not comply with COMPOSITION NEUTRAL ELEMENT either.

Formal constraints analysis for embedding, composition, and similarity functions.

On the other hand, pairwise average vector ( F Avg ) is sensitive to the addition of null information (e.g., avg (4,0) = 2 < 4). Therefore, it does not satisfy Composition Neutral Element ( Property 3 ). It also fails to satisfy Composition Norm Monotonicity ( Property 5 ).

Because the composite embedding corresponds to the joint IC in F Joint , it does not comply with Composition Norm Lowerbound ( Property 4 ).

None of the previous composition functions satisfy all properties simultaneously. The following theorem establishes the range of values of λ and μ for which all formal properties are satisfied. This theorem implies that none of the previous functions ( F Joint , F Ind , F Sum , and F Avg ) satisfies the three composition properties simultaneously. However, the functions F Joint and F Ind are just at the boundaries (see proof in Section A.1 ):

The generalized composition function satisfies Composition Neutral Element ( Property 3 ) , Composition Norm Lowerbound ( Property 4 ) , Composition Norm Monotonicity ( Property 5 ) , and Sensitivity to Structure ( Property 6 ) if and only if λ = 1 and μ ∈ 0 , min ( ∥ v → 1 ∥ , ∥ v → 2 ∥ ) max ( ∥ v → 1 ∥ , ∥ v → 2 ∥ ) ⁠ .

Thus, any other choice of parameter values within the formal range will give us a function where the IC of “hello hello” will be between the IC of “hello” and the double of this value. At the moment, we have no information beyond that contained in the embedding vectors, so no pragmatic aspect is considered in the composition. However, the theory leaves room for including this kind of information, so that the IC of “hello hello” would be larger than the IC of “hello.” Since we do not currently have a criterion for what the IC difference should be between these two cases, for our F Inf function we are left with the value at the extreme end of the theoretical range, and it does not increase the amount of IC.

In conclusion, we can state that, unlike the most common composition functions in the literature, the proposed function F Inf is the only one that simultaneously satisfies all four properties.

5.3 Similarity Functions

The most popular vector similarity functions in the literature are Euclidean distance, cosine similarity, and the inner product. Table 2 summarizes the formal properties of these similarity functions.

Euclidean distance is monotonic regarding both Angular Distance Similarity Monotonicity ( Property 7 ) and Orthogonal Embedding Similarity Monotonicity ( Property 8 ). However, it is invariant with respect to the origin of coordinates. Therefore, it does not satisfy Equidistant Embedding Similarity Monotonicity ( Property 9 ). On the other hand, cosine similarity focuses exclusively on angular distance, satisfying Angular Distance Similarity Monotonicity ( Property 7 ) but not Orthogonal Embedding Similarity Monotonicity ( Property 8 ). Equidistant Embedding Similarity Monotonicity ( Property 9 ) is satisfied given that the cosine of two close points increases as they move away from the origin of coordinates if the Euclidean distance between them is maintained.

The third typical similarity approach is the inner product ( 〈 v → 1 , v → 2 〉 = ∥ v → 1 ∥ ∥ v → 2 ∥ cos ( v → 1 , v → 2 ) ) ⁠ , which has a correspondence with PMI in distributional models. The inner product is monotonic regarding the angle between vectors, and it is sensitive to vector norms. It lacks the property of Orthogonal Embedding Similarity Monotonicity . The reason for that is that orthogonal vectors achieve zero similarity regardless of their norm.

We will refer to this similarity function as the Vector-based ICM . Interestingly, ICM β V is equivalent to the inner product when β = 1, and it is equivalent to the Euclidean distance when β = 2. In addition, ICM β V is the only one that satisfies the three similarity constraints (see Table 2 and formal proof in Section A.2 ):

The Vector-based Information Contrast Model satisfies Angular Distance Similarity Monotonicity ( Property 7 ) , Orthogonal Embedding Similarity Monotonicity ( Property 8 ), and Equidistant Embedding Similarity Monotonicity ( Property 9 ) whenever β ∈ (1,2).

The estimator β ^ is designed to satisfy similarity formal properties, to give an average similarity value of zero (neutral), and also to give neutral similarity to document pairs with average single and joint Information Content.

5.4 Linguistic Structures

If the composition function satisfies Sensitivity to Structure (all excepting F Sum and the global average), the linguistic structure and the order in which linguistic units are composed affects the composite embedding. Regarding the literature, past linguistic theories have discussed the sequential versus hierarchical processing of linguistic data (Frank, Bod, and Christiansen 2012 ).

For instance, in a sequential right-to-left manner, an example sentence would be structured as: Just (Robert (showed (up (for (the (party)))))) . A left-to-right composition would be: ((((Just Robert) showed) up) for) the) party . On the other hand, a hierarchical structure could be: (Just Robert) ((showed up) (for (the party)))) .

Many structural constraints have been granted to the development of languages that may express and share intrinsic features of text semantics but also, some of these attributions to languages may have just appeared as a simple-to-explain aggregation of elements that do not relate directly to the Compositional Theorem of textual units. Therefore, an order must be specified to decide which child node is the next one to modify the parent via composition. In this work, we have explored a right-to-left and left-to-right order; but more complex approaches, such as a rule-based system considering the type of relations in dependency parsing or constituent types, could also be used. In this article, we consider three different linguistic structures. The first is the sequential composition of the data in which composition is based on a temporal order, assuming that the text is humanly understood either by the forward or backward composition/propagation of its components semantics. The second structure is constituency parsing, in which the role of sequential and component relations is more balanced. A third option is the dependency tree. In this case, sequential order plays a minor role in the composition and dependencies among elements are what drive the main structural process.

The purpose of the experiments carried out in this article is to study to what extent the satisfaction of the described formal properties affects the effectiveness of meaning representation models. In this sense, the combination of the Word2Vec embedding function, the F Inf combining function, and the ICM similarity, which satisfy all formal properties, should equal or improve the rest of approaches. Our experiments have been sensitive enough to capture the positive effect of the properties Angular Isometry , Composition Neutral Element , Sensitivity to Structure , and Equidistant Embedding Similarity Monotonicity .

Because most of the composition functions are not associative, they are sensitive to the linguistic structure that guide the representation composition. That is, considering different linguistic structures lead to different word groupings in the composition process. In this article we compare the most common linguistic structures in NLP tools, namely, bag of words (no structure), sequential, constituent, and dependency parsing.

Our main findings are:

In the absence of supervision, contextual embedding functions (BERT and GPT-2) perform worse than non-contextual (static) word embedding functions (Word2Vec). Note that, according to the literature, the contextual embedding functions have limitations in terms of Angular Isometry due to the representation degradation problem (Ethayarajh 2019 ; Gao et al. 2019 ; Li et al. 2020 ; Wu et al. 2020 ; Cai et al. 2021 ).

Composition functions based on Information Content ( F Ind , F Joint , and F Inf ) perform better than traditional composition functions ( F Sum , F Avg ), suggesting that Composition Neutral Element and Sensitivity to Structure are relevant properties.

Using models trained on similarity tasks such as S-BERT improves non-supervised similarity measures on tasks similar to those on which they have been trained. However, the effectiveness decreases substantially when applied to data of a different nature. This corroborates the limitations of current contextual embedding models as meaning representation methods in non-supervised scenarios ( Angular Isometry ).

The ICM similarity function generalizes both the Euclidean distance and the dot product (see Section 5.3 ). Unlike these, ICM satisfies simultaneously Orthogonal Embedding Similarity Monotonicity and Equidistant Embedding Similarity Monotonicity . The experiments show that ICM outperforms its components in isolation. However, the cosine similarity, which simply ignores the vector norms ( Orthogonal Embedding Similarity Monotonicity ), achieves similar or even better results than ICM. Unfortunately, this suggests that our datasets are not sensitive to the effect of the IC (i.e., vector norm) of the compared utterances.

Although considering structures improves the results ( Sensitivity to Structure ) with respect to bag of words-based approaches (sum, global average), we did not observe any substantial and consistent difference between linguistic structures (sequential, constituent, and dependency).

6.1 Evaluation Metric

Models of distributional representation have been evaluated in multiple ways throughout the literature. In many cases, extrinsic evaluation is applied, studying the effectiveness of the model for the performance of a particular task for which training data are available. However, this evaluation criterion may be biased by the task itself and also captures aspects that are not specific to the representation, such as behavior against training samples.

Other evaluation metrics focus on the characteristics of the model as a representation system. Some of them focus on the sensitivity of the model such as non-conflation or Demonstration of Multifacetedness (Wang et al. 2019 ). In view of the literature, it is not an aspect that discriminates too much between models.

Another common criterion is analogy , which states that the representation model must maintain a certain coherence in how the words relate to each other. That is, if there is a relationship between two words, for example, France-Paris , then from Germany , it should be possible to infer Berlin . Rogers, Drozd, and Li ( 2017 ) distinguish between four types of analogies: inflectional morphology, derivational morphology, lexicographic semantics, and encyclopedic. The Bigger Analogy Test Set contains 15 types of linguistic relationships (Gladkova and Drozd 2016 ). Analogy as an evaluation criterion was referenced when word embedding models became popular (Mikolov et al. 2013 ) and in subsequent work (Baroni, Dinu, and Kruszewski 2014 ; Levy and Goldberg 2014 ; Schnabel et al. 2015 ; Rogers, Drozd, and Li 2017 ; Wang et al. 2019 ). Furthermore, analogy has been exploited in different NLP tasks such as word sense disambiguation, search, and so forth (Rogers, Drozd, and Li 2017 ). However, this assessment criterion is not appropriate in our context, since it focuses on lexical representation. Note that we are interested in assessing the embedding compositionality.

On the other hand, contextuality metrics quantify the extent to which the representation model captures the context in which words occur. Some of the measures proposed in the literature are the average cosine distance between representations of the same word in different contexts, or between the representation of a sentence and the set of its words. Ethayarajh ( 2019 ) studied the variance of principal components of the matrix formed by the embeddings of a word in different contexts. However, this evaluation criterion is also lexical-oriented.

Isotropy is another criterion for assessing representation. It is assumed that the distribution of embeddings in the space must have certain a priori characteristics, that is, the distribution of embeddings should look the same regardless of the reference point. Some measures that capture this aspect are the distribution of scalar products with the average vector (Mimno and Thompson 2017 ), the average dot product with unit vectors (Mu and Viswanath 2018 ), the average distance to their nearest neighbor of frequent versus infrequent terms (Li et al. 2020 ), or the average cosine distance between embeddings (Cai et al. 2021 ). Isotropy is related to the capacity of the representation model to use the full dimensional space. It has been shown to have some positive effect on the effectiveness of representation models, but in this article we will focus on less indirect aspects.

In our experiments, we will focus on isometry . That is, there must exist a similarity function over the embedding space that correlate with the proximity of meanings. Several authors directly compute the correlation between embedding cosine and lexical similarity annotated by humans (Gladkova and Drozd 2016 ; Qiu et al. 2018 ; Wang et al. 2019 ). Some available lexical similarity datasets are Wordsim-353 (Finkelstein et al. 2001 ), MEN (Bruni, Tran, and Baroni 2014a ), SimLex-999 (Hill, Reichart, and Korhonen 2015 ), or SimVErb-3500 (Gerz et al. 2016 ). There are also some variants in which instead of graded similarity values, the embedding proximity is compared against synonyms or term categories (Baroni, Dinu, and Kruszewski 2014 ). For instance, Mu and Viswanath ( 2018 ) evaluated clustering algorithms over word representations and Cohenen et al. clustered representations according to their sense and quantified the discriminative power of distance to sense centroids (Mu and Viswanath 2018 ). Some word/concept datasets are Almuhareb ( 2006 ), ESSLLI 2008 Distributional Semantic Workshop shared task dataset (Baroni, Evert, and Lenci 2008 ), or the Batting test set (Baroni and Lenci 2010 ). Detecting an outlier in a term set is another isometry-based metric (Camacho-Collados and Navigli 2016 ). Another approach consists of measuring the component-wise correlation between word vectors from a word embedding model and manually constructed linguistic word vectors in the SemCor dataset (Miller et al. 1994 ), that is, Qvec (Tsvetkov et al. 2015 ).

But all the previous metrics are lexically oriented. Isometry can be extrapolated to sentence representation via semantic text similarity annotations (Zhang et al. 2020 ; Poerner, Waltinger, and Schütze 2020 ; Li et al. 2020 ; Yang et al. 2019 ). Note that testing isometry is distinct from using textual similarity as an extrinsic task-oriented metric. That is, a neural distributional model might not provide a match between representation and meaning spaces (isometry), and yet provide very good results in supervised textual similarity prediction with training data. This is the case of BERT (Devlin et al. 2019 ). It should also be noted that in our experiments, the term isometry is not entirely accurate from a formal point of view, since both the semantic similarity itself and some similarity functions (e.g., PMI, ICM) do not necessarily fulfill the properties of a metric space (Amigó et al. 2020 ; Amigó and Gonzalo 2022 ). For simplicity, we will use this term in the article to refer to our evaluation criterion.

In this article we will focus on the notion of non-supervised isometry, evaluating the correspondence between embedding distances and meaning similarity of sentences and paragraphs. This evaluation methodology checks simultaneously the suitability of the embedding, composition, and similarity functions.

6.2 Datasets

We have selected different publicly available datasets that annotate the similarity scores of two pieces of text based on a certain target.

The most popular semantic similarity dataset, STS (Cer et al. 2017 ), corresponds to the evaluation set from the corpus of English STS shared task data (2012–2017). It contains 1,102 pairs of sentences annotated with their meaning similarity ranging from 0 to 5, with 0 being two completely dissimilar sentences and 5 being two completely equivalent sentences. This dataset is composed of multiple subsets ( MSRpar , MSRvid , answer-answer , track5 , and images ).

In general, the STS dataset is oriented to ambiguity and lexical semantic proximity rather than linguistic structures, and many sentence pairs consist of the substitutions of terms with synonyms. Marelli et al. ( 2014 ) developed SICK, a dataset created from STS specifically to test Compositional Distributional Semantic models given that the pairs of sentences encode compositional knowledge.

A common feature of the above two corpora is that their sentence pairs tend to have a high lexical overlap. In order to experiment on purely semantic proximity, we have developed DEF2DEF, where the texts compared consist of definitions of words. For this purpose, we take word pairs from the MEN Test Collection. MEN consists of 2,993 pairs of words annotated with their lexical-semantic similarity ranging from 0 to 50 (Bruni, Tran, and Baroni 2014b ). We replace the words with their first definition in the WordNet set of synsets, which corresponds to the most frequent use of the word and assume that the semantic similarity between their respective definitions does not differ greatly from the original annotations. To corroborate this assumption, we have carried out a parallel annotation in which two human experts identify between two pairs of definitions, the pair that has the highest similarity between its definitions; 88.5% of the 200 annotated pairs matched the corresponding word similarities from the original MEN corpus. We used the following steps to sample comparison pairs from MEN corpus: (i) Define number of samples to annotate (200) and a relatedness difference threshold (20). (ii) Extract pairs of words from the MEN corpus and their relatedness score RS 1 . (iii) For each original pair of words, randomly select another pair of words from the MEN corpus and their relatedness score RS 2 such that abs ( RS 1 − RS 2 ) ≥ 20. Note that repeated pairs are avoided at this step. (iv) The new gold standard becomes 1 if RS 1 > RS 2 , else 0. Hypothetically, word-level comparisons should match results of definition level comparisons. The model performance in all previous datasets is evaluated in our experiments via the Pearson correlation between the predicted and groundtruth similarities.

In all previous datasets, sentence pairs have a similar semantic specificity or Information Content; for instance, “short sentence” to “short sentence” or “paragraph” to “paragraph” in STS and SICK datasets. In contrast, the MEN corpus contains word pairs with similar semantic specificity. Following the methodology introduced by Kartsaklis, Sadrzadeh, and Pulman ( 2012 ), we have built WORD2DEF, a custom benchmark based on proximity between words and their definitions. A good composition function should result in representations that are closer to the defined words rather than the other alternatives (excluding synonyms and ambiguously defined words acting as noise). Note that the candidate list includes words with a great deal of diversity in terms of specificity. In order to produce positive and negative similarity instances, we have followed a series of careful steps to avoid variants of the target words, stopwords, multiwords, and so forth, that may introduce noise and deviate the results: (i) Select the set of SINGLE words in the Word2Vec vocabulary (without digits) that do not appear in the MEN dataset and have at least one WordNet synset (80,058 tokens); (ii) Lemmatize the available words according to their grammatical tagging (NOUN, VERB, ADVERB, and ADJECTIVE) (44,642 tokens); (iii) Filter out those lemmas that are stopwords, or have no associated synset in Wordnet, or vector in Word2Vec (43,469 tokens); (iv) Keep 30,000 random words from the filtered lemmas (using a pre-defined seed for reproducibility purposes) (30,000 tokens); (v) Collect the 20,183 different words appearing in MEN; (vi) For each word in MEN, consider the word and its corresponding definition as positive similarity instances (20,183 positive instances). Thus, they are represented by a maximum similarity value of 1; (vii) For each word in MEN, consider their corresponding definition (and we set a maximum similarity among them) and 100 random tokens out of the 30,000 tokens batch collected in step (4) as negative similarity samples (201,830 negative instances). In this second case, they are represented by a similarity value of 0; and (viii) For each similarity function, receiver operator characteristic (ROC) area under the curve (AUC) score is computed taking the 0–1 ground truth labels and the similarity predictions as input.

6.3 Semantic Representation Systems in Comparison

This subsection briefly explains the reasons behind all considered instantiations of the functions composing an ICDS representation system (see Section 5 for the theoretical details and constraints of such systems). These components correspond to the embedding π, composition ⊙ and the similarity function δ. Table 3 illustrates all possible configuration set-ups that are considered in our experiments.

Configurations considered for the experiments.

We consider both static (non-contextual) and contextual embedding functions. The former is instantiated with Word2Vec representations, although other well-known alternatives such as GloVe or FastText (Mikolov et al. 2018 ) could also apply. In addition, the embedding function in the experiments is also instantiated by pre-trained contextualized representation models based on Transformer Encoder/Decoder architectures: BERT (Devlin et al. 2019 ) and GPT-2 (Brown et al. 2020 ). Following the suggestions given by Ethayarajh ( 2019 ), we take the second layer in GPT-2. This author suggests that upper layers of contextualizing models produce more context-specific representations.

We also include S-BERT (Reimers and Gurevych 2019 ) in the experiments. S-BERT is a supervised BERT variant representation approach that provides embeddings at the sentence level by re-training the model on SNLI and NLI annotated corpora (Bowman et al. 2015 ), or even on STS training datasets. 6 In terms of our theoretical framework, S-BERT consists of the selected BERT version (BERT, distilBERT, RoBERTa…with re-trained weights) as the embedding function π, followed by a simple composition function ⊙ (vector global average). S-BERT and similar supervised approaches have reported state-of-the-art results in semantic textual similarity, jointly performing the processes of vector space projection and semantic composition. We are interested in analyzing the relative performance of unsupervised and supervised BERT as well as in determining how task-specific this fine-tuned model is.

Regarding the composition function ⊙, in our experiments, we consider traditional sum ( F Sum ) and pairwise average ( F Avg ) operations, as well as three other configurations of the Generalized Composition Function: F Ind , F Joint , and F Inf . In addition, with regard to the BERT model, we consider max pooling 7 and CLS token extraction 8 as two alternatives to combine the contextualized token representations. All of these composition functions are applied on top of all embedding functions except for S-BERT.

Note that S-BERT simply uses the global average on top of the BERT architecture as a single composition function option.

Concerning the linguistic structures and composition order used in our experiments (see Section 5.4 ), in the case of BERT and GPT-2, we consider the sequential structure in both L2R and R2L directions. Note that F sum , Max Pooling, Global Average, and CLS token are composition criteria that do not comply with Property 6 ( Sensitivity to Structure ). Therefore, the structure does not affect the composite representation. In the case of static embeddings (Word2Vec), we use sequential, constituency, and dependency parsing. In order to extract the proper linguistic structures from the whole set of texts, we used the Stanford Core NLP API, including both tokenizer and parsers for extracting constituent and dependency trees. These tree representations were later flattened into nested lists following a right-to-left (R2L) or left-to-right (L2R) direction for a more efficient computation of the compositions. A visualization of such flattening output is presented in Table 4 .

Example of the resulting flattened linguistic structures.

Finally, the similarity functions included in our experiments are cosine similarity, dot product, ICM β , and Euclidean similarity. Their definitions, correspondences, and formal properties are described in Section 5 .

6.4 Results

For clarity of exposition, the complete table of results will be available in the Github repository. 9 This section shows aggregated data in bar charts that allow us to analyze the results and draw conclusions. The most important empirical results that we have been able to derive from the experiments are described below.

6.4.1 Contextual vs. Static Embedding Functions

According to previous studies (Ethayarajh 2019 ; Gao et al. 2019 ; Li et al. 2020 ), although Contextual Neural Language Models such as BERT and GPT have a high prediction power as Language Models, they do not distribute words isometrically. That is, there is no isometry between the angular position of utterance embeddings and their expected semantic similarity according to humans ( Angular Isometry ). Consequently, applying these embedding functions may affect the effectiveness of similarity functions, at least in a non-supervised representation oriented experiment such as this. We can observe this effect in our experiments.

Figure 6 shows the Pearson correlation between estimated and real similarities for the datasets STS, SICK, and DEF2DEF. In this experiment, we compare the performance of embedding functions. In every approach, we apply the sequential structure SEQ (selecting the best processing direction, L2R or R2L, in each case), the best composition function (within F sum , F avg , F ind , F joint , and F inf ), and cosine as similarity functions. In the case of BERT-CLS and S-BERT, the linguistic structure and the composition function is given directly by the Neural Language Model. Table 5 shows the result of the same experiment over the WORD2DEF test set and ROC-AUC as an evaluation measure.

Contextual vs. Static Embedding function performance in terms of ROC-AUC score at the WORD2DEF dataset using the cosine similarity, a sequential structure, and the best configuration of the remaining components.

Contextual vs. Static Embedding function performance in terms of Pearson correlation across several semantic similarity prediction datasets using the cosine similarity, a sequential structure, and the best configuration of the remaining components.

As Table 5 and Figure 6 show, unsupervised contextual embedding functions (pre-trained BERT and GPT-2) achieve a lower correlation than static embeddings (Word2Vec) in every dataset. For example, differences of around 50% are observed between BERT and Word2Vec in STS-MSRvid or between GPT-2 and Word2Vec in DEF2DEF. In fact, the results obtained by a two-tailed Steiger’s test (Steiger 1980 ) show a significant statistical difference when comparing Word2Vec to GPT-2 and BERT with a p-value below 2% for all datasets except STS-MSRpar and STS-answer-answer. The correlation values obtained for BERT using the CLS field are even lower and the statistical test shows a significant difference with respect to Word2Vec in all datasets with a p-value of 0%. The differences between contextual and static embedding are more evident in the WORD2DEF dataset where a word is compared to a complete sentence ( Table 5 ).

On the other hand, contextual embeddings perform better when applying a supervised transfer learning from paraphrasing datasets. This is the case of S-BERT, whose correlation value exceeds those obtained with Word2Vec word embeddings. Nevertheless, the statistical test is only able to detect a significant improvement with respect to static embeddings for the STS-images dataset where, similar to other STS subsets and the SICK dataset, sentence pairs are comparable and words overlap. On the contrary, S-BERT falls behind Word2Vec performance in the DEF2DEF corpus with a p-value of 0. In the case of WORD2DEF, S-BERT achieves an ROC-AUC score of 0.88 while Word2Vec achieves 0.94. Consistent with previous studies (Raffel et al. 2020 ; Yogatama et al. 2019 ), this result suggests that transferring learning from supervised tasks can bias the system performance.

In sum, the experiments suggest that static embedding functions are more effective as text representation models in nonsupervised scenarios . In our opinion, this is related to the lack of Angular Isometry due to the representation degradation problem in contextual embedding functions.

6.4.2 Composition Functions.

In order to analyze the performance of the composition functions, we evaluate static embeddings (Word2Vec) with the best linguistic structure for each dataset and cosine similarity. Figure 7 and Table 6 show the results.

Composition function performance in terms of ROC-AUC score at the WORD2DEF dataset using static embeddings, cosine similarity, and the best configuration of the remaining components.

Composition function performance in terms of Pearson correlation across several semantic similarity prediction datasets using static embeddings, cosine similarity, and the best configuration of the remaining components.

First, we observe that in general, F avg performs worse than other functions. In particular, Steiger’s method on Pearson correlations indicates that F avg is statistically outperformed by F ind , F inf , and F joint with a p-value below 3% on 5 out of 7 datasets with the exception of STS-MSRvid and STS-answer-answer. This finding suggests that Composition Neutral Element is a relevant property (see Table 2 ). Note that F avg has the counter-intuitive effect of reducing the magnitude (Information Content) of the embedding as we add information by compositing it with other shorter vectors.

In the datasets DEF2DEF and WORD2DEF, without word overlap between sentences, F sum obtains similar results to the non-associative functions F ind , F joint , or F inf . These results suggest that the Sensitivity to Text Structure in those datasets is not crucial when composing embeddings. However, in the case of sentence-level datasets with a high overlap (SICK and 3 out of 6 STS datasets), the structure-sensitive composite embedding functions, F ind , F joint , and F inf outperform F sum with a p-value close to 0%. In sum, as can be guessed, satisfying Sensitivity to Text Structure in the composition function is especially relevant when comparing sentences with high word overlap.

We do not find a consistent and substantial difference across datasets between F inf and the parameterizations at the boundaries of the theoretical area in which Composition Norm Lowerbound and Composition Norm Monotonicity are satisfied (see Table 2 ). However, it should be noted that F joint and F ind are just at the boundaries (see Section 5.2 ).

6.4.3 Similarity Functions.

In the previous analysis, the similarity function was fixed to be the cosine similarity in order to consistently explore the composition function behavior. Now, we explore the alternative similarity functions. Table 7 and Figure 8 contain the average performance across datasets for S-BERT and Word2Vec with different composition functions. In the case of static embedding (Word2Vec), we selected the best linguistic structure per dataset. Note that in order to average Pearson correlations, scores are transformed with Fisher Z-transformation, averaged and transformed back to the original space.

Similarity function performance in terms of ROC-AUC score at the WORD2DEF dataset per composition function on static embeddings after selecting their best configuration per dataset. S-BERT is added for comparison purposes (contextual embeddings and state-of-the-art performance).

Similarity function average performance in terms of Pearson correlation over all presented semantic similarity datasets per composition function on static embeddings after selecting their best configuration per dataset. S-BERT is added for comparison purposes (contextual embeddings and state-of-the-art performance).

First, the results provide evidence for the importance of satisfying the formal constraints Orthogonal Embedding Similarity Monotonicity and Equidistant Embedding Similarity Monotonicity . Note that ICM falls within the limits of the theoretical range corresponding to the dot product (ICM with β = 1) and Euclidean distance (ICM with β = 2). In all static embedding cases, ICM outperforms the dot product and Euclidean distance. To be more precise, when ICM is paired with F inf so that all considered properties are satisfied, a statistical significant difference is achieved in all 7 datasets with respect to dot product and 4 datasets when comparing it against Euclidean distance.

However, the cosine distance obtains greater results than ICM with the theoretically estimated parameter with statistical significance in 3 to 4 datasets out of the 7 datasets included in Figure 8 , depending on the composition function used to compute final embeddings. Based on our embedding space interpretation (see Section 3.1 ), this result suggests that the Information Content or specificity of the compared texts are not crucial in our semantic similarity datasets. In other words, the annotations in at least these textual similarity datasets are more focused on angular distance (pragmatic similarity) than vector magnitudes (literal similarity which takes into account the amount of information provided by texts).

The difference between both similarity functions is really subtle in the case of WORD2DEF lying on both directions depending on which composition function is presented, and thus no clear conclusions can be drawn. One possible reason why both similarities are on par with each other in this scenario when previous datasets presented significant differences could be the increasing importance of Orthogonal Embedding Similarity Monotonicity when specificity between each pair of texts differs in larger quantities.

6.4.4 Linguistic Structures.

One of the premises of this research work is the fact that following a structure in the composition stage of an ICDS representation system should benefit performance with respect to traditional orderless baselines such as the sum of vectors. This statement was proven to be true in Section 6.4.2 where the sum operation was outperformed by all other unsupervised composition functions that followed a certain structure. The next experiment aims to investigate whether there are significant performance changes when alternating from one linguistic structure to another among all of the options presented in Section 5.4 . The corresponding results for each structure across datasets are shown in Figure 9 , fixing the model configuration to Word2Vec as the representation system, F inf as the composition function, and cosine as the similarity function. The ROC-AUC results for the dataset WORD2DEF are 0.95847, 0.95967, and 0.95941 for the sequential structure, syntactic constituents, and dependency trees, respectively.

Similarity prediction power of our main proposed baseline (Word2Vec as the representation system, F Inf as the composition function and cosine as the similarity function) in terms of Pearson correlation across all presented semantic similarity datasets with respect to the selected linguistic structure to follow during composition.

Unfortunately, no substantial difference can be found among the tested linguistic structures. The effect of each linguistic structure depends strongly on the dataset and task. For instance, both DEP and SEQ achieve a significant statistical difference with respect to CONST at STS-MSRvid with a p-value below 5%; but SEQ is outperformed significantly by DEP and CONST at SICK with a p-value of 0% and both DEP and CONST are significantly outperformed by SEQ, with a p-value below 5% at STS-answer-answer. No other significant differences can be extracted from the statistical analysis for the other 4 datasets. Nonetheless, the importance and benefits of following a structure instead of associative composition functions has been illustrated previously.

In this article, we have formalized the concept of an Information Theory–based Compositional Distributional Semantics as a tuple of three functions, namely, embedding, composition, and similarity. The main contribution is to have established a bridge between representation space and Information Theory, not only for embeddings but also for composition and similarity functions. The proposed framework includes a set of nine formal constraints based on Information Content boundaries and monotonicity in embedding, composition, and similarity functions.

Regarding embedding function, our experiments show that adding embeddings from internal layers of contextual neural networks such as GPT-2 or BERT does not outperform the use of static embeddings when performing a textual semantic composition process, at least in terms of the semantic similarity prediction. These results are consistent with the Representation Degradation Problem identified in the literature (Ethayarajh 2019 ; Gao et al. 2019 ; Li et al. 2020 ). It is true that the S-BERT model based on supervised transfer learning manages to increase the effectiveness in certain datasets. However, its effectiveness decreases in problems that are not similar to the tasks on which it has been trained, as in the case of word-definition comparison.

Our formal analysis shows that popular, well-known baseline composition and similarity functions do not satisfy all the constraints simultaneously. In order to overcome this formal limitation, we have defined two parameterizable approaches that generalize standard composition (sum) and similarity (Euclidean distance or dot product) functions. Our theoretical study shows that these functions satisfy all constraints within certain parameter ranges. Our empirical study suggests that the generalized composition function within the theoretical parameter range outperforms standard functions such as the sum or the global average.

The same occurs in the case of the similarity functions, where the Euclidean distance or the scalar product fall outside the range. However, our results also show that the cosine distance, while not sensitive to the vector norm ( Property 8 , Orthogonal embedding similarity Monotonicity ), is a robust similarity estimate. According to our embedding space interpretation, this means that the similarity annotations in our datasets are more closely related with the pragmatic context (i.e., vector direction) than with literal utterance similarity . That is, it does not take into account how much information the message provides in relation to its meaning in context (i.e., vector norm).

Finally, according to our experiments, the election of linguistic structures does not affect the performance substantially. Sequential text processing achieves results similar to those of more complex structures. However, the results also show that considering structure (sequential, constituents, or dependencies) outperforms those approaches that do not consider word order (i.e. sum, or global average of embeddings). That is, we have not been able to contribute new results to the discussion between the sequential or hierarchical nature of language, although we have been able to confirm that considering structures is more effective than representing texts as a bag of words. It should be noted that this effect becomes tangible in the case of similarity between sentences with some overlap of words (STS and SICK corpora), whereas in our definition vs. definition or definition vs. word comparison, we have not managed to observe this effect.

In view of our analysis, our recommendation for unsupervised text representation is to combine static (non-contextual) embeddings following a sequential structure using the proposed composition function F inf , and using ICM with either the estimated β value or cosine as a similarity function. Note that this recommendation applies only to non-supervised scenarios. The literature has demonstrated repeatedly the predictive power of contextual models in supervised tasks.

As future work we can identify several research lines. Firstly, to address the problem of the degeneration of representation in contextual approaches. This direction has already been explored in multiple studies (Ethayarajh 2019 ; Gao et al. 2019 ; Li et al. 2020 ). Also, in line with previous work, we see the need to design semantic similarity datasets over more heterogeneous cases and scenarios when comparing general purpose representation models. A third line of work is the development of datasets that capture the effect of specificity and the amount of information on the estimation of semantic similarity between textual representations, as has been observed in experiments within cognitive sciences (Tversky 1977 ; Amigó et al. 2020 ). A fourth and perhaps the most ambitious line of research is to close the circle of Neuro-Symbolic Artificial Intelligence in the context of distributional semantics. In this article we have addressed the representation phase in which symbolic information (text) is translated into a continuous space. The pending step is extraction , in which the representation in this continuous space is translated into a symbolic domain. There are several authors who have focused on this challenge (Sarker et al. 2021 ).

A.1 Proof for Theorem 1

Let us analyze the Composition Norm Lowerbound property. The square of the composite embedding magnitude (information content) is ∥ F λ , μ ( v → 1 , v → 2 ) ∥ 2 = λ ( ∥ v → 1 ∥ 2 + ∥ v → 2 ∥ 2 ) − μ 〈 v → 1 , v → 2 〉 = λ ( ∥ v → 1 ∥ 2 + ∥ v → 2 ∥ 2 ) − μ ( ∥ v → 1 ∥ · ∥ v → 2 ∥ · cos ( v → 1 , v → 2 ) ) ⁠ .

A.2 Formal Proof for Theorem 2

In order to satisfy Angular isometricity , the expression must be monotonic regarding cos ( v → 1 , v → 2 ) and, therefore, β must be strictly greater than zero. In order to satisfy Composition norm Monotonicity , I C M β V must decrease with ∥ v → 1 ∥ and ∥ v → 2 ∥ when cos ( v → 1 , v → 2 ) = 0 ⁠ . Therefore β > 1.

The authors would like to thank anonymous reviewers for their valuable suggestions to improve the quality of the article. This research was partially supported by the Spanish Ministry of Science and Innovation Project FairTransNLP (PID2021-124361OB-C32); by MCI/AEI/FEDER, UE DOTT-HEALTH Project under grant PID2019-106942RB-C32; and LyrAIcs project through the European Research Council (ERC), under the Research and Innovation Program Horizon2020 under grant agreement number 964009. The work has been carried out in the framework of the following projects: MISMIS project (PGC2018-096212-B), funded by Ministerio de Ciencia, Innovación y Universidades (Spain) and CLiC SGR (2027SGR341), funded by AGAUR (Generalitat de Catalunya).

In this article we will use the terms word and utterance . The notion of word used in this article can be generalized to any atomic element on which composition or distributional analysis is applied by the computational linguistic model. This ranges from morphemes in synthetic languages such as German to multiwords or collocations. On the other hand, we will use the term utterance to refer to the compound expressions we want to represent.

Note that dimensionality reduction compresses the representation by taking into account the interdependence between terms.

Note that I D F ( t ) = log ( N ∣ { d ∈ D : t ∈ d } ∣ ) ⁠ , where D represents the document collection and N represents its size, has a direct correspondence with its specificity of Information Content I C ( t ) = − log ( P ( t ) ) according to the Shannon’s Information Theory.

The difference between pairwise and global averaging lies on the composition procedure for utterances of N components with N being bigger than 2. The former applies pairwise compositions until the ROOT node is reached (2 components are considered at each step) while the latter composes all terms in a single averaging step.

Note that: ∥ π ( x ) ∥ 2 + ∥ π ( y ) ∥ 2 − 〈 π ( x ) , π ( y ) 〉 ≃ − log ( P ( x ) ) − log ( P ( y ) ) − log P ( x , y ) P ( x ) · P ( y ) = − log ( P ( x , y ) )

These datasets were created to identify a certain semantic similarity/relation between pairs of sentences.

Max Pooling is a technique commonly used in Computer Vision to reduce the dimension of the image while maintaining the spatial relation of pixels as much as possible. In this NLP operation, we apply it over the dimensions of the token embeddings.

CLS token, also known as Classification Token, is a token added at the beginning of each text input during the pre-processing step to be used in classification tasks adopting the role of representing the whole text.

Source code, data used in the experiments, and the complete table of results are publicly available to encourage reproducible research: https://github.com/alarca94/ICDS .

Author notes

Email alerts, related articles, related book chapters, affiliations.

• Online ISSN 1530-9312
• Print ISSN 0891-2017

A product of The MIT Press

Mit press direct.

• About MIT Press Direct

Information

• Accessibility
• For Authors
• For Customers
• For Librarians
• Direct to Open
• Open Access
• Media Inquiries
• Rights and Permissions
• For Advertisers
• About the MIT Press
• The MIT Press Reader
• MIT Press Blog
• Seasonal Catalogs
• MIT Press Home
• Give to the MIT Press
• Direct Service Desk
• Terms of Use
• Privacy Statement
• Crossref Member
• COUNTER Member  
• The MIT Press colophon is registered in the U.S. Patent and Trademark Office

This Feature Is Available To Subscribers Only

Sign In or Create an Account