A Compositional Semantics for Graphics

Abstract

In this paper a theory for developing "intelligent" interactive graphic systems is detailed. The Fregean compositionality principle is enunciated for graphical representations. Geometrical symbols and relations receive semantic interpretations which are expressed as first order relations in the first order logical language. These interpretations are introduced with the help of deictic expressions. Deictic expressions constitute one associative mechanism between analogical representational systems used in graphics, and functional representational systems commonly used in AI applications. Interpretations of graphical symbols and geometrical relations between them constitute an ontology upon which complex linguistic interpretations are assigned to graphics. A concept of graphical grammar is introduced. Frege's compositionality principle, and the notion of graphical grammar lead to a concept of meaningful drawing. The graphical grammar constitutes a second associative mechanism between the two representational systems that have been mentioned. The truth conditions for relations in the graphical grammar are computed through geometrical knowledge. Computational geometry algorithms are associated with the high level representational system by means of the graphical grammar. Semantic interpretations of graphics are useful in carrying out natural language-like dialogue about graphical representations. Dialogues refer to true facts in particular interactive states, A notion of interactive state as a function of time and situation is then developed. One example of how this theory can be used in linking interactive graphics with AI applications is given. In the example, the semantic interpretation of a geographical map is constructed. This theory has been tested with an experimental program called GRAFLOG. The program is implemented in PROLOG and GKS.