In case you came here by an unexpected route, this
an introduction to the part of my site whose entry page is
That's the "Home" link in the navigation bar above.
I wrote this and the other pages to raise money
for research I'm doing, and for a web-based
program to demonstrate category theory. I'll say more about
A good way to start is with the image I put onto
my PayPal subscription form. I uploaded an
image containing this:
But unfortunately, PayPal cropped it and then
proceeded to wreck the quality as below.
One good thing, however, is that it gives me an
excuse to introduce the topics I'll probably write
most of my articles about: art, meaning, maths, and
cartoons. I'll start with
the need for more sophisticated
image processing, and then explain how this relates to the objects
depicted in the image. The common themes are semiotics (the study
and category theory (a branch of mathematics which I and others
have been applying to semiotics). Towards the end,
I'll also explain why I am, unusually for my website,
asking readers to pay.
Think of downscaling an image as pouring its meaning from a big container into a smaller. If the second container is too small to hold all of the meaning, what can we throw away in order to do the least damage to that meaning?
Pocket cartoonists — Matt in the Daily Telegraph is an excellent example — face this problem every time they draw a face. Their newspapers don't give them much room, so if they drew a person with normal proportions, the expression would be unreadable. They therefore make the face bigger. Often much bigger, perhaps even as large as the torso. Likewise, cartoonists exaggerate hands and hand gestures.
Faces and hands, we might say, are critical zones. When space is restricted, they should get priority at the expense of less important parts of the figure. Graphics programs such as Photoshop would be much more useful if they could identify such zones and preferentially preserve their meaning. To squish pixels together regardless of content is as destructive to meaning as summarising a scientific paper by dumping every second word into the bin.
True, these programs provide geometric transformations such as rotation, reflection, translation (i.e. movement from one point to another), and scaling. These affect points in the image in well-defined ways, familiar to anyone who has programmed with matrices. But what we really need is programs that provide semiotic transformations. Since semiotics is the study of meaning, semiotic transformations are those that affect meaning in well-defined ways. The transformation I suggested above — identifying and preferentially enhancing critical zones — is one such.
But before we can implement such transformations, we need to understand how images convey meaning. This might seem obvious: surely it's just about the objects in the image? But that isn't enough. What about non-representational imagery, such as abstract paintings and decorative motifs on clothes? What about moody skies and threatening clouds? Why did the illustrators want the coat and the cape to blow in the wind? In fact, why do they like their heroes to wear capes?
One aspect of image semiotics is demonstrated by the figure at the bottom centre of my diagram. This shows a boteh — paisley motif — photographed from a Jake velvet jacket and rotated into four different orientations. To me, each rotation has its own unique "feel". This probably comes about because my brain is trying to interpret the curved lines as facial expressions. Some seem to smile, and some to frown:
There's more about this in my first article, "The Power of Line". Briefly stated, the images don't denote anything: that is, don't stand for an object or an idea. But they do connote something: they evoke some kind of feeling. It's hard to say precisely what, but proving it's there is simple. Looking at feels different from looking at . That is,
Connotation( ) ≠ Connotation( )
But for two things to differ, at least one must be different from zero (and from anything else you care to name). Therefore, at least one of the connotations is non-zero.
Much more can be written about image semiotics, and I discuss some of it in "The Power of Line" and my second article, "The Use of Line". But I now want to turn to mathematics. Semioticians have, unfortunately, produced some remarkably obscure writing. Daniel Chandler in "Semiotics for Beginners" remarks that
The writings of semioticians have a reputation for being dense with jargon: as one critic put it, 'Semiotics tells us things we already know in a language we will never understand' (Paddy Whannel, cited in Seiter 1992: 1).
The trouble is that semiotics involves some very complicated concepts, and these cannot be written down intelligibly without mathematics. A parallel from the history of algebra is this word problem and its solution, from this passage in al-Khwārizmī's Algebra, c. 830 AD:
What must be the amount of a square, which when twenty-one dirhems are added to it, becomes equal to the equivalent of ten roots of that square?
Halve the number of roots; the moiety is five. Multiply this by itself; the product is twenty-five. Subtract from this the twenty-one which are connected with the square; the remainder is four. Extract its root; it is two. Subtract this from the moiety of the roots, which is five; the remainder is three. This is the root of the square which you required and the square is nine …
How much easier had al-Khwārizmī had modern algebraic notation, and could simply have written 𝑥2 + 21 = 10𝑥. How much easier if semioticians could do the like.
And they can. As I said at the outset, the branch of mathematics known as category theory has been applied to semiotics. I'll discuss this in future articles, so for the moment, let's just see an experiment from my own work. It's still preliminary, but is inspired by category theory — in particular, the view that the relations between objects are more important than the objects themselves. Can we formulate artistic style in terms of relations?
Suppose we depict an object, such as a boy's head, in two different styles. One is a flat style using colour to distinguish different regions. The other is a cartoony black-and-white style which emphasises outline rather than fill: in this case, the minimalistic style of Punch cartoonist J. W. Taylor. This style has trouble with blond hair, because if lines are used to depict individual locks of hair, they are necessarily dark. Hence, cartoonists tend to imply the main mass of blond hair, leaving it undrawn and using lines only for locks on the boundary:
Each of the two diagrams has two parts. On the left is what we can call a "schema" or "blueprint". This is a realistic rendering of a head and one of its parts. On the right are renderings of the head and the same part, but in both styles.
Now consider the parallelogram of arrows in the right-hand half of the left-hand diagram. At the top is the head in colour-fill style; at the bottom is an ear in J. W. Taylor style. What I intend the arrows to say is that we can get to the Taylor ear in two ways. We can take the colour-fill head; grab its ear; and restyle that. Or we can take the colour-fill head; restyle it; and then grab the ear from the result. In symbols:
Taylorise( ear-of( ) ) = ear-of( Taylorise( ) )
However, this doesn't work with the parallelogram in the second diagram. In the colour-fill style, we can find a region that corresponds to the patch of hair above the ear. But in the Taylor style, we can't. This is because a patch of hair is represented only by (some of) its boundary. There are no marks depicting the hair within the boundary, so without context, we can't tell whether this inner region does represent hair. Taylor's style is not context-free; we can't reliably translate between it and the colour-fill style.
I'll explain this further in my third article, Artistic Style, Mirror-Earth, and the Twelve O'Clock Rule. For now, let's just say that mathematics is the most precise language humanity has, and that the part of it called category theory seems to be ideal for formulating semiotics.
But why bother? Semiotician Graham Joncas answers this in a highly readable blog post explaining the work of the person who (as far as I know) first applied category theory to semiotics, Joseph Goguen. Amongst other things, he shows how Goguen used category theory to model how metaphors get their meaning. This is Joncas' conclusion:
It's clear that the tools are in place for a formal science of signs. Goguen's algebraic semiotics was developed with working examples implemented in OBJ* code. The main barrier has simply been that experts in semiotics have never even heard of ideas like colimits† or universal algebra‡. Again, all of this is realizable right now — all that's missing is someone willing to do the dirty work.
Radical ideas like 'cognitive ergonomics' are often tossed around for selling snake oil, but Goguen opens up the tantalizing thought that foundations for this could truly exist. We can speculate on an algebraic semiotics software added to design workflows like a debugger, optimizing user experience and potentially avoiding disastrous design flaws. We can imagine a semiotic branch of numerous sciences, such as computational biosemiotics giving us algebraic models of animal communication.
Overwhelmingly, semiotics is used as an academic acrolect to ensure that people can 'talk the talk', as well as dressing up insipid research to sound radical and profound. It's time for semiotics to finally live up to its potential, as the kind of unified theory that gives post-structuralists nightmares.
My own take on this is that it should be possible to develop a "semiotic compiler". A compiler translates computer programs into instructions that can be run by the computer's hardware. By analogy, a semiotic compiler would take high-level descriptions of some semiotic object such as an artwork, or task such as summarising a text or shrinking an image, and translate it into runnable code that can create the object or perform the task. As these tasks so often involve creativity, we might equally well speak of a "creativity compiler".
However, I don't expect this to be easy. To see one reason why, we need to look at biology. I have already mentioned one quirk of brain design, namely the tendency to see faces where there are none. There are many others; and because brains and sensory organs are not designed by engineers for ease of comprehension, they won't be easy to chop up into nice modular specifications.
I can't emphasise this
enough, because few researchers
seem to think of aesthetics and semiotics
as biological disciplines. So I'll
co-opt one of my favourite stories to demonstrate.
It's "The Digital Dictator", by
mathematician, science-fiction writer, and science
populariser Ian Stewart:
In Ian's story, protagonist Oliver Gurney has found a novel way to simulate human personalities on the computer. Unlike many attempts at artificial intelligence, this doesn't try to break the mind down into comprehensible parts and understand how it works. Instead, it's more like photography, which simulates a scene without analysing it into pieces. Gurney's point is that such analysis will give little insight into how the brain works, because of the way evolution grabs at any design going, no matter how Heath Robinson:
"Hmp," said Olly skeptically. "Oh, some insight, I suppose. "But evolution is a terrible opportunist. If it can cannibalize existing components, or make one structure perform several tasks, it will. That produces the damnedest designs."
Oliver is rather like a rusty tap. It takes quite an effort to turn the flow on — but once it is on, it's almost impossible to turn it off again. "Look," he said, "let me give you a simple analogy. Imagine an evolving house. One day, the bedroom decides it would be nice to have a door to block up the hole that you enter it by, so it evolves one. What do the other rooms do?"
"They evolve doors too," I said.
"Ah. In a designed system, maybe. In an evolved one, they'll just as likely come up with the door-sharing mechanism that transports the door round the house to whichever rooms needs it. And once the transporter has evolved, the house may discover that it functions pretty well with only one window and one ceiling. You get some sort of dynamic entity, constantly reshuffling itself into the currently desirable form.
"After all," he went on, "look at genetics. Overlapping genes; genes that come in several pieces — damn it, overlapping genetic codes! It's a designer's nightmare! Sure, it all works — thanks to a billion years of R&D. But do you really think that a designed system can mimic that kind of structure? Never!"
Such evolved messiness means that capturing semiotics with hard-edged formulations will never be entirely possible. Thus, Joseph Goguen in "An Introduction to Algebraic Semiotics" notes that although we can't expect our semiotics models to be perfect, a precise description that's somewhat wrong is better than a description so vague that no-one can tell whether it's wrong.
The point applies to models generally, of course. As writer — and ex–marine-mammal biologist — Peter Watts says:
And the virtues of reducing model complexity were hammered home to me on Day One by my old doctoral mentor Carl Walters, who never let us forget that any model as complex as Nature is going to be as difficult to understand as Nature. In a lot of cases, hi-def realism is the last thing you want in your models; they have [to] be caricatures if they're going to generate useful insights.
Category theory is a great source of organising principles, as Goguen noted in his "A Categorical Manifesto". It can help with formulating definitions and theories; and as point four in the link says, can help achieve independence from the often overwhelmingly complex details of how things are represented or implemented. This is what we need. Not hi-def realism, but precise caricatures that we can understand well enough to tell when they're wrong.
This leads me on to the figure at the
top-right of my opening image. It's
part of a diagram, of a kind typically
used by mathematicians to represent the colimits that
Joncas alludes to. Here's an extreme
a phenomenon called "universality".
I animated it in 3D to make it
That diagram, and the one in my opening image, were generated by a category-theory demonstrator I wrote and put onto the web. Users describe a category-theoretic construct that they'd like an example of, and the program calculates the rest of the example, then displays it as a diagram and a collection of mathematical elements. This is useful, because as with any field of mathematics, students of category theory need to work through many examples in order to get a feel for the subject. As mathematical physicist John Baez said about the program's calculations:
Maybe they're "low-level" — but maybe that's great! A lot of people think category theory is "too abstract". Seeing limits§ worked out in a very low-level concrete way‖ should correct any misimpression that they're some sort of airy nonsense.
My demonstrator will help students who want to apply category theory to semiotics, but it's equally useful to those learning it for other reasons. When I launched it, it attracted lots of enthusiastic comments, some submitted to the Graphical Category Theory Demonstrations thread at The n-Category Café blog. Baez's was one of those.
Unfortunately, although requests for improvements were plentiful, money wasn't. I never found anyone to fund the demonstrator, so I pay for the server out of my own resources. I would love to have money to move it to a faster server, to improve the diagrams and layout, and to add new calculations of the kind suggested in the thread. It would — let's be honest here — be more pleasurable than the stuff I do to earn a living, and I'd feel that I was contributing something rarer, more valuable, and that better uses my skills.
When I think what could have been done in the years since the launch, this is incredibly disappointing.
So that's one thing I'd use subscriptions and donations raised here for: enhancing the category-theory demonstrator. Another is implied by Graham Joncas:
The main barrier has simply been that experts in semiotics have never even heard of ideas like colimits or universal algebra. Again, all of this is realizable right now — all that's missing is someone willing to do the dirty work.
The point here is that as well as knowing some category theory, I'm a cartoonist. So I know and feel the semiotics of images from the inside, as it were, because of my experience with artistic techniques. I suspect from their writing that not all semioticians have this. So I can help bridge the gap, explaining category theory to semioticians, and semiotics to category theorists.
In fact, I am writing a book on category theory and semiotics. In other words, doing the dirty work. Subscriptions and donations would pay for that too.
I hope this introduction has made sense. It's tremendously difficult to explain why a branch of mathematics is useful, to someone who doesn't already understand it. You have to have fought your way through examples, and then worked with the maths for long enough to breed a feeling of success, before you gain a gut conviction of its worth.
The same goes for art. If you don't draw or paint, you may not understand why art techniques and the semiotics of images interest me. Nevertheless, you surely have interests of your own. So even if you don't enjoy mine, you should be able to imagine transferring your own feelings of enjoyment from subjects you like to a subject I like.
And these are probably the best I can do as a populariser. I can't pack the moral equivalent of a treatise on quantum mechanics into nine pictures and a webpage; but I can show that there are some nifty ideas, hint at applications, try to persuade that these ideas are worth taking further, and show that there's a gap to be bridged.
* OBJ is a program-specification language: that is, it describes what a program should do rather than how. Being close to mathematics notationally, it's easy to reason about; and it's modular, meaning big OBJ programs can be built from smaller pieces that can be developed independently. It is based heavily on category theory.
† Colimit is a category-theoretic analogue to addition, useful for constructing descriptions of systems from those of their components. The systems can be anything: electronic; economic; even computational. OBJ uses them to put modules together. What's relevant here is that colimits also apply to semiotic systems — systems of signs and meanings. A classic example is that Goguen's work on metaphors used colimits to generate the metaphors' meanings from those of their parts.
‡ Universal algebra is a field of mathematics closely related to category theory.
§ Limits are structurally the same as colimits, but in diagrams, their arrows all point the other way. A colimit builds a description of a system from descriptions of its parts plus specifications of which parts are shared; a limit calculates the behaviour of a system from the behaviours of its parts plus specifications of how the parts restrict each other when connected.
‖ Using the set-theoretic notions of set, function, Cartesian product, disjoint union, and quotient. If you know these, you know enough to calculate the limit and colimit of a network of sets.
Daniel Chandler (1994) "Semiotics for Beginners", http://web.pdx.edu/~singlem/coursesite/begsem.html .
Joseph Goguen (1991) "A Categorical Manifesto", Mathematical Structures in Computer Science Volume 1 Number 1, http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.13.362 .
Joseph Goguen (1998) "An Introduction to Algebraic Semiotics, with Applications to User Interface Design", https://cseweb.ucsd.edu/~goguen/pps/as.pdf .
Jocelyn Ireson-Paine (2008) "Category Theory Demonstrations", http://www.j-paine.org/cgi-bin/webcats/webcats.php .
Jocelyn Ireson-Paine (2014) "Jocelyn's Cartoons", http://www.jocelyns-cartoons.co.uk/ .
Jocelyn Ireson-Paine (2021) "The Medium Wrecks the Message: Describing Artistic Style Using a Relational View of Art", Polish Journal of Aesthetics Issue 62 (3/2021), https://pjaesthetics.uj.edu.pl/en_GB/archives/-/journal_content/56_INSTANCE_r1lnMup4DOPq/138618288/149774426 .
Jocelyn Ireson-Paine, with replies by John Baez, Aaron Denney, Urs Schreiber, Jacques Distler, Mike Stay, Jürgen Koslowski, and Tom Ellis (thread starting 7.4.2009) "Graphical Category Theory Demonstrations", The n-Category Café, https://golem.ph.utexas.edu/category/2009/04/graphical_category_theory_demo.html .
Graham Joncas (26.12.2020) "Algebraic Semiotics: Joseph Goguen's Semiotic Morphisms", Oneironomics, https://gjoncas.github.io/posts/2020-12-26-algebraic-semiotics.html .
Mike Lynch (18.5.2008) "J.W. Taylor", Mike Lynch Cartoons, http://mikelynchcartoons.blogspot.com/2008/05/jw-taylor.html .
"Matt" (n.d.) "Matt cartoons", Daily Telegraph, https://www.telegraph.co.uk/news/matt/ .
nLab (n.d.) "A Categorical Manifesto", nLab, https://ncatlab.org/nlab/show/A+Categorical+Manifesto .
Ian Stewart (2021) "The Digital Dictator", in Message from Earth and other SF stories, available via https://ianstewartjoat.weebly.com/ . Originally published in Analog August 1982.
Peter Watts (25.3.2021) "What Dreams May Come: Interrogating the Dream", No Moods, Ads or Cutesy Fucking Icons, https://www.rifters.com/crawl/?p=9844 .
My PayPal subscriptions-page header. Assembled from a variety of images, including some from "The Power of Line" and one from my Polish Journal of Aesthetics paper. The figure at the top right is part of a results page from the category-theory demonstrator.
Film poster for 2008 film Jumper, https://en.wikipedia.org/wiki/Jumper_(2008_film) .
"Military Leader standing triumphant with cape blowing in the wind", https://www.bigstockphoto.com/image-142924484/stock-photo-military-leader-standing-triumphant-with-cape-blowing-in-the-wind .
Jake cropped velvet jacket hanging from tree.
Boteh motif from the jacket, in different orientations.
Figures from the Polish Journal of Aesthetics paper.
First two pages of "The Digital Dictator" from Analog. Scan kindly supplied by Ian Stewart.
Animated diagram from the category-theory demonstrator, generated in VRML (Virtual Reality Modeling Language).