Unmaking Sense

We pick a text and compare it with four others texts using embedding.

From episode 58 of series 8 an embedding is a way of comparing so I thought that I would do a little bit of experimentation on texts similar and dissimilar from different texts to fraudulent texts that are manufactured and see what the results were, and the results were exactly what you would hope.

The ways in which we encode the natural language text that we use in order to communicate with a chatbot are fundamental to its effectiveness and its efficiency. Unfortunately, terminology like “embedding” and “encoding” tends to get thrown around with a certain amount of abandon as though the two are the same, when they are not. We start exploring the differences and seeing under what circumstances each of these techniques is used.

This is the 300th episode of a Unmaking Sense: Living the Present without Mortgaging the Future. Thanks to all my listeners who have stayed with me on this long journey. We celebrate by considering the possibility that by customising a chat bot like chatGPT with our personal characteristics as a final fine-tuned wrapper, we might achieve a kind of immortality.

This episode is in two parts: in the first we discuss the nature of fine-tuning, how it’s done and why it’s necessary; in the second part we discuss how at we as end-users can use our own data to customise the whole system to suit and serve better the particular interests that we have in our amateur or professional lives. We briefly discuss the costs of this process and how to do it.

In episode 54, we talked about propensities in human beings and in chatbots, and I suggested that the chatbots will embody to some extent certain characteristics that are reflected in the way they’ve been trained, on the choices that have been made by those training them. Extend that a little, and you can start to see how different chat bots from different parts of the world trained by different groups of people are likely to have different characteristics and so different personalities. So human beings, when they come to decide which chatbots to use, will be affected by those factors in just the same way that they are when they choose their own human partners, friends and collaborators. But the fact that we can wrap these chatbots with fine-tuning means that we can also tailor them to our own personal or corporate or collective interests in order to try to persuade them to give better responses to the things that we are interested in. But where does that start and where does it finish, and how safe are we from the misappropriation of chatbot technology by those who would wrap them in skins and give them personalities of a kind that we would deplore and justifiably find quite alarming, dangerous and frightening? Fine tuning will be in 8.56.

How the way a chatbot is trained imputes opinions and values that mean that it will take on the characteristics of its culture through its training.

Our artificial intelligence models should not be thought of as if they are trying to reproduce whatever it was that produced the phenomena that they are modelling. So if they produce something that looks vaguely like Shakespeare, that does not mean that they have recreated Shakespeare’s brain. If they produce reliable predictions of earthquakes, it doesn’t mean that they have reproduced a working theory that is adequate to the seismology of the Earth, all that matters is that their predictions are reliable or interesting or entertaining. My experience with Andrew Karpathy’s GPT model from YouTube, “Let’s build GPT: from scratch, in code, spelled out.”

The “Attention is All You Need” paper lies behind Andrew Karpathy’s excellent YouTube video “Let’s build GPT: from scratch, in code, spelled out”. We discuss some implications.

This episode might well be skipped by anyone who doesn’t like mathematical things, particularly the idea of thinking involved in three dimensions but if you’re remotely interested in what tensors have got to do with machine learning and therefore with chatGPT and large language models, it may be of interest. It’s very imperfect, and I am conscious that it may well be a bit of a peculiar episode and in the context of this series. I will come back to something much more down-to-earth episode 52.