Existing approaches to Explainable AI try to dive into the inner workings of AI models and, for model developers, there is indeed a lot you can do to give more insight into that. However, for non-technical people, who just want to use the model’s output for further decision-making, insight into these inner workings may actually be very confusing. For them, AI models will always remain black boxes, no matter how hard you try to open them up. Even approximating the output of black box models with a simpler white box model, like linear regression, can still be too difficult for many people. I also think it’s a bad idea but that’s another story (discussed in this post).
Why do we insist on giving full insight into the working of AI? As humans, we have been surrounded by technology for decades. We have no clue how it really works but we’re totally fine relying on it for everything we do in our lives. The main reason why technology is so successful is that it works with layers of abstraction. We all use computers on a daily basis, e.g., a laptop running Windows, but the average user has no clue how the Windows operating system works. How do these menus and windows get displayed on screen exactly? That’s a layer too low for normal users. Operating system developers (those guys working for Microsoft) know all about the Windows OS but they, in turn, have no clue about device drivers and the hardware the control. That’s another layer deeper. Hardware engineers know all about the hardware but they’re not physicists who understand the physics of electrons. That’s again another layer. Technology works because, if the layer below you has been designed properly, you don’t need to worry about it. You just focus on the layer you’re good at. And if you do that right, the person in the layer above you doesn’t have to worry about your layer. She can just work in a her own layer of abstraction. That’s technology.
And now we have AI and suddenly people start to panic and worry that they don’t understand every detail that going on inside and AI’s mind. We insist on opening up the black box and showing what going on. I tell you that this is an illusion. You may be able to uncover one or two layers of abstraction below you but it’s impossible to go all the way down and understand things on a physical or even quantum-mechanical level. Perhaps we shouldn’t want to.
If we look at humans, do we know exactly what going on in their minds? We may think we do but in the end we really don’t. We just make assumptions and these assumptions are often wrong. How do we try to explain human behavior then? Well, we have psychology don’t we?
Could we not also use a similar approach to explain AI behavior? We already do that to some extent by assigning certain human-like properties to AI by saying
- The AI was confused
- The AI made a mistake
- The AI failed to
- The AI managed to
- The AI was ignoring
- The AI got lost
- The AI was discriminating against
When you Google “psychology of AI” you get some hits but these are mostly related to (a) the interaction between humans and AI or (b) the application of AI technology in psychology. Nobody talks about treating AI, or explaining its behavior, from a psychological perspective. There are many psychology subfields but behavioral psychology seems a field that particularly matches our AI problem. Behavioral psychology tries to explain and predict what humans do by looking at their actions and behaviors. Basically, this means treating the human mind as a black box and just focusing on the (sensory, historical, genetic) inputs and outputs (actions and behaviors). Of course, we often hear a similar complaint about AI, especially deep learning, that it’s a black box technology that gives no insight into its inner workings. Could we not approach the explanation of AI predictions in the same way as behavioral psychology does for humans, by focussing on inputs and outputs only?
I’ll try to describe in this post how we might approach this.
Behavioral psychology in humans
In humans, behavioral psychology is mostly used to change (mostly undesirable) behavior. Think of Ivan Pavlov who used the stimulus of a ringing bell to make dogs salivate because the ringing of the bell was always followed by the administration of food. The intended purpose is to make the dogs salivate after ringing the bell without administering food. That’s the behavioral change.
Behavioral psychology applied to AI models
In AI we have different goal. We’re not trying to change its behavior because AI is unable to learn from new data. It can only learn from training data that includes ground truth information. So instead of changing AI behavior, we want to understand its behavior. Like behavioral psychology that focuses on stimuli and behavior within the context of knowledge, we can perhaps explain AI models by focusing on their inputs and outputs within the context of their knowledge.
Any AI model has a body of knowledge that is encapsulated by the data it was trained on. When you give the AI some new (previously unseen) input, it will predict an output. How could we explain this output to a non-technical person who has no insight whatsoever in the inner workings of the AI model? There are two scenarios.
In the first scenario the AI is uncertain about its output. In that case, it should be able to explain why it’s uncertain by comparing the input with the data it was trained on and showing that the input is far removed from anything it has seen during training. It’s extrapolating and that causes it to be uncertain.
In the second scenario the AI is certain about its output. In that case, it should be able to explain why it’s certain by comparing the input with the data it was trained on and showing that the input is, in fact, very similar to the training data.
There are various ways to show that an input is either similar or dissimilar to the training data. This is a more of an unsupervised clustering problem. Look at this paper in Nature [1].
References