A couple weeks ago I briefly explored self-driving cars in the context of consent to information-gathering. This week I want to revisit self-driving cars from a different lens: the moral dilemma presented by the trolley problem. The trolley problem is a classic thought experiment designed to explore whether moral dilemmas hinge essentially on processes or outcomes. The situation is often presented like this:
A runaway trolley is speeding down a length of track toward two-way division controlled by a switch, which you are standing next to. If the trolley maintains its current trajectory, it will strike and kill five people. If, however, you pull the switch, it will switch tracks and kill a single person. What do you do?- but wait! We haven’t reached the dilemma yet. Most versions of the thought experiment agree that sacrificing one to save five is considered the best outcome possible. We’ll revisit this assumption later, but for now let’s adopt it and move on to the first variant. Rather than pulling the switch to divert the train, imagine that you and another person are on a bridge over the track and you can save the group of five people below by pushing your fellow human over the edge and into the path of the train. He or she will die, but five others will be saved.
This is the proposed moral dilemma: does the process matter if the outcome is the same? If you pull the switch in the first scenario, one dies and five live. If you push the person over the edge in the second scenario, one dies and five live. Clearly a different process, but the same outcome, some contend. We can certainly argue convincingly that the ‘outcome’ of both scenarios is not the same in any real sense, but let’s focus solely on the ratio of life to death, which is the same.
I find that the power of this thought experiment lies in its ability to reveal tendencies and inconsistencies in our anticipated behavior by constructing multiple variations. For example, consider changing around your relation to each person. What if you know the entire group of five, but not the individual person? Or what if the individual is your soulmate and the five are strangers? Or perhaps the group of five are your coworkers and one of them covered your shift, so now they are on the track instead of you. The possible variations are endless.
Relations is simply one category. You could: 1) change the race, ethnicity, age, or gender of the people; 2) presume that the single person wishes to die to save five others, or that the group of five unanimously agrees to be sacrificed for the individual; 3) change how much information you have, how certain you are, or whether the predicted outcome is guaranteed to occur; or 4) throw non-human life into the mix. It may be instructive to explore the degree to which any or all of your constructions influence your anticipated behavior. And don’t forget to explore the difference in process- that is, between throwing the switch and pushing someone to their doom.
Now let’s consider the thought experiment in the context of self-driving cars. Certainly there is an argument to be made that A.I. can better anticipate and react to potential accidents and will therefore reduce their frequency. This may well be validated when the technology is scaled up, although I suspect that in the process, the consequences of having relieved humans of the responsibility of being fully present during periods of intense interpersonal coordination through shared spaces will be largely ignored until it becomes apparent that the process of driving actually contributed to some extinct form of social cohesion. Setting this speculation aside and assuming that the overall frequency or intensity of accidents will decline, the rate will certainly not reach zero, and the potential for the trolley dilemma remains.
The presence of A.I. adds another dimension to the original thought experiment. If a computer is determining whether to pull the switch, push a person, or do nothing (that is, act through inaction), your role shifts from decider to observer. In a sense, the choice that was made by some people at some point to imbue technology with decision-making power often fades into the ever-expanding background of ‘simply how things are’. What choice did we have, really? Such is the nature of progress, and we must learn to cope with the reality that our technology will face moral dilemmas and therefore must be programmed to choose between certain processes and outcomes associated with them. When this happens, we might ask ourselves what qualities are needed to make such a choice? Reason, or course, and computers are quite efficient reasoners. But what else? Are wisdom, compassion, empathy, or love needed to make moral decisions? If so, can these capacities emerge in computers, even the most ‘advanced’ ‘self-learning’ A.I.? And if these capacities are not needed to make moral decisions, what are they for? And what is needed for moral decision-making?
Up to this point, these considerations have been in the context of the original assumption of the thought experiment: sacrificing one life to save five is the best possible outcome. I think it is likely that the situational context of the moral dilemma might shake loose this dogma in certain instances- but again, we can set this aside for now. I do want to draw attention to the assumption though, because this seems to be relevant when we consider how the normative prescriptions from a certain era perpetuate over time, and how it is that cultures with different notions of progress can be compared and determined to be more or less advanced than the others.
As with all technology, from language to the Mars rover, it does not create itself independent of human action. In terms of the dynamics of living systems, our technologies are not alive because they are not systems that are self-organizing, maintaining, and reproducing. Even though they are (like living systems) structurally open to flows of energy and matter, they are more or less produced and programmed by humans, not by themselves. This is an essential difference between a frog and a self-driving car: a frog’s structure contributes to the reproduction of its organizational pattern, whereas a self-driving car’s structure perpetuates its function, not its form. This means that a self-driving car’s moral (and directional) compass is ultimately determined by the people who designed it, even if its algorithms are programed for continuous contextual “learning”.
I certainly hope I am never faced with having to choose my way through a real-life trolley dilemma. But if I do, would rather the choice be mine than than that of an anonymous team of programmers embedded in a particular corporate identity that developed in a specific culture with its own histories and trajectories. Why would I rather be forced to choose than relieved of this responsibility? There are several reasons worth considering, many of which are common currency for discussion: agency, cultural reflexivity, and the essence of humanity, to name a few.
But one that is perhaps less obvious is the idea that humans may need to flex their morality muscles to keep them sharp, so to speak. What happens when we choose to transfer more and more of our moral decision-making powers to our technologies? Might we lose some of the depth of our human experiences that are closely tied to considerations of the fate of others? It seems that not all moral decisions have equal weight or significance, and it is worth considering what happens when we only wade into the shallow waters of morality, letting our technologies take the deeper dives. As with many thought experiments regarding moral dilemmas, considering the underlying assumptions of the question itself can be just as instructive- and morally relevant- as pursuing the intended dilemma.
About the Author: Matt Nock is a third-year PhD student in the School of Sustainability at Arizona State University. His research is at the intersection of sustainability education and public pedagogy, and focuses on the ways in which the dominant discourses of social and environmental oppression are reproduced or challenged through formal and informal learning processes.