On the legibility of humans
Ezra Klein recently described a strange anxiety taking hold in Silicon Valley, that many technologists are increasingly concerned about improving the AI "legibility" of their lives.1 The irony is that LLMs are quite illegible themselves. Understandability remains an unsolved problem. Accountability is difficult not only because no human made the decision, but because we have no idea why the model reached its conclusion. And the emergence of intelligence from statistical token prediction seems more a philosophical debate more than a scientific one. Some are opting in willingly, creating digital clones of themselves to increase efficiency. But most of us are opted-in without consent. Corporations have used our data to maximize engagement and sell our attention, now it's being used by states to model human behavior, nominally to inform policy decisions.
Legibility is a necessary step in management: from corporations to states, now with our humanity. But the map is not the territory and attempts to put legibility first frequently lead to bad outcomes: unfriendly cities through high modernist urbanism, ecosystem collapse through monoculture. Legibility is the first step in the pursuit of goals, but we must remember that targets are based on simplifications.
All of this is to say that we've put the cart before the horse. As the recent Vatican encyclical warns: technology should be a tool that works for human dignity, not the other way around.