I originally posted this on 1/5/25 as the 2nd of a two-part post. This topic is so important that it needs a dedicated post. I cannot emphasize too much how important this new AI-powered front in propaganda war is. This is a lightly edited version of the original.
*****************************************************************
There is a new front in the war on human minds by unscrupulous marketers, ideologues, plutocrats, theocrats, politicians and other morally debased opportunists! On Dec. 30, 2024, the HSDR (Harvard Data Science Review) posted an article about this nascent new war. Viewed from my cognitive biology and social behavior point of view, this constitutes an entirely new political, religious and economic gold rush. The HSDR calls it the intention economy. The HSDR sees this new mind war about the same way I see it, scary or worse.
Long story short: Artificial intelligence gives the opportunists a way to discover and manipulate your intentions and motives to do or not do something, e.g., to buy or not buy a product or to vote for or against a political candidate. This new mind war is truly scary:
ABSTRACTThe rapid proliferation of large language models (LLMs) invites the possibility of a new marketplace for behavioral and psychological data that signals intent. This brief article [it's not brief, it's well over 4,500 words] introduces some initial features of that emerging marketplace. We survey recent efforts by tech executives to position the capture, manipulation, and commodification of human intentionality as a lucrative parallel to—and viable extension of—the now-dominant attention economy, which has bent consumer, civic, and media norms around users’ finite attention spans since the 1990s. We call this follow-on the intention economy. We characterize it in two ways. First, as a competition, initially, between established tech players armed with the infrastructural and data capacities needed to vie for first-mover advantage on a new frontier of persuasive technologies. Second, as a commodification of hitherto unreachable levels of explicit and implicit data that signal intent, namely those signals borne of combining (a) hyper-personalized manipulation via LLM-based sycophancy, ingratiation, and emotional infiltration and (b) increasingly detailed categorization of online activity elicited through natural language.This new dimension of automated persuasion draws on the unique capabilities of LLMs and generative AI more broadly, which intervene not only on what users want, but also, to cite Williams, “what they want to want” (Williams, 2018, p. 122). We demonstrate through a close reading of recent technical and critical literature (including unpublished papers from ArXiv) that such tools are already being explored to elicit, infer, collect, record, understand, forecast, and ultimately manipulate, modulate, and commodify human plans and purposes, both mundane (e.g., selecting a hotel) and profound (e.g., selecting a political candidate).CONCLUSIONThe possibility for harm made feasible by a large-scale, multiparty intention economy merits sustained scholarly, civic, and regulatory scrutiny. In whatever way these data partnerships turn out in practice, the ambition of making conversational interfaces and generative AI systems unavoidable mediators of human–computer interaction signals a turn from the attention economy, whereby access to the limited resource of human attention is traded through advertising exchanges, to the intention economy, whereby commercial and political actors bid on signals that forecast human intent. This transition would empower diverse actors to intervene in new ways on shaping human actions. This ambition must be considered in light of the likely impact such a marketplace would have on other human aspirations, including free and fair elections, a free press, fair market competition, and other aspects of democratic life.
All of that is going on right now. As far as I can tell, the mainstream media has not picked up on this yet. It constitutes a vast new playing field for bad people and interests to (i) manipulate minds and beliefs, (ii) create false realities, and scarier yet, (iii) implant desires and motivations in a person's mind.
Q: How well known is the concept of the intention economy, as exemplified in this article: https://hdsr.mitpress.mit.edu/pub/ujvharkk/release/1 ?A: The concept of the intention economy is gaining traction in academic and tech circles but remains relatively niche in broader public discourse.Attention vs. Intention: There's a distinction between the attention economy, which focuses on capturing user engagement, and the intention economy, which aims to predict and influence user decisions. Some discussions mistakenly conflate the two, but the intention economy goes beyond mere attention to actively shape consumer behavior.
Ethical Concerns: The intention economy raises significant ethical questions about privacy, autonomy, and manipulation. Researchers warn that without proper regulation, this economy could undermine fundamental societal values, including free and fair elections, a free press, and fair market competition.
Marketplace for Intentions: The intention economy envisions a marketplace where AI systems not only capture attention but also anticipate and influence user intentions. This could lead to a scenario where our decisions are influenced before we consciously make them, creating a new commercial frontier.
Hm, Perplexity sees it about the same way that HSDR and I see it. Well brain war fans, there we have it. A whole new front against us and our democracy has opened up and we are clueless.
Of course, this could also be used for good. But think about our current political, social, commercial, financial and religious situation. Where does the balance of power lie, with super wealthy, self-serving special interests and ideologies or public interest-serving ones?
Q1: What array of forces and finances is most likely to more aggressively and more deeply apply this new kind of mental manipulation, (1) morally bankrupt for profit commercial interests and various kinds of authoritarians seeking power and wealth, or (2) nice people wanting to do good things for people and society?
Q2: Which political party is most likely going to oppose regulating the intention economy for concentration wealth and power, the Repubs, the Dems, or both about equally?
Q3: Can you see the potential for a further shift in the balance of power from the seriously weakened public interest to the already more powerful special interests?