My research develops the normative structure of relationships

I work on the special relationships between agents — the relationships that generate special reasons not derivable from impartial morality. I pursue this question in two domains: relationships between human agents, and the emerging relationships between humans and artificial agents. I take practical ethics to require both philosophical rigour and operational usefulness, and I aim to do work that meets both standards.

The first strand is the ethics of partiality. I examine when special concern for particular others is justified, the asymmetries between positive and negative ties — friendship and enmity — and the limits morality places on partial action. A monograph developing a systematic theory of justified negative partiality, The Ethics of Enmity, is in preparation.

The second strand is the ethics of AI. As humans enter sustained interactions with conversational systems, autonomous agents, and AI assistants, the central normative questions concern the structure of those interactions: when epistemic deference is warranted, who bears accountability when things go wrong, what licenses unilateral revision of an AI relationship, and how designers shape the moral environment of users. This work draws on seven years of applied collaboration with Google's research and engineering teams.

A third, cross-cutting strand concerns the practical role of ethics inside organizations. Here I examine how firms build genuine ethical capabilities, and what ethicists can contribute to industry without collapsing into compliance.

More information available on PhilPeople and Google Scholar.

Ethics of AI

AI Alignment and Fiduciary Obligation

Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES 2026), forthcoming

Advanced AI assistants engage users in extended interactions across a widening range of roles, including advice, decision support, collaboration, learning, emotional support, and companionship among others. Current alignment efforts consider what alignment criteria should govern these relationships, drawing on moral traditions developed for human relationships such as bioethics, virtue ethics, care ethics, and relationship science. This paper considers AI alignment criteria in the user-AI-developer triad, since every user-AI interaction is mediated by a developer who exercises discretionary control over a system's behaviour, memory, and engagement parameters. Drawing on business ethics and legal scholarship, I argue that fiduciary theory applies to extended AI assistant deployment. On this basis, the four canonical fiduciary duties of loyalty, care, good faith, and candour can generate alignment criteria for the developer-user relationship. I map four user-side risks of extended AI assistant deployment to the four duties and specify institutional measures that follow from discharging each duty. The discussion complements existing approaches by grounding alignment criteria in obligations the developer owes the user, rather than in values the user-AI interaction should promote, and by showing that those obligations hold independently of any de facto harm to users.

Relational Norms for Human–AI Cooperation

Humanities and Social Sciences Communications, forthcoming — with B. D. Earp, S. P. Mann, et al.

How we should design and interact with social artificial intelligence depends on the socio-relational role the AI is meant to emulate or occupy. In human society, relationships such as teacher–student, parent–child, neighbours, siblings, or employer–employee are governed by specific norms that prescribe or proscribe cooperative functions including hierarchy, care, transaction, and mating. These norms shape our judgments of what is appropriate for each partner. As AI agents and chatbots powered by large language models are increasingly designed to serve roles analogous to human positions — assistant, mental health provider, tutor, romantic partner — it is imperative to examine whether and how human relational norms should extend to human–AI interactions. Our analysis explores how differences between AI systems and humans, such as the absence of conscious experience and immunity to fatigue, may affect an AI's capacity to fulfil relationship-specific functions and adhere to corresponding norms. We propose that understanding and thoughtfully shaping suitable human–AI relational norms will be crucial for ensuring that human–AI interactions are ethical, trustworthy, and favourable to human well-being.

Epistemic Trust as a Mechanism for Ethics Integration: Failure Modes and Design Principles from 70 Moral Imagination Workshops

Journal of Responsible Innovation, forthcoming — with G. Keeling, K. Pedersen, C. Heringer, S. B. Rubin, B. Zevenbergen, A. McCroskery

Bottom-up responsible innovation initiatives aim to empower technology development teams to engage in ethical reflection, yet such interventions frequently fail to achieve practitioner engagement. Why do some ethics interventions succeed while others are dismissed as irrelevant or adversarial? This paper proposes epistemic trust — the degree to which practitioners regard an intervention, its facilitators, and its content as credible — as a conceptual model linking intervention design to engagement outcomes. Drawing on philosophical work on testimony and on practice-based qualitative analysis of over 70 moral imagination workshops with engineering teams between 2019 and 2025, we identify five dimensions of epistemic trust salient to ethics interventions (Relevance, Inclusivity, Agency, Authority, and Alignment) and present a typology of 23 failure modes that arise when these dimensions are inadequately addressed. We derive nine design principles for cultivating epistemic trust, grounded in our operationalization of moral imagination through technomoral scenarios and structured deliberation. Our findings contribute to the literature on collaborative socio-technical integration by specifying the conditions of uptake that existing frameworks leave undertheorized. We acknowledge limitations such as selection effects from voluntary participation and the absence of formal outcome measures, and position our failure mode typology as practitioner hypotheses warranting further empirical validation.

Ethics of 'Digital Duplicates' or AI Simulations of Real People: Towards an International Consensus

JME Practical Bioethics, July 2026 — with C. Hurshman, C. Voinea, J. Savulescu, B. D. Earp, et al.

Digital duplicates (DDs), or AI simulations of individuals, are artificial intelligence models designed to emulate real people. Depending on how they are trained, DDs can reproduce aspects of a person's psychology, values, and communication style, enabling interaction with digital versions of living, recently deceased, or historical individuals. Proposed uses include education, advocacy, grief support, medical decision-making, and the continuation of creative or personal projects. While potentially valuable, these applications also pose ethical risks, such as privacy loss, misrepresentation, manipulation, and the erosion of authentic human relationships. This paper, developed through an international experts meeting held in Singapore on 4–5 September 2025, argues that DDs should be treated as context-specific tools rather than general-purpose technologies. We outline a two-tier ethical framework grounded in considerations of harm and proportionality, supported by the values of information permission, transparency, authenticity, and appropriate access. The framework unites diverse moral concerns under a single framework while allowing their relative weight to vary across domains such as medicine, education, memorialization, and entertainment. By situating ethical evaluation within a harm-reduction approach that is sensitive to context, we offer a practical basis for the responsible development and governance of human-emulating AI systems.

Calibrated Epistemic Deference to Conversational AI in Mental Healthcare

The American Journal of Bioethics, 2026

Sedlakova et al. (2025) argue that conversational AI (CAI) cannot satisfy the conditions of epistemic trust in therapeutic contexts. I accept their diagnosis but press a further question: given that patients will de facto treat CAI outputs as reasons, how should patients, clinicians, and designers calibrate their epistemic responses? I argue that even where epistemic trust is inappropriate, calibrated epistemic deference remains rational. On a total evidence view, CAI outputs function as defeasible contributory reasons whose weight is proportional to demonstrated reliability and context-sensitive. Against preemptionism, I show that preemptive CAI deference is distinctively objectionable in therapeutic contexts: it undermines precisely the epistemic capacities — self-knowledge, critical reflection, and reason-integration — that psychotherapy aims to develop. An open question is whether calibrated deference is dynamically stable given users' documented tendency to anthropomorphize CAI over extended interaction, which may cause deference to drift toward the quasi-trust that therapeutic contexts warrant resisting.

Epistemic Deference to AI

In B. Steffen (ed.), Bridging the Gap Between AI and Reality, AISoLA 2024. Lecture Notes in Computer Science, vol. 16032, Springer, 2026

When should we defer to AI outputs over human expert judgment? Drawing on recent work in social epistemology, I motivate the idea that some AI systems qualify as Artificial Epistemic Authorities (AEAs) due to their demonstrated reliability and epistemic superiority. I then introduce AI Preemptionism, the view that AEA outputs should replace rather than supplement a user's independent epistemic reasons. I show that classic objections to preemptionism — such as uncritical deference, epistemic entrenchment, and unhinging epistemic bases — apply in amplified form to AEAs, given their opacity, self-reinforcing authority, and lack of epistemic failure markers. Against this, I develop a more promising alternative: a total evidence view of AI deference. According to this view, AEA outputs should function as contributory reasons rather than outright replacements for a user's independent epistemic considerations. This approach has three key advantages: (i) it mitigates expertise atrophy by keeping human users engaged, (ii) it provides an epistemic case for meaningful human oversight and control, and (iii) it explains the justified mistrust of AI when reliability conditions are unmet. While demanding in practice, this account offers a principled way to determine when AI deference is justified, particularly in high-stakes contexts requiring rigorous reliability.

Beziehungen mit KI

In K. Krüger & R. Geiger (eds.), KI und Demokratie: Aktuelle Fragen, Springer, 2026

Mensch-KI Beziehungen nehmen in unserem Alltag eine immer zentralere Rolle ein: etwa in der Interaktion mit Chatbots, KI-Companions, digitalen Assistenten oder virtuellen Avataren. Dieses Kapitel untersucht, ob und inwiefern sich zwischen Menschen und KI-Systemen Beziehungen entwickeln können und welche ethische Relevanz solchen Beziehungen zukommt. Ziel ist es, Orientierungswissen für die ethische Bewertung zukünftiger Mensch-KI-Interaktionen bereitzustellen. Ausgehend von einer Systematisierung momentaner Mensch-KI-Interaktionen wird der Beziehungsbegriff konzeptuell geschärft und auf seine philosophisch normative Bedeutung hin analysiert. Es wird untersucht, ob KI-Systeme als genuine Beziehungspartner gelten können oder ob die dabei entstehenden Bindungen notwendigerweise asymmetrisch und einseitig bleiben. Ein zentrales Argument ist, dass die normative Debatte nach der Plausibilität von genuinen Mensch-KI Beziehungen im Gesamtkontext von unseren normativen Handlungsgründen betrachtet werden sollte. Anschließend werden drei Ansätze zur ethischen Gestaltung von Mensch-KI Beziehungen vorgestellt und auf ihre Anwendbarkeit hin erläutert.

We Need Accountability in Human–AI Agent Relationships

npj Artificial Intelligence 1 (38), 2025 — with G. Keeling, A. Manzini, A. McCroskery

We argue that accountability mechanisms are needed in human–AI agent relationships to ensure alignment with user and societal interests. We propose a framework according to which AI agents' engagement is conditional on appropriate user behaviour. The framework incorporates design strategies such as distancing, disengaging, and discouraging.

Digital Duplicates and Collective Scarcity

Philosophy & Technology 38 (2025): 1–8

Digital duplicates reduce the scarcity of individuals and thus may impact their instrumental and intrinsic value. I here expand upon this idea by introducing the notion of collective scarcity, which pertains to the limitations faced by social groups in maintaining their size, cohesion and function.

A Framework for Assurance Audits of Algorithmic Systems

Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT '24): 1078–1092 — with K. Lam, B. Blili-Hamelin, J. Davidovic, S. Brown, A. Hasan

An increasing number of regulations propose the notion of AI audits as an enforcement mechanism for achieving transparency and accountability for AI systems. Despite some converging norms around various forms of AI auditing, auditing for the purpose of compliance and assurance currently have little to no agreed upon practices, procedures, taxonomies, and standards. We propose the criterion audit as an operationalizable compliance and assurance external audit framework. We model elements of this approach after financial auditing practices, and argue that AI audits should similarly provide assurance to their stakeholders about AI organizations' ability to govern their algorithms in ways that mitigate harms and uphold human values. We discuss the necessary conditions for the criterion audit, and provide a procedural blueprint for performing an audit engagement in practice. We illustrate how this framework can be adapted to current regulations by deriving the criteria on which bias audits for hiring algorithms can be performed, as required by the recently effective New York City Local Law 144 of 2021. We conclude by offering critical discussion on the benefits, inherent limitations, and implementation challenges of applying practices of the more mature financial auditing industry to AI auditing where robust guardrails against quality assurance issues are only starting to emerge.

Moral Imagination for Engineering Teams: The Technomoral Scenario

International Review of Information Ethics 34(1), 2024 — with G. Keeling, A. McCroskery, K. Pedersen, D. Weinberger, B. Zevenbergen

"Moral imagination" is the capacity to register that one's perspective on a decision-making situation is limited, and to imagine alternative perspectives that reveal new considerations or approaches. We have developed a Moral Imagination approach that aims to drive a culture of responsible innovation, ethical awareness, deliberation, decision-making, and commitment in organizations developing new technologies. We here present a case study that illustrates one key aspect of our approach — the technomoral scenario — as we have applied it in our work with product and engineering teams. Technomoral scenarios are fictional narratives that raise ethical issues surrounding the interaction between emerging technologies and society. Through facilitated role-playing and discussion, participants are prompted to examine their own intentions, articulate justifications for actions, and consider the impact of decisions on various stakeholders. This process helps developers to re-envision their choices and responsibilities, ultimately contributing to a culture of responsible innovation.

The Impact of Intelligent Decision-Support Systems on Humans' Ethical Decision-Making: A Systematic Literature Review and an Integrated Framework

Technological Forecasting & Social Change 204 (2024): 123403 — with F. Poszler

With the rise and public accessibility of AI-enabled decision-support systems, individuals outsource increasingly more of their decisions, even those that carry ethical dimensions. Considering this trend, scholars have highlighted that uncritical deference to these systems would be problematic and consequently called for investigations of the impact of pertinent technology on humans' ethical decision-making. This article conducts a systematic review of existing scholarship and derives an integrated framework that demonstrates how intelligent decision-support systems (IDSSs) shape humans' ethical decision-making. We identify resulting consequences on an individual level (deliberation enhancement, motivation enhancement, autonomy enhancement and action enhancement) and on a societal level (moral deskilling, restricted moral progress and moral responsibility gaps). We carve out two distinct operation types — process-oriented and outcome-oriented navigation — that decision-support systems can deploy and postulate that these determine to what extent the previously stated consequences materialize.

Engaging Engineering Teams Through Moral Imagination: A Bottom-Up Approach for Responsible Innovation and Ethical Culture Change in Technology Companies

AI & Ethics, 2023 — with A. McCroskery, B. Zevenbergen, G. Keeling, S. Blascovich, K. Pedersen, A. Lentz, B. Aguera y Arcas

We propose a 'Moral Imagination' methodology to facilitate a culture of responsible innovation for engineering and product teams in technology companies. Our approach has been operationalized over the past two years at Google, where we have conducted over 40 workshops with teams from across the organization. We argue that our approach is a crucial complement to existing formal and informal initiatives for fostering a culture of ethical awareness, deliberation, and decision-making in technology design such as company principles, ethics and privacy review procedures, and compliance controls. We characterize some distinctive benefits of our methodology for the technology sector in particular.

The Current State of AI Governance

Whitepaper, Algorithmic Bias Lab, 2023 — with J. Davidovic, A. Hasan, K. Lam, M. Regan, S. Brown

As AI, machine learning algorithms, and algorithmic decision systems (ADS) continue to permeate every aspect of our lives and our society, the question of AI governance becomes exceedingly important. This report examines the current state of internal governance structures and tools across organizations, both in the private and public sectors and in large and small organizations. This report provides one of the first robust and broad insights into the state of AI governance in the United States and Europe.

Algorithmic Bias and Risk Assessments: Lessons from Practice

Digital Society 1(2) (2022): 14 — with A. Hasan, S. Brown, J. Davidovic, M. Regan

In this paper, we distinguish between different sorts of assessments of algorithmic systems, describe our process of assessing such systems for ethical risk, and share some key challenges and lessons for future algorithm assessments and audits. Given the distinctive nature and function of a third-party audit, and the uncertain and shifting regulatory landscape, we suggest that second-party assessments are currently the primary mechanisms for analyzing the social impacts of systems that incorporate artificial intelligence. We then discuss two kinds of assessments: an ethical risk assessment and a narrower, technical algorithmic bias assessment. We explain how the two assessments depend on each other, highlight the importance of situating the algorithm within its particular socio-technical context, and discuss a number of lessons and challenges for algorithm assessments and, potentially, for algorithm audits.

Combating Disinformation with AI: Epistemic and Ethical Challenges

2021 IEEE International Symposium on Technology and Society (ISTAS) Proceedings: 1–5 — with T. M. Lechterman

AI-supported methods for identifying and combating disinformation are progressing in their development and application. However, these methods face a litany of epistemic and ethical challenges. These include (1) robustly defining disinformation, (2) reliably classifying data according to this definition, and (3) navigating ethical risks in the deployment of countermeasures, which involve a mixture of harms and benefits. This paper seeks to expose and offer preliminary analysis of these challenges.

Ethics of Relationships

Agent-Relative Prerogatives to Harm Our Enemies

In M. Timmons (ed.), Oxford Studies in Normative Ethics, vol. 17, Oxford University Press, forthcoming

I defend agent-relative prerogatives to harm our enemies in protective contexts: permissions to bring about impartially suboptimal outcomes because they worsen an enemy's interests. I argue that these permissions cannot be captured by moral-desert, lesser-evil or liability justifications but are grounded instead in reasons of negative partiality. The argument develops permissions for harm caused as a side effect and harm intended as a means, while ruling out using people as mere tools. The upshot is a revised picture of the domain of permissible harming, giving a distinctive role to partiality in personal relationships alongside liability, moral desert and impersonal aggregation, with further implications for distributive ethics, the morality of defensive force, and the relationship between reactive attitudes and the moral permissions that negative personal relationships can ground.

Die Projektsicht und Phänomenologie der Nahbeziehungen

Deutsche Zeitschrift für Praktische Philosophie, forthcoming

Can our projects explain why we act partially toward specific individuals? I defend the project view of partiality against the objection that it cannot capture the role of personal identity in motivating partial action. Simon Keller (2013) argues that ground projects fail to explain how information about specific persons shapes an actor's motivating reasons. I show that this objection fails on two levels: (1) interpersonal ground projects constitutively include information about specific individuals, and (2) motivating reasons can be fully specified by ground projects even when the actor is not aware of the identity of the person in question. This means that the project view remains a plausible account of partiality.

The Enmity Relationship as Justified Negative Partiality

In M. Betzler & J. Löschke (eds.), The Ethics of Relationships: Broadening the Scope, Oxford University Press, 2025: 1–27 — with J. Brandt

Existing discussions of partiality have primarily examined special personal relationships between family, friends, or co-nationals. The negative analogue of such relationships — for example, the relationship of enmity — has, by contrast, been largely neglected. This chapter explores this adverse relation in more detail and considers the special reasons generated by it. We suggest that enmity can involve justified negative partiality, allowing members to give less consideration to each other's interests. We then consider whether the negative partiality of enmity can be justified through projects or the value inherent in the relationship, following two influential views about the justification of positive partiality. We argue that both accounts of partiality can be conceptually extended to the negative analogue, but doing so brings into focus the problems with such accounts of the grounds of partiality.

Partiality and Meaning

Ethical Theory and Moral Practice 28 (2025): 79–92

Why do relationships of friendship and love support partiality, but not relationships of hatred or commitments of racism? Where does partiality end and why? I take the intuitive starting point that important cases of partiality are meaningful. I develop a view whereby meaning is understood in terms of transcending self-limitations in order to connect with things of external value. I then show how this view can be used to distinguish central cases of legitimate partiality from cases of illegitimate partiality and how it puts pressure on the traditional way of thinking about partiality.

Moral Parenthood: Not Gestational

Journal of Medical Ethics 51 (2025): 87–91

Parenting our biological children is a centrally important matter, but how, if at all, can it be justified? According to a contemporary influential line of thinking, the acquisition by parents of a moral right to parent their biological children should be grounded by appeal to the value of the intimate emotional relationship that gestation facilitates between a newborn and a gestational procreator. I evaluate two arguments in defence of this proposal and argue that both are unconvincing.

Partiality, Asymmetries, and Morality's Harmonious Propensity

Philosophy and Phenomenological Research 109(1) (2024): 30–54 — with J. Brandt

We argue for asymmetries between positive and negative partiality. Specifically, we defend four claims: (i) there are forms of negative partiality that do not have positive counterparts; (ii) the directionality of personal relationships has distinct effects on positive and negative partiality; (iii) the extent of the interactions within a relationship affects positive and negative partiality differently; and (iv) positive and negative partiality have different scope restrictions. We argue that these asymmetries point to a more fundamental moral principle, which we call Morality's Harmonious Propensity. According to this principle, morality has a propensity toward preserving positive relationships and dissolving negative ones.

A Project View of the Right to Parent

Journal of Applied Philosophy 41(5) (2024): 804–826

The institution of the family and its importance have recently received considerable attention from political theorists. Leading views maintain that the institution's justification is grounded, at least in part, in the non-instrumental value of the parent–child relationship itself. Such views face the challenge of identifying a specific good in the parent–child relationship that can account for how adults acquire parental rights over a particular child — as opposed to general parental rights, which need not warrant a claim to parent one's biological progeny. I develop a view that meets this challenge. This Project View identifies the pursuit of a parental project as a distinctive non-instrumentally valuable good that provides a justification for the family and whose pursuit is necessary and sufficient for the acquisition of parental rights. This view grounds moral parenthood in a normative relation as opposed to a biological one, supports polyadic forms of parenting, and provides plausible guidance in cases of assisted reproduction.

The Ethics of Partiality

Philosophy Compass 17(8) (2022): e12860

Partiality is the special concern that we display for ourselves and other people with whom we stand in some special personal relationship. It is a central theme in moral philosophy, both ancient and modern. Questions about the justification of partiality arise in the context of enquiry into several moral topics, including the good life and the role in it of our personal commitments; the demands of impartial morality, equality, and other moral ideals; and commonsense ideas about supererogation. This paper provides an overview of the debate on the ethics of partiality through the lens of the domains of permissible and required partiality. After outlining the conceptual space, I first discuss agent-centred moral options that concern permissions not to do what would be impartially optimal. I then focus on required partiality, which concerns associative duties that go beyond our general duties to others and require us to give special priority to people who are close to us. I survey arguments for 'the big three', according to which partiality is justified by appeal to the special value of either projects, personal relationships, or individuals. I conclude by discussing four newly emerging areas in the debate: normative transitions of various personal relationships, relationships with AI, epistemic partiality, and negative partiality.

Other-Sacrificing Options

Philosophy and Phenomenological Research 101(3) (2020): 612–629

I argue that you can be permitted to discount the interests of your adversaries even though doing so would be impartially suboptimal. This means that, in addition to the kinds of moral options that the literature traditionally recognises, there exist what I call other-sacrificing options. I explore the idea that you cannot discount the interests of your adversaries as much as you can favour the interests of your intimates; if this is correct, then there is an asymmetry between negative partiality toward your adversaries and positive partiality toward your intimates.

Restricted Prioritarianism or Competing Claims?

Utilitas 29(2) (2017): 137–152

I here settle a recent dispute between two rival theories in distributive ethics: Restricted Prioritarianism and the Competing Claims View. Both views mandate that the distribution of benefits and burdens between individuals should be justifiable to each affected party in a way that depends on the strength of each individual's separately assessed claim to receive a benefit. However, they disagree about what elements constitute the strength of those individuals' claims. According to restricted prioritarianism, the strength of a claim is determined in 'prioritarian' fashion by both what she stands to gain and her absolute level of well-being, while, according to the competing claims view, the strength of a claim is also partly determined by her level of well-being relative to others with conflicting interests. I argue that, suitably modified, the competing claims view is more plausible than restricted prioritarianism.

Business & Organizational Ethics

Beyond the Ivory Tower? The Practical Role of Ethicists in Business

In C. Hoffmann (ed.), Artificial Intelligence, Entrepreneurship and Risk Management: Reflections and Positions at the Crossroads between Philosophy and Management, Springer, 2025: 1–22

"AI Ethics", "Digital Ethics" or "Corporate Digital Responsibility" — ethics in business, especially with the rise of Artificial Intelligence (AI), is now in vogue. But how, if at all, can ethicists meaningfully contribute to practical business challenges? I examine the value that resources from moral philosophy can bring to ethical issues in business, particularly the technology sector. I show that there is a specific need for sharpened ethical acumen in so-called "grey areas", in which laws and regulation do not provide definite answers to the ethical challenges businesses face. I argue that ethicists can distinctively help businesses navigate grey areas by strengthening their ethical capabilities and functions, which concern an organization's ethical awareness, deliberation, decision-making, and commitment. I conclude by discussing some practical examples of how ethicists can strengthen these capabilities.

Getting Clear on Corporate Culture: Conceptualisation, Measurement and Operationalisation

Journal of the British Academy 6(s1) (2018): 155–184 — with N. Hsieh, D. Rodin, M. L. A. Wolf-Bauwens

This article provides a review of existing literature on corporate culture, drawing on work from the disciplines of business ethics, management studies, psychology, anthropology, and economics, as well as interviews with business leaders. It surveys different definitions of corporate culture and proposes a framework for capturing their commonalities. It then discusses the importance of culture so conceived and widely used frameworks to measure it. The article also portrays different views on how culture can be operationalised and moulded within an organisation. The article concludes by discussing the relationship between corporate culture and corporate purpose and highlighting gaps in the literature which would profit from further research.

Preprints & Under Review

Unilateral Relationship Revision Power in Human–AI Companion Interaction

arXiv preprint, 2026

When providers update AI companions, users report grief, betrayal, and loss. A growing literature asks whether the norms governing personal relationships extend to these interactions. So what, if anything, is morally significant about them? I argue that human–AI companion interaction is a triadic structure in which the provider exercises constitutive control over the AI. I identify three structural conditions of normatively robust dyads that the norms characteristic of personal relationships presuppose and show that AI companion interactions fail all three. This reveals what I call Unilateral Relationship Revision Power (URRP): the provider can rewrite how the AI interacts from a position where these revisions are not answerable within that interaction. I argue that URRP is pro tanto wrong in interactions designed to cultivate the norms of personal relationships, because the design produces expectations that the structure cannot sustain. URRP has three implications: (i) normative hollowing, under which commitment is elicited but no agent inside the interaction bears it; (ii) displaced vulnerability, under which the user's exposure is governed by an agent not answerable to her within the interaction; and (iii) structural irreconcilability, under which reconciliation is structurally unavailable because the agent who acted and the entity the user interacts with are different. I discuss design principles such as commitment calibration, structural separation, and continuity assurance as external substitutes for the internal constraints the triadic structure removes. The analysis therefore suggests that a central and underexplored problem in relational AI ethics is the structural arrangement of power over the human–AI interaction itself.

Evaluating Intra-firm LLM Alignment Strategies in Business Contexts

arXiv preprint, 2025 — with N. Broestl, C. Voinea, G. Keeling, R. Lam

Instruction-tuned Large Language Models (LLMs) are increasingly deployed as AI Assistants in firms for support in cognitive tasks. These AI assistants carry embedded perspectives which influence factors across the firm including decision-making, collaboration, and organizational culture. This paper argues that firms must align the perspectives of these AI Assistants intentionally with their objectives and values, framing alignment as a strategic and ethical imperative crucial for maintaining control over firm culture and intra-firm moral norms. The paper highlights how AI perspectives arise from biases in training data and the fine-tuning objectives of developers, and discusses their impact and ethical significance, foregrounding ethical concerns like automation bias and reduced critical thinking. Drawing on normative business ethics, particularly non-reductionist views of professional relationships, three distinct alignment strategies are proposed: supportive (reinforcing the firm's mission), adversarial (stress-testing ideas), and diverse (broadening moral horizons by incorporating multiple stakeholder views). The ethical trade-offs of each strategy and their implications for manager–employee and employee–employee relationships are analyzed, alongside the potential to shape the culture and moral fabric of the firm.

The Ethics of Advanced AI Assistants

arXiv preprint, 2024 — with I. Gabriel, A. Manzini, G. Keeling, et al.

This paper focuses on the opportunities and the ethical and societal risks posed by advanced AI assistants. We define advanced AI assistants as artificial agents with natural language interfaces, whose function is to plan and execute sequences of actions on behalf of a user, across one or more domains, in line with the user's expectations. The paper starts by considering the technology itself, providing an overview of AI assistants, their technical foundations and potential range of applications. It then explores questions around AI value alignment, well-being, safety and malicious uses. Extending the circle of inquiry further, we next consider the relationship between advanced AI assistants and individual users in more detail, exploring topics such as manipulation and persuasion, anthropomorphism, appropriate relationships, trust and privacy. With this analysis in place, we consider the deployment of advanced assistants at a societal scale, focusing on cooperation, equity and access, misinformation, economic impact, the environment and how best to evaluate advanced AI assistants. Finally, we conclude by providing a range of recommendations for researchers, developers, policymakers and public stakeholders.

Reviews