Papers

Limits to Predicting Online Speech Using Large Language Models Mina Remeli; Moritz Hardt; Robert Williamson We study the predictability of online speech on social media, and whether predictability improves with information outside a user's own posts. Recent work suggests that the predictive information contained in posts written by a user's peers can surpass that of the user's own posts. Motivated by the success of large language models, we empirically test this hypothesis. We define unpredictability as a measure of the model's uncertainty, i.e., its negative log-likelihood on future tokens given context. As the basis of our study, we collect a corpus of 6.25M posts from more than five thousand X (previously Twitter) users and their peers. Across three large language models ranging in size from 1 billion to 70 billion parameters, we find that predicting a user's posts from their peers' posts performs poorly. Moreover, the value of the user's own posts for prediction is consistently higher than that of their peers'. Across the board, we find that the predictability of social media posts remains low without additional context.

Stochastic Concept Bottleneck Models Moritz Vandenhirtz; Sonia Laguna; Ričards Marcinkevičs; Julia E Vogt Concept Bottleneck Models (CBMs) have emerged as a promising interpretable method whose final prediction is based on intermediate, human-understandable concepts rather than the raw input. Through time-consuming manual interventions, a user can correct wrongly predicted concept values to enhance the model's downstream performance. We propose *Stochastic Concept Bottleneck Models* (SCBMs), a novel approach that models concept dependencies. In SCBMs, a single-concept intervention affects all correlated concepts. Leveraging the parameterization, we derive an effective intervention strategy based on the confidence region. We show empirically on synthetic tabular and natural image datasets that our approach improves intervention effectiveness significantly. Notably, we showcase the versatility and usability of SCBMs by examining a setting with CLIP-inferred concepts, alleviating the need for manual concept annotations.

Why Would You Suggest That? Human Trust in Language Model Responses Manasi Sharma; Ho Chit Siu; Rohan R Paleja; Jaime Daniel Pena The emergence of Large Language Models (LLMs) has revealed a growing need for human-AI collaboration, especially in creative decision making scenarios where trust and reliance are paramount. Through human studies and model evaluations on the open-ended News Headline Generation task from the LaMP benchmark, we analyze how the framing and presence of explanations affect user trust and model performance. Overall, we provide evidence that adding an explanation in the model response to justify its reasoning significantly increases self-reported user trust in the model when the user has the opportunity to compare various responses. Position and faithfulness of these explanations are also important factors. However, these gains disappear when users are shown responses independently, suggesting that humans trust all model responses, including deceptive ones, equitably when they are shown in isolation. Our findings urge future research to delve deeper into the nuanced evaluation of trust in human-machine teaming systems.

Mitigating Exposure Biases in Personalized Timelines through Agent-based Models Nathan Bartley; Keith Burghardt; Kristina Lerman Recommender systems are ubiquitous in online social networks. Studying how these systems expose people to information at scale is difficult to do as one cannot assume each user is subject to the same feed condition and building evaluation infrastructure is costly. We present an agent-based model comparing personalization algorithms in how they skew users' network perception, and we demonstrate that a greedy algorithm based on network properties is effective at creating less biased feeds. This underscores the influence that network structure has in determining the effectiveness of recommender systems and offers a tool for mitigating perception biases through algorithmic feed construction.

"Learning the eye of the beholder: Statistical modeling and estimation for personalized color perception" Xuanzhou Chen; Austin Xu; Jingyan Wang; Ashwin Pananjady Color perception has long remained an intriguing topic in vision and cognitive science. It is a common practice to classify a person as either "color-normal" or "color-blind", and that there are a few prevalent types. However, empirical evidence has repeatedly suggested that at best, categories for colorblindness only serve as approximations to real manifestations of it. To better understanding individual-level color perception, we propose a color perception model that unifies existing theories for color-normal and color-blind people, which posits a low-dimensional structure in color space according to which any given user distinguishes colors. We design an algorithm to learn this low-dimensional structure from user queries, and prove statistical guarantees on its performance. Taking inspiration from these guarantees, we design a novel data collection paradigm based on perceptual adjustment queries (PAQs) that efficiently infers a user’s color distinguishability profile from a small number of cognitively lightweight responses. In a host of simulations, PAQs offer significant advantages over the de facto method of collecting comparison-based similarity queries.

Strategic Linear Contextual Bandits Thomas Kleine Buening; Aadirupa Saha; Christos Dimitrakakis; Haifeng Xu Motivated by the phenomenon of strategic agents gaming a recommender system to maximize the number of times they are recommended to users, we study a strategic variant of the linear contextual bandit problem, where the arms can strategically misreport their privately observed contexts to the learner. We treat the algorithm design problem as one of mechanism design under uncertainty and propose the Optimistic Grim Trigger Mechanism that incentivizes the agents (i.e., arms) to report their contexts truthfully while simultaneously minimizing regret. We also show that failing to account for the strategic nature of the agents results in linear regret. However, a trade-off between mechanism design and regret minimization appears to be unavoidable. More broadly, this work aims to provide insight into the intersection of online learning and mechanism design.

The Role of Learning Algorithms in Collective Action Omri Ben-Dov; Jake Fawkes; Samira Samadi; Amartya Sanyal Collective action in Machine Learning is the study of the control that a coordinated group can have over machine learning algorithms. While previous research has concentrated on assessing the impact of collectives against Bayes optimal classifiers, this perspective is limited in that classifiers seldom achieve Bayes optimality, and are influenced by the choice of learning algorithms along with their inherent biases. In this work, we initiate the study of how the choice of the learning algorithm plays a role in the success of a collective in practical settings. Specifically, we focus on distributionally robust algorithms (DRO), popular for improving a worst group error, and on the ubiquitous stochastic gradient descent (SGD), due to its inductive bias for "simpler" functions. Our empirical results, supported by a theoretical foundation, show that the effective size and success of the collective are highly dependent on properties of the learning algorithm. This highlights the necessity of taking the learning algorithm into account when studying the impact of collective action in machine learning.

Optimizing Machine Learning Explanations for Properties Hiwot Belay Tadesse; Yaniv Yacoby; Weiwei Pan; Finale Doshi-Velez There are explanation methods, as well as works that quantify the extent to which these explanations satisfy properties, like faithfulness or robustness. For instance, SmoothGrad \cite{smilkov_smoothgrad_2017} encourages robustness by averaging gradients around an input, whereas LIME \cite{ribeiro_why_2016} encourages fidelity by fitting a linear approximation of a function. However, we demonstrate that these forms of encouragement do not consistently target their desired properties. In this paper, we \emph{directly optimize} explanations for desired properties. We show that, compared to SmoothGrad and LIME, we are able to: (1) produce explanations that are more optimal with respect to chosen properties (2) manage trade-offs between properties more explicitly and intuitively.

Is a Good Description Worth a Thousand Pictures? Reducing Multimodal Alignment to Text-Based, Unimodal Alignment Amin Memarian; Touraj Laleh; Irina Rish; Ardavan S. Nobandegani Generative AI systems (ChatGPT, Llama, etc.) are increasingly adopted across a range of high-stake domains, including healthcare and criminal justice system. This rapid adoption indeed raises moral and ethical concerns. The emerging field of AI alignment aims to make AI systems that respect human values. In this work, we focus on evaluating the ethics of multimodal AI systems involving both text and images --- a relatively under-explored area, as most alignment work is currently focused on language models. Specifically, here we investigate whether the multimodal alignment problem (i.e., the problem of aligning a multimodal system) could be effectively reduced to the (text-based) unimodal alignment problem, wherein a language model would make a moral judgment purely based on a description of an image. Focusing on GPT-4 and LLaVA as two prominent examples of multimodal systems, here we demonstrate, rather surprisingly, that this reduction can be achieved with a relatively small loss in moral judgment performance in the case of LLaVa, and virtually no loss in the case of GPT-4.

Predictive Performance Comparison of Decision Policies Under Confounding Luke Guerdan; Amanda Lee Coston; Ken Holstein; Steven Wu Predictive models are often introduced to decision-making tasks under the rationale that they improve performance over an existing decision-making policy. However, it is challenging to compare predictive performance against an existing decision-making policy that is generally under-specified and dependent on unobservable factors. These sources of uncertainty are often addressed in practice by making strong assumptions about the data-generating mechanism. In this work, we propose a method to compare the predictive performance of decision policies under a variety of modern identification approaches from the causal inference and off-policy evaluation literatures (e.g., instrumental variable, marginal sensitivity model, proximal variable). Key to our method is the insight that there are regions of uncertainty that we can safely ignore in the policy comparison. We develop a practical approach for finite-sample estimation of regret intervals under no assumptions on the parametric form of the status quo policy. We verify our framework theoretically and via synthetic data experiments. We conclude with a real-world application using our framework to support a pre-deployment evaluation of a proposed modification to a healthcare enrollment policy.

To Give or Not to Give? The Impacts of Strategically Withheld Recourse Yatong Chen; Andrew Estornell; Yevgeniy Vorobeychik; Yang Liu Individuals often aim to reverse undesired outcomes in interactions with automated systems, like loan denials, through system-recommended actions (recourse) or manipulation actions (e.g., misreporting feature values). While providing recourse benefits users and enhances system utility, it also increases transparency, enabling more strategic exploitation by individuals, especially when groups share information. We show that this tension could potentially lead systems to strategically withhold recourse, challenging assumptions about universal recourse provision in current literature. We propose a framework to investigate the interplay of transparency, recourse, and manipulation and demonstrate that rational utility-maximizing systems frequently withhold recourse, leading to decreased population utility, particularly impacting sensitive groups. To mitigate these effects, we explore the role of recourse subsidies, finding them effective in increasing the provision of recourse actions by rational systems.

Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learning Jonathan Cook Cultural accumulation drives the open-ended progress in capabilities spanning human history. It builds an expanding body of knowledge and skills by combining individual exploration with inter-generational information transmission. Despite its widespread success among humans, the capacity for artificial learning agents to accumulate culture remains under-explored. In particular, approaches to reinforcement learning typically strive for improvements over only a single lifetime. Generational algorithms that do exist fail to capture the open-ended, emergent nature of cultural accumulation, which allows individuals to trade-off innovation and imitation. Building on the previously demonstrated ability for reinforcement learning agents to perform social learning, we find that training setups which balance this with independent learning give rise to cultural accumulation. To the best of our knowledge, this work is the first to present general models that achieve emergent cultural accumulation in reinforcement learning, presenting new routes to more open-ended learning systems, as well as new opportunities for modelling human culture.

Post-processing fairness with minimal changes Federico Di Gennaro; Thibault Laugel; Vincent Grari; Xavier Renard; Marcin Detyniecki In this paper, we introduce a novel post-processing algorithm that is both model-agnostic and does not require the sensitive attribute at test time. In addition, our algorithm is explicitly designed to enforce minimal changes between biased and debiased predictions—a property that, while highly desirable, is rarely prioritized as an explicit objective in fairness literature. Our approach leverages a multiplicative factor applied to the logit value of probability scores produced by a black-box classifier. We demonstrate the efficacy of our method through empirical evaluations, comparing its performance against other four debiasing algorithms on two widely used datasets in fairness research.

"You just can’t go around killing people'' Explaining Agent Behavior to a Human Terminator Uri Menkes; Ofra Amir; Assaf Hallak Consider a setting where a pre-trained agent is operating in an environment and a human operator can decide to temporarily terminate its operation and take-over for some duration of time. These kind of scenarios are common in human-machine interactions, for example in autonomous driving, factory automation and healthcare. In these settings, we typically observe a trade-off between two extreme cases -- if no take-overs are allowed, then the agent might employ a sub-optimal, possibly dangerous policy. Alternatively, if there are too many take-overs, then the human has no confidence in the agent, greatly limiting its usefulness. In this paper, we formalize this setup and propose an explainability scheme to help optimize the number of human interventions.

Bias Transmission in Large Language Models: Evidence from Gender-Occupation Bias in GPT-4 Kirsten Morehouse; Weiwei Pan; Juan Manuel Contreras; Mahzarin R. Banaji Recent advances in generative AI are poised to reduce the burden of important and arduous tasks, including drafting job application materials. In this paper, we examine whether GPT-4 produces job cover letters that systematically advantage some users and disadvantage others. To test this, we introduce a novel method designed to probe LLMs for gender-occupation biases. Using our method, we show that GPT-4, like humans, possesses strong gender-occupation associations (e.g., surgeon = male, nurse = female). However, surprisingly, we find that biased associations do not necessarily translate into biased results. That is, we find that GPT-4 can (a) produce reasonable evaluations of cover letters, (b) evaluate information written by men and women equally, unlike humans, and (c) generate equally strong cover letters for male and female applicants. Our work calls for more systematic studies of the connection between association bias and outcome bias in generative AI models.

Machine Learning Without True Probabilities Benedikt Höltgen; Robert Williamson Drawing on scholarship across disciplines, we argue that probabilities are constructed rather than discovered and show how this is important for Machine Learning, especially in social settings. We criticise the conventional notion of datapoints as sampled from a true distribution and propose an alternative mathematical framework that allows to analyse learning. We highlight problematic aspects of common reasoning and rhetoric about probabilities in the context of social predictions. We also strengthen the case that (probabilistic) Machine Learning models cannot be separated from the choices that went into their construction and from the task they were meant for.

Evaluating Fairness in Black-box Algorithmic Markets: A Case Study of Ride Sharing in Chicago Yuhan Liu; Yuhan Zheng; Siyuan Zhang; Lydia T. Liu This study examines fairness within the rideshare industry, focusing on both drivers' wages and riders' trip fares. Through quantitative analysis, we found that drivers' hourly wages are significantly influenced by factors such as race/ethnicity, health insurance status, tenure to the platform, and working hours. Despite platforms' policies not intentionally embedding biases, disparities persist based on these characteristics. For ride fares, we propose a method to audit the pricing policy of a proprietary algorithm by replicating it; we conduct a hypothesis test to determine if the predicted rideshare fare is greater than the taxi fare, taking into account the approximation error in the replicated model. Challenges in accessing data and transparency hinder our ability to isolate discrimination from other factors, underscoring the need for collaboration with rideshare platforms and drivers to enhance fairness in algorithmic wage determination and pricing.

CoBo: Collaborative Learning via Bilevel Optimization Diba Hashemi; Lie He; Martin Jaggi Clients in collaborative learning aim to improve model quality through jointly training. Identifying helpful clients, however, presents challenging and often introduces significant overhead. In this paper, we model $ extit{client-selection}$ and $ extit{model-training}$ as two interconnected optimization problems, proposing a novel bilevel optimization problem for collaborative learning. We introduce CoBo, an efficient SGD-type alternating optimization algorithm that addresses collaborative learning with theoretical convergence guarantees. Moreover, CoBo presents strong empirical performances, outperforming all other algorithms in terms of model quality and fairness.

Bayesian Collaborative Bandits for Time Slot Inference in Maternal Health Programs Arpan Dasgupta; Arun Suggala; Karthikeyan Shanmugam; Aparna Taneja; Milind Tambe Mobile health programs have gained a lot of popularity recently due to the widespread use of mobile phones, particularly in underserved communities. However, call records from one such maternal mHealth program in India indicate that different beneficiaries have different time preferences, due to their availability during the day as well as limited access to a phone. This makes selection of the best time slot to call a beneficiary an important problem for the program. Prior work has formalized this as a collaborative bandit problem, where the assumption of a low-rank call pickup matrix allows for more efficient exploration across arms. We propose a novel Bayesian solution to the collaborative bandit problem using Stochastic Gradient Langevin Dynamics (SGLD) and Thompson Sampling for selection of time slots. We show that this method is able to perform better in scarce data situations where there are limited time steps for exploration, and has the ability to utilize prior knowledge about arms to its advantage. We also propose a faster version of the algorithm using alternative sampling which can potentially scale to a very large number of users such that it may be potentially deployable in the real world. We evaluate the algorithm against existing methods on simulated data inspired from real-world data.

User-Creator Feature Dynamics in Recommender Systems with Dual Influence Tao Lin; Kun Jin; Andrew Estornell; Xiaoying Zhang; Yiling Chen; Yang Liu Recommender systems present relevant content to users and help content creators reach their target audience. The dual nature of these systems influences both users and creators: users' preferences are altered by the items they are recommended, while creators are incentivized to alter their content such that it is recommended more frequently. We define a model, called user-creator feature dynamics, to capture the dual influences of recommender systems. We prove that a recommender system with dual influence is guaranteed to polarize, causing diversity loss in the system. We then investigate, both theoretically and empirically, approaches for mitigating polarization and promoting diversity in recommender systems. Unexpectedly, we find that common diversity-promoting approaches do not work in the presence of dual influence, while relevancy-optimizing methods like top-$k$ recommendation can prevent polarization and improve diversity of the system.

The Limitations of Model Retraining in the Face of Performativity Anmol Kabra; Kumar Kshitij Patel We study stochastic optimization in the context of performative shifts, where the data distribution changes in response to the deployed model. We demonstrate that naive retraining can be provably suboptimal even for simple distribution shifts. The issue worsens when models are retrained given a finite number of samples at each retraining step. We show that adding regularization to retraining corrects both of these issues, attaining provably optimal models in the face of distribution shifts. Our work advocates rethinking how machine learning models are retrained in the presence of performative effects.

Nearly-tight Approximation Guarantees for the Improving Multi-Armed Bandits Problem Avrim Blum; Kavya Ravichandran We give nearly-tight upper and lower bounds for the improving multi-armed bandits problem. An instance of this problem has $k$ arms, each of whose reward function is a concave and increasing function of the number of times that arm has been pulled so far. This models decision-making scenarios where performance at a task improves with practice, but the performance curves are unknown to the agent a priori. We show that for any randomized online algorithm, there exists an instance on which it must suffer at least an $\Omega(\sqrt{k})$ approximation factor relative to the optimal reward. We then provide a randomized online algorithm that guarantees an $O(\sqrt{k})$ approximation factor, if it is told the maximum reward achievable by the optimal arm in advance. We then show how to remove this assumption at the cost of an extra $O(\log k)$ approximation factor, achieving an overall $O(\sqrt{k} \log k)$ approximation.

Designing Experimental Evaluations of Algorithmic Interventions with Human Decision Makers In Mind Inioluwa Deborah Raji; Lydia T. Liu Automated decision systems (ADS) are broadly deployed to inform or support human decision-making across a wide range of consequential contexts. An emerging approach to the assessment of such systems is through experimental evaluation, which aims to measure the causal impacts of the ADS deployment on decision making and outcomes. However, various context-specific details complicate the goal of establishing meaningful experimental evaluations for algorithmic interventions. Notably, current experimental designs rely on simplifying assumptions about human decision making in order to derive causal estimates. In reality, cognitive biases of human decision makers induced by experimental design choices may significantly alter the observed effect sizes of the algorithmic intervention. In this paper, we formalize and investigate various models of human decision-making in the presence of a predictive algorithmic aid. We show that each of these behavioral models produces dependencies across decision subjects and results in the violation of existing assumptions, with consequences for treatment effect estimation.

Bridging the Gap between AI Developers and Social Practitioners: A Fairness Framework for Designing AI Systems Mírian Silva; Mariano G. Beiró; Marisa Affonso Vasconcelos; Ana Couto Fairness interventions are a key focus in most Artificial Intelligence (AI) ethics research fields. When biases related to some features (e.g., race, sex, age, religion) are identified in AI systems that contribute to discrimination outcomes, developers, engineers, or stakeholders must choose how and when to intervene. However, given the plethora of available options, a lack of standardization in the intervention process prevails, making it challenging to determine the suitable option for a given context. In this work, we propose a developmental framework to explore different types of measures based on non-discrimination criteria aimed at filling the gap between AI developers and social practitioners . We then construct a framework to analyze the performance of the interventions over AI models in terms of statistical non-discrimination fairness criteria.

Conformal Prediction Sets Improve Human Decision Making Jesse C. Cresswell; Yi Sui; Bhargava Kumar; Noël Vouitsis In response to everyday queries, humans explicitly signal uncertainty and offer alternative answers when they are unsure. Conformal prediction produces calibrated prediction sets that mimic this human behaviour since larger sets signal greater uncertainty while providing alternatives. In this work, we study the usefulness of conformal prediction sets as an aid for human decision making by conducting a pre-registered randomized controlled trial with conformal prediction sets provided to human subjects. With statistical significance, we find that when humans are given conformal prediction sets their accuracy on tasks improves compared to fixed-size prediction sets with the same coverage guarantee.

Challenging the Human-in-the-loop in Algorithmic Decision-making Sebastian Tschiatschek; Eugenia Stamboliev; Timothée Schhmude; Mark Coeckelbergh; Laura Koesten We discuss the role of humans in algorithmic decision-making (ADM) for socially relevant problems, highlighting tensions arising from the misalignment of the humans with each other and with the algorithms involved. To this end, we assume that a supervisor introduces ADM to achieve strategic goals and that the algorithms’ recommended actions are overseen by agents who makes the final decisions. While the agents should be a corrective, they can counteract the realization of the supervisor’s goals because of misalignment and unmet information needs. This impacts the distribution of power between the stakeholders, and we emphasize the overseeing agents’ implied role as potential political and ethical decision-makers. On a machine learning benchmark dataset we illustrate the significant impact overseeing agents’ decisions can have even if they are constrained to performing only few corrections to the algorithms’ recommendations. Our findings emphasize the need for an in-depth discussion of the role and power of the agents and challenge the often-taken view that just including a human-in-the-loop in ADM ensures its ‘correct’ and ‘ethical’ functioning.

From Individual Experience to Collective Evidence: An Incident-Based Framework for Identifying Systemic Discrimination Jessica Dai; Paula Gradu; Inioluwa Deborah Raji; Benjamin Recht When an individual reports a personal negative experience, how can we confirm this as part of any broader, systemic pattern of discrimination? In this work, we study the incident database problem, where individual reports of adverse events are aggregated over time. In such a model, reports arrive sequentially; our goal is to identify whether some subgroup, defined by any combination of relevant features, experiences adverse events disproportionately often. We propose an algorithm to conduct this assessment via sequential hypothesis testing; we efficiently identify marginalized subgroups while handling multiple testing with a possibly-exponential number of hypotheses. We then demonstrate our method on real-world datasets including mortgage decisions and vaccine side effects; on each, our method (re-)identifies subgroups known to experience disproportionate harm using only a fraction of the data that was initially used to discover them.

Causal models predict average outcomes, not individual effects Benedikt Höltgen; Robert Williamson Consequential decisions need to be based on causally robust predictions. Causality is usually analysed in such contexts through Rubin Causal Models, although they are based on overly strong assumptions and make unverifiable predictions. In this work, we develop a weaker framework for causality with assumptions that are more realistic and directly verifiable. We demonstrate its applicability to different inference methods such as RCTs, Machine Learning, and Exact Matching.

What is the Right Notion of Distance between Predict-then-Optimize Tasks? Paula Rodriguez-Diaz; Kai Wang; David Alvarez-Melis; Milind Tambe Optimal transport-based dataset distances are a principled way to measure task similarity, informing tasks like domain adaptation and transfer learning, typically assessed by prediction error minimization. However, in Predict-then-Optimize (PtO) frameworks, success is measured by decision regret minimization. We show that feature- and label-based distances lack informativeness in PtO and propose a new decision-aware distance that effectively captures adaptation success in PtO.

The Missing Link: Allocation Performance in Causal Machine Learning Unai Fischer-Abaigar; Christoph Kern; Frauke Kreuter Automated decision-making (ADM) systems are being deployed across a diverse range of critical problem areas such as social welfare and healthcare. Recent work highlights the importance of causal ML models in ADM systems, but implementing them in complex social environments poses significant challenges. Research on how these challenges impact the performance in specific downstream mph{decision-making} tasks is limited. Addressing this gap, we make use of a comprehensive real-world dataset of jobseekers to illustrate how the performance of a single CATE model can vary significantly across different decision-making scenarios and highlight the differential influence of challenges such as distribution shifts on predictions and allocations.

Controlling Counterfactual Harm in Decision Support Systems Based on Prediction Sets Eleni Straitouri; Suhas Thejaswi; Manuel Gomez Rodriguez Decision support systems based on prediction sets help humans solve multiclass classification tasks by narrowing down the set of potential label values to a subset of them, namely a prediction set, and asking them to always predict label values from the prediction sets. While this type of systems have been proven to be effective at improving the average accuracy of the predictions made by humans, by restricting human agency, they may cause harm—a human who has succeeded at predicting the ground-truth label of an instance on their own may have failed had they used these systems. In this paper, our goal is to control how frequently a decision support system based on prediction sets may cause harm, by design. To this end, we start by characterizing the above notion of harm using the theoretical framework of structural causal models. Then, we show that, under a natural monotonicity assumption, we can estimate how frequently a system may cause harm using only predictions made by humans on their own. Building upon this assumption, we introduce a computational framework to design decision support systems based on prediction sets that are guaranteed to cause harm less frequently than a user-specified value using conformal risk control. We validate our framework using real human predictions from a human subject study and show that, in decision support systems based on prediction sets, there is a trade-off between accuracy and counterfactual harm.

A Sim2Real Approach for Identifying Task-Relevant Properties in Interpretable Machine Learning Eura Nofshin; Esther Brown; Brian Lim; Weiwei Pan; Finale Doshi-Velez Existing user studies suggest that different tasks may require explanations with different properties. However, user studies are expensive. In this paper, we introduce XAIsim2real, a generalizable, cost-effective method for identifying task-relevant explanation properties in silico, which can guide the design of more expensive user studies. We use XAIsim2real to identify relevant proxies for three example tasks and validate our simulation with real user studies.

Adaptive Algorithmic Interventions for Escaping Pessimism Traps in Dynamic Sequential Decisions Emily Diana; Alexander Tolbert; Kavya Ravichandran; Avrim Blum In this paper, we relate the philosophical literature on pessimism traps to information cascades, a formal model derived from the economics and mathematics literature. A pessimism trap is a social pattern in which individuals in a community, in situations of uncertainty, begin to copy the sub-optimal actions of others, despite their individual beliefs. This maps nicely onto the concept of an information cascade, which involves a sequence of agents making a decision between two alternatives, with a private signal of the superior alternative and a public history of others' actions. Key results from the economics literature show that information cascades occur with probability one in many contexts, and depending on the strength of the signal, populations can fall into the incorrect cascade very easily and quickly. Once formed, in the absence of external perturbation, a cascade cannot be broken -- therefore, we derive an intervention that can be used to nudge a population from an incorrect to a correct cascade and, importantly, maintain the cascade once the subsidy is discontinued. We study this both theoretically and empirically.

Risk Scores in Algorithmic Decision-Making as Statistical Fatalism Sebastian Zezulka; Konstantin Genin A fundamental problem in algorithmic fairness is determining whether machine learning algorithms will reproduce or exacerbate structural inequalities reflected in their training data. Addressing this challenge requires two key steps. First, we must evaluate fairness interventions on predictions in algorithmic decision-making by examining the causal effect their deployment has on the distribution of relevant social goods. Second, we propose the framework of extit{prospective fairness}, which necessitates anticipating these effects before implementing algorithmic policies. Extending this line of work, we advocate shifting the focus from predicting (fair) risk scores to estimating extit{potential outcomes} under available policy decisions.

Smooth Ambiguity-Averse Preferences and Bayesian Nonparametrics for Data-Driven Distributionally Robust Optimization Nicola Bariletto; Nhat Ho Training machine learning and statistical models often involves optimizing a data-driven risk criterion. The risk is usually computed with respect to the empirical data distribution, but this may result in poor and unstable out-of-sample performance due to distributional uncertainty. In the spirit of distributionally robust optimization, we propose a novel robust criterion by combining insights from a recent decision-theoretic model of smooth ambiguity-averse preferences and Bayesian nonparametric statistics. The optimization procedure provably enjoys finite-sample and asymptotic statistical performance guarantees. Moreover, the smoothness of the criterion and the properties of the employed Dirichlet process prior allow for easy-to-optimize approximations. The method also achieves promising empirical results as to improving and stabilizing the out-of-sample performance of popular statistical learning algorithms.

From Unknown to Known: An AI Coaching Problem in Open-World Environments Xuejie Liu; Anji Liu; Zihao Wang; Zhengxinyue; Liwen Zhu; Haobo Fu; Yitao Liang Large language models have been the state of the art for many tasks. Yet, whether their own competence can be beneficial to human learning of those tasks remains uncertain. We hypothesize the key is whether we can successfully infer the unknown-to-known reasoning process behind completing those tasks. We further ground the helping into two modules, router design and active helper. Tested on the popular open-world sandbox game Minecraft, our method consistently surpasses the performance of commonly used large language models.

Reconciling Predictive Multiplicity in Practice Tina Behzad; Sílvia Casacuberta; Emily Diana; Alexander Tolbert Many machine learning applications focus on predicting individual probabilities; for example, the probability that an individual develops a certain illness. Since these probabilities are inherently unknowable, a fundamental question that arises is how to resolve the (common) scenario where different models trained on the same dataset obtain different predictions on certain individuals. A well-known instance of this problem is the so-called model multiplicity (MM) phenomenon, in which a collection of comparable models present inconsistent predictions. Recently, Roth, Tolbert, and Weinstein proposed a reconciliation procedure (called the "Reconcile algorithm") as a solution to this problem- given two disagreeing models, they show how this disagreement can be leveraged to falsify and improve at least one of the two models. In this paper, we perform an empirical analysis of the Reconcile algorithm on three well-known fairness datasets- COMPAS, Communities and Crime, and Adult. We clarify how Reconcile fits within the model multiplicity literature, and compare it to the main solutions proposed in the MM setting, demonstrating the efficacy of the Reconcile algorithm. Lastly, we demonstrate ways of improving the Reconcile algorithm in theory and in practice.

The Impact of Recommender Systems and Homophily in Information Diffusion on Online Social Media João N. Fonseca; Fernando P. Santos Online social media platforms fundamentally impact information transmission in our societies. In order to understand phenomena such as political polarization, misinformation spreading or even large-scale collective action, it is important to understand what drives the spread of information in online platforms. The impact of algorithmic recommendations in online information diffusion remains poorly understood. Here, we present a preliminary model to test how different forms of content recommendation might impact information diffusion patterns, in heterogeneous populations where groups might be connected with arbitrary homophily levels. We observe that content-based recommenders can increase the size of information cascades and affect the possibility that minority groups trigger large cascade events.

Measuring Fairness in Large-Scale Recommendation Systems with Missing Labels Yulong Dong; Kun Jin; Xinghai Hu; Yang Liu Despite the commercial success of large-scale recommendation systems, people have recently raised concerns about the social responsibility of them, where fairness is one of the most important aspects. Accurate measurement of fairness metrics is vital for trustworthy fairness monitoring and diagnosis. But since most of these large recommendation systems do not have ground truths on the users' preferences on items never recommended to them, the systems suffer from the prevalence of missing ground truth labels on user-item pairs, and it poses significant challenges to accurate fairness metric measurements. Our work proposes a natural and efficient approach that addresses these issues caused by such missing labels, where we leverage the random traffic as a probe to the dataset with missing labels. We show both theoretically and numerically on real-world data that our approach is efficient and necessary.

Do causal predictors generalize better to new domains? Vivian Yvonne Nastl; Moritz Hardt We study how well machine learning models trained on causal features generalize across domains. We consider 16 prediction tasks on tabular datasets covering applications in health, employment, education, social benefits, and politics. Each dataset comes with multiple domains, allowing us to test how well a model trained in one domain performs in another. For each prediction task, we select features that have a causal influence on the target of prediction. Our goal is to test the hypothesis that models trained on causal features generalize better across domains. Without exception, we find that predictors using all available features, regardless of causality, have better in-domain and out-of-domain accuracy than predictors using causal features. Moreover, even the absolute drop in accuracy from one domain to the other is no better for causal predictors than for models that use all features. In addition, we show that recent causal machine learning methods for domain generalization do not perform better in our evaluation than standard predictors trained on the set of causal features. Likewise, causal discovery algorithms either fail to run or select causal variables that perform no better than our selection. Extensive robustness checks confirm that our findings are stable under variable misclassification.

AnonFair: A Flexible Toolkit for Algorithmic Fairness Eoin D. Delaney; Zihao Fu; Sandra Wachter; Brent Mittelstadt; Chris Russell We present AnonFair, a new open source toolkit for enforcing algorithmic fairness. Compared to existing toolkits: (i) We support NLP and Computer Vision classification as well as standard tabular problems. (ii) We support enforcing fairness on validation data, making us robust to a wide-range of overfitting challenges. (iii) Our approach can optimize any measure that is a function of True Positives, False Positive, False Negatives, and True Negatives. This makes it easily extendable, and much more expressive than existing toolkits. It supports 9/9 and 10/10 of the group metrics of two popular review papers. AnonFair is compatible with standard ML toolkits including sklearn, Autogluon and pytorch and is available online.

When Does Homogenization Reduce Competition in Algorithmic Personalized Pricing? Nathanael Jo; Ashia C. Wilson; Kathleen Creel; Manish Raghavan This paper explores the implications for market competition of increasing homogenization between personalized pricing algorithms. Our analysis reveals that higher homogenization (correlated outcomes) diminishes consumer welfare. Furthermore, as consumers become more price sensitive, firms are increasingly incentivized to compromise on the accuracy of their predictions in exchange for coordination. Our results underscore the potential anti-competitive effects of algorithmic pricing and highlight the need for refined antitrust approaches in the era of digital markets.

Fairness in Ranking under Disparate Uncertainty Richa Rastogi; Thorsten Joachims Ranking is a ubiquitous method for focusing the attention of human evaluators on a manageable subset of options. Its use as part of human decision-making processes ranges from surfacing potentially relevant products on an e-commerce site to prioritizing college applications for human review. While ranking can make human evaluation more effective by focusing attention on the most promising options, we argue that it can introduce unfairness if the uncertainty of the underlying relevance model differs between groups of options. Unfortunately, such disparity in uncertainty appears widespread, often to the detriment of minority groups for which relevance estimates can have higher uncertainty due to a lack of data or appropriate features. To address this fairness issue, we propose Equal-Opportunity Ranking (EOR) as a new fairness criterion for ranking and show that it corresponds to a group-wise fair lottery among the relevant options even in the presence of disparate uncertainty. EOR optimizes for an even cost burden on all groups, unlike the conventional \emph{Probability Ranking Principle}, and is fundamentally different from existing notions of fairness in rankings, such as \emph{demographic parity} and \emph{proportional Rooney rule} constraints that are motivated by proportional representation relative to group size. To make EOR ranking practical, we present an efficient algorithm for computing it in time $O(n \log(n))$ and prove its close approximation guarantee to the globally optimal solution. In a comprehensive empirical evaluation, we find that the algorithm reliably guarantees EOR fairness while providing effective rankings.

Models That Prove Their Own Correctness Noga Amit; Shafi Goldwasser; Orr Paradise; Guy N. Rothblum How can we trust the correctness of a learned model on a particular input of interest? Model accuracy is typically measured *on average* over a distribution of inputs, giving no guarantee for any fixed input. This paper proposes a theoretically-founded solution to this problem: to train *Self-Proving models* that prove the correctness of their output to a verification algorithm $V$ via an Interactive Proof. We devise a generic method for learning Self-Proving models, and we prove convergence bounds under certain assumptions. As an empirical exploration, our learning method is used to train a Self-Proving transformer that computes the Greatest Common Divisor (GCD) *and* proves the correctness of its answer.

Learning Graph Neural Networks from Biased Outcome Data Sidhika Balachandar; Shuvom Sadhuka; Bonnie Berger; Emma Pierson; Nikhil Garg Graph neural networks (GNNs) are widely used to make predictions on graph-structured data -- e.g., in spatiotemporal forecasting applications, GNNs are used to predict extreme weather events and traffic flows. However, data from graph nodes is frequently noisy or missing; nodes have a true latent state (e.g. a neighborhood is flooded) that is observed via a report (e.g. a resident reports the flood). We propose a GNN-based model to predict both the true latent state of nodes and the observed reports. Estimating the latent state is challenging as there is a lack of ground truth data. However, we often have sparse data we can use to inform the model about the latent state. We apply our model to a case study of urban reporting from New York City 311 complaints with latent state data sourced from government inspections. We show that by jointly modeling the latent state and reporting rates across neighborhoods and incident types we are able to generalize to unobserved neighborhoods, types, and time periods. Our analysis reveals a widely applicable approach for using GNNs and sparse data to identify latent states.

Exploring Desiderata for Individual Fairness Shai Ben-David; Tosca Lechner; Ruth Urner Algorithmic fairness for automated decision making systems has received much attention in recent years, with studies falling broadly into one of two camps: notions of (statistical) group fairness (GF), and notions of individual fairness (IF) - fairness as a right to be guaranteed to individuals. In this work, we review the latter notion for classification tasks and propose a formal framework for distinguishing individual from group fairness notions. We take an "axiomatic" approach, and identify a list of desirable properties for such a notion. We analyze relationships between these requirements, showing that some of them are mutually exclusive. We discuss some of the existing approaches to individual fairness from the perspective of our framework. In particular, we address the common view of IF as a Lipschitzness requirement ("similar individuals should be treated similarly") and discuss some of its concerning drawbacks.

Attaining Human's Desirable Outcomes in Human-AI Interaction via Structural Causal Games Anjie Liu; Jianhong Wang; Haoxuan Li; Xu Chen; Jun Wang; Samuel Kaski; Mengyue Yang In human-AI interaction, a prominent goal is to attain human's desirable outcome with the assistance of AI agents, which can be ideally delineated as a problem of seeking the optimal Nash Equilibrium that matches the human's desirable outcome. However, reaching the outcome is usually challenging due to the existence of multiple Nash Equilibria that are related to the assisting task but do not correspond to the human's desirable outcome. To tackle this issue, we employ a theoretical framework called structural causal game (SCG) to formalize the human-AI interactive process. Furthermore, we introduce a strategy referred to as pre-policy intervention on the SCG to steer AI agents towards attaining the human's desirable outcome. In more detail, a pre-policy is learned as a generalized intervention to guide the agents' policy selection, under a transparent and interpretable procedure determined by the SCG. To make the framework practical, we propose a reinforcement learning-like algorithm to search out this pre-policy. The proposed algorithm is tested in both gridworld environments and realistic dialogue scenarios with large language models, demonstrating its adaptability in a broader class of problems and potential effectiveness in real-world situations.

Uncertainty-Aware Fair Regularization Under Datasets With Incomplete Sensitive Information andreas athanasopoulos; Christos Dimitrakakis We consider the challenge of algorithmic fairness for datasets with partially annotated sensitive information. Many existing methods simply use imputation models to infer sensitive attributes as a preprocessing step. We argue that the inherent uncertainty in imputation significantly influences the bias mitigation process, particularly in scenarios with limited annotations. We adopt a Bayesian viewpoint and propose two methods based on common fairness metrics. The first minimises the expected deviation from fairness under the current belief. The second instead uses the epistemic value-at-risk, in order to robustify the algorithm's fairness properties. In practice, we implement this approach through an ensemble of neural networks. The results show that explicitly incorporating uncertainty about the individual imputed labels as well as the imputation models leads to significantly improved fairness properties and overall performance.

Privacy-Efficacy Tradeoff of Clipped SGD with Decision-dependent Data Qiang LI; Michal Yemini; Hoi To Wai This paper studies the privacy-efficacy tradeoff of clipped SGD algorithms when there is an interplay between the data distribution and the model deployed by the algorithm during training, also known as the performative prediction setting. Our contributions are two-fold. First, we show that the projected clipped SGD (**PCSGD**) algorithm may converge to a biased solution bounded away from the performative stable point. We quantify the lower and upper bound for the bias magnitude and demonstrate a *bias amplification* phenomenon where the bias grows with the sensitivity of the data distribution. Second, we suggest remedies to trade-off between the clipping bias and privacy guarantee using an asymptotically optimal step size design for **PCSGD**. Numerical experiments are presented to verify our analysis.

Query Design for Crowdsourced Clustering: Effect of Cognitive Overload and Contextual Bias Yi Chen; Ramya Korlakai Vinayak Crowdsourced clustering leverages human input to group items into clusters. The design of tasks for crowdworkers, specifically the number of items presented per query, impacts answer quality and cognitive load. This work investigates the trade-off between query size and answer accuracy, revealing diminishing returns beyond 4-5 items per query. Crucially, we identify contextual bias in crowdworker responses – the likelihood of grouping items depends not only on their similarity but also on the other items present in the query. This structured noise contradicts assumptions made in existing noise models. Our findings underscore the need for more nuanced noise models that account for the complex interplay between items and query context in crowdsourced clustering tasks.

An Electoral Approach to Diversify LLM-based Multi-Agent Collective Decision-Making Xiutian Zhao; Ke Wang; Wei Peng Modern large language models (LLMs) have demonstrated cooperative synergy on complex task-solving, and collective decision-making (CDM) is a pivotal component in LLM-based multi-agent collaboration frameworks. Surprisingly, our survey on 52 recent such systems reveals a severe lack of diversity and heavy reliance on dictatorial and plurality voting for CDM. Using social choice theory as a lens, we critically examine widely-adopted CDM methods and identify their limitations. To enrich current monotonous and limited landscape of LLM-based CDM, we introduce 8 ordinal preferential voting mechanisms that can be easily integrated with various multi-agent frameworks. Our empirical case study on MMLU benchmark demonstrates that incorporating certain CDM methods alone can enhance the reasoning performance and robustness of some state-of-the-art LLMs, without any complex system designs. Furthermore, some CDM mechanisms generate positive synergies with as few as three agents, foreshadowing a profitable computation trade-off.

Do LLM Agents Have Regret? A Case Study in Online Learning and Games Chanwoo Park; Xiangyu Liu; Asuman E. Ozdaglar; Kaiqing Zhang Despite Large language models' (LLMs) emerging successes, the performance of LLM agents in decision-making has not been fully investigated through quantitative metrics, especially in the multi-agent setting when they interact with each other, a typical scenario in real-world LLM-agent applications. To better understand the limits of LLM agents in these interactive environments, we propose to study their interactions in benchmark decision-making settings in online learning and game theory, through the performance metric of mph{regret}. We first empirically study the no-regret behaviors of LLMs in canonical (non-stationary) online learning problems, as well as the emergence of equilibria when LLM agents interact through playing repeated games. We then provide some theoretical insights into the no-regret behaviors of LLM agents, under certain assumptions on the supervised pre-training and the rationality model of human decision-makers who generate the data. Notably, we also identify (simple) cases where advanced LLMs such as GPT-4 fail to be no-regret. To promote the no-regret behaviors, we propose a novel mph{unsupervised} training loss of mph{regret-loss}, which, in contrast to the supervised pre-training loss, does not require the labels of (optimal) actions. Finally, we establish the mph{statistical} guarantee of generalization bound for regret-loss minimization, and more importantly, the mph{optimization} guarantee that minimizing such a loss may mph{automatically} lead to known no-regret learning algorithms. Our further experiments demonstrate the effectiveness of our regret-loss, especially in addressing the above ``regrettable'' cases.

Multi-Agent Imitation Learning: Value is Easy, Regret is Hard Jingwu Tang; Gokul Swamy; Fei Fang; Steven Wu We study a multi-agent imitation learning (MAIL) problem where we take the perspective of a learner attempting to coordinate a group of agents in a Markov Game by recommending actions based on demonstrations of an expert doing so. Most prior work in MAIL essentially reduces the problem to matching the behavior of the expert *within* the support of the demonstrations. While doing so is sufficient to drive the *value gap* between the learner and the expert to zero under the assumption that agents are non-strategic, it does not guarantee robustness to deviations by strategic agents. Intuitively, this is because strategic deviations can depend on a counterfactual quantity: the coordinator's recommendations outside of the state distribution their recommendations induce. In response, we initiate the study of an alternative objective for MAIL in Markov Games we term the *regret gap* that explicitly accounts for potential deviations by agents in the group. We first perform an in-depth exploration of the relationship between the value and regret gaps. First, we show that while the value gap can be efficiently minimized via a direct extension of single-agent IL algorithms, even *value equivalence* can lead to an arbitrarily large regret gap. This implies that achieving regret equivalence is harder than achieving value equivalence in MAIL. We then provide a pair of efficient reductions to no-regret online convex optimization that are capable of minimizing the regret gap *(a)* under a coverage assumption on the expert (MALICE) or *(b)* with access to a queryable expert (BLADES).

Towards Safe Large Language Models for Medicine Tessa Han; Aounon Kumar; Chirag Agarwal; Himabindu Lakkaraju As large language models (LLMs) develop ever-improving capabilities and are applied in real-world settings, their safety is critical. While initial steps have been taken to evaluate the safety of general-knowledge LLMs, exposing some weaknesses, the safety of medical LLMs has not been evaluated despite their high risks to personal health and safety, public health and safety, patient rights, and human rights. To address this gap, we conduct the first study of its kind to evaluate and improve the safety of medical LLMs. We find that 1) current medical LLMs do not meet standards of general or medical safety, as they readily comply with harmful requests and that 2) fine-tuning medical LLMs on safety demonstrations significantly improves their safety. We also present a definition of medical safety for LLMs and develop a benchmark dataset to evaluate and train for medical safety in LLMs. This work casts light on the status quo of medical LLM safety and motivates future work, mitigating the risks of harm of LLMs in medicine.

Generative AI Misuse: A Taxonomy of Tactics and Insights from Media Data Nahema Marchal; Rachel Xu; Rasmi Elasmar; Iason Gabriel; Beth Goldberg; William Isaac Generative, multimodal artificial intelligence(GenAI) offers transformative potential across industries, but its misuse poses significant risks. While prior research has shed light on the potential of advanced AI systems to be exploited for malicious purposes, we still lack a concrete under-standing of how GenAI models are specifically exploited or abused in practice, including the tac-tics employed to inflict harm. In this paper, we present a taxonomy of GenAI misuse tactics, in-formed by existing academic literature and a qualitative analysis of approximately 200 observed incidents of misuse reported between January 2023and March 2024. Through this analysis, we illuminate key and novel patterns in misuse during this time period, including potential motivations, strategies, and how attackers leverage and abuse system capabilities across modalities (e.g. image, text, audio, video) in the wild. Notably, we find that manipulation of human likeness (i.e., impersonation and sockpuppeting) and falsification of evidence underlie the most common tactics used in real-world cases of misuse. We further show that the majority of reported misuse cases leverage easily accessible GenAI capabilities that require minimal technical expertise, rather than relying on complex attacks or advanced system manipulation.

A Baseline that Matters: Categorical Prioritization as Benchmark for Social Policy Algorithms Benedikt Stroebl; Rajiv Krishna Swamy; Lydia T. Liu In this paper, we investigate the impact of algorithmic decision-making on social policy, focusing on "algorithms of care" designed to enhance well-being and equitable access to services. We compare algorithmic prioritization (AP), which uses machine learning models, with categorical prioritization (CP), social policy’s current status quo method for decision-making in social policy. By establishing CP as a robust baseline, we provide a comparative analysis framework to evaluate AP against CP, using the Rashomon set to explore the spectrum of nearly optimal AP models. Our study investigates the efficacy and fairness of AP models relative to CP, utilizing a Rashomon set analysis to explore a spectrum of nearly optimal AP models. By conducting a detailed case study on student dropout prediction at the Polytechnic Institute of Portalegre (IPP) in Portugal, we demonstrate how well-designed CP rules serve not only as effective benchmarks but also potentially surpass AP in terms of fairness and operational simplicity. We emphasize the need for careful deliberation when choosing between AP and CP in social policy.

Fair Allocation in Dynamic Mechanism Design Alireza Fallah; Michael Jordan; Annie S Ulichney We consider a dynamic mechanism design problem where an auctioneer sells an indivisible good to two groups of buyers in every round, for a total of $T$ rounds. The auctioneer aims to maximize their discounted overall revenue while adhering to a fairness constraint that guarantees a minimum average allocation for each group. We begin by studying the static case ($T=1$) and establish that the optimal mechanism involves two types of subsidization: one that increases the overall probability of allocation to all buyers, and another that favors the group which otherwise has a lower probability of winning the item. We then extend our results to the dynamic case by characterizing a set of recursive functions that determine the optimal allocation and payments in each round. Notably, our results establish that in the dynamic case, the seller, on one hand, commits to a participation reward to incentivize truth-telling, and on the other hand, charges an entry fee for every round. Moreover, the optimal allocation once more involves subsidization in favor of one group, where the extent of subsidization depends on the difference in future utilities for both the seller and buyers when allocating the item to one group versus the other. Finally, we present an approximation scheme to solve the recursive equations and determine an approximately optimal and fair allocation efficiently.

AIs Favoring AIs: Large Language Models Favor Their Own Generated Content Walter Laurito; Benjamin Davis; peli grietzer; Tomáš Gavenčiak; Ada Böhm; Jan Kulveit Are large language models (LLMs) biased towards text generated by LLMs over text authored by humans, leading to possible anti-human bias? Utilizing a classical experimental design inspired by employment discrimination studies, we tested widely-used LLMs, including GPT-3.5 and GPT-4, in binary-choice scenarios. These involved LLM-based agents selecting between products and academic papers described either by humans or LLMs under identical conditions. Our results show a consistent tendency for LLM-based AIs to prefer LLM-generated content. This suggests the possibility of AI systems implicitly discriminating against humans, giving AI agents an unfair advantage.

A Generative Port of Mars Oliver Slumbers; Joel Z Leibo; Marco Janssen Collective risk social dilemmas (CRSD) highlight a tradeoff between individual preferences and the need for all to contribute toward achieving a group objective. Problems such as climate change are in this category, and so it is critical to understand their social underpinnings. However, rigorous CRSD methodology often demands large-scale human experiments but it is difficult to guarantee sufficient power and heterogeneity over socio-demographic factors. Generative AI offers a potential way to circumvent this problem. By replacing human participants with large language models (LLM), allowing for a scalable empirical framework. This work focuses on the validity of this approach and whether it is feasible to represent a large-scale human-like experiment with sufficient diversity using LLM. In particular, where previous literature has focused on political surveys, virtual towns and classical game-theoretic examples, we focus on a complex CRSD used in the institutional economics and sustainability literature known as Port of Mars.

Improving Centrality Fairness in Algorithmic Link-Recommendations Madhura Manish Pawar; Fariba Karimi; Fernando P. Santos Link recommendation algorithms are used in online social networks to recommend new connections (e.g., friends or followees) to users. These algorithms can reduce the visibility of certain demographic groups. Recent approaches aim at adapting embedding-based methods to create unbiased network representations which, in turn, can be used to recommend connections in a fairer way. It remains unclear how these methods affect the network centrality of groups in social networks. Here, we evaluate how recommendations based on Fairwalk (a well-known method to generate fair graph embeddings) impact groups' betweenness centrality. We find that Fairwalk only ensures fair betweenness centrality for a narrow combination of group homophily. We propose a new method (Adaptive-alpha) that ensures fair centrality of various sensitive groups while maintaining similar utility when evaluated on synthetic networks and an empirical social network.