Hevelius, a web-based computer mouse test, measures arm movement and has been shown to accurately evaluate severity for patients with Parkinson’s disease and ataxias. A Hevelius session produces 32 numeric features, which may be hard to interpret, especially in time-constrained clinical settings. This work aims to support clinicians (and other stakeholders) in interpreting and connecting Hevelius features to clinical concepts. Through an iterative design process, we developed a visualization tool (Hevelius Report) that (1) abstracts six clinically relevant concepts from 32 features, (2) visualizes patient test results, and compares them to results from healthy controls and other patients, and (3) is an interactive app to meet the specific needs in different usage scenarios. Then, we conducted a preliminary user study through an online interview with three clinicians who were not involved in the project. They expressed interest in using Hevelius Report, especially for identifying subtle changes in their patients’ mobility that are hard to capture with existing clinical tests. Future work will integrate the visualization tool into the current clinical workflow of a neurology team and conduct systematic evaluations of the tool’s usefulness, usability, and effectiveness. Hevelius Report represents a promising solution for analyzing fine-motor test results and monitoring patients’ conditions and progressions.

Affective computing has potential to enrich the development lifecycle of Graphical User Interfaces (GUIs) and of intelligent user interfaces by incorporating emotion-aware responses. Yet, affect is seldom considered to determine whether a GUI design would be perceived as good or bad. We study how physiological signals can be used as an early, effective, and rapid affective assessment method for GUI design, without having to ask for explicit user feedback. We conducted a controlled experiment where 32 participants were exposed to 20 good GUI and 20 bad GUI designs while recording their eye activity through eye tracking, facial expressions through video recordings, and brain activity through electroencephalography (EEG). We observed noticeable differences in the collected data, so we trained and compared different computational models to tell good and bad designs apart. Taken together, our results suggest that each modality has its own “performance sweet spot” both in terms of model architecture and signal length. Taken together, our findings suggest that is possible to distinguish between good and bad designs using physiological signals. Ultimately, this research paves the way toward implicit evaluation methods of GUI designs through user modeling.

Though images are ubiquitous across Wikipedia, it is not obvious that the image choices optimally support learning. When well selected, images can enhance learning by dual coding, complementing, or supporting articles. When chosen poorly, images can mislead, distract, and confuse. We developed a large dataset containing 470 questions & answers to 94 Wikipedia articles with images on a wide range of topics. Through an online experiment (n=704), we determined whether the images displayed alongside the text of the article are effective in helping readers understand and learn. For certain tasks, such as learning to identify targets visually (e.g., "which of these pictures is a gujia?"), article images significantly improve accuracy. Images did not significantly improve general knowledge questions (e.g., "where are gujia from?"). Most interestingly, only some images helped with visual knowledge questions (e.g., "what shape is a gujia?"). Using our findings, we reflect on the implications for editors and tools to support image selection.

Imagine if AI decision-support tools not only complemented our ability to make accurate decisions, but also improved our skills, boosted collaboration, and elevated the joy we derive from our tasks. Despite the potential to optimize a broad spectrum of such human-centric objectives, the design of current AI tools remains focused on decision accuracy alone. We propose offline reinforcement learning (RL) as a general approach for modeling human-AI decision-making to optimize human-AI interaction for diverse objectives. RL can optimize such objectives by tailoring decision support, providing the right type of assistance to the right person at the right time. We instantiated our approach with two objectives: human-AI accuracy on the decision-making task and human learning about the task and learned decision support policies from previous human-AI interaction data. We compared the optimized policies against several baselines in AI-assisted decision-making. Across two experiments (N=316 and N=964), our results demonstrated that people interacting with policies optimized for accuracy achieve significantly better accuracy -- and even human-AI complementarity -- compared to those interacting with any other type of AI support. Our results further indicated that human learning was more difficult to optimize than accuracy, with participants who interacted with learning-optimized policies showing significant learning improvement only at times. Our research (1) demonstrates offline RL to be a promising approach to model human-AI decision-making, leading to policies that may optimize human-centric objectives and provide novel insights about the AI-assisted decision-making space, and (2) emphasizes the importance of considering human-centric objectives beyond decision accuracy in AI-assisted decision-making, opening up the novel research challenge of optimizing human-AI interaction for such objectives.

The aesthetic appeal of a website has strong effects on users’ reactions, appraisals, and even behaviors. However, evaluating website aesthetics through user ratings is resource intensive, and extant models to predict website aesthetics are limited in performance and ability. We contribute a novel and more precise approach to predict website aesthetics that considers rating distributions. Moreover, we use this approach as a baseline model to illustrate how future research might be conducted using predictions instead of participants. Our approach is based on a deep convolutional neural network model and uses innovations in the field of image aesthetic prediction. It was trained with the dataset from Reinecke and Gajos [2014] and was validated using two independent large datasets. The final model reached an unprecedented cross-validated correlation between the ground truth and predicted rating of LCC = 0.752. We then used the model to successfully replicate prior findings and conduct original research as an illustration for AI-based research.

Strong end-user security practices benefit both the user and hosting platform, but it is not well understood how companies communicate with their users to encourage these practices. This paper explores whether web companies and their platforms use different levels of language formality in these communications and tests the hypothesis that higher language formality leads to users’ increased intention to comply. We contribute a dataset and systematic analysis of 1,817 English language strings in web security and privacy interfaces across 13 web platforms, showing strong variations in language. An online study with 512 participants further demonstrated that people perceive differences in the language formality across platforms and that a higher language formality is associated with higher self-reported intention to comply. Our findings suggest that formality can be an important factor in designing effective security and privacy prompts. We discuss implications of these results, including how to balance formality with platform language style. In addition to being the first piece of work to analyze language formality in user security, these findings provide valuable insights into how platforms can best communicate with users about account security.

The “Reading the Mind in the Eyes” Test (Eyes Test) is a widely used assessment of “theory of mind.” The NIMH Research Domain Criteria recommends it as one of two tests for “understanding mental states.” Previous studies have demonstrated an on-average female advantage on the Eyes Test. However, it is unknown whether this female advantage exists across the lifespan and across a large number of countries. Thus, we tested sex and age differences using the English version of the Eyes Test in adolescents and adults across 57 countries. We also tested for associations with sociodemographic and cognitive/personality factors. We leveraged one discovery dataset ( N = 305,726) and three validation datasets ( N s = 642; 5,284; and 1,087). The results show that: i) there is a replicable on-average female advantage in performance on the Eyes Test; ii) performance increases through adolescence and shallowly declines across adulthood; iii) the on-average female advantage is evident across the lifespan; iv) there is a significant on-average female advantage in 36 out of 57 countries; v) there is a significant on-average female advantage on translated (non-English) versions of the Eyes Test in 12 out of 16 countries, as confirmed by a systematic review; vi) D-scores, or empathizing-systemizing, predict Eyes Test performance above and beyond sex differences; and vii) the female advantage is negatively linked to “prosperity” and “autonomy,” and positively linked to “collectivism,” as confirmed by exploratory country-level analyses. We conclude that the on-average female advantage on the Eyes Test is observed across ages and most countries.

Website aesthetics play an important role in attracting users and customers, as well as in enhancing user experience. In this work, we propose a tool that performs automatic evaluation of website aesthetics using deep learning models that display high correlation to human perception. These models were developed using two different datasets. The first dataset was created by employing a rating-based ranking approach and contains user judgments on websites in the form of an explicit numerical value on a scale. Using the first dataset, we developed models following three different approaches and managed to outperform previous works. In addition, we created a new dataset by employing a comparison-based ranking approach, which is a more reliable dataset in the sense that it follows a more “natural” data collection method. In this case, users were asked to compare two websites at a time and choose which is more attractive. Data collection was performed via a web application especially designed and developed for this purpose. In the experiments conducted, we evaluated each model and compared the two data collection methods. This work aims to illustrate the effectiveness of deep learning as a solution to the problem as well as to highlight the importance of comparison-based ranking in order to achieve reliable results. In order to further promote our work, we also developed a tool that scores the aesthetics of a website, simply by providing the website URL. We argue that such a tool will serve as a reliable guide in the hands of designers and developers during the design process.

Aesthetics is a central consideration in user interface design. It is known to affect end-user behavior and perception, in particular the first impression of graphical user interfaces. However, what users perceive as pleasant or good design is highly subjective. We contribute a computational model that estimates the visual appeal of a given webpage for several common cohorts, or user groups, including gender, age, and education level. Our model, a convolutional neural network trained on 418 webpage screenshots having 771k aesthetic scores (in a 1–9 Likert scale) from 32k users, achieves high accuracy and is always less than 1 point off from ground-truth ratings. Designers can use our model to anticipate how people would rate their webpage, offer personalized designs according to the visual preferences of their users, and support rapid evaluations of webpage design prototypes for specific cohorts.

Design biases in NLP systems, such as performance differences for different populations, often stem from their creator's positionality, i.e., views and lived experiences shaped by identity and background. Despite the prevalence and risks of design biases, they are hard to quantify because researcher, system, and dataset positionality is often unobserved. We introduce NLPositionality, a framework for characterizing design biases and quantifying the positionality of NLP datasets and models. Our framework continuously collects annotations from a diverse pool of volunteer participants on LabintheWild, and statistically quantifies alignment with dataset labels and model predictions. We apply NLPositionality to existing datasets and models for two tasks -- social acceptability and hate speech detection. To date, we have collected 16,299 annotations in over a year for 600 instances from 1,096 annotators across 87 countries. We find that datasets and models align predominantly with Western, White, college-educated, and younger populations. Additionally, certain groups, such as non-binary people and non-native English speakers, are further marginalized by datasets and models as they rank least in alignment across all tasks. Finally, we draw from prior literature to discuss how researchers can examine their own positionality and that of their datasets and models, opening the door for more inclusive NLP systems.

Museums have widely embraced virtual exhibits. However, relatively little attention is paid to how sound may create a more engaging experience for audiences. To begin addressing this lacuna, we conducted an online experiment to explore how sound influences the interest level, emotional response, and engagement of individuals who view objects within a virtual exhibit. As part of this experiment, we designed a set of different soundscapes, which we presented to participants who viewed museum objects virtually. We then asked participants to report their felt affect and level of engagement with the exhibits. Our results show that soundscapes customized to exhibited objects significantly enhance audience engagement. We also found that more engaged audience members were more likely to want to learn additional information about the object(s) they viewed and to continue viewing these objects for longer periods of time. Taken together, our findings suggest that virtual museum exhibits can improve visitor engagement through forms of customized soundscape design.

As dating websites are becoming an essential part of how people meet intimate and romantic partners, it is vital to design these systems to be resistant to, or at least do not amplify, bias and discrimination. Instead, the results of our online experiment with a simulated dating website, demonstrate that popular dating website design choices, such as the user of the swipe interface (swiping in one direction to indicate a like and in the other direction to express a dislike) and match scores, resulted in people racially biases choices even when they explicitly claimed not to have considered race in their decision-making. This bias was significantly reduced when the order of information presentation was reversed such that people first saw substantive profile information related to their explicitly-stated preferences before seeing the profile name and photo. These results indicate that currently-popular design choices amplify people’s implicit biases in their choices of potential romantic partners, but the effects of the implicit biases can be reduced by carefully redesigning the dating website interfaces.

When people receive advice while making difficult decisions, they often make better decisions in the moment and also increase their knowledge in the process. However, such incidental learning can only occur when people cognitively engage with the information they receive and process this information thoughtfully. How do people process the information and advice they receive from AI, and do they engage with it deeply enough to enable learning? To answer these questions, we conducted three experiments in which individuals were asked to make nutritional decisions and received simulated AI recommendations and explanations. In the first experiment, we found that when people were presented with both a recommendation and an explanation before making their choice, they made better decisions than they did when they received no such help, but they did not learn. In the second experiment, participants first made their own choice, and only then saw a recommendation and an explanation from AI; this condition also resulted in improved decisions, but no learning. However, in our third experiment, participants were presented with just an AI explanation but no recommendation and had to arrive at their own decision. This condition led to both more accurate decisions and learning gains. We hypothesize that learning gains in this condition were due to deeper engagement with explanations needed to arrive at the decisions. This work provides some of the most direct evidence to date that it may not be sufficient to include explanations together with AI-generated recommendation to ensure that people engage carefully with the AI-provided information. This work also presents one technique that enables incidental learning and, by implication, can help people process AI recommendations and explanations more carefully.

The combination of tree search and neural networks has achieved super-human performance in challenging domains. We are interested in transferring to humans the knowledge these learning systems generate. We hypothesize the process in which neural-guided tree search algorithms learn how to solve a set of problems can be used to generate curricula for helping human learners. In this paper we show how the Bootstrap learning system can be modified to learn curricula for humans in a puzzle domain. We evaluate our system in two curriculum learning settings. First, given a small set of problem instances, our system orders the instances to ease the learning process of human learners. Second, given a large set of problem instances, our system returns a small ordered subset of the initial set that can be presented to human learners. We evaluate our curricula with a user study where participants learn how to solve a class of puzzles from the game `The Witness.' The user-study results suggest one of the curricula our system generates compares favorably with simple baselines and is competitive with the curriculum from the original `The Witness' game in terms of user retention and effort.

As conversational agents and digital assistants become increasingly pervasive, understanding their synthetic speech becomes increasingly important. Simultaneously, speech synthesis is becoming more sophisticated and manipulable, providing the opportunity to optimize speech rate to save users time. However, little is known about people’s abilities to understand fast speech. In this work, we provide an extension of the first large-scale study on human listening rates, enlarging the prior study run with 453 participants to 1,409 participants and adding new analyses on this larger group. Run on LabintheWild, it used volunteer participants, was screen reader accessible, and measured listening rate by accuracy at answering questions spoken by a screen reader at various rates. Our results show that people who are visually impaired, who often rely on audio cues and access text aurally, generally have higher listening rates than sighted people. The findings also suggest a need to expand the range of rates available on personal devices. These results demonstrate the potential for users to learn to listen to faster rates, expanding the possibilities for human-conversational agent interaction.

With disease-modifying approaches under evaluation in ataxia-telangiectasia and other ataxias, there is a need for objective and reliable biomarkers of free-living motor function. In this study, we test the hypothesis that metrics derived from a single wrist sensor worn at home provide accurate, reliable, and interpretable information about neurological disease severity in children with A-T. A total of 15 children with A-T and 15 age- and sex-matched controls wore a sensor with a triaxial accelerometer on their dominant wrist for 1 week at home. Activity intensity measures, derived from the sensor data, were compared with in-person neurological evaluation on the Brief Ataxia Rating Scale (BARS) and performance on a validated computer mouse task. Children with A-T were inactive the same proportion of each day as controls but produced more low intensity movements (p < 0.01; Cohen’s d = 1.48) and fewer high intensity movements (p < 0.001; Cohen’s d = 1.71). The range of activity intensities was markedly reduced in A-T compared to controls (p < 0.0001; Cohen’s d = 2.72). The activity metrics correlated strongly with arm, gait, and total clinical severity (r: 0.71–0.87; p < 0.0001), correlated with specific computer task motor features (r: 0.67–0.92; p < 0.01), demonstrated high reliability (r: 0.86–0.93; p < 0.00001), and were not significantly influenced by age in the healthy control group. Motor activity metrics from a single, inexpensive wrist sensor during free-living behavior provide accurate and reliable information about diagnosis, neurological disease severity, and motor performance. These low-burden measurements are applicable independent of ambulatory status and are potential digital behavioral biomarkers in A-T.

Prior work in cross-cultural psychology and neuroscience has shown robust variations in visual attention patterns. People from East Asian societies, in which a holistic thinking style predominates, have been found to attend to contextual information in scenes more than Westerners, whose tendency to think analytically expresses itself in greater attention to foreground objects. This paper applies these findings to website design, using an online study to evaluate whether Japanese (N=65) remember more and are faster at finding contextual website information than US Americans (N=84). Our results do not support this hypothesis. Instead, Japanese overall took significantly longer to find information than US participants—a difference that was exacerbated by an increase in website complexity—suggesting that Japanese may holistically take in a website before engaging with detailed information. We discuss implications of these findings for website design and cross-cultural research.

In the ongoing COVID-19 pandemic, public health experts have produced guidelines to limit the spread of the coronavirus, but individuals do not always comply with experts’ recommendations. Here, we tested whether a specific psychological belief—identification with all humanity—predicts cooperation with public health guidelines as well as helpful behavior during the COVID-19 pandemic. We hypothesized that peoples’ endorsement of this belief—their relative perception of a connection and moral commitment to other humans—would predict their tendencies to adopt World Health Organization (WHO) guidelines and to help others. To assess this, we conducted a global online study ( N = 2537 participants) of four WHO-recommended health behaviors and four pandemic-related moral dilemmas that we constructed to be relevant to helping others at a potential cost to oneself. We used generalized linear mixed models (GLMM) that included 10 predictor variables (demographic, contextual, and psychological) for each of five outcome measures (a WHO cooperative health behavior score, plus responses to each of our four moral, helping dilemmas). Identification with all humanity was the most consistent and consequential predictor of individuals’ cooperative health behavior and helpful responding. Analyses showed that the identification with all humanity significantly predicted each of the five outcomes while controlling for the other variables ( P range < 10 −22 to < 0.009). The mean effect size of the identification with all humanity predictor on these outcomes was more than twice as large as the effect sizes of other predictors. Identification with all humanity is a psychological construct that, through targeted interventions, may help scientists and policymakers to better understand and promote cooperative health behavior and help-oriented concern for others during the current pandemic as well as in future humanitarian crises.

We conducted an online study with 165 participants in which we tested their search efficiency and information recall. We confirm that the visual complexity of a website has a significant negative effect on search efficiency and information recall. However, the search efficiency of those who preferred simple websites was more negatively affected by highly complex websites than those who preferred high visual complexity. Our results suggest that diverse visual preferences need to be accounted for when assessing search response time and information recall in HCI experiments, testing software, or A/B tests.

Abstract While online experiments have shown tremendous potential to study larger and more diverse participant samples than is possible in the lab, the uncontrolled online environment has prohibited many types of psychophysical studies due to difficulties controlling the viewing distance and stimulus size. We introduce the Virtual Chinrest, a method that measures a participant’s viewing distance in the web browser by detecting a participant’s blind spot location. This makes it possible to automatically adjust stimulus configurations based on an individual’s viewing distance. We validated the Virtual Chinrest in two laboratory studies in which we varied the viewing distance and display size, showing that our method estimates participants’ viewing distance with an average error of 3.25 cm. We additionally show that by using the Virtual Chinrest we can reliably replicate measures of visual crowding, which depends on a precise calculation of visual angle, in an uncontrolled online environment. An online experiment with 1153 participants further replicated the findings of prior laboratory work, demonstrating how visual crowding increases with eccentricity and extending this finding by showing that young children, older adults and people with dyslexia all exhibit increased visual crowding, compared to adults without dyslexia. Our method provides a promising pathway to web-based psychophysical research requiring controlled stimulus geometry.

Abstract Background Objective assessments of movement impairment are needed to support clinical trials and facilitate diagnosis. The objective of the current study was to determine if a rapid web‐based computer mouse test (Hevelius) could detect and accurately measure ataxia and parkinsonism. Methods Ninety‐five ataxia, 46 parkinsonism, and 29 control participants and 229,017 online participants completed Hevelius. We trained machine‐learning models on age‐normalized Hevelius features to (1) measure severity and disease progression and (2) distinguish phenotypes from controls and from each other. Results Regression model estimates correlated strongly with clinical scores (from r = 0.66 for UPDRS dominant arm total to r = 0.83 for the Brief Ataxia Rating Scale). A disease change model identified ataxia progression with high sensitivity. Classification models distinguished ataxia or parkinsonism from healthy controls with high sensitivity (≥0.91) and specificity (≥0.90). Conclusions Hevelius produces a granular and accurate motor assessment in a few minutes of mouse use and may be useful as an outcome measure and screening tool. © 2019 The Authors. Movement Disorders published by Wiley Periodicals, Inc. on behalf of International Parkinson and Movement Disorder Society.

Current story writing or story editing systems rely on human judgments of story quality for evaluating performance, often ignoring the subjectivity in ratings.We analyze the effect of author and reader characteristics and story writing setup on the quality of stories in a short storytelling task.To study this effect, we create and release STORIESINTHEWILD, containing 1,630 stories collected on a volunteerbased crowdsourcing platform.Each story is rated by three different readers, and comes paired with the author's and reader's age, gender, and personality.Our findings show significant effects of authors' and readers' identities, as well as writing setup, on story writing and ratings.Notably, compared to younger readers, readers age 45 and older consider stories significantly less creative and less entertaining.Readers also prefer stories written all at once, rather than in chunks, finding them more coherent and creative.We also observe linguistic differences associated with authors' demographics (e.g., older authors wrote more vivid and emotional stories).Our findings suggest that reader and writer demographics, as well as writing setup, should be accounted for in story writing evaluations.

Emotional Intelligence, Social Intelligence and Spiritual Intelligence have been proposed as the strong contenders over cognitive concepts of intelligence. The definitions of emotional, social and spiritual intelligences vary among the theorists and have many overlaps on the dimensions. Present study aims to study relationship among emotional, social and spiritual intelligence. It was hypothesized that there will be strong positive correlations between the three intelligences. This correlational study used three online tools Matrix Emotional Intelligence Test, Social Intelligence Test and Spiritual Intelligence Test and found out scores of 23 girls between the age group 20-25 years pursuing post-graduation in psychology from Department of Psychology, The Maharaja Sayajirao University of Baroda, Vadodara. Pearson Product Moment Correlations reveals distinct pattern of relationship among the three intelligences – Emotional, Social, and Spiritual. The implications of the findings are discussed in the light of conceptual overlap of the three intelligences.

As reading increasingly shifts from paper to online media, many web browsers now provide a "Reader View,'' which modifies web page layout and design for better readability. However, research has yet to establish whether Reader Views are effective in improving readability and how they might change the user experience. We characterize how Mozilla Firefox's Reader View significantly reduces the visual complexity of websites by excluding menus, images, and content. We then conducted an online study with 391 participants (including 42 who self-reported having been diagnosed with dyslexia), showing that compared to standard websites the Reader View increased reading speed by 5% for readers on average, and significantly improved perceived readability and visual appeal. We suggest guidelines for the design of websites and browsers that better support people with varying reading skills.

Abstract As web has become the most popular media to attract users and customers worldwide, webpage aesthetics plays an increasingly important role for engaging users online and impacting their user experience. We present a novel method using deep learning to automatically compute and quantify webpage aesthetics. Our deep neural network, named as Webthetics, which is trained from the collected user rating data, can extract representative features from raw webpages and quantify their aesthetics. To improve the model performance, we propose to transfer the knowledge from image style recognition task into our network. We have validated that our method significantly outperforms previous method using hand-crafted features such as colorfulness and complexity. These promising results indicate that our method can serve as an effective and efficient means for providing objective aesthetics evaluation during the design process.

Color names facilitate the identification and communication of colors, but may vary across languages. We contribute a set of human color name judgments across 14 common written languages and build probabilistic models that find different sets of nameable (salient) colors across languages. For example, we observe that unlike English and Chinese, Russian and Korean have more than one nameable blue color among fully-saturated RGB colors. In addition, we extend these probabilistic models to translate color terms from one language to another via a shared perceptual color space. We compare Korean-English translations from our model to those from online translation tools and find that our method better preserves perceptual similarity of the colors corresponding to the source and target terms. We conclude with implications for visualization and future research.

There are few large-scale empirical studies with people with disabilities or older adults, mainly because recruiting partici­pants with specific characteristics is even harder than recruit­ing young and/or non-disabled populations. Analyzing four online experiments on LabintheWild with a total of 355,656 participants, we show that volunteer-based online experiments that provide personalized feedback attract large numbers of participants with diverse disabilities and ages and allow ro­bust studies with these populations that replicate and extend the findings of prior laboratory studies. To find out what mo­tivates people with disabilities to take part, we additionally analyzed participants' feedback and forum entries that discuss LabintheWild experiments. The results show that participants use the studies to diagnose themselves, compare their abilities to others, quantify potential impairments, self-experiment, and share their own stories -- findings that we use to inform design guidelines for online experiment platforms that adequately support and engage people with disabilities.

As conversational agents and digital assistants become increasingly pervasive, understanding their synthetic speech becomes increasingly important. Simultaneously, speech synthesis is becoming more sophisticated and manipulable, providing the opportunity to optimize speech rate to save users time. However, little is known about people's abilities to understand fast speech. In this work, we provide the first large-scale study on human listening rates. Run on LabintheWild, it used volunteer participants, was screen reader accessible, and measured listening rate by accuracy at answering questions spoken by a screen reader at various rates. Our results show that blind and low-vision people, who often rely on audio cues and access text aurally, generally have higher listening rates than sighted people. The findings also suggest a need to expand the range of rates available on personal devices. These results demonstrate the potential for users to learn to listen to faster rates, expanding the possibilities for human-conversational agent interaction.

The ubiquity of internet-based nutrition information sharing indicates an opportunity to use social computing platforms to promote nutrition literacy and healthy nutritional choices. We conducted a series of experiments with unpaid volunteers using an online Nutrition Knowledge Test. The test asked participants to examine pairs of photographed meals and identify meals higher in a specific macronutrient (e.g., carbohydrate). After each answer, participants received no feedback on the accuracy of their answers, viewed proportions of peers choosing each response, received correctness feedback from an expert dietitian with or without expert-generated explanations, or received correctness feedback with crowd-generated explanations. The results showed that neither viewing peer responses nor correctness feedback alone improved learning. However, correctness feedback with explanations (i.e., modeling) led to significant learning gains, with no significant difference between explanations generated by experts or peers. This suggests the importance of explanations in social computing-based casual learning about nutrition and the potential for scaling this approach via crowdsourcing.

One of the problems adaptive interfaces must solve is the issue of stability---users must be able to complete a familiar task reliably. Split Adaptive Interfaces, where a limited part of the screen contains copies of the interface elements predicted to be of immediate use, are one technique for resolving this difficulty. While prior work demonstrated that Split Adaptive Interfaces improve performance on average, the results of our study demonstrate systematic individual differences in the utilization of the adaptive features, which correlate with the stable user traits of Need for Cognition and Extraversion. Specifically, higher Need for Cognition (a willingness to undertake difficult mental activities) is correlated with increased utilization rates, while higher Extraversion (a general orientation towards seeking gratification from the external world) is negatively correlated with utilization rates. Our results also demonstrate a significant negative correlation between cognitive load induced by a secondary task and the utilization of the adaptive features. This effect, however, is very small (less than two percentage points). Together, these results provide additional evidence of the usefulness of the split adaptive interface approach and a negligible effect of additional cognitive load, but also demonstrate that the approach does not benefit all users equally.

As data become more pervasive and computing power increases, the opportunity for transformative use of data grows. Collecting data from individuals can be useful to the individuals (by providing them with personalized predictions) and the data collectors (by providing them with information about populations). However, collecting these data is costly: answering survey items, collecting sensed data, and computing values of interest deplete finite resources of time, battery, life, money, etc. Dynamically ordering the items to be collected, based on already known information (such as previously collected items or paradata), can lower the costs of data collection by tailoring the information-acquisition process to the individual. This thesis presents a framework for an iterative dynamic item ordering process that trades off item utility with item cost at data collection time. The exact metrics for utility and cost are application-dependent, and this frame- work can apply to many domains. The two main scenarios we consider are (1) data collection for personalized predictions and (2) data collection in surveys. We illustrate applications of this framework to multiple problems ranging from personalized prediction to questionnaire scoring to government survey collection. We compare data quality and acquisition costs of our method to fixed order approaches and show that our adaptive process obtains results of similar quality at lower cost. For the personalized prediction setting, the goal of data collection is to make a prediction based on information provided by a respondent. Since it is possible to give a reasonable prediction with only a subset of items, we are not concerned with collecting all items. Instead, we want to order the items so that the user provides information that most increases the prediction quality, while not being too costly to provide. One metric for quality is prediction certainty, which reflects how likely the true value is to coincide with the estimated value. Depending whether the prediction problem is continuous or discrete, we use prediction interval width or predicted class probability to measure the certainty of a prediction. We illustrate the results of our dynamic item ordering framework on tasks of predicting energy costs, student stress levels, and device identification in photographs and show that our adaptive process achieves equivalent error rates as a fixed order baseline with cost savings up to 45%. For the survey setting, the goal of data collection is often to gather information from a population, and it is desired to have complete responses from all samples. In this case, we want to maximize survey completion (and the quality of necessary imputations), and so we focus on ordering items to engage the respondent and collect hopefully all the information we seek, or at least the information that most characterizes the respondent so imputed values will be accurate. One item utility metric for this problem is information gain to get a “representative” set of answers from the respondent. Furthermore, paradata collected during the survey process can inform models of user engagement that can influence either the utility metric ( e.g., likelihood therespondent will continue answering questions) or the cost metric (e.g., likelihood the respondent will break off from the survey). We illustrate the benefit of dynamic item ordering for surveys on two nationwide surveys conducted by the U.S. Census Bureau: the American Community Survey and the Survey of Income and Program Participation.

Prior work on creativity support tools demonstrates how a computational semantic model of a solution space can enable interventions that substantially improve the number, quality and diversity of ideas. However, automated semantic modeling often falls short when people contribute short text snippets or sketches. Innovation platforms can employ humans to provide semantic judgments to construct a semantic model, but this relies on external workers completing a large number of tedious micro tasks. This requirement threatens both accuracy (external workers may lack expertise and context to make accurate semantic judgments) and scalability (external workers are costly). In this paper, we introduce IdeaHound, an ideation system that seamlessly integrates the task of defining semantic relationships among ideas into the primary task of idea generation. The system combines implicit human actions with machine learning to create a computational semantic model of the emerging solution space. The integrated nature of these judgments allows IDEAHOUND to leverage the expertise and efforts of participants who are already motivated to contribute to idea generation, overcoming the issues of scalability inherent to existing approaches. Our results show that participants were equally willing to use (and just as productive using) IDEAHOUND compared to a conventional platform that did not require organizing ideas. Our integrated crowdsourcing approach also creates a more accurate semantic model than an existing crowdsourced approach (performed by external crowds). We demonstrate how this model enables helpful creative interventions: providing diverse inspirational examples, providing similar ideas for a given idea and providing a visual overview of the solution space.

Users frequently experience situations in which their ability to differentiate screen colors is affected by a diversity of situations, such as when bright sunlight causes glare, or when monitors are dimly lit. However, designers currently have no way of choosing colors that will be differentiable by users of various demographic backgrounds and abilities and in the wide range of situations where their designs may be viewed. Our goal is to provide designers with insight into the effect of real-world situational lighting conditions on people's ability to differentiate colors in applications and imagery. We therefore developed an online color differentiation test that includes a survey of situational lighting conditions, verified our test in a lab study, and deployed it in an online environment where we collected data from around 30,000 participants. We then created ColorCheck, an image-processing tool that shows designers the proportion of the population they include (or exclude) by their color choices.

Mealtimes are a cherished part of everyday life around the world. Often centered on family, friends, or special occasions, sharing meals is a practice embedded with traditions and values. However, as mobile phone adoption becomes increasingly pervasive, tensions emerge about how appropriate it is to use personal devices while sharing a meal with others. Furthermore, while personal devices have been designed to support awareness for the individual user (e.g., notifications), little is known about how to support shared awareness in acceptability in social settings such as meals. In order to understand attitudes about mobile phone use during shared mealtimes, we conducted an online survey with 1,163 English-speaking participants. We find that attitudes about mobile phone use at meals differ depending on the particular phone activity and on who at the meal is engaged in that activity, children versus adults. We also show that three major factors impact participants' attitudes: 1) their own mobile phone use; 2) their age; and 3) whether a child is present at the meal. We discuss the potential for incorporating social awareness features into mobile phone systems to ease tensions around conflicting mealtime behaviors and attitudes.

Information graphics, or infographics, combine elements of data visualization with design and have become an increasingly popular means for disseminating data. While several studies have suggested that aesthetics in visualization and infographics relate to desirable outcomes like engagement and memorability, it remains unknown how quickly aesthetic impressions are formed, and what it is that makes an infographic appealing. We address these questions by analyzing 1,278 participants' ratings on appeal after seeing infographics for 500ms. Our results establish that: 1) people form a reliable first impression of the appeal of an infographic based on a mere exposure effect, 2) this first impression is largely based on colorfulness and visual complexity, and 3) age, gender, and education level influence the preferred level of colorfulness and complexity. More generally, these findings suggest that outcomes such as engagement and memorability might be determined much earlier than previously thought.

Website aesthetics have been recognized as an influential moderator of people's behavior and perception. However, what users perceive as "good design" is subject to individual preferences, questioning the feasibility of universal design guidelines. To better understand how people's visual preferences differ, we collected 2.4 million ratings of the visual appeal of websites from nearly 40 thousand participants of diverse backgrounds. We address several gaps in the knowledge about design preferences of previously understudied groups. Among other findings, our results show that the level of colorfulness and visual complexity at which visual appeal is highest strongly varies: Females, for example, liked colorful websites more than males. A high education level generally lowers this preference for colorfulness. Russians preferred a lower visual complexity, and Macedonians liked highly colorful designs more than any other country in our dataset. We contribute a computational model and estimates of peak appeal that can be used to support rapid evaluations of website design prototypes for specific target groups.

Users make lasting judgments about a website's appeal within a split second of seeing it for the first time. This first impression is influential enough to later affect their opinions of a site's usability and trustworthiness. In this paper, we demonstrate a means to predict the initial impression of aesthetics based on perceptual models of a website's colorfulness and visual complexity. In an online study, we collected ratings of colorfulness, visual complexity, and visual appeal of a set of 450 websites from 548 volunteers. Based on these data, we developed computational models that accurately measure the perceived visual complexity and colorfulness of website screenshots. In combination with demographic variables such as a user's education level and age, these models explain approximately half of the variance in the ratings of aesthetic appeal given after viewing a website for 500ms only.

Psychological research has identified different patterns individuals have while making decisions, such as vigilance (making decisions after thorough information gathering), hypervigilance (rushed and anxious decision-making), and buckpassing (deferring decisions to others). We examine whether these decision-making patterns affect peoples’ engagement with AI-generated information in decision-making. In an online experiment with 810 participants tasked with distinguishing food facts from myths, we found that a higher buckpassing tendency was positively correlated with the likelihood of seeking AI information and reported reliance on AI, while being negatively correlated with the time spent reading AI explanations. In contrast, the higher a participant tended towards vigilance, the more carefully they scrutinized the AI’s information, as indicated by an increased time spent looking through the AI’s explanations. These findings suggest that a person’s decision-making pattern plays a significant role in their interactions with AI suggestions, which provides a new understanding of individual differences in AI-assisted decision-making.

How culture shapes the design and use of technology—and how we can resist the one-size-fits-all approach to technology design

To design accessible technology, it is vital in both academia and industry to solicit information from people with disabilities. Gathering information from those with cognitive disabilities is particularly important since the impact of these disorders differs by individual and over time. However, traditional recruiting methods, through gate-keepers such as local organizations, jeopardize generalizability due to the small numbers of such participants. Therefore, some researchers turn to online experiments, including those with volunteers, to increase participation and diversity. However, there is little understanding of whether volunteer-based online experiments are suitable for studying those with cognitive disabilities: Can we attract sufficient numbers of such volunteers? Can these studies be conducted rigorously? Can we provide benefits and support in return for volunteer participation? In this dissertation, I address the trade-off between needing more diverse and larger numbers of participants and needing more control to conduct rigorous studies that target cognitive disabilities. Towards the goal, I (1) demonstrate the viability of volunteer-based online experiments in studying people with cognitive disabilities at scale, (2) identify how volunteer-based online studies related to cognitive disabilities are currently perceived by different stakeholders, i.e., participants and healthcare professionals, and (3) demonstrate the feasibility of using online experiments as a method to rigorously conduct studies that inform design guidelines of technologies for cognitive disabilities, in particular, dyslexia. Together, this work demonstrates the thesis statement: Volunteer-based online studies can be conducted with people with cognitive disabilities in a way that is large-scale, self-motivating, helpful for participants, and

Roughly 1 in 3 people around the world are affected by cognitive or mental disabilities at some point in their lives, yet people often face a variety of barriers when seeking support and receiving diagnosis from healthcare professionals. While prior work found that people with such disabilities assess themselves using online tests and assessments, it remains unknown whether and how effectively these tests fill gaps in healthcare and general support systems. To find out, we interviewed 17 adults with cognitive or mental disabilities about their motivation for and experience using online tests. We learned that online tests act as an important resource that address the shortcomings in support systems for people with professionally diagnosed or suspected cognitive or mental disabilities. In particular, online tests can lower barriers to a professional diagnosis, provide valuable information about the nuances of a disability, and support people in forming a disability identity – an invaluable step towards a positive acceptance of oneself. Our results also uncovered challenges and risks that prevent people with known or suspected health conditions from fully taking advantage of online tests. Based on these findings, we discuss how online tests can be better leveraged to support people with cognitive or mental disabilities before and after professional diagnosis.

Participant engagement in online studies is key to collecting reliable data, yet achieving it remains an often discussed challenge in the research community. One factor that might impact engagement is the formality of language used to communicate with participants throughout the study. Prior work has found that language formality can convey social cues and power hierarchies, affecting people's responses and actions. We explore how formality influences engagement, measured by attention, dropout, time spent on the study and participant performance, in an online study with 369 participants on Mechanical Turk (paid) and LabintheWild (volunteer). Formal language improves participant attention compared to using casual language in both paid and volunteer conditions, but does not affect dropout, time spent, or participant performance. We suggest using more formal language in studies containing complex tasks where fully reading instructions is especially important. We also highlight trade-offs that different recruitment incentives provide in online experimentation.

OBJECTIVE: Digital devices are now broadly accessible and have the capacity to measure aspects of human behavior with high precision and accuracy, in a standardized manner. The purpose of this article is to characterize opportunities and barriers for modern digital neuropsychology, particularly those that are unique to digital assessment. METHODS: We provide a critical overview of the state-of-the-art in digital neuropsychology, focusing on personal digital devices. RESULTS: We identify three major barriers associated with digital neuropsychology, which affect both the interpretation of test scores and test norms: (1) variability in the perceptual, motor and cognitive demands of the same test across digital device classes (e.g. personal computer, tablet and smartphone); (2) hardware and software variability between devices within the same class that affect stimulus presentation and measurement and (3) rapid changes over time in hardware, software and device ownership, which can lead to rapid obsolescence of particular tests and test norms. We offer specific recommendations to address these barriers and outline new opportunities to understand and measure neuropsychological functioning over time and in everyday environments. CONCLUSIONS: Digital neuropsychology provides new approaches for measuring and monitoring neuropsychological functioning, informed by an understanding of the limitations and potential of digital technology.

There are few large-scale empirical studies with people with disabilities or older adults, mainly because recruiting partici­pants with specific characteristics is even harder than recruit­ing young and/or non-disabled populations. Analyzing four online experiments on LabintheWild with a total of 355,656 participants, we show that volunteer-based online experiments that provide personalized feedback attract large numbers of participants with diverse disabilities and ages and allow ro­bust studies with these populations that replicate and extend the findings of prior laboratory studies. To find out what mo­tivates people with disabilities to take part, we additionally analyzed participants' feedback and forum entries that discuss LabintheWild experiments. The results show that participants use the studies to diagnose themselves, compare their abilities to others, quantify potential impairments, self-experiment, and share their own stories -- findings that we use to inform design guidelines for online experiment platforms that adequately support and engage people with disabilities.

This full-day workshop focuses on Massive Online Experiments (MOEs). MOEs have transformative potential, as they effectively allow researchers to run hundreds of experiments simultaneously (cf. Hartshorne & Germine, 2015; Reinecke & Gajos, 2014). The goal of the workshop is to help a broad cross-section of cognitive scientists begin to incorporate MOEs into their research.

Online experiments allow researchers to collect data from large, demographically diverse global populations. Unlike in-lab studies, however, online experiments often fail to inform participants about the research to which they contribute. This paper is the first to investigate barriers that prevent researchers from providing such science communication in online experiments. We found that the main obstacles preventing researchers from including such information are assumptions about participant disinterest, limited time, concerns about losing anonymity, and concerns about experimental bias. Researchers also noted the dearth of tools to help them close the information loop with their study participants. Based on these findings, we formulated design requirements and implemented Digestif, a new web-based tool that supports researchers in providing their participants with science communication pages. Our evaluation shows that Digestif's scaffolding, examples, and nudges to focus on participants make researchers more aware of their participants' curiosity about research and more likely to disclose pertinent research information.

Online experimentation with volunteers relies on participants' non-financial motivations to complete a study, such as to altruistically support science or to compare oneself to others. Researchers rely on these motivations to attract study participants and often use incentives, like performance comparisons, to encourage participation. Often, these study incentives are advertised using a slogan (e.g., "What is your thinking style?''). Research on framing effects suggests that advertisement slogans attract people with varying demographics and motivations. Could the slogan advertisements for studies risk attracting only specific users? To investigate the existence of potential sample biases, we measured how different slogan frames affected which participants self-selected into studies. We found that slogan frames impact recruitment significantly; changing the slogan frame from a 'supporting science' frame to a 'comparing oneself to others' frame lead to a 9% increase in recruitment for some studies. Additionally, slogans framed as learning more about oneself attract participants significantly more motivated by boredom compared to other slogan frames. We discuss design implications for using frames to improve recruitment and mitigate sources of sample bias in online research with volunteers.

Understanding whether and how motivation affects participation in online experiments is critical because who contributes and how they contribute can affect the validity of findings. Analyzing data from 7,674 participants across three different studies on the volunteer-based online experiment platform LabintheWild, we identified five motivation types for participating: boredom, comparison, fun, science, and self-learning. We found that these motivation types affect study selection, attention, and dropouts. Participants who were highly motivated by boredom paid less attention and were more likely to dropout than those who were motivated by the possibility of contributing to science. We additionally show that motivation can impact study results and suggest how researchers can take participants' motivation into account when designing and analyzing data from volunteer-based online experiments.

As an alternative to online labor markets, several platforms recruit unpaid online volunteers to participate in behavioral experiments that provide personalized feedback. These platforms rely on word-of-mouth sharing by previous participants for recruitment of new participants. We analyzed the impact of performance feedback provided at the end of an experiment on 81,131 participants' sharing behavior. We show that higher performing participants share significantly more. We also show that self-verification has a moderating effect: people who expected to do poorly are not affected by a high score, but people who expected to do as well as others or better, are. In a second experiment, we evaluate three distinct social comparison designs for the presentation of the results. As expected, the design that most emphasized participants' relative success led to most sharing. Contrary to our expectations, people who expected to do poorly benefited from the most optimistic social comparison more than participants who expected to do better than others.

Online experimentation with volunteers could be described as a form of citizen science in which participants take part in behavioral studies without financial compensation. However, while citizen science projects aim to improve scientific understanding, volunteer-based online experiment platforms currently provide minimal possibilities for research involvement and learning. The goal of this paper is to uncover opportunities for expanding participant involvement and learning in the research process. Analyzing comments from 8,288 volunteers who took part in four online experiments on LabintheWild, we identified six themes that reveal needs and opportunities for closer interaction between researchers and participants. Our findings demonstrate opportunities for research involvement, such as engaging participants in refining experiment implementations, and learning opportunities, such as providing participants with possibilities to learn about research aims. We translate these findings into ideas for the design of future volunteer-based online experiment platforms that are more mutually beneficial to citizen scientists and researchers.

We compared the data reliability on a subjective task from two platforms: Amazon's Mechanical Turk (MTurk) and LabintheWild. MTurk incentivizes participants with financial compensation while LabintheWild provides participants with personalized feedback. LabintheWild was found to produce higher data reliability than MTurk. Our findings suggested that online experiment platforms providing feedback in exchange for study participation could produce more reliable data in subjective preference tasks than those offering financial compensation.

We present LabintheWild, an online experiment platform that provides participants with the opportunity to learn about themselves and compare themselves to others. In the past four years, LabintheWild has attracted approximately 3.5 million participants of diverse ages and education levels who came from more than 200 countries and regions. Leveraging our experience with LabintheWild, we show how researchers can design engaging and robust online experiments that provide personalized feedback. In particular, our interactive tutorial highlights challenges and best-practice guidelines for designing volunteer-based online experiments for diverse participant samples.

If you are interested in conducting your research with larger and more diverse participant samples, or if you already have a bunch of experience that you are able to share, then this workshop is for you! The majority of research findings in the social sciences are based on studies with American undergraduate samples -- samples that have been criticized for being too Western, Educated, Industrialized, Rich, and Democratic, or “WEIRD”. As a result, some of the knowledge derived from these usually small and locally-recruited student sample populations has been found to be non-generalizable to other demographic and geographic groups. One possibility to recruit larger and more diverse participant pools than previously feasible in laboratory studies is to crowdsource online experiments on platforms such as Mechanical Turk, LabintheWild, or TestMyBrain. The main objective of this workshop is to set the agenda for conducting such studies in the social sciences to replicate, verify, and extend existing research findings and to conduct novel experiments with more diverse sample populations. Part of the workshop will include hands-on activities to design robust online experiments that are capable of attracting diverse participant samples.

Web-based experimentation with uncompensated and unsupervised samples has the potential to support the replication, verification, extension and generation of new results with larger and more diverse sample populations than previously seen. We introduce the experimental online platform LabintheWild, which provides participants with personalized feedback in exchange for participation in behavioral studies. In comparison to conventional in-lab studies, LabintheWild enables the recruitment of participants at larger scale and from more diverse demographic and geographic backgrounds. We analyze Google Analytics data, participants' comments, and tweets to discuss how participants hear about the platform, and why they might choose to participate. Analyzing three example experiments, we additionally show that these experiments replicate previous in-lab study results with comparable data quality.

For synergistic interactions between humans and artificial intelligence (AI) systems, AI outputs often need to be explainable to people. Explainable AI (XAI) systems are commonly tested in human user studies. However, whether XAI researchers consider potential cultural differences in human explanatory needs remains unexplored. We highlight psychological research that found significant differences in human explanations between many people from Western, commonly individualist countries and people from non-Western, often collectivist countries. We argue that XAI research currently overlooks these variations and that many popular XAI designs implicitly and problematically assume that Western explanatory needs are shared cross-culturally. Additionally, we systematically reviewed over 200 XAI user studies and found that most studies did not consider relevant cultural variations, sampled only Western populations, but drew conclusions about human-XAI interactions more generally. We also analyzed over 30 literature reviews of XAI studies. Most reviews did not mention cultural differences in explanatory needs or flag overly broad cross-cultural extrapolations of XAI user study results. Combined, our analyses provide evidence of a cultural bias toward Western populations in XAI research, highlighting an important knowledge gap regarding how culturally diverse users may respond to widely used XAI systems that future work can and should address.

Since the 1990s, economic research has played an increasingly important role in the practical organization and design of markets. In many of the successful cases of market design, such as labor market clearinghouses, formal procedures for student assignment to public schools, centralized systems for the allocation of courses, theoretical and experimental research have complemented each other and influenced the design of market institutions. In this chapter, I will survey a number of experimental methods used in matching market experiments, starting with laboratory experiments, where the experimenter has the most control over the environment, followed by lab-in-the-field experiments, and ending with field experiments. Before separately examining each of the three types of experiments, I briefly mention some common features of these experiments and experimental research.

With the continuously increasing number and variety of devices, the study of visual design of their Graphical User Interfaces grows in importance and scope, particularly for new devices, including smartphones, tablets, and large screens. Conducting a visual design experiment typically requires defining and building a GUI dataset with different resolutions for different devices, computing visual design measures for the various configurations, and analyzing their results. This workflow is very time- and resource-consuming, therefore limiting its reproducibility. To address this problem, we present UiLab, a cloud-based workbench that parameterizes the settings for conducting an experiment on visual design of Graphical User Interfaces, for facilitating the design of such experiments by automating some workflow stages, and for fostering their reproduction by automating their deployment. Based on requirements elicited for UiLab, we define its conceptual model to delineate the borders of services of the software architecture to support the new workflow. We exemplify it by demonstrating a system walkthrough and we assess its impact on experiment reproducibility in terms of design and development time saved with respect to a classical workflow. Finally, we discuss potential benefits brought by this workbench with respect to reproducing experiments in GUI visual design and existing shortcomings to initiate future avenues. We publicly release UiLab source code on a GitHub repository.

Because of the COVID-19 pandemic, researchers are facing unprecedented challenges that affect our ability to run in-person experiments. With mandated social distancing in a controlled laboratory environment, many researchers are searching for alternative options to conduct research, such as online experimentation. However, online experimentation comes at a cost; learning online tools for building and publishing psychophysics experiments can be complicated and time-consuming. This learning cost is unfortunate because researchers typically only need to use a small percentage of these tools’ capabilities, but they still have to deal with these systems’ complexities (e.g., complex graphical user interfaces or difficult programming languages). Furthermore, after the experiment is built, researchers often have to find an online platform compatible with the tool they used to program the experiment. To simplify and streamline the online process of programming and hosting an experiment, I have created SimplePhy. SimplePhy can save researchers’ time and energy by allowing them to create a study in just a few clicks. All researchers have to do is select among a few experiment settings and upload the stimuli. SimplePhy is able to run most psychophysical perception experiments that require mouse clicks and button presses. In addition to collecting online behavioral data, SimplePhy can also collect information regarding the estimated viewing distance between the participant and the monitor, the screen size, and the experimental trial’s timing—features not always offered in other online platforms. Overall, SimplePhy is a simple, free, open-source tool (code can be found here: https://gitlab.com/malago/simplephy) aimed to help labs conduct their experiments online.

Once a fixture of research in the social and behavioral sciences, volunteer subjects are now only rarely used in human subjects research. Yet volunteers are a potentially valuable resource, especially for research conducted online. We argue that online volunteer laboratories are able to produce high-quality data comparable to that from other online pools. The scalability of volunteer labs means that they can produce large volumes of high-quality data for multiple researchers, while imposing little or no financial burden. Using a range of original tests, we show that volunteer and paid respondents have different motivations for participating in research, but have similar descriptive compositions. Furthermore, volunteer samples are able to replicate classic and contemporary social science findings, and produce high levels of overall response quality comparable to paid subjects. Our results suggest that online volunteer labs represent a potentially significant untapped source of human subjects data.

Half of the world's population has internet access. In principle, researchers are no longer limited to subjects they can recruit into the laboratory. Any study that can be run on a computer or mobile device can be run with nearly any demographic anywhere in the world, and in large numbers. This has allowed scientists to effectively run hundreds of experiments at once. Despite their transformative power, such studies remain rare for practical reasons: the need for sophisticated software, the difficulty of recruiting so many subjects, and a lack of research paradigms that make effective use of their large amounts of data, due to such realities as that they require sophisticated software in order to run effectively. We present Pushkin: an open-source platform for designing and conducting massive experiments over the internet. Pushkin allows for a wide range of behavioral paradigms, through integration with the intuitive and flexible jsPsych experiment engine. It also addresses the basic technical challenges associated with massive, worldwide studies, including auto-scaling, extensibility, machine-assisted experimental design, multisession studies, and data security.

Classic ways of gathering data on human behaviour are time-consuming, costly and are subject to limited participant pools. Crowdsourcing offers a reduction in operating costs and access to a diverse and large participant pool; however issues arise concerning low worker pay and questions about data quality. Gamification provides a motivation to participate, but also requires the development of specialized, research-question specific games that can be costly to produce. Our solution combines gamification and crowdsourcing in a smartphone-based system that emulates the popular Freemium model of play to motivate voluntary participation through in-game rewards, using a robust framework to study multiple unrelated research questions within the same system. We deployed our game on the Android store and compared it to a gamified laboratory version and a non-gamified laboratory version, and found that players who used the in-game rewards were motivated to do experimental tasks. There was no difference between the systems for performance on a motor task; however, performance on the cognitive task was worse for the crowdsourced game. We discuss options for improving performance on tasks requiring attention.