Designing, Running, and Analyzing Experiments

Go to Course: https://www.coursera.org/learn/designexperiments

Introduction

**Course Review: Designing, Running, and Analyzing Experiments on Coursera** In today's fast-paced digital environment, understanding user experience (UX) is more crucial than ever. "Designing, Running, and Analyzing Experiments," a course offered on Coursera, provides invaluable training for anyone looking to explore experiments in user-centered research. This comprehensive course is designed to equip learners with practical skills in designing, implementing, and analyzing experiments effectively to assess and improve user experiences. ### Course Overview This course delivers a robust curriculum focusing on the essentials of experimental design, statistical analysis, and the methodologies relevant to user experience research. The course navigates through essential experiments within the domains of UX, Interaction Design (IxD), and Human-Computer Interaction (HCI). By working through real-world examples, students will gain experience that is both theoretically rich and practically applicable. ### Key Modules & Content Analysis 1. **Basic Experiment Design Concepts**: The course begins by grounding learners in basic concepts vital to understanding experiments, such as variance and statistical significance. This is crucial for anyone new to experimental design or in need of a refresher. 2. **Tests of Proportions**: With a focus on user preferences, students will dive into analysis using R and RStudio. This module provides learners with essential statistical tests to analyze data effectively—key for anyone conducting user-related research. 3. **The T-Test**: Understanding A/B testing is crucial in the UX field. This module robustly covers the design and analysis of such tests, enabling students to confidently report their findings using t-tests. 4. **Validity in Design and Analysis**: Ensuring the validity of data is paramount. This module teaches rigorous standards for controlling confounding variables, allowing students to enhance the integrity of their findings. 5. **One-Factor Between-Subjects and Within-Subjects Experiments**: Expanding upon experimental designs, these modules cover both between-subject and within-subject frameworks, complete with statistical tools needed for analysis. 6. **Factorial Experiment Designs**: This module dives into hybrid experiments, where students learn how to manage multiple factors, further enhancing their analytical capabilities. 7. **Generalizing the Response**: One of the standout features of the course is its inclusion of Generalized Linear Models (GLM), providing learners with insights into analyzing different types of data responses effectively. 8. **The Power of Mixed Effects Models**: Concluding with advanced modeling techniques allows students to address complex data scenarios. This is invaluable for researchers looking to maximize the use of their data. ### Learning Outcomes By the end of this course, participants will have a solid understanding of how to: - Design user-centered experiments that consider practical constraints. - Employ statistical software (R and RStudio) for data analysis. - Engage with various statistical tests and models to accurately interpret data. - Apply learned concepts to real-world experiments and user experiences in a thoughtful manner. ### Recommendation I highly recommend the "Designing, Running, and Analyzing Experiments" course on Coursera to UX researchers, interaction designers, product managers, and analytics professionals. The course structure is logical and builds progressively, making the complex subject matter accessible even to beginners while also challenging for more experienced practitioners. The hands-on nature of the projects, combined with expert insights and real-world applications, makes this course a valuable investment for anyone serious about enhancing their experimental research skills. Whether you are looking to improve your user testing capabilities or seek a deeper understanding of data analysis, this course is an exceptional resource. ### Final Thoughts In a world where user experience can make or break products, investing the time to understand and master experimental design will undoubtedly yield dividends in your career. "Designing, Running, and Analyzing Experiments" offers an unparalleled cornerstone of knowledge that will aid your journey into effective user research and beyond. Enroll today and reshape your approach to UX research with confidence.

Syllabus

Basic Experiment Design Concepts

In this module, you will learn basic concepts relevant to the design and analysis of experiments, including mean comparisons, variance, statistical significance, practical significance, sampling, inclusion and exclusion criteria, and informed consent. You’ll also learn to think of an experiment in terms of usability, its participants, apparatus, procedure, and design & analysis. This module covers lecture videos 1-2.

In this module, you will learn how to analyze user preferences (or other tallies) using tests of proportions. You will also get up and running with R and RStudio. Topics covered include independent and dependent variables, variable types, exploratory data analysis, p-values, asymptotic tests, exact tests, one-sample tests, two-sample tests, Chi-Square test, G-test, Fisher’s exact test, binomial test, multinomial test, post hoc tests, and pairwise comparisons. This module covers lecture videos 3-9.

In this module, you will learn how to design and analyze a simple website A/B test. Topics include measurement error, independent variables as factors, factor levels, between-subjects factors, within-subjects factors, dependent variables as responses, response types, balanced designs, and how to report a t-test. You will perform your first analysis of variance in the form of an independent-samples t-test. This module covers lecture videos 10-11.

In this module, you will learn about how to ensure that your data is valid through the design of experiments, and that your analyses are valid by understanding and testing for certain assumptions. Topics include how to achieve experimental control, confounds, ecological validity, the three assumptions of ANOVA, data distributions, residuals, normality, homoscedasticity, parametric versus nonparametric tests, the Shapiro-Wilk test, the Kolmogorov-Smirnov test, Levene’s test, the Brown-Forsythe test, and the Mann-Whitney U test. This module covers lecture videos 12-15.

In this module, you will learn about one-factor between-subjects experiments. The experiment examined will be a between-subjects study of task completion time with various programming tools. You will understand and analyze data from two-level factors and three-level factors using the independent-samples t-test, Mann-Whitney U test, one-way ANOVA, and Kruskal-Wallis test. You will learn how to report an F-test. You will also understand omnibus tests and how they relate to post hoc pairwise comparisons with adjustments for multiple comparisons. This module covers lecture videos 16-18.

In this module, you will learn about one-factor within-subjects experiments, also known as repeated measures designs. The experiment examined will be a within-subjects study of subjects searching for contacts in a smartphone contacts manager, including the analysis of times, errors, and effort Likert-type scale ratings. You will learn counterbalancing strategies to avoid carryover effects, including full counterbalancing, Latin Squares, and balanced Latin Squares. You will understand and analyze data from two-level factors and three-level factors using the paired-samples t-test, Wilcoxon signed-rank test, one-way repeated measures ANOVA, and Friedman test. This module covers lecture videos 19-23.

In this module, you will learn about experiments with multiple factors and factorial ANOVAs. The experiment examined will be text entry performance on different smartphone keyboards while sitting, standing, and walking. Topics include mixed factorial designs, interaction effects, factorial ANOVAs, and the Aligned Rank Transform as a nonparametric factorial ANOVA. This module covers lecture videos 24-27.

In this module, you will learn about analyses for non-normal or non-numeric responses for between-subjects experiments using Generalized Linear Models (GLM). We will revisit three previous experiments and analyze them using generalized models. Topics include a review of response distributions, nominal logistic regression, ordinal logistic regression, and Poisson regression. This module covers lecture videos 28-29.

In this module, you will learn about mixed effects models, specifically Linear Mixed Models (LMM) and Generalized Linear Mixed Models (GLMM). We will revisit our prior experiment on text entry performance on smartphones but this time, keeping every single measurement trial as part of the analysis. The full set of analyses covered in this course will also be reviewed. This module covers lecture videos 30-33.