Overview

Output from summary(lmerMod)

Compare ANOVAs and LMER for an individual simulation

Descriptives

Plot Type

By-Item ANOVA

Aggregated by Items

By-Subjects ANOVA

Aggregated by Subject

LMER

The p-value for the main effect in LMER will be near-identical to the by-stimuli ANOVA if the random slope for the main effect is set to zero (i.e., where between-subject variation in the effect of stimulus type is 0).

Not Aggregated

Compare False Positive Rate and Power

This function will run the number of simulations with the parameters you've set and report the proportion of runs that gave a significant effect of condition (given the alpha you set). It will also report the false positive rate for the same simulations with the main effect of condition set to 0. If you set the main effect of condition to 0, then power will be equal to the false positive rate.

It is not an error that the false positive rate for the by-subjects ANOVA is very high. With this type of within-subjects, between-items design, you can get very high false positive rates if items have some variation in their mean DV (i.e., where faces tend to vary in expressiveness). For this type of design (no between-subject factors), the by-items ANOVA will have a closer-to-nominal false positive rate, but will have a more inflated false positive rate for designs with between-subject factors where subjects have some random variation in their mean repsonses.

If you set the Item Intercept SD to 0, you will see that the by-subjects ANOVA has a false positive rate closer to the nominal alpha (defaults to 0.05). However, this models a very unrealistic situation where the variation in expressiveness of faces is 0.

Simulated Effect Size Distribution

Introduction

This app is a companion to Understanding mixed effects models through data simulation by Lisa M. DeBruine and Dale J. Barr.

Preprint on PsyArXiv | Example R code | Code for this app

Set the parameters in the sidebar menu for a crossed design where raters (subjects) classify the emotional expression of faces (items) as fast as possible. Faces are either from an ingroup or an outgroup category (X_i). The hypothesis is that people will classify the emotions of ingroup faces more quickly than outgroup faces. Learn more about what these parameters mean below.

Click on Simulating LMER in the sidebar menu to view the output of the lmer summary and see how the parameters you specified affect the output. Click on Compare ANOVA & LMER to compare the results of the mixed effect model with by-subject and by-item aggregated ANOVA. Click on Power & False Positives to run a power analysis using your parameters and compare power and false positive rate between lmer and ANOVA.

RTsi = β0 + T0s + O0i + (β1 + T1s) * Xi + esi

The response time for subject s on item i (RTsi) is decomposed into:

  • an intercept β0 (population grand mean)
  • a fixed slope β1 (effect of face category)
  • a by-subject random intercept T0s
  • a by-subject random slope T1s
  • T0s and T1s are correlated rho
  • a by-item random intercept O0i
  • a trial-level residual esi
  • the numeric value of the predictor Xi (ingroup = -0.5, outgroup = +0.5)

Our data-generating process is fully determined by seven parameters: two fixed effects (intercept beta_0 and slope beta_1), four variance parameters governing the random effects (tau_0, tau_1, rho, and omega_0), and one parameter governing the trial level variance (sigma).

The parameters β0 and β1 are fixed effects: they characterize properties of the population of encounters between subjects and stimuli. The grand mean, or intercept (beta_0) is the mean RT for a typical subject encountering a typical stimulus. The main effect of category, or slope (beta_1) is how much faster RT is for ingroup than outgroup faces, on average.

Our predictor is categorical, so we need to assign a numeric value to the levels ingroup and ourgroup. In this app, we will use deviation coding where ingroup = -0.5 and outgroup = +0.5. See coding categorical predictor variables in factorial designs for further discussion.

Subjects are not identical in their response characteristics: some will be faster than average, and some slower. We can characterize the difference from the grand mean (beta_0) for each subject s in terms of a random effect T0s. In other words, we assign each subject a unique random intercept.

The actual values for T0s in our sampled dataset will depend on which subjects we happened to have sampled from their respective populations. Although the individual values will differ for each sample, we can set a fixed standard deviation (tau_0) for the population to use when we sample subjects.

Likewise, it is unrealistic to assume that it is equally easy to categorize emotional expressions across all faces in the dataset; some will be easier than others. We incorporate this assumption by including by-item random intercepts O0i. Again, although the individual values will differ for each sample, we can set a fixed standard deviation (omega_0) for the population to use when we sample items

The random slope T1s is an estimate of how much faster or slower subject s is in categorizing ingroup/outgroup faces than the population mean effect beta_1. Again, we can set a fixed standard deviation (tau_1) for the population to use when we sample subjects.

Note that we are sampling two random effects for each subject s, a random intercept T0s and a random slope T1s. It is possible for these values to be correlated, in which case we should not sample them independently. For instance, perhaps people who are faster than average overall (negative random intercept) also show a smaller than average of the ingroup/outgroup manipulation (negative random slope) due to allocating less attention to the task. We can capture this by allowing for a small correlation between the two factors, rho.

Finally, we need to characterize the trial-level noise in the study (esi) in terms of its standard deviation, sigma.

Set the number of subjects and faces per group.