Statistical power analysis for sample size estimation and understanding risks in experiments with users

Diane Kelly (University of North Carolina, Chapel Hill)

CSIRO ICT IR and friends

DATE: 2014-12-01
TIME: 16:00:00 - 17:00:00
LOCATION: RSISE Seminar Room, ground floor, building 115, cnr. North and Daley Roads, ANU
CONTACT: JavaScript must be enabled to display this email address.

One critical decision that researchers must make when designing experiments with users is how many participants to study. In our field, the determination of sample size is often based on heuristics and limited by practical constraints such as time and finances. As a result, many studies are underpowered and it is common to see researchers make statements like "With more participants significance might have been detected," but what does this mean? What does it mean for a study to be underpowered? How does this effect what we are able to discover about information search behavior, how we interpret study results and how we make choices about what to study next? How does one determine an appropriate sample size? What does it even mean for a sample size to be appropriate?

In this talk, I will discuss the use of statistical power analysis for sample size estimation in experiments. Statistical power analysis does not necessarily give researchers a magic number, but rather allows researchers to understand the risks of Type I and Type II errors given an expected effect size. In discussing this topic, the issues of effect size, Type I and Type II errors and experimental design, including choice of statistical procedures, will also be addressed. I hope this talk will function as a conversation starter about issues related to sample size in experimental interactive information retrieval.
Diane Kelly is an Associate Professor at the School of Information and Library Science at the University of North Carolina at Chapel Hill. Her research and teaching interests are in interactive information search and retrieval, information search behavior, and research methods. Kelly was recently awarded the Association for Information Science and Technology (ASIST) Research Award. She is the recipient of the 2013 British Computer Societyas IRSG Karen SpArck Jones Award, the 2009 ASIST/Thomson Reuters Outstanding Information Science Teacher Award and the 2007 SILS Outstanding Teacher of the Year Award. She is the current ACM SIGIR treasurer and served as conference program committee co-chair in 2013. She serves on the editorial boards of Information Processing & Management, Information Retrieval Journal and Foundations and Trends in IR. Kelly received a Ph.D., M.L.S. and a graduate certificate in cognitive science from Rutgers University and a B.A. from the University of Alabama.

Updated:  25 November 2014 / Responsible Officer:  JavaScript must be enabled to display this email address. / Page Contact:  JavaScript must be enabled to display this email address. / Powered by: Snorkel 1.4