0

I work on a website that gets around 150,000 unique visitors a month.

I am proposing to sample one in 1,000 people visiting the site with a pop-up survey as described in this question and answer about calculating credible intervals for survey data. When we ran this survey in the past, giving it to all our users for a week, about 1 in 5 answered the question and 4 in 5 dismissed the pop-up without answering.

My Program Manager and my Director want to know what the justification for sampling 1 in 1,000 is (I was hoping this would minimize user annoyance by sampling at a low rate, and if the rate is low I would not need to set "supercookies" to keep track of who has taken the survey already, I'd just survey them again if they use the site so much.) They also question why, in the previous question and answer, this issue of size of the population the sample is from does not figure into the calculation of the credible interval.

(How) do sampling rate (1% sampled vs. 1 per thousand) and completion rate affect the quantification of how certain we are about our results and conclusions? Or is it just the number of samples that matter without regard to knowing the total population size?

Community
  • 1
Brian Tingle
  • 153
  • Brian, when you say "why, in the previous question and answer, this issue of sample size does not figure into the calculation of the credible interval", that is not true. The concentration of the Dirichlet distribution in that answer is determined by the sum of its parameters. You can check that this sum is equal to the sample size plus $k$. So, "automatically", when the sample size increases, you get a more concentrated posterior distribution that gives you shorter credible intervals. – Zen Oct 19 '12 at 01:22
  • Sorry, I guess I mean "total population size" or "size of the sample relative to the total population" I'll edit it. – Brian Tingle Oct 19 '12 at 01:41
  • http://www.resample.com/content/text/28-Chap-24.pdf by Resampling: The New Statistics (1997) Julian L. Simon CHAPTER 24: How Large a Sample? "Eventually we found out that, even though there are some fairly rational ways of fixing the sample size, most sample sizes in most studies are fixed simply (and irrationally) by the amount of money that is available or by the sample size that similar pieces of research have used in the past." Then ss to do cost/benefit analysis. – Brian Tingle Oct 21 '12 at 22:39

0 Answers0