Simply Statistics - Latest Comments

Re: When does replication reveal fraud?

Keith O'Rourke — Tue, 21 May 2013 08:31:02 -0000

Maybe I am missing something here but

1. Differences in statistical significance aren't significant (Andrew Gelman phrase)

2. From wiki entry on meta-analysis http://en.wikipedia.org/wiki/M...

"In statistics, a meta-analysis refers to methods focused on _contrasting_ and combining results from different studies, in the hope of identifying patterns among study results, _sources of disagreement_ among those results, or other interesting relationships that may come to light in the context of multiple studies".
So the question to me, sounds like asking for the sound of one hand clapping...

Re: When does replication reveal fraud?

Desmond — Mon, 20 May 2013 14:42:52 -0000

What is the answer?

Re: Sunday data/statistics link roundup (5/19/2013)

usinoz — Sun, 19 May 2013 13:47:22 -0000

Given all the positive press and books around "data science" and big data, I wonder if they asked if people trusted data science instead, would the results still be the same.

Re: When does replication reveal fraud?

Daniel Scharfstein — Sat, 18 May 2013 20:40:39 -0000

Nope.

Re: When does replication reveal fraud?

Peter Klaren — Sat, 18 May 2013 11:46:42 -0000

I'd first like to know group sizes and effect sizes of both studies before drawing some conclusion as to the big F.

Re: When does replication reveal fraud?

Steven Salzberg — Sat, 18 May 2013 07:54:58 -0000

My conclusion is that Study A appears in Science or Nature. Study B appears in ArXiv and after 3 rejections finally gets published a year later in PLoS ONE. Investigators learn once again that there's no point in trying to publish negative results.

Ergo: for every negative result that gets published, another 5-10 negative results get tossed in the bin. Thus a negative result carries more weight.

Re: When does replication reveal fraud?

Chuck Rohde — Sat, 18 May 2013 07:38:19 -0000

You first need to define what evidence is!

Re: When does replication reveal fraud?

Howard Johnson — Fri, 17 May 2013 21:51:54 -0000

I would look first at a power estimate of study A and the related estimate of the variance explained. Would that not provide some evidence of the probability of #3. Beyond that, I would suspect to find something in the methodology. There are so many possibilities to be found there. My favorite cynical reference on methodology is Paul Meehl's 1990, Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195-244

Re: When does replication reveal fraud?

Patrick Rogers — Fri, 17 May 2013 20:47:39 -0000

A thought experiment helps a bit. Assume three things: (1) the p-value estimated in study A was exactly .05, (2) the estimated value of D happened to be exactly accurate--that is, the estimate produced in study A was the exact value fo D in real life, finally, (3) that samples of D are normally distributed--that is there's no bias in how it's measured, only error. If that's the case then half of the replication tests would fail a null of D=0 at a 95% significance level.

A single replication failure is not evidence of fraud or even of Type I Error.

Re: When does replication reveal fraud?

Henrik Kristensen — Fri, 17 May 2013 20:32:09 -0000

The description doesn't mention anything about assumptions, methods, etc. It's hard to judge the end results if you don't know what they're based on. That aside, graphs of the results (e.g., showing actual distributions) might help figure out what was really going on here?

Re: When does replication reveal fraud?

Andy Mitchell — Fri, 17 May 2013 20:25:13 -0000

My initial guess is choice 3. I think it is more likely that the first study revealed a false positive. Maybe there were subtle differences in the two experiments which Jane can use to shed light on unspoken assumptions by Joe.

Re: When does replication reveal fraud?

Ken — Fri, 17 May 2013 17:23:17 -0000

Type 2 error is only conditional on a given effect size and sample size so you don't know this. The actual power may be not much above 0.05 to find any reasonable difference.

Given the way most science works, I would expect that study A was a type I error. What I would want to know is the power for study B given an effect size that is consistent with study A, and then could determine if study B was likely to be type II error. Would need a lot more evidence to accuse anyone of fraud.

Re: When does replication reveal fraud?

Roger Peng — Fri, 17 May 2013 13:37:37 -0000

Dan, I'm not in charge of that anymore, remember?

Re: When does replication reveal fraud?

Dan Scharfstein — Fri, 17 May 2013 11:26:57 -0000

why don't you put this on the qualifying exam?

Re: When does replication reveal fraud?

Matthew — Fri, 17 May 2013 11:14:46 -0000

I agree with Eric. We conventionally allow a smaller type 1 error rate than type 2 error rate, so my first instinct would be that study B committed a type 2 error.

Re: When does replication reveal fraud?

Eric Yount — Fri, 17 May 2013 09:52:22 -0000

I'd said _most_ likely, given a standard alpha=.05 and beta=.2, would be 4. I'd immediately rule out fraud, unless there was more evidence than given above. That would give #3 a 1/20 chance (alpha) of being the case, if D is not significantly different from 0 or #4 a 1/5 chance (beta) of being the case, if D is different from 0.

Re: Fitbit, why can't I have my data?

Gillian Morris-Talbot — Thu, 16 May 2013 17:03:13 -0000

Hexoskin (www.hexoskin.com) is a more open self-monitoring device. The guys behind it designed it specifically to be open data. You can access your body metrics as you work out, via our free iPhone app (https://itunes.apple.com/ca/ap..., then get all the detail you want via our web portal (http://www.hexoskin.com/en/hex..., which allows you to drill down with our tools, but also to export the raw data to manipulate the way you want.

Re: Sunday data/statistics link roundup (5/12/2013, Mother's Day!)

Karl Broman — Mon, 13 May 2013 11:12:11 -0000

I'm overly sensitive. ;)

I read the subtext to be, “Karl, why have you still not tried healthvis?”

Re: Sunday data/statistics link roundup (5/12/2013, Mother's Day!)

jtleek — Mon, 13 May 2013 10:53:47 -0000

I hope people know I was kidding. Nacho is awesome and Clickme is great. I was just trying to be clever, maybe it fell flat. But Prasad - the student running healthvis - is pretty great too!

Re: Statistics project ideas for students (part 2)

Carl — Mon, 13 May 2013 10:53:42 -0000

Thank you. James.

Re: Sunday data/statistics link roundup (5/12/2013, Mother's Day!)

Karl Broman — Mon, 13 May 2013 09:39:46 -0000

Added.

(Nacho is a bioinformatics graduate student, while you healthvis guys have considerable clout.)

Re: I wish economists made better plots

Chris Stehlik — Mon, 13 May 2013 02:25:35 -0000

As you mentioned, what stands out is not the Excel error (though that's bad) but the amount of variability. A good number of countries at the 90% level had economic growth over even the corrected 2.2% amount (much less the -0.9%). The 90% debt is not a death knell it was being made out to be.

(Got to your blog from Prof Leek's earlier Coursera course.)

Re: A Shiny web app to find out how much medical procedures cost in your state.

Vivek Patil — Fri, 10 May 2013 09:05:06 -0000

Jeff, Thanks for that data analysis class.I was one of over 100k students there. I developed another shiny app with this data, but focusing specifically on my city. I've discussed that along with some other analysis of this data at: http://analyticsandvisualizati...

Re: A Shiny web app to find out how much medical procedures cost in your state.

Jean — Fri, 10 May 2013 09:02:56 -0000

Working for me now, too. Excellent.

Re: A Shiny web app to find out how much medical procedures cost in your state.

paulsef11 — Fri, 10 May 2013 02:13:49 -0000

Would love to see this crossed with how the hospitals rank in terms of procedure success rates. Who has the best rate for the lowest cost?