Comments on: How To Not Screw Up A / B Testing Your App http://www.insidefacebook.com/2008/07/11/how-to-not-screw-up-a-b-testing-your-app/ Tracking Facebook and the Facebook Platform for Developers and Marketers Mon, 13 Feb 2012 23:28:38 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: Tim Levine http://www.insidefacebook.com/2008/07/11/how-to-not-screw-up-a-b-testing-your-app/comment-page-1/#comment-10216 Tim Levine Tue, 19 Aug 2008 21:23:58 +0000 http://www.insidefacebook.com/?p=981#comment-10216 ajaypopat, Absolutely you may use it 'blindly'. Here's a page from whence the formula came. http://www.jerrydallal.com/LHSP/p.htm Look under 'Significance Tests' and then 'Comparing Two Proportions'. If you set n1=n2=n and solve for n then you'll get the above where z^2=2.7. If you're at all worried about making type I and type II errors, then double the number of impressions you run per cell and stop thinking about it. Get on with testing. Thanks, Tim ajaypopat,

Absolutely you may use it ‘blindly’. Here’s a page from whence the formula came.

http://www.jerrydallal.com/LHSP/p.htm

Look under ‘Significance Tests’ and then ‘Comparing Two Proportions’. If you set n1=n2=n and solve for n then you’ll get the above where z^2=2.7.

If you’re at all worried about making type I and type II errors, then double the number of impressions you run per cell and stop thinking about it. Get on with testing.

Thanks,

Tim

]]> By: ajaypopat http://www.insidefacebook.com/2008/07/11/how-to-not-screw-up-a-b-testing-your-app/comment-page-1/#comment-9886 ajaypopat Thu, 07 Aug 2008 09:05:49 +0000 http://www.insidefacebook.com/?p=981#comment-9886 Hi Tim, I've struggled with this A/B testing question before so this post is great for me. One question: Where'd you get that formula following "How many ad views do we need to declare a statistically significant difference between a CTR of 0.5% and one of 0.575%?". I'm talking about the 2.7*p1(1-p1) ... Can i blindly use this formula for my own experiments or I need to come up with my own? Thanks. Hi Tim,

I’ve struggled with this A/B testing question before so this post is great for me. One question: Where’d you get that formula following “How many ad views do we need to declare a statistically significant difference between a CTR of 0.5% and one of 0.575%?”. I’m talking about the 2.7*p1(1-p1) …

Can i blindly use this formula for my own experiments or I need to come up with my own?

Thanks.

]]> By: How many A:B tests do I have to run before it is meaningful? « Lightspeed Venture Partners Blog http://www.insidefacebook.com/2008/07/11/how-to-not-screw-up-a-b-testing-your-app/comment-page-1/#comment-9378 How many A:B tests do I have to run before it is meaningful? « Lightspeed Venture Partners Blog Mon, 21 Jul 2008 06:32:17 +0000 http://www.insidefacebook.com/?p=981#comment-9378 [...] by jeremyliew in A:B testing, product management. trackback Inside Facebook has a good post about how to not screw up your A:B testing that is a useful reminder about how many tests you need to run before you know that the results are [...] [...] by jeremyliew in A:B testing, product management. trackback Inside Facebook has a good post about how to not screw up your A:B testing that is a useful reminder about how many tests you need to run before you know that the results are [...]

]]>