Welcome to the most in-depth book on statistical methods in online controlled experiments, a.k.a. A/B testing! It is a culmination of many years of continuous work by the author Georgi Georgiev and aims to bring much needed clarity and context to the application of statistical tools for business decision-making and risk-management through experimentation. The book is still a work in progress, but you are encouraged to sign up to be notified when the book is out for sale.
"This is the book I wish I had when I started learning A/B testing stats!"
- Georgi Georgiev, the Author
Based on the content covered in chapters in the book as currently planned
Risk-management & Decision-making
The role of statistics in business decision-making. The art and science of balancing risk & reward through optimal test design.
Types of Statistical Hypotheses
Testing for superiority, non-inferiority, and strong superiority. One-sided versus two-sided hypotheses.
Statistical Models in A/B Testing
Statistical assumptions, assessing model adequacy, nominal vs. actual error rates. External validity of experiment outcomes.
Multivariate tests a.k.a. A/B/N tests
Understanding MVTs. Statistical adjustments for p-values and confidence intervals. Complications in power and sample size calculation.
P-values & Confidence Intervals
Calculation and interpretation of p-values and confidence intervals. What is and what isn't warranted by a statistically-significant outcome.
Statistical tests and decision-making in the presence of continuous monitoring. Group sequential tests with alpha and beta spending.
Statistical Power & Sample Size Calculations
Calculation and understanding of the concept of statistical power and the false negative error rate. Sample size formulas.
& Much more
Handling segmentation, multiple KPIs, percentage change as KPI, running multiple tests at the same time, communicating statistical results...
Learn how to gain business insights and control risk through statistical methods
This book will help you navigate the statistical jungle of jargon & formulas. You will understand the advantages and limitations of statistical approaches for assessing the uncertainty of data from online controlled experiments. You will learn how to design efficient statistical tests and to adapt statistical models in a number of practical scenarios, primarily from e-commerce A/B testing.
Who is this book for?
This book is best suited for professionals working in:
- Conversion Rate Optimization
- Analysis of experimental data
- Website & mobile app A/B testing
- User Experience testing
- Digital marketing testing
Subscribe Now to Get Notified
When the Book is Up for Sale
Enter your name and e-mail to receive e-mail updates and get notified once the book is published.
Georgi Z. Georgiev
Founder of Analytics-Toolkit.com
Georgi is a veteran digital marketer, data analyst, and statistician specializing in statistical methods for online controlled experiments. He is the author of the popular blog at Analytics-Toolkit.com, several technical whitepapers on statistical methods in A/B testing, a comprehensive glossary of A/B testing statistical terms, and primary contributor to the OneSided project. He has developed a number of statistical tools used by the A/B testing industry. Georgi is a lecturer at several industry events and the author of the CXL Institute course "Statistics for A/B Testing".