The book

Welcome to the most in-depth book on statistical methods in online controlled experiments, a.k.a. A/B testing! It is a culmination of many years of continuous work by the author Georgi Georgiev and aims to bring much needed clarity and context to the application of statistical tools for business decision-making and risk-management through experimentation. The expected release date is late September, 2019. You are encouraged to sign up to be notified when the book is out for sale.

"This is the book I wish I had when I started learning A/B testing stats!"

- Georgi Georgiev, the Author

Topics covered

Based on the content covered in chapters in the book as currently planned
Risk-management & Decision-making

The role of statistics in business decision-making. The art and science of balancing risk & reward through optimal test design.

Types of Statistical Hypotheses

Testing for superiority, non-inferiority, and strong superiority. One-sided versus two-sided hypotheses.

Statistical Models in A/B Testing

Statistical assumptions, assessing model adequacy, nominal vs. actual error rates. External validity of experiment outcomes.

Multivariate tests a.k.a. A/B/N tests

Understanding MVTs. Statistical adjustments for p-values and confidence intervals. Complications in power and sample size calculation.

P-values & Confidence Intervals

Calculation and interpretation of p-values and confidence intervals. What is and what isn't warranted by a statistically-significant outcome.

Sequential Testing

Statistical tests and decision-making in the presence of continuous monitoring. Group sequential tests with alpha and beta spending.

Statistical Power & Sample Size Calculations

Calculation and understanding of the concept of statistical power and the false negative error rate. Sample size formulas.

& Much more

Handling segmentation, multiple KPIs, percentage change as KPI, running multiple tests at the same time, communicating statistical results...

Learn how to gain business insights and control risk through statistical methods

This book will help you navigate the statistical jungle of jargon & formulas. You will understand the advantages and limitations of statistical approaches for assessing the uncertainty of data from online controlled experiments. You will learn how to design efficient statistical tests and to adapt statistical models in a number of practical scenarios, primarily from e-commerce A/B testing.

Who is this book for?

This book is best suited for professionals working in:

  • Conversion Rate Optimization
  • Analysis of experimental data
  • Website & mobile app A/B testing
  • User Experience testing
  • Digital marketing testing

Subscribe Now to Get Notified
When the Book is Up for Sale

Enter your name and e-mail to receive e-mail updates and get notified once the book is published.

About the author

Georgi Georgiev

Georgi Z. Georgiev

Founder of Analytics-Toolkit.com

Georgi is a veteran digital marketer, data analyst, and statistician specializing in statistical methods for online controlled experiments. He is the author of the popular blog at Analytics-Toolkit.com, several technical whitepapers on statistical methods in A/B testing, a comprehensive glossary of A/B testing statistical terms, and primary contributor to the OneSided project. He has developed a number of statistical tools used by the A/B testing industry. Georgi is a lecturer at several industry events and the author of the CXL Institute course "Statistics for A/B Testing".

Sign Up for Book Updates Now

Receive an e-mail once the book is out for sale