Hi, we've been running a few experiments using Split and noticed nearly all of our variants were losing over the control version by pretty hefty margins. So we set up a test to see how Split performed when splitting an 'experiment' where the control and the variant were the exact same experience. This test found the variant losing to the control by nearly 20% with a 95% confidence interval. Any suggestions as to what might be causing this or how we can improve the accuracy of our results? This was using the default splitting algorithm. Would switching to the block randomization algorithm help us out at all?
Hi, we've been running a few experiments using Split and noticed nearly all of our variants were losing over the control version by pretty hefty margins. So we set up a test to see how Split performed when splitting an 'experiment' where the control and the variant were the exact same experience. This test found the variant losing to the control by nearly 20% with a 95% confidence interval. Any suggestions as to what might be causing this or how we can improve the accuracy of our results? This was using the default splitting algorithm. Would switching to the block randomization algorithm help us out at all?