"Insufficient confidence" with 1400+ participants, 60+ completions and 2 variants

[dingyu-tourcandy]

We started this A/B test about three weeks ago to find out which of the two form designs has the best conversion rate. So far we have 1400+ participants and 60+ completions, but split still gives us a "Insufficient confidence".

Is it normal? If not, what could be the cause?

[andrehjr]

Split calculates the ZScore based on the participant count and conversion rate. Only displays it when confidence is >= 90%

So I think the confidence level is still too low(<90%). If the conversion rate(completed count) increases a bit, it should work. For example, if the conversion rate on the second alternative hits 6.2% it should display 90% confidence (if my quick math here is not wrong)

https://github.com/splitrb/split/blob/main/lib/split/dashboard/helpers.rb#L29-L44

[dingyu-tourcandy]

@andrehjr Thanks for the explanation. But the conversion rate is still a bit far from 6.2% and probably never will be. Is there any other way to have a reasonable confidence level?