How to Turn an A/B Test into an Automated Report in Two Minutes — Step-by-Step Guide with soffi.io

Situation
Because of every experiment, an analyst opens Excel again and again, calculates the p-value, draws slides, and then rushes to send a presentation to management. Two hours of routine clicks instead of analyzing ideas for the next sprint.

soffi.io solves this pain: the service automatically pulls results from Google Sheets or Optimizely-CSV, calculates statistics, creates a brief summary, and puts the ready table straight into the boss's inbox. Below is the recipe: set up the process once and forget about boring calculations.

1. Prepare the data

What you need	Where it’s stored	How to name columns
Experiment results	Google Sheets `ab_test_results` or CSV-link from Optimizely	`variant`, `sessions`, `conversions`

Tip: if you export CSV, put the link to the file in a separate column on the "Sources" sheet — this way it’s easier to update.

2. Create a task in soffi.io

1.
Open Soffi and click "New Task" → "Google Sheets / CSV".
2.
Specify the path to the table or paste the CSV URL.
3.
Add a task description (natural language supported, but be precise):

			Compare variants A and B: 
— for each, calculate conversion = conversions / sessions; 
— compute relative lift (B vs A), 
— perform a z-test at α = 0.05, 
— report p-value, 95% CI, and verdict Accept / Reject H₀. 
If significant, suggest next steps; otherwise — Need more data. 
Create a brief summary + a table for the slide.

1.
Save the task, set the schedule to "When new rows appear" or "At experiment end" (if Optimizely sends a webhook).

3. How Soffi calculates statistics (so you know what’s inside)

1.
Conversion
$\hat{p} = \dfrac{\text{conversions}}{\text{sessions}}$
2.
Difference and lift
$\Delta = \hat{p}_B - \hat{p}_A,\qquad \text{lift} = \frac{\Delta}{\hat{p}_A}$
3.
z-test (two-sided)
$z = \dfrac{\hat{p}_B - \hat{p}_A}{\sqrt{ \hat{p}(1-\hat{p})(\frac{1}{n_A}+\frac{1}{n_B}) }}$,
where $\hat{p}$ is the combined conversion.
4.
p-value — area of the tails of the standard normal distribution.
5.
Confidence interval — Wilson method (more accurate with small samples).

These formulas are built into the engine; you don’t need to enter them in the task.

4. Set up output format

Soffi will offer a Markdown table by default, but you can choose Google Slides or PDF:

			output:
  type: slides
  theme: clean
  sections:
    - title: "A/B Test: Result"
      bullets:
        - "{{summary}}"
    - table: "{{result_table}}"

Need just text? Choose Plain → paste into Confluence or Notion.

5. Automatic recommendations

In the template above, the word {{summary}} is replaced as follows:

•
Accept H₀ → "No differences detected, continue collecting data for another N sessions."
•
Reject H₀ → "Variant B ↑ +7% conversion, z = 2.32, p = 0.02. We recommend:
1. 1.
  roll out to 100% of traffic; 2) record the change; 3) plan a test with a new hypothesis."

6. Sending to management

1.
In the "Delivery" block, select recipients: CEO, product, marketing team.
2.
Check "Copy me in Slack" — convenient for quick access to numbers during a standup.
3.
Optionally add a condition: "if p-value < 0.05 — send immediately, else — at the end of the day."

7. Verification: run a test trial

•
Click Run now.
•
In 5–10 seconds you'll receive a report by email.
•
Compare the figures with what you have in Excel — they should match up to the fourth decimal.

8. What to do next

Scenario	Action
p < 0.05, lift positive	Roll out the winner to 100%, add a hypothesis for the next step to the backlog.
p < 0.05, lift negative	Roll back, record lessons learned, collect ideas for a new variant.
p ≥ 0.05	Increase the sample, refine segments, review the effect size.

9. Save time and nerves

•
1–2 hours are freed up for the analyst on each experiment.
•
Management gets a clear verdict right after "Stop Experiment".
•
Product and marketing iterations go faster — ideas are tested almost in real time.

Conclusion

Once you set up a task in soffi.io, you no longer open a calculator or draw boring slides. The service fills in the numbers, checks statistical significance, and suggests what to do next. Meanwhile, you have time to come up with your next hypothesis, order a coffee, and persuade the team to try a new idea — while competitors are still building pivot tables.

How to Turn an A/B Test into an Automated Report in Two Minutes — Step-by-Step Guide with soffi.io

Situation
Because of every experiment, an analyst opens Excel again and again, calculates the p-value, draws slides, and then rushes to send a presentation to management. Two hours of routine clicks instead of analyzing ideas for the next sprint.

1. Prepare the data

What you need	Where it’s stored	How to name columns
Experiment results	Google Sheets `ab_test_results` or CSV-link from Optimizely	`variant`, `sessions`, `conversions`

Tip: if you export CSV, put the link to the file in a separate column on the "Sources" sheet — this way it’s easier to update.

2. Create a task in soffi.io

1.
Open Soffi and click "New Task" → "Google Sheets / CSV".
2.
Specify the path to the table or paste the CSV URL.
3.
Add a task description (natural language supported, but be precise):

			Compare variants A and B: 
— for each, calculate conversion = conversions / sessions; 
— compute relative lift (B vs A), 
— perform a z-test at α = 0.05, 
— report p-value, 95% CI, and verdict Accept / Reject H₀. 
If significant, suggest next steps; otherwise — Need more data. 
Create a brief summary + a table for the slide.

1.
Save the task, set the schedule to "When new rows appear" or "At experiment end" (if Optimizely sends a webhook).

3. How Soffi calculates statistics (so you know what’s inside)

1.
Conversion
$\hat{p} = \dfrac{\text{conversions}}{\text{sessions}}$
2.
Difference and lift
$\Delta = \hat{p}_B - \hat{p}_A,\qquad \text{lift} = \frac{\Delta}{\hat{p}_A}$
3.
z-test (two-sided)
$z = \dfrac{\hat{p}_B - \hat{p}_A}{\sqrt{ \hat{p}(1-\hat{p})(\frac{1}{n_A}+\frac{1}{n_B}) }}$,
where $\hat{p}$ is the combined conversion.
4.
p-value — area of the tails of the standard normal distribution.
5.
Confidence interval — Wilson method (more accurate with small samples).

These formulas are built into the engine; you don’t need to enter them in the task.

4. Set up output format

Soffi will offer a Markdown table by default, but you can choose Google Slides or PDF:

			output:
  type: slides
  theme: clean
  sections:
    - title: "A/B Test: Result"
      bullets:
        - "{{summary}}"
    - table: "{{result_table}}"

Need just text? Choose Plain → paste into Confluence or Notion.

5. Automatic recommendations

In the template above, the word {{summary}} is replaced as follows:

•
Accept H₀ → "No differences detected, continue collecting data for another N sessions."
•
Reject H₀ → "Variant B ↑ +7% conversion, z = 2.32, p = 0.02. We recommend:
1. 1.
  roll out to 100% of traffic; 2) record the change; 3) plan a test with a new hypothesis."

6. Sending to management

1.
In the "Delivery" block, select recipients: CEO, product, marketing team.
2.
Check "Copy me in Slack" — convenient for quick access to numbers during a standup.
3.
Optionally add a condition: "if p-value < 0.05 — send immediately, else — at the end of the day."

7. Verification: run a test trial

•
Click Run now.
•
In 5–10 seconds you'll receive a report by email.
•
Compare the figures with what you have in Excel — they should match up to the fourth decimal.

8. What to do next

Scenario	Action
p < 0.05, lift positive	Roll out the winner to 100%, add a hypothesis for the next step to the backlog.
p < 0.05, lift negative	Roll back, record lessons learned, collect ideas for a new variant.
p ≥ 0.05	Increase the sample, refine segments, review the effect size.

9. Save time and nerves

•
1–2 hours are freed up for the analyst on each experiment.
•
Management gets a clear verdict right after "Stop Experiment".
•
Product and marketing iterations go faster — ideas are tested almost in real time.

A/B test without routine: automatic report with p-value in 2 minutes

How to Turn an A/B Test into an Automated Report in Two Minutes — Step-by-Step Guide with soffi.io

1. Prepare the data

2. Create a task in soffi.io

3. How Soffi calculates statistics (so you know what’s inside)

4. Set up output format

5. Automatic recommendations

6. Sending to management

7. Verification: run a test trial

8. What to do next

9. Save time and nerves

Conclusion

A/B test without routine: automatic report with p-value in 2 minutes

How to Turn an A/B Test into an Automated Report in Two Minutes — Step-by-Step Guide with soffi.io

1. Prepare the data

2. Create a task in soffi.io

3. How Soffi calculates statistics (so you know what’s inside)

4. Set up output format

5. Automatic recommendations

6. Sending to management

7. Verification: run a test trial

8. What to do next

9. Save time and nerves

Conclusion

Stop wasting your time!