EventFlow A/B Testing Proposal

Experimentation Framework for State Machines

Version: Draft 0.1 Date: December 2024 Status: To Be Explored

---

Overview

This proposal explores built-in A/B testing and experimentation support for EventFlow state machines.

Motivation

Product teams often need to:

• Test alternative flows (new checkout vs old checkout)
• Measure conversion rate differences between variants
• Make data-driven decisions about feature releases

Currently, this requires external tools. A native EventFlow solution could provide tighter integration with funnel analytics.

Approaches to Explore

Approach A: Dedicated Experiment Block

A structured approach with explicit experiment lifecycle:

analytics:
  experiment "New Checkout Flow"
    variants:
      control: 50%
      treatment: 50%

    assign by $user_id

    track :checkout
      segment by variant

    measure conversion to #fulfilled
      compare variants
      significance: 95%

    duration: 14 days
    minimum_sample: 1000

Pros:

• Explicit experiment lifecycle management
• Built-in statistical significance calculation
• Clear variant assignment rules

Cons:

• More verbose syntax
• Additional keywords to learn

Approach B: Segment-Based Analysis

A simpler approach using existing segmentation:

analytics:
  track :checkout
    segment by $ab_variant

  funnel "Purchase Flow"
    segment by $ab_variant

Pros:

• Uses existing segment by syntax
• No new concepts
• Flexible

Cons:

• No built-in statistical significance
• Manual experiment lifecycle

Key Questions

1. Should EventFlow handle variant assignment or rely on external systems?
2. Is statistical significance calculation in scope?
3. How does this interact with feature flags?
4. Should experiments pause automatically when significance is reached?

Related Proposals

• Analytics Proposal - Funnel metrics that experiments would measure

---

This proposal is a placeholder for future exploration. Content moved from Analytics Proposal for focused discussion.