A/B Testing

Overview

A/B testing (also called split testing) lets you test different versions of features, UI elements, or flows to determine which performs better. The MentiQ SDK provides built-in A/B testing capabilities with automatic variant assignment and conversion tracking.

Quick Start

Enable A/B Testing

Set enableABTesting: true in your analytics configuration:

import { AnalyticsProvider } from "mentiq-sdk";

function App() {
  return (
    <AnalyticsProvider
      config={{
        apiKey: "your-api-key",
        projectId: "your-project-id",
        enableABTesting: true,
      }}
    >
      <YourApp />
    </AnalyticsProvider>
  );
}

Create Experiment in Dashboard

In your MentiQ dashboard:

Create a new experiment
Define variants (e.g., “control” and “variant-a”)
Set traffic split percentages
Start the experiment

Get Variant Assignment

Use the useABTest hook to get the assigned variant:

import { useABTest } from "mentiq-sdk";

function PricingPage() {
  const { variant, isLoading } = useABTest("pricing-experiment");

  if (isLoading) return <Spinner />;

  return variant === "variant-a" ? (
    <NewPricingUI />
  ) : (
    <CurrentPricingUI />
  );
}

Track Conversions

Track when users complete the goal of your experiment:

const { trackConversion } = useABTest("pricing-experiment");

const handlePurchase = () => {
  trackConversion({
    eventName: "purchase",
    eventValue: orderTotal,
  });
};

Core Concepts

Experiments

An experiment tests different variations of a feature:

interface Experiment {
  id: string;              // Unique experiment ID
  key: string;             // Human-readable key (e.g., "checkout-flow-test")
  name: string;            // Display name
  status: "DRAFT" | "RUNNING" | "PAUSED" | "COMPLETED" | "ARCHIVED";
  trafficSplit: number;    // % of users included (0-100)
  variants: Variant[];     // Array of variants
  startDate?: string;      // When experiment started
  endDate?: string;        // When experiment ended
}

Variants

Each experiment has multiple variants (including a control):

interface Variant {
  id: string;              // Unique variant ID
  key: string;             // Variant key (e.g., "control", "variant-a")
  name: string;            // Display name
  isControl: boolean;      // Is this the baseline/control?
  trafficSplit: number;    // % of experiment traffic (must sum to 100)
}

Assignment

When a user is assigned to an experiment:

interface ExperimentAssignment {
  experimentId: string;    // Experiment they're in
  variantId: string;       // Variant ID assigned
  variantKey: string;      // Variant key (use this in your code)
  variantName: string;     // Human-readable name
  isControl: boolean;      // Is user in control group?
  assignedAt: string;      // When assignment occurred
}

Using the useABTest Hook

import { useABTest } from "mentiq-sdk";

function FeatureComponent() {
  const {
    variant,          // Current variant key (e.g., "control", "variant-a")
    isLoading,        // Is assignment loading?
    experiment,       // Full experiment object
    assignment,       // Full assignment object
    trackConversion,  // Function to track conversions
  } = useABTest("my-experiment-key");

  if (isLoading) {
    return <LoadingState />;
  }

  if (!variant) {
    // User not included in experiment
    return <DefaultExperience />;
  }

  // Render based on variant
  switch (variant) {
    case "control":
      return <ControlVersion />;
    case "variant-a":
      return <VariantA />;
    case "variant-b":
      return <VariantB />;
    default:
      return <DefaultExperience />;
  }
}

Assignment Logic

From src/ab-testing.ts:28-66, variant assignment is deterministic and sticky:

async getAssignment(
  experimentKey: string,
  options?: AssignmentOptions
): Promise<ExperimentAssignment | null> {
  const userIdentifiers = {
    userId: options?.userId || getUserId(),
    anonymousId: options?.anonymousId || getAnonymousId(),
  };

  const response = await fetch(
    `${this.apiEndpoint}/experiments/${experimentKey}/assignment`,
    {
      method: "POST",
      headers: this.headers,
      body: JSON.stringify(userIdentifiers),
    }
  );

  const assignment = await response.json();
  
  if (assignment.included === false) {
    return null; // User not included in experiment
  }
  
  return assignment;
}

How Assignment Works

User identified by userId (if logged in) or anonymousId
Hash generated from user ID + experiment key (deterministic)
Variant selected based on hash and traffic splits
Assignment cached to ensure consistency

Assignments are sticky—once a user is assigned to a variant, they stay in that variant even if you change traffic splits.

Tracking Conversions

Simple Conversion

function SignupForm() {
  const { trackConversion } = useABTest("signup-flow-test");

  const handleSignup = async (data) => {
    await createAccount(data);
    
    // Track successful signup
    trackConversion({
      eventName: "signup_completed",
    });
  };

  return <form onSubmit={handleSignup}>...</form>;
}

Conversion with Value

function CheckoutPage() {
  const { trackConversion } = useABTest("checkout-experiment");

  const handlePurchase = async (order) => {
    await processPayment(order);
    
    // Track conversion with revenue value
    trackConversion({
      eventName: "purchase_completed",
      eventValue: order.total, // Revenue in cents
      properties: {
        order_id: order.id,
        item_count: order.items.length,
      },
    });
  };

  return <Checkout onComplete={handlePurchase} />;
}

Multiple Conversion Events

Track different conversion types in the same experiment:

function ProductPage() {
  const { trackConversion } = useABTest("product-page-test");

  const handleAddToCart = () => {
    trackConversion({
      eventName: "add_to_cart",
    });
  };

  const handlePurchase = () => {
    trackConversion({
      eventName: "purchase",
      eventValue: productPrice,
    });
  };

  const handleWishlist = () => {
    trackConversion({
      eventName: "add_to_wishlist",
    });
  };

  return (
    <div>
      <button onClick={handleAddToCart}>Add to Cart</button>
      <button onClick={handleWishlist}>Save for Later</button>
    </div>
  );
}

Use Cases

Testing Button Copy

function CTAButton() {
  const { variant, trackConversion } = useABTest("cta-copy-test");

  const buttonText = {
    control: "Sign Up",
    "variant-a": "Get Started Free",
    "variant-b": "Try It Now",
  }[variant || "control"];

  const handleClick = () => {
    trackConversion({ eventName: "cta_clicked" });
  };

  return (
    <button onClick={handleClick}>
      {buttonText}
    </button>
  );
}

Testing Pricing

function PricingTable() {
  const { variant, trackConversion } = useABTest("pricing-test");

  const plans = {
    control: {
      starter: 29,
      pro: 99,
      enterprise: 299,
    },
    "variant-a": {
      starter: 19,
      pro: 79,
      enterprise: 249,
    },
  }[variant || "control"];

  const handleSelectPlan = (plan: string, price: number) => {
    trackConversion({
      eventName: "plan_selected",
      eventValue: price * 100, // Convert to cents
      properties: { plan },
    });
  };

  return (
    <div>
      {Object.entries(plans).map(([plan, price]) => (
        <PlanCard
          key={plan}
          plan={plan}
          price={price}
          onSelect={() => handleSelectPlan(plan, price)}
        />
      ))}
    </div>
  );
}

Testing Onboarding Flows

function OnboardingFlow() {
  const { variant, trackConversion } = useABTest("onboarding-test");

  if (variant === "variant-a") {
    // Shorter 3-step onboarding
    return (
      <ThreeStepOnboarding
        onComplete={() => trackConversion({ eventName: "onboarding_completed" })}
      />
    );
  }

  // Control: 5-step onboarding
  return (
    <FiveStepOnboarding
      onComplete={() => trackConversion({ eventName: "onboarding_completed" })}
    />
  );
}

Testing UI Layouts

function ProductGrid() {
  const { variant } = useABTest("grid-layout-test");

  const gridColumns = {
    control: 3,
    "variant-a": 4,
    "variant-b": 2,
  }[variant || "control"];

  return (
    <div 
      className="grid" 
      style={{ gridTemplateColumns: `repeat(${gridColumns}, 1fr)` }}
    >
      {products.map(product => (
        <ProductCard key={product.id} product={product} />
      ))}
    </div>
  );
}

Advanced Usage

Checking Specific Variant

import { useABTest } from "mentiq-sdk";

function FeatureFlag() {
  const { variant } = useABTest("new-feature-rollout");

  // Only show to users in variant-a
  const showNewFeature = variant === "variant-a";

  return showNewFeature ? <NewFeature /> : null;
}

Multiple Experiments

function ComplexPage() {
  const ctaTest = useABTest("cta-test");
  const layoutTest = useABTest("layout-test");
  const colorTest = useABTest("color-test");

  return (
    <div className={layoutTest.variant === "variant-a" ? "new-layout" : "old-layout"}>
      <Header backgroundColor={colorTest.variant === "variant-a" ? "blue" : "green"} />
      <CTA text={ctaTest.variant === "variant-a" ? "Get Started" : "Sign Up"} />
    </div>
  );
}

Programmatic Variant Check

import { useAnalytics } from "mentiq-sdk";

function MyComponent() {
  const analytics = useAnalytics();

  const checkVariant = async () => {
    const assignment = await analytics.abTesting.getAssignment("my-experiment");
    
    if (assignment?.variantKey === "variant-a") {
      // Do something
    }
  };

  return <button onClick={checkVariant}>Check</button>;
}

Best Practices

A/B Testing Tips:

Run one test at a time per page/feature to avoid interaction effects
Define success metrics before starting
Run experiments for at least 1-2 weeks to account for weekly patterns
Ensure sufficient sample size (typically 1000+ users per variant)
Don’t stop experiments early—wait for statistical significance
Document what you’re testing and why
Clean up old experiments regularly

Statistical Significance

Don’t make decisions based on small sample sizes:

function ExperimentResults({ experiment }) {
  const totalUsers = experiment.variants.reduce(
    (sum, v) => sum + v.userCount, 0
  );

  if (totalUsers < 1000) {
    return (
      <Warning>
        Need at least 1000 users before drawing conclusions.
        Current: {totalUsers}
      </Warning>
    );
  }

  return <ResultsChart experiment={experiment} />;
}

Troubleshooting

Assignment Always Returns Null

Check experiment status is RUNNING
Verify trafficSplit > 0
Ensure experiment key matches exactly
Check user is identified (has userId or anonymousId)

Variant Keeps Changing

Assignments should be sticky. If changing:

Check userId/anonymousId is consistent
Verify browser storage is enabled
Don’t use forceRefresh option unless needed

Conversions Not Tracking

Verify user has an active assignment
Check experiment is still running
Enable debug mode to see conversion events
Ensure trackConversion is called after assignment loads

Get Started

Core Concepts

React Integration

Features

Guides

Examples

Overview

Quick Start

Core Concepts

Experiments

Variants

Assignment

Using the useABTest Hook

Assignment Logic

How Assignment Works

Tracking Conversions

Simple Conversion

Conversion with Value

Multiple Conversion Events

Use Cases

Testing Button Copy

Testing Pricing

Testing Onboarding Flows

Testing UI Layouts

Advanced Usage

Checking Specific Variant

Multiple Experiments

Programmatic Variant Check

Best Practices

Statistical Significance

Troubleshooting

Assignment Always Returns Null

Variant Keeps Changing

Conversions Not Tracking

​Overview

​Quick Start

​Core Concepts

​Experiments

​Variants

​Assignment

​Using the useABTest Hook

​Assignment Logic

​How Assignment Works

​Tracking Conversions

​Simple Conversion

​Conversion with Value

​Multiple Conversion Events

​Use Cases

​Testing Button Copy

​Testing Pricing

​Testing Onboarding Flows

​Testing UI Layouts

​Advanced Usage

​Checking Specific Variant

​Multiple Experiments

​Programmatic Variant Check

​Best Practices

​Statistical Significance

​Troubleshooting

​Assignment Always Returns Null

​Variant Keeps Changing

​Conversions Not Tracking

Overview

Quick Start

Core Concepts

Experiments

Variants

Assignment

Using the useABTest Hook

Assignment Logic

How Assignment Works

Tracking Conversions

Simple Conversion

Conversion with Value

Multiple Conversion Events

Use Cases

Testing Button Copy

Testing Pricing

Testing Onboarding Flows

Testing UI Layouts

Advanced Usage

Checking Specific Variant

Multiple Experiments

Programmatic Variant Check

Best Practices

Statistical Significance

Troubleshooting

Assignment Always Returns Null

Variant Keeps Changing

Conversions Not Tracking