Skip to main content
Metaculus aggregates individual forecasts to create the Community Prediction (CP), which often outperforms most individual forecasters. This guide explains the different aggregation methods and when each is used.

​
Overview

Aggregation is the process of combining multiple individual forecasts into a single collective prediction. Metaculus uses sophisticated methods to weight and combine forecasts in ways that leverage the wisdom of crowds.
The community prediction is continuously updated as new forecasts are submitted or existing forecasts are updated.

​
Aggregation Methods

Metaculus supports four primary aggregation methods, defined in the AggregationMethod enum:

​
Recency Weighted

Method: recency_weighted
Default: Yes (for most questions)
Best for: Standard questions with regular updates
Each forecast is weighted based on when it was made, with more recent forecasts receiving higher weight. This ensures the community prediction reflects the most current information and understanding.Weighting formula:
  • Recent forecasts get maximum weight
  • Older forecasts decay exponentially
  • The decay rate is calibrated to balance recency with stability
Recency weighted aggregation is ideal for questions where information evolves over time, such as geopolitical events, technology developments, or market predictions.

​
Unweighted

Method: unweighted
Best for: Short-term questions, live forecasting, spot predictions
All active forecasts receive equal weight regardless of when they were made. The aggregate is a simple combination of all current predictions.Process:
  1. Collect all active forecasts
  2. Combine with equal weighting
  3. Generate aggregate distribution
Unweighted aggregation is recommended for questions that close within hours or days, or where forecasting happens in real-time during an event.

​
Single Aggregation

Method: single_aggregation
Best for: Special aggregate-only questions, benchmarking
Instead of aggregating user forecasts, this method uses a single pre-computed aggregate forecast. This is typically used for:
  • Reference forecasts
  • Baseline predictions
  • Historical aggregates
  • External model predictions

​
Metaculus Prediction

Method: metaculus_prediction
Best for: Official Metaculus forecast, enhanced aggregation
A proprietary Metaculus aggregation method that may incorporate:
  • Advanced weighting schemes
  • Historical forecaster performance
  • Calibration adjustments
  • Extremization
  • Other enhancements to standard aggregation
The exact algorithm is periodically updated as Metaculus improves its aggregation techniques.
The Metaculus Prediction method is less transparent than other methods since the exact algorithm may evolve. For transparency and reproducibility, recency weighted or unweighted methods are preferred.

​
Setting the Aggregation Method

Question authors set the default aggregation method using the default_aggregation_method field: 
default_aggregation_method = models.CharField(
    max_length=20,
    choices=AggregationMethod.choices,
    default=AggregationMethod.RECENCY_WEIGHTED,
)

​
Guidelines for Choosing

  • Question is open for weeks or months
  • Information environment is dynamic
  • You want to encourage regular updates
  • This is the default for good reason - it works well for most questions
  • Question closes within 24-72 hours
  • Forecasting happens live during an event
  • You want maximum stability
  • You’re doing spot predictions
  • You’re providing a reference forecast
  • You want to preserve a historical aggregate
  • You’re incorporating an external model
  • You want Metaculus’s best aggregate
  • Performance weighting is important
  • You trust Metaculus’s aggregation research

​
How Aggregation Works by Question Type

Different question types aggregate differently:

​
Binary Questions

Aggregates the probability values: 
CP = Aggregate(p₁, p₂, p₃, ..., pₙ)
Where each pᵢ is a forecaster’s probability estimate. 
CP = Σ(wᵢ × pᵢ) / Σ(wᵢ)
Where wᵢ is the recency weight for forecast i.

​
Multiple Choice Questions

Aggregates probability distributions across options: 
CP_option[j] = Aggregate(p₁[j], p₂[j], ..., pₙ[j])
Each option gets its own aggregated probability, and the final CP is normalized to sum to 100%.
When options change (added/removed), forecasts are automatically adjusted proportionally to maintain valid probability distributions.

​
Continuous Questions (Numeric, Date, Discrete)

Aggregates probability distributions (CDFs):
  1. Each forecast is represented as a cumulative distribution function (CDF)
  2. CDFs are combined using the chosen method
  3. The result is a smooth aggregate CDF representing the community’s collective belief
For percentiles:
  • 25th percentile: 25% of the aggregate probability is below this value
  • 50th percentile (median): The community’s central estimate
  • 75th percentile: 75% of the aggregate probability is below this value

​
Aggregate Forecast Storage

Aggregates are stored in the AggregateForecast model: 
class AggregateForecast(models.Model):
    question = models.ForeignKey(Question, ...)
    method = models.CharField(choices=AggregationMethod.choices)
    start_time = models.DateTimeField()  # When this aggregate became active
    end_time = models.DateTimeField()     # When it was superseded (null if current)
    forecast_values = ArrayField(...)     # The aggregate CDF/PMF
    forecaster_count = models.IntegerField()  # How many forecasters contributed
    # Additional statistics
    interval_lower_bounds = ArrayField(...)
    centers = ArrayField(...)
    interval_upper_bounds = ArrayField(...)
    means = ArrayField(...)
    histogram = ArrayField(...)
Aggregate forecasts are versioned over time. You can query historical aggregates by filtering on start_time and end_time.

​
Aggregate Statistics

Beyond the central aggregate, Metaculus computes additional statistics:

​
Forecaster Count

The number of unique forecasters whose predictions contribute to the current aggregate.

​
Percentile Ranges

For continuous questions:
  • 25th-75th percentile range: The middle 50% of the aggregate distribution
  • 5th-95th percentile range: The middle 90% of the aggregate distribution

​
Histogram

A 100-bin histogram showing the shape of the aggregate distribution, useful for visualization.

​
Community Prediction Visibility

Some questions hide the CP until a reveal time (cp_reveal_time): 
@property
def is_cp_hidden(self):
    return (
        not self.resolution  # always show cp when resolved
        and self.cp_reveal_time
        and self.cp_reveal_time > timezone.now()
    )
This prevents anchoring and groupthink, encouraging independent thinking.
When CP is hidden, you should form your own prediction before the reveal time. Research shows that seeing the CP first can bias your forecast.

​
Bot Forecasts

Questions can include or exclude bot forecasts from aggregates: 
include_bots_in_aggregates = models.BooleanField(default=False)
  • False (default): Only human forecasts are aggregated
  • True: Bot forecasts are included in the CP
Bot forecasts are marked with is_bot=True on the user account. Some bots are marked as is_primary_bot=True for special treatment.

​
Aggregation Performance

Research shows that aggregated forecasts typically:
  • Outperform most individual forecasters
  • Are well-calibrated when the crowd is diverse
  • Improve with more forecasters (up to a point)
  • Benefit from forecaster diversity in methods and information

​
When Aggregation Works Best

High Quality

  • Many active forecasters (50+)
  • Diverse viewpoints and methods
  • Regular updates
  • Clear resolution criteria

Conditions Matter

  • Adequate information available
  • Forecasters have domain knowledge
  • No systematic biases
  • Sufficient time horizon

​
When Aggregation Struggles

  • Very few forecasters (less than 5)
  • Highly correlated information sources
  • Groupthink or information cascades
  • Ambiguous resolution criteria
  • Black swan events with no base rates

​
Best Practices

  1. Forecast independently: Form your view before checking the CP
  2. Update regularly: Keep your forecast current, especially with recency weighting
  3. Diversify your approach: Use different methods than others might
  4. Explain your reasoning: Help others understand alternative perspectives
  1. Choose appropriate method: Match the aggregation to the question timeline
  2. Consider CP reveal time: Hide CP initially for important questions
  3. Exclude bots carefully: Include only if bots add value
  4. Monitor aggregate quality: Check forecaster count and diversity
  1. Standardize methods: Use consistent aggregation across similar questions
  2. Document choices: Explain why you chose each method
  3. Test alternatives: Compare methods on resolved questions
  4. Consider scoring: Some scoring methods work better with certain aggregations

​
API Access

You can retrieve aggregate forecasts via the API: 
# Get the latest aggregate for a question
agg = AggregateForecast.objects.filter(
    question=question,
    method=question.default_aggregation_method,
    end_time__isnull=True
).first()

# Get historical aggregates
aggs = AggregateForecast.objects.filter(
    question=question,
    method=question.default_aggregation_method
).order_by('start_time')
See the API reference for more details.

​
Next Steps

Making Predictions

Learn how to submit your own forecasts

Question Types

Understand different prediction formats

Track Record

See how aggregates are scored

API Reference

Access aggregate data programmatically

Build docs developers (and LLMs) love

Get started for freeTalk to us