Skip to main content
The benchmark module provides comprehensive tools for training, evaluating, and comparing multiple AVM models. It handles the complete modeling workflow from variable selection to ensemble prediction.

​
Classes

​
BenchmarkResults

Container for benchmark results across multiple models. Attributes:
  • df_time (pd.DataFrame): Timing information for model execution
  • df_stats_test (pd.DataFrame): Statistics for the holdout test set
  • df_stats_test_post_val (pd.DataFrame): Statistics for post-valuation-date test set only
  • df_stats_full (pd.DataFrame): Statistics for the full universe
  • test_empty (bool): Whether test set contains no records
  • full_empty (bool): Whether full set contains no records
  • test_post_val_empty (bool): Whether post-valuation test set contains no records

​
__init__

BenchmarkResults(
    df_time: pd.DataFrame,
    df_stats_test: pd.DataFrame,
    df_stats_test_post_val: pd.DataFrame,
    df_stats_full: pd.DataFrame
)
​
df_time
pd.DataFrame
DataFrame containing timing data for model execution
​
df_stats_test
pd.DataFrame
DataFrame with test set statistics
​
df_stats_test_post_val
pd.DataFrame
DataFrame with test set statistics (post-valuation-date only)
​
df_stats_full
pd.DataFrame
DataFrame with full universe statistics

​
print()

Return a formatted string summarizing the benchmark results.
​
return
str
A formatted string including timings, test set stats, and universe set stats

​
MultiModelResults

Container for results from multiple models along with their benchmark. Attributes:
  • model_results (dict[str, SingleModelResults]): Dictionary mapping model names to their results
  • benchmark (BenchmarkResults): Benchmark results computed from the model results
  • df_univ_orig (pd.DataFrame): Original universe DataFrame
  • df_sales_orig (pd.DataFrame): Original sales DataFrame

​
__init__

MultiModelResults(
    model_results: dict[str, SingleModelResults],
    benchmark: BenchmarkResults,
    df_univ: pd.DataFrame,
    df_sales: pd.DataFrame
)
​
model_results
dict[str, SingleModelResults]
Dictionary of individual model results
​
benchmark
BenchmarkResults
Benchmark results
​
df_univ
pd.DataFrame
Universe DataFrame
​
df_sales
pd.DataFrame
Sales DataFrame

​
add_model()

Add a new model’s results and update the benchmark. 
add_model(model: str, results: SingleModelResults)
​
model
str
The model name
​
results
SingleModelResults
The results for the given model

​
Functions

​
try_variables

Experiment with variables to determine which are most useful for modeling. 
try_variables(
    sup: SalesUniversePair,
    settings: dict,
    verbose: bool = False,
    plot: bool = False,
    do_report: bool = False
)
​
sup
SalesUniversePair
The SalesUniversePair containing sales and universe data
​
settings
dict
Settings dictionary
​
verbose
bool
default:"False"
Whether to print verbose output
​
plot
bool
default:"False"
Whether to generate plots
​
do_report
bool
default:"False"
Whether to generate a PDF report

​
get_variable_recommendations

Determine which variables are most likely to be meaningful in a model. This function examines sales and universe data, applies feature selection via correlations, elastic net regularization, R², p-values, t-values, and VIF, and produces a set of recommended variables along with a written report. 
get_variable_recommendations(
    df_sales: pd.DataFrame,
    df_universe: pd.DataFrame,
    vacant_only: bool,
    settings: dict,
    model_group: str,
    variables_to_use: list[str] | None = None,
    tests_to_run: list[str] | None = None,
    do_cross: bool = True,
    do_report: bool = False,
    do_plots: bool = False,
    verbose: bool = False,
    t: TimingData = None
) -> dict
​
df_sales
pd.DataFrame
The sales data
​
df_universe
pd.DataFrame
The parcel universe data
​
vacant_only
bool
Whether to consider only vacant sales
​
settings
dict
The settings dictionary
​
model_group
str
The model group to consider
​
variables_to_use
list[str] | None
default:"None"
A list of variables to use for feature selection. If None, variables are pulled from modeling section
​
tests_to_run
list[str] | None
default:"None"
A list of tests to run. If None, all tests are run. Legal values are “corr”, “r2”, “p_value”, “t_value”, “enr”, and “vif”
​
do_cross
bool
default:"True"
Whether to perform cross-validation
​
do_report
bool
default:"False"
If True, generates a report of the variable selection process
​
do_plots
bool
default:"False"
If True, prints correlation plots
​
verbose
bool
default:"False"
If True, prints additional debugging information
​
t
TimingData
default:"None"
TimingData object
​
return
dict
A dictionary with keys:
  • "variables": the best variables list
  • "report": the generated report
  • "df_results": DataFrame with detailed results

​
run_models

Runs predictive models on the given SalesUniversePair. This function takes detailed instructions from the provided settings dictionary and handles all the internal details like splitting the data, training the models, and saving the results. It performs basic statistic analysis on each model, and optionally combines results into an ensemble model. 
run_models(
    sup: SalesUniversePair,
    settings: dict,
    save_params: bool = False,
    use_saved_params: bool = True,
    save_results: bool = False,
    verbose: bool = False,
    run_main: bool = True,
    run_vacant: bool = True,
    run_hedonic: bool = True,
    run_ensemble: bool = True,
    do_shaps: bool = False,
    do_plots: bool = False
)
​
sup
SalesUniversePair
Sales and universe data
​
settings
dict
The settings dictionary
​
save_params
bool
default:"False"
Whether to save model parameters
​
use_saved_params
bool
default:"True"
Whether to use saved model parameters
​
save_results
bool
default:"False"
Whether to save model results
​
verbose
bool
default:"False"
If True, prints additional information
​
run_main
bool
default:"True"
Whether to run main (non-vacant) models
​
run_vacant
bool
default:"True"
Whether to run vacant models
​
run_hedonic
bool
default:"True"
Whether to run hedonic models
​
run_ensemble
bool
default:"True"
Whether to run ensemble models
​
do_shaps
bool
default:"False"
Whether to compute SHAP values
​
do_plots
bool
default:"False"
Whether to plot scatterplots
​
return
MultiModelResults
The MultiModelResults containing all model results and benchmarks

​
run_one_model

Run a single model based on provided parameters and return its results. 
run_one_model(
    df_sales: pd.DataFrame,
    df_universe: pd.DataFrame,
    vacant_only: bool,
    model_group: str,
    model_name: str,
    model_entries: dict,
    settings: dict,
    dep_var: str,
    dep_var_test: str,
    best_variables: list[str],
    fields_cat: list[str],
    outpath: str,
    save_params: bool,
    use_saved_params: bool,
    save_results: bool,
    verbose: bool = False,
    hedonic: bool = False,
    test_keys: list[str] | None = None,
    train_keys: list[str] | None = None
) -> SingleModelResults | None
​
df_sales
pd.DataFrame
Sales DataFrame
​
df_universe
pd.DataFrame
Universe DataFrame
​
vacant_only
bool
Whether to use only vacant sales
​
model_group
str
Model group identifier
​
model_name
str
Model’s unique identifier
​
model_entries
dict
Dictionary of model configuration entries
​
settings
dict
Settings dictionary
​
dep_var
str
Dependent variable for training
​
dep_var_test
str
Dependent variable for testing
​
best_variables
list[str]
List of best variables selected
​
fields_cat
list[str]
List of categorical fields
​
outpath
str
Output path for saving results
​
save_params
bool
Whether to save parameters
​
use_saved_params
bool
Whether to use saved parameters
​
save_results
bool
Whether to save results
​
verbose
bool
default:"False"
If True, prints additional information
​
hedonic
bool
default:"False"
Whether to use hedonic pricing
​
test_keys
list[str] | None
default:"None"
Optional list of test keys (will be read from disk if not provided)
​
train_keys
list[str] | None
default:"None"
Optional list of training keys (will be read from disk if not provided)
​
return
SingleModelResults | None
SingleModelResults if successful, else None

Build docs developers (and LLMs) love

Get started for freeTalk to us