Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/LauraSilRu/exoplanet-profiler/llms.txt

Use this file to discover all available pages before exploring further.

03_preprocessing.ipynb transforms the raw TESS extract into a clean, scaled, and imputed matrix ready for PCA. It implements a strict completeness filter, applies log1p transforms to skewed variables, scales with RobustScaler, and imputes missing values with KNNImputer. The notebook works exclusively with the 12 candidate features agreed in notebooks 01 and 02 — no further variable selection is performed here.

​
Input Contract

Before running this notebook, the following conditions must be satisfied:
  • File: data/raw/exoplanets.csv must exist
  • Identifier: pl_name column must be present with no nulls and no duplicates
  • Features: all 12 FEATURE_COLUMNS must be present in the file
  • Auxiliary column: disc_year must be present (used in completeness audit)
  • Domain constraints: all numeric variables must satisfy physical bounds — pl_orbeccen ∈ [0, 1]; pl_orbper, pl_orbsmax, pl_rade, pl_bmasse, pl_insol, and stellar parameters must be positive where applicable
In the validated TESS extract no domain violations were found.

​
Completeness Filter

With several variables carrying high null rates — pl_insol missing in 57.58% of rows, pl_bmasse in 31.76%, pl_orbeccen in 31.32% — a row-level completeness gate is applied before imputation. Only planets with at least 8 of 12 features observed are retained. This ensures that KNNImputer never has to reconstruct more than four values per row, which would make the imputed values unreliable.
MetricValue
Rows in raw file910
Rows retained731 (80.44%)
Rows excluded179
Complete cases (0 nulls)249
Cells imputed within retained set~9.99%
Exoplanets discovered in 2026 have a retention rate of only 33.92%, well below the overall 80.44%. These are likely recent entries that have not yet received full observational follow-up. The resulting dataset is biased toward planets with more consolidated measurements. Any time-series or discovery-year analysis should account for this imbalance.
The per-planet include/exclude decision is recorded in preprocessing_row_audit.csv so that exclusions are fully auditable.
Even after the completeness filter, pl_insol (insolation flux) remains missing in 51.71% of the retained rows. It is kept by project decision because it is an important thermal descriptor, but any interpretation of pl_insol values — especially imputed ones — should be made with caution. High imputation rates increase uncertainty in the reconstructed values for this variable.

​
Transformations

Six of the 12 features are right-skewed positive variables. log1p (i.e., log(1 + x)) is applied to each of them to compress the right tail and reduce the influence of extreme values on downstream distance calculations. 
# log1p applied to positive, right-skewed variables
log1p_features = [
    'pl_orbper',   # Orbital period
    'pl_orbsmax',  # Semi-major axis
    'pl_rade',     # Planetary radius
    'pl_bmasse',   # Planetary mass
    'pl_insol',    # Insolation flux
    'st_rad',      # Stellar radius
]

# Not transformed:
# pl_orbeccen  — bounded [0, 1]; log transform not applicable
# st_logg      — already on a logarithmic scale
# Remaining six features remain on their functional scale before scaling

​
Pipeline Order

The preprocessing steps are executed in a fixed order. Changing this order would produce different results.
1

Log1p Transforms

Applied column-by-column to the six skewed features listed above. This step must occur first, before any scaling, so that the scaler operates on the already-compressed distributions.
2

RobustScaler

Centers each feature on its median and scales by its interquartile range (IQR). Because it uses robust statistics rather than mean and standard deviation, it is less sensitive to extreme values than StandardScaler.Scaling must precede KNN imputation because KNNImputer computes Euclidean distances between rows. Without scaling, variables with larger absolute ranges would dominate the distance metric and distort neighbor selection.
3

KNNImputer (n_neighbors=5, weights='distance')

Fills the remaining missing values using the five nearest observed neighbors, weighted by inverse distance so that closer neighbors contribute more. Applied after scaling so that all 12 features contribute equally to the distance calculation used to find neighbors.

​
KNN vs Median Imputation

The choice of KNNImputer over simple median imputation was validated empirically. Using the 249 complete cases, 10% of values were randomly hidden (seed 42) and then reconstructed by each method. Error was measured in the transformed-and-scaled space.
MethodRMSEMAE
Median0.71680.5651
KNN (5 neighbors, distance weights)0.40260.2422
KNN achieved lower error on both metrics. The benchmark does not eliminate all uncertainty — pl_insol in particular has very high missingness — but it provides a reproducible, quantitative justification for the choice.

​
Output Contract

The output file data/processed/exoplanets_preprocessed.csv must satisfy all of the following conditions before notebook 04 can proceed:
  • Exactly 731 rows
  • Columns: pl_name + the 12 FEATURE_COLUMNS in the agreed order
  • Zero nulls
  • Zero infinities
  • Zero duplicate identifiers
The notebook performs these checks automatically and will raise an assertion error if any condition is violated.

​
Output Artifacts

All files are written to data/processed/.
FileDescription
exoplanets_preprocessed.csvClean 731 × 12 feature matrix — primary input for notebook 04
exoplanets_selected_raw.csvOriginal (untransformed) values for the 731 retained rows
preprocessing_row_audit.csvPer-planet include/exclude decision with reason
preprocessing_quality_summary.csvQuality metrics summary (null rates, zero counts, shape)
preprocessing_scaler_comparison.csvDescriptive comparison of StandardScaler vs RobustScaler
preprocessing_imputer_comparison.csvMedian vs KNN benchmark results (RMSE, MAE)
preprocessing_pipeline.joblibFitted sklearn pipeline (log1p → RobustScaler → KNNImputer)
preprocessing_metadata.jsonFull pipeline contract: feature list, order, parameters, row counts

​
Running the Notebook

jupyter notebook notebooks/03_preprocessing.ipynb
After running, verify the final output matches the expected contract: 
Shape: (731, 13)      # pl_name + 12 feature columns
Nulls: 0
Infinities: 0
Duplicate pl_name: 0

​
Applying to New Data

To transform a new batch of planets using the same fitted pipeline — for example, when NASA updates the TESS table — load the saved artifact rather than re-fitting. 
import joblib
import pandas as pd

pipeline = joblib.load('data/processed/preprocessing_pipeline.joblib')
new_df = pd.read_csv('new_planets.csv')

X_transformed = pipeline.transform(new_df[FEATURE_COLUMNS])
Never re-fit the pipeline on new data. Calling pipeline.fit_transform() would compute new median, IQR, and KNN neighbor statistics from the new batch, making the output incompatible with the PCA model trained in notebook 04. Always use the saved .joblib artifact to ensure consistent scaling and imputation across all data.

Build docs developers (and LLMs) love

Get started for freeTalk to us