Overview
Normalizes translation text by standardizing line endings and ensuring translation markers appear on their own lines. This is essential for reliable parsing of LLM translation responses.Function Signature
Parameters
Raw translation text that may contain inconsistent line endings or merged markers.
Returns
Normalized text with standardized line endings and properly separated translation markers.
Usage
Basic Example
Handling Line Ending Variations
Splitting Merged Markers
What It Does
The normalization process performs several operations:- Line Ending Normalization: Converts all line endings (\r\n, \r) to \n
-
Merged Marker Splitting: Inserts newlines before markers that are merged with preceding text
helloP1 - text→hello\nP1 - textword P2 - text→word\nP2 - text
-
Escaped Bracket Handling: Removes backslash escaping from
\[→[
When to Use
UsenormalizeTranslationText when:
- Processing raw LLM translation responses before parsing
- Preparing text for ID extraction or translation parsing
- Handling text from different platforms with varying line endings
- Ensuring consistent marker formatting across inputs
parseTranslations and extractTranslationIds, which expect properly normalized input.
Related Functions
normalizeTranslationTextWithMap- Returns normalized text with character position mappingextractTranslationIds- Extract IDs from normalized textparseTranslations- Parse normalized text into a Map