SIL TECkit is a text encoding conversion toolkit used to move legacy byte-encoded language material into Unicode. If you need a TECkit tutorial, a legacy font encoding conversion plan, a way to compile a TECkit mapping, or guidance on Unicode migration quality assurance, the first decision is not which command to run. It is determining exactly what character each original byte or sequence was supposed to represent. Without that evidence, an apparently successful conversion can rewrite an archive into plausible but incorrect text.
TECkit’s conversion engine operates with compiled mapping tables built from human-readable mapping descriptions. It is particularly useful when a linguistic archive, dictionary project or document collection relies on an old local encoding that ordinary modern software does not understand. Source bytes, Unicode code points and display fonts are separate layers. The conversion must handle all relevant input, including spaces, line endings, punctuation and unusual sequences, and should be validated by people who understand the language as well as by technical tests.
This world-facing guide explains how to identify legacy encodings, design and compile maps, process sample data, detect replacement characters, test round trips, protect structural fields and preserve source provenance. It includes practical migration situations, a review checklist and troubleshooting advice for translation teams and community archives. The official SIL TECkit reference should be used for exact mapping syntax and tool flags because installed packages and platform binaries vary. The goal is trustworthy Unicode text that remains searchable, editable and understandable.
- Preserve unmodified source files and record the probable legacy encoding.
- Distinguish byte values, Unicode characters, font glyphs and language meaning.
- Compile documented mapping descriptions and validate all input classes.
- Treat U+FFFD replacement characters and damaged whitespace as stop signals.
- Test normalization, reverse conversion and database import separately.
- Have speakers approve sample words, names and uncommon orthographic sequences.
- Keep golden-corpus results, source provenance and reproducible build evidence.
A conversion is only correct when the receiving text is trustworthy
A useful conversion report answers four questions: which original data was processed, what mapping was applied, what content could not be resolved, and whether the target system preserved language and document structure. A pleasing screenshot answers none of these completely. Treat conversion as a carefully controlled transfer of meaning-bearing characters and evidence, especially when source material may be difficult or impossible to recollect.
1. Recognize a legacy encoding before editing its text
A document that displays letters correctly is not necessarily Unicode. Older keyboards and fonts often reused ordinary byte values for unrelated script characters. A file may look legible only while a particular font is selected. Copying it into a modern website can reveal Latin punctuation, boxes or unrelated letters.
Collect sample originals and identify the application, keyboard and font that produced each file. Ask whether a national character encoding standard, a custom font encoding or a mixed scheme was used. Record observations before opening files in an editor that may silently reinterpret the bytes.
Preserve the original binary data untouched. Make working copies, record file hashes where suitable and label the assumed encoding as a hypothesis until sample words and a byte inventory support it. A conversion plan based only on visual resemblance is weak evidence.
Capture a source-file fingerprint, software origin, visible word sample and probable font before doing any conversion. Ask an experienced reader to identify an ordinary phrase in the original application. This becomes the reference for confirming that the new Unicode result preserved meaning rather than merely changing its appearance.
2. Know where TECkit belongs in the workflow
SIL TECkit is a low-level conversion toolkit for applications converting text, particularly from legacy encodings into Unicode. Its engine uses compiled mapping tables created from human-readable mapping descriptions. It is a transformation component, not a language translator or a general text-repair oracle.
A typical migration separates discovery, map authorship, compilation, sample conversion, linguistic review and controlled import into the new system. The project may use TECkit directly or through another application that supports its mapping format. Establish who owns each stage before converting thousands of records.
Check the current project documentation and installed distribution for exact tool names and options. TECkit distributions include command-line utilities such as teckit_compile and txtconv, but a command that exists on one platform may not be installed in the same location on another.
Write a one-page migration diagram with original file, mapping source, compiled converter, Unicode output and receiving application. Assign an owner to each step. Separate the encoding change from translation, spelling review and publication approval, since one person’s successful conversion does not validate all the others.
3. Distinguish bytes, Unicode characters and fonts
In a byte encoding, the numerical byte value represents a character according to the local scheme. In Unicode, the text stores assigned code points. A font then selects glyph shapes for displaying those code points. Conflating the three makes a migration fail even if the converted document is superficially readable.
Build a small table with source byte sequence, intended character, Unicode code point and example word. For ligatures or multi-character sequences, include the entire mapping rather than inventing a single code point. Note any punctuation and whitespace values that the legacy scheme reassigned.
Validate the table against actual written words and qualified speakers. A source encoding may also use contextual behavior, especially if a custom keyboard replaced characters as people typed. One source byte can be insufficient to infer meaning without neighboring bytes.
For five representative source characters, record the raw bytes, font-dependent old glyph, proposed Unicode code point and word context. If the same byte seems to represent two letters, investigate source versions or contextual rules. Do not merge contradictory evidence into one mapping entry.
4. Compile maps from readable descriptions
TECkit uses mapping descriptions that can be compiled into binary tables for the conversion engine. The readable file is the document reviewers should understand, because the binary result does not explain why one code point was chosen over another.
Store every mapping description in version control with explanatory comments, sample inputs and the target Unicode inventory. Compile only after checking the intended source and destination code spaces. Use the version-matched TECkit language documentation to express multi-pass behavior correctly.
Include compiler diagnostics in the build log. A successful binary build is necessary but not enough; it does not prove that the map respects the community orthography or that every old document used the same source encoding.
Rebuild the TECkit mapping from its readable source on a clean machine and preserve compiler diagnostics. Document the installed toolkit version and the test file used. If the compiled map cannot be reproduced without an undocumented local folder, fix project packaging before a mass conversion.
5. Map whitespace and control codes explicitly
One unusually important TECkit warning is that byte-to-Unicode maps generally replace unmapped input with a default character, often U+FFFD. If the author maps only visible letters, ordinary spaces, tabs and line endings may also be damaged. The source project’s FAQ calls this out because it is easily overlooked.
Write tests containing spaces, tabs, carriage returns, line feeds, punctuation, digits and all frequent letters. Make a deliberate decision for each byte or sequence that may appear in the source. Never assume that a byte-to-Unicode mapping automatically preserves familiar ASCII values.
After a trial conversion, compare paragraph boundaries, tabular alignment and counts of replacement characters. A document with beautiful script words but destroyed field separators is not a successful migration. Check round-trip results in the target application too.
Convert a sample with spaces, tabs, CR, LF, punctuation and digits, then count U+FFFD characters and compare line boundaries. A table whose letters look accurate but whose separators are replaced can destroy a dictionary export. Reject that map until structural codes are accounted for.
6. Account for multiple conversion passes
Some mappings require intermediate stages because the source contains contextual sequences or mixed structural conventions. TECkit documentation explains code-space compatibility: output from one pass must match the input space expected by the next. A mismatch can trigger a compiler error.
Use as few passes as necessary. Describe what each stage does: byte normalization, sequence recognition, Unicode output or a targeted text transformation. If a pass changes byte values before Unicode interpretation, make that explicit in the technical record.
Test each stage separately if tools and workflow allow it. When a mapping reports a code-space mismatch, inspect pass boundaries rather than randomly editing individual character mappings. Complexity should solve a documented source behavior, not conceal a missing inventory.
Draw a flow showing each pass’s input and output code spaces. When a compiler reports a mismatch, inspect the boundary, not random character entries. Test a minimal sequence for each stage before adding another transformation; unnecessary passes make future correction and archival explanation harder.
7. Avoid a deceptive one-to-one character chart
A handwritten alphabet chart is a starting point, not a complete encoding specification. Some source sequences represent one character; some represent multiple characters; others change meaning according to the preceding byte. Punctuation may be overloaded as a control for a glyph substitution.
Build representative phrase-level tests containing frequent letter combinations, capitalization, numbers, mark sequences and borrowed words. Check how a legacy keyboard created the observed output. If the font draws a glyph for two distinct source byte values, decide whether the distinction is semantic or accidental.
Have language reviewers examine the converted text without the legacy font. Record uncertain characters as unresolved issues rather than choosing the nearest-looking Unicode glyph. A mistaken mapping can damage thousands of dictionary entries at once.
Select challenging words containing legacy glyph combinations rather than relying on one alphabet chart. Compare the proposed mapping with actual typed files and ask speakers to approve distinctions that font appearance conceals. Mark rare or disputed forms as unresolved instead of guessing their intended Unicode identity.
8. Use a character inventory before batch conversion
Begin by counting distinct byte values and recurrent sequences in a sample that includes different document ages and authors. The inventory reveals whether the putative map covers all data, and whether multiple encodings may be mixed in one collection.
Measure how many instances would be mapped, replaced or left ambiguous under the planned table. Review the most common unmapped values first; they may be field separators or very frequent marks. Inspect small samples around every unusual pattern.
Never let frequency alone decide a character’s identity. Names and uncommon grammatical forms may be rare but vital. Provide a way to flag rare patterns for linguistic review rather than discarding them as noise.
Run a byte-frequency inventory on samples from several authors and dates. Investigate unusually frequent unmapped codes, then inspect rare codes in names and grammatical examples. Report coverage by files and record types, because a map that handles most prose may still corrupt crucial dictionary keys.
9. Treat Unicode normalization as a separate decision
After byte conversion, some scripts have canonically equivalent representations, especially when combining marks are involved. Normalization can help interoperability, but it must not be confused with correcting a spelling, transliterating a script or merging distinct words.
Select a normalization policy appropriate to the language and receiving systems. Preserve a copy of the mapped output before further normalization. In test records, compare code points before and after each operation so reviewers can trace exactly which stage changed a sequence.
Review multiple marks, decomposed sequences and rare forms. Search and collation tools may behave differently with equivalent strings, so test the actual database and publishing system rather than assuming a single normalization step fixes everything.
Save both the raw TECkit-mapped output and any subsequently normalized text. Inspect combining-mark sequences and run search tests in the destination system. If characters remain canonically equivalent but a database behaves differently, fix that comparison layer without changing the documented original mapping.
10. Understand why reverse conversion may lose information
A map may convert multiple legacy sequences to the same Unicode character. That means a perfect reversal is impossible without extra information. Similarly, the target Unicode text may contain characters absent from the old encoding. A reverse map can be useful but must not be advertised as lossless without proof.
Create explicit reverse tests for every mapped character and for complete words. Count collisions and document characters with no representation in the old code space. Make a policy for unrepresentable characters rather than silently omitting them.
Compare reconstructed bytes with the archived original, not merely the visual appearance. If the converter cannot preserve a distinction, explain the limitation before teams use the output for archival or legal material.
Identify every source sequence whose Unicode destination is shared with another source. Test inverse conversion and produce a collision list. If byte-exact recovery fails, state that plainly and preserve original files; visual similarity on a reconstructed page cannot prove archival reversibility.
11. Build a golden corpus with linguists
A golden corpus is a small but carefully checked set of source files and approved Unicode results. It should include ordinary text, challenging orthographic patterns, punctuation, whitespace, names, numbers and any layout structure the importer cares about.
Ask language experts to review final Unicode words and sentences, while engineers verify exact code points and field integrity. Keep the corpus small enough for manual understanding but varied enough to represent the data collection. Record the approval date and responsible reviewers.
Run the corpus after every mapping revision. If a change improves one passage and damages another, the regression should be visible immediately. This is more reliable than inspecting a few random pages after each batch.
Build a golden corpus containing real approved words, names, numbers, boundaries and difficult mark sequences. Keep its expected Unicode output under review by language experts. Every mapping revision should rerun that corpus and explain changes, rather than silently replacing previously approved dictionary entries.
12. Translate text only after decoding it correctly
Conversion and translation are separate tasks. An incorrect character mapping may turn a valid source word into nonsense, causing a human translator or a machine system to draw the wrong conclusion. Translate after the source text has been decoded and validated.
Set a quality gate: no large translation memory import or terminology extraction until encoding issues have been reviewed. Keep uncertain legacy passages marked for specialist checking. Retain a link to original files so translators can recheck suspicious words.
If a reviewer sees a strange phrase, distinguish an original typo, a conversion defect and an unfamiliar dialect form. Fixing all three through a single automated find-and-replace operation destroys useful evidence.
Before importing converted material into a translation memory, ask reviewers to inspect examples known to be sensitive to character mistakes. Maintain categories for original typo, mapping error and legitimate variant. This distinction keeps automated spell checks from rewriting historical or dialectal language evidence.
13. Plan data import without destroying structure
Dictionary files, spreadsheets, database exports and archived newsletters may contain delimiters, quoted fields and line endings. A mapping that changes these characters can shift columns, merge records or break XML and CSV syntax even when the human-readable words appear improved.
Identify which bytes are payload text and which are structural syntax. Convert only fields or segments that are truly in the legacy encoding where possible. Validate record counts, keys, escaping and imported values on a staging copy before production migration.
Compare output against expected row counts and foreign-key relationships. Never overwrite the original database. A repeatable import with a rollback route is more valuable than one unrecorded successful demonstration.
Verify the CSV or XML structure after conversion by counting records, keys and field delimiters. When possible, convert only fields known to contain legacy text. Preserve original database dumps and rehearse rollback; a Unicode migration that shifts definitions to wrong headwords is worse than a delayed import.
14. Document provenance and permissions
Historical documents may contain unpublished community stories, personal names, sacred material or restricted lexicons. A conversion changes format, not the underlying right to disclose material. The conversion team needs authority to access and publish what it processes.
Keep metadata for source owner, date, probable encoding, map version, operator and reviewer decisions. Separate public examples from restricted archives. When working with personal information, prefer synthetic examples for technical documentation.
Record whether a map is general-purpose, community-specific or derived from proprietary software. Permissions and licensing can affect whether the map itself, source fonts or documents may be redistributed.
Create public test data from consented or synthetic words and preserve any restrictions on source documents. Record whether the mapping itself may be redistributed. Technical access to a manuscript or community story does not automatically authorize publication of its contents in a software example.
15. Assess search and collation after migration
Unicode conversion does not automatically deliver correct sorting or search for every language. Different platforms may use different collation rules, normalization choices and treatment of combining marks. A converted dictionary may therefore contain accurate characters but unusable lookup behavior.
Prepare a list of culturally and linguistically meaningful sorting pairs and expected search results. Test with real database collation and indexing settings. For mixed-language publications, include entries with diacritics and variant spellings.
Do not edit the mapping merely to force a bad sorting algorithm to behave. Preserve correct Unicode text and fix collation at the appropriate layer; this keeps the data portable to future tools.
Prepare language-specific collation and lookup pairs, including accents, borrowed words and variants. Test these in the final dictionary, not only in the converter output. Correct Unicode spelling should remain stable even if the application’s sorting or search engine requires separate configuration.
16. Make migration results reproducible
Migration often needs to be rerun when new files arrive or a map defect is discovered. Without a stable map, input inventory and conversion log, different batches quietly diverge. Readers then see apparently identical words encoded differently.
Store source hashes, map files, compiler version, command settings, timestamps and output identifiers. Use a staging environment and compare golden-corpus results before processing a new batch. Report replacements, unmapped values and exceptions in a concise summary.
Ask another operator to rebuild the converter and reproduce a known result. A migration is not complete until the receiving team understands how to maintain new Unicode content and how to recognize leftover legacy data.
Store original file hashes, mapping revision, converter binary, tool version, imported record counts and unresolved cases in a migration manifest. Give that manifest to a new operator and ask them to reproduce a sample conversion. A handoff that cannot explain what was changed is incomplete.
Twelve realistic legacy-text migration cases
Each scenario illustrates a different failure class. The response must follow evidence in the real corpus; no generic mapping table can prove the identity of characters from an unknown old system. Written samples and language review remain central.
Case 1: An old dictionary uses a decorative font
The dictionary looks correct in its original word processor, but pasted headwords become Latin symbols. The archive has three copies made years apart.
Compare byte inventories and sample words, identify the exact custom font encoding and document each version. Build a conversion map on a copy, not on the only archive.
Approve sample headwords, definitions and example sentences with fluent speakers. Check search results and record counts after import.
Retain screenshots from the original editor and a byte-based sample of several headwords. Confirm map choices with speakers before importing. An old font can make the original appear correct while leaving every copied word technically incompatible with Unicode workflows.
Case 2: Spaces disappear in converted paragraphs
A test maps every visibly unusual glyph but omits ordinary byte values. Converted paragraphs contain replacement characters where spaces and tabs should be.
Audit complete byte coverage and the default behavior of byte-to-Unicode mapping. Add intentional handling for structural and whitespace codes using the version-specific mapping language.
Compare before-and-after paragraph breaks, tabs and U+FFFD counts. Do not release merely because alphabetic words can be recognized.
Count replacement characters before release and compare normal spacing, tabs and line endings against the preserved source. A conversion with thousands of U+FFFD characters deserves investigation even when the most visible alphabetic words are legible.
Case 3: Two documents use different keyboard generations
A field worker changed keyboard software halfway through a project. The same byte represents different marks in early and late files.
Segment the corpus by provenance or detectable patterns. Use separate maps if evidence demands it and record the source hypothesis for each batch.
Test ambiguous examples with a language expert. Do not force all historical material through one mapping simply for convenience.
Separate the documents by source date and keyboard version when possible. Test ambiguous byte patterns in each group and retain the original classification evidence. If a document’s encoding cannot be identified confidently, hold it for manual review instead of guessing.
Case 4: A reverse map cannot restore original bytes
The target Unicode representation merges two older spellings that were entered through different legacy byte sequences.
Calculate mapping collisions and decide whether reverse conversion is necessary. Preserve originals alongside modern text rather than presenting reconstruction as archival restoration.
A comparison report should list cases where output has no unique reverse source, including any losses of punctuation distinctions.
Present the collision list to the archival custodian before approving reverse export. If several old byte sequences map to one modern character, the converter cannot recover the original without additional metadata. Preserve the original files as the authoritative historical witnesses.
Case 5: A CSV import shifts columns
After conversion, the new dictionary has fewer rows and many definitions appear beside the wrong headwords.
Determine whether commas, tabs, quotation marks or newlines were affected. Separate transport syntax from payload conversion and rerun on a staging export.
Compare record counts, field lengths, unique IDs and representative definitions. Reject an import with unexplained structural differences.
Compare CSV or database row counts, unique identifiers and field boundaries. Test a reversible staging import and reject any batch with unexplained record movement. Language reviewers should then check that a definition still belongs to its intended headword.
Case 6: A combining mark is visually plausible but wrong
Converted words look acceptable in one font, yet searching for a known spelling does not find them. A code-point inspection reveals the wrong combining sign.
Review the intended script and Unicode sequence with language experts. Correct the mapping, not the display font, then rerun the approved sample corpus.
Test searching, sorting and copy-paste in the final platform. Keep the original byte sample in the defect record.
Capture the old byte sequence, selected Unicode combining mark and the final target rendering. Confirm the mark’s identity through official character data and language expertise. A visually plausible near-match must not be accepted as a successful conversion.
Case 7: A mixed Unicode and legacy file appears
One document contains genuine Unicode text pasted from email beside legacy-encoded passages typed years earlier.
Identify boundaries before mapping. A blind whole-file conversion can corrupt already-correct Unicode text; use field-level or segment-level processing with documented assumptions.
Inspect a sample at every suspected boundary and retain the untouched source for later correction. Declare uncertain sections unresolved.
Use segment-level diagnosis to protect already-correct Unicode material. Label boundaries and retain uncertain spans for later review. A blind transformation of an entire mixed file can make a good modern section as unusable as the legacy section.
Case 8: An archive contains restricted narratives
A research team can technically convert traditional stories but does not have permission to place them in a public demonstration dataset.
Separate processing rights from publication rights. Use synthetic or approved test strings for maps; secure originals and record access rules.
Review release artifacts and logs for unintended excerpts. Technical QA should not become an accidental disclosure channel.
Keep restricted materials outside public regression repositories and redact source examples in support tickets. Build tests from approved substitutes that still exercise the byte patterns. The ability to run a converter should never override archival consent conditions.
Case 9: A spelling checker flags thousands of words
The converted corpus contains unexpected forms and the team proposes mass correction from a standardized word list.
First determine whether the cause is source variation, conversion errors or legitimate dialect forms. Run precise diagnostics rather than broad replacement.
Have qualified reviewers label a sample across the three categories. Keep any language-standardization project separate from encoding conversion.
Ask reviewers to categorize a sample of flagged words before any bulk correction. Distinguish unfamiliar morphology, historical spelling and corrupted mapping. A spelling-checker disagreement is evidence for review, not permission to silently standardize the corpus.
Case 10: A new system silently normalizes input
A database import succeeds but an export produces different code point sequences than the converter originally emitted.
Track normalization at every stage, including conversion, import, editing and export. Compare exact sequences and define acceptable canonical equivalence.
Check that meaningful distinctions survive, searches work and round-trip behavior is recorded. Do not claim byte-identical reversibility when only visual equivalence remains.
Compare pre-normalization and post-import code points and record any canonical changes. A useful output can be search-equivalent without being byte-identical. Avoid promising reversible conversion where intermediate applications change strings.
Case 11: A migration must be audited later
Months after a large conversion, a reviewer challenges one mapping choice. The original operator has left and no build notes exist.
Use archived map versions, sample corpora and source hashes to reconstruct the specific decision. If evidence is missing, mark the result as unverified rather than inventing it.
Preserve a correction log showing which outputs are affected by each map revision. A change is a data migration, not merely a software patch.
Keep every mapping revision linked to its affected document batches. When a reviewer reports a suspect character, the team should identify exactly which outputs need correction and rerun only the relevant migration, retaining previous results for audit.
Case 12: A school moves from a legacy keyboard
Teachers need to access old handouts while introducing a new Unicode keyboard for future material.
Convert approved handout samples with TECkit, provide modern keyboard training separately and explain that conversion cannot automatically settle spelling disputes.
Test old-to-new document compatibility, searching and printing. Give staff a path to report unusual characters without manually substituting lookalikes.
Give teachers both converted example handouts and instructions for a modern Unicode keyboard. Test classroom copying and printing across devices. Explain that the new keyboard helps future input while TECkit serves a separate task of interpreting the old files.
A pre-import quality gate for TECkit conversions
- Original data preserved and encoding hypothesis documented.
- Readable map source and compiled output tracked together.
- Whitespace, controls, punctuation and ordinary text tested.
- Replacement characters counted and unresolved values investigated.
- Language experts reviewed rare words, names and mark combinations.
- Round-trip limitations and normalization behavior recorded.
- CSV, XML or database structure validated after transformation.
- Privacy and publication rights checked for source material.
- Recovery, rollback and repeatable migration instructions available.
Passing this gate should require evidence rather than optimism. Keep unresolved bytes, unknown script variants and questionable source segments visible to readers of the audit. A converter has one job: implement a mapping faithfully. Choosing a faithful mapping and interpreting historical language data require human expertise, especially when the old input scheme was never formally documented.
Frequently asked questions
What is TECkit used for?
It converts text using defined mapping tables, especially where an existing legacy encoding must be imported into a Unicode workflow. It does not translate meanings between natural languages.
Write the source encoding and intended Unicode destination on the project brief. A TECkit mapping establishes character correspondence; a translator still needs the content in a correct, human-readable language form before interpreting meaning.
Does TECkit repair any unreadable document automatically?
No. The team must identify the source encoding and build or obtain an appropriate mapping. Unidentified bytes, multiple encodings and damaged files demand investigation.
Collect the old application, font information and several verified words before trying tools. Without an encoding hypothesis, identical-looking symbols could correspond to different byte sequences. Escalate uncertain material to a specialist.
Why does U+FFFD appear?
It is the Unicode replacement character, commonly used when an input cannot be mapped or decoded. Count and inspect occurrences rather than deleting them to make the output look clean.
Inspect the source byte at each U+FFFD position and compare it with the mapping table. Fix missing coverage where the intended character is known. Do not erase replacement symbols indiscriminately, because they indicate unresolved information loss.
Do spaces and line endings need mapping?
In byte-to-Unicode mappings, yes: TECkit explicitly warns that unmapped spaces, tabs and line endings can turn into replacement characters. Test them even if your main interest is the alphabet.
Run a whitespace and punctuation test before ordinary prose review. The TECkit guidance explicitly identifies unmapped spaces and line endings as a common issue in byte-to-Unicode maps. Preserve document structure as carefully as letters.
Is a font conversion the same as text conversion?
No. A font changes visual glyphs; conversion changes character representation. Source font evidence may help identify an old encoding but is not a substitute for its mapping.
Convert text and render it with an appropriate modern font as two separate acceptance steps. A legacy font can provide clues about the encoding, but its outline shapes are not a Unicode conversion table.
Can the converter work backwards?
Some conversions can be reversed, but only under documented conditions. Many-to-one mappings and unrepresentable Unicode characters cause information loss.
Document one-to-many and many-to-one mapping behavior before declaring any reverse process safe. Keep original bytes for archives; not every Unicode character can be represented in the legacy source code space.
Should I normalize Unicode immediately?
Choose a policy, test it and preserve intermediate output. Normalization can improve comparisons but is not a universal cure for incorrect character assignments.
Retain a raw mapped output and a separately normalized output. Compare exact code points and the language’s preferred representations before choosing a normalization policy. Equivalent display does not settle every data-processing need.
What is the safest first migration?
Convert a small representative corpus on a copy, check its byte coverage, compare source and Unicode examples with language experts, and validate structural fields before a batch run.
Pilot the mapping on small, varied documents and verify structural fields in a staging system. A representative sample should include proper names and rare forms, not only high-frequency letters likely to convert easily.
What makes a mapping maintainable?
Readable source maps, documented decisions, test data, compiled artifacts, version numbers and named review authority. A binary converter file alone is not sufficient.
Keep editable mapping source and approved corpus examples with the compiled converter. Ask a new maintainer to reproduce a sample result; their ability to do so is stronger evidence of long-term preservation than a binary file alone.
Can we use the same map for every document?
Only if the source encoding is the same. Mixed archives and successive keyboard versions may require segmentation and multiple map variants.
Record an encoding hypothesis per file or homogeneous batch. If authors used different keyboard generations, compare byte inventories and assign different maps where justified rather than applying one universal rule.
What should a migration report include?
Source inventory, mapping revision, converter version, output file list, replacement counts, unresolved cases, structural validation and human-language approval.
Report unresolved byte values and replacement counts alongside successful records. A high-level percentage can conceal serious problems in proper names or structural fields that occur infrequently but matter greatly.
Where can I find documentation?
SIL’s TECkit site provides the toolkit overview, downloads and mapping-language documentation. Use the documentation that matches the installed release.
Use the official TECkit mapping and tools documentation for syntax and command options. Package differences between Windows, macOS and Linux mean copied commands must be checked on the actual installed environment.
Official documentation and related reading
Begin with SIL’s official TECkit documentation and downloads. The TECkit language and tool references are linked on that page. For modern input after migration, read Translate Like a Pro 12380 — Keyman Developer. For Unicode font testing and rendering, the separate HarfBuzz shaping guide explains why display tests cannot establish correct input sequences.
The most important habit is to protect evidence. Keep old files, understand their byte system, map only what you can justify, and never conceal uncertain conversions beneath a fluent-looking display. When legacy material enters a Unicode dictionary, an educational app or a translation archive, it should arrive with a traceable history. That is how language preservation moves forward without losing the ability to question or improve earlier decisions.