Character Encoding

Overview

A GEDCOM file includes a CHAR tag in the HEAD section that indicates the character encoding used in the file. The valid options as of GEDCOM v5.5.1 are ANSEL, ASCII, UNICODE, and UTF-8. Unfortunately, some programs specify the wrong character encoding, some use an invalid option such as ANSI, and some files are edited by text editors that change the character encoding of the file without changing the CHAR value. For those reasons, GedSite detects the character encoding using both the CHAR value and character encoding detection techniques.

GedSite opens the GEDCOM file once to detect if the file has a Unicode "byte order mark" (BOM). If GedSite detects a Unicode BOM, it ignores the CHAR value and uses the encoding associated with the BOM to read the file.

GedSite re-opens the GEDCOM file to read the CHAR tag and several other GEDCOM tags from the HEAD section. If GedSite did not find a Unicode BOM, GedSite uses an ASCII encoding to read the file for the tag preview.

After reading the CHAR tag, if GedSite did not find a Unicode BOM, GedSite chooses an encoding based on the CHAR tag value.

If your genealogy program supports writing a GEDCOM file using the UTF-8 encoding, choose that option for the best results with GedSite.

When the character encoding in the GEDCOM file is set to "ASCII", GedSite will accept characters in the Windows-1252 encoding. Windows-1252 is a superset of ASCII.

Challenges

Unfortunately, there are character encoding issues that make it difficult or impossible to detect the encoding automatically.

GEDCOM file specifies UTF-8, but file is not UTF-8

If a GEDCOM file self-identifies as UTF-8 by including a 1 CHAR UTF-8 record, GedSite may or may not be able to detect if the file is actually in some other format. For example, if the file is actually a Windows text file with encoding "Windows 1252", a common text file format on PCs running MS Windows, then GedSite cannot tell that the file is not in UTF-8 format. Non-accented characters will display correctly in the resulting site, but many accented characters will not. The solution is to change the Database.CHAR Value property to "ASCII".

ANSEL

ANSEL was an ANSI standard used to encode text, but as of 14 February 2013, the standard has been withdrawn. The Family History Department of the Church of Jesus Christ of Latter-day Saints recommended an extended version of ANSEL for use in GEDCOM files.

Fortunately, modern genealogy software programs are not limited to writing ANSEL-encoded GEDCOM files, and you should configure your software to write in another format, preferably UTF-8. Still, to process files that are encoded in ANSEL, GedSite includes support for that encoding. GedSite's support is based on the information sources listed below.

The tables below describe how certain ANSEL code points map to Unicode. These tables were constructed based on information from these sources:

1. The GEDCOM Standard, Draft Release 5.5.1, "Appendix C, ANSEL Character Set"
2. The GEDCOM Standard, Release 5.5, "Appendix D, ANSEL Character Set"
3. The GEDCOM Standard, Release 4.0, "Chapter 6, Specification for GEDCOM Character Sets"
4. The Character Name Index on The Unicode Consortium
5. GEDCOM ANSEL Table by Tamura Jones, especially for corrections and examples

The GEDCOM standards listed above were prepared by the Family History Department of the Church of Jesus Christ of Latter-day Saints.

Spacing Characters

GedSite does not support LDS extensions "e in middle of line" or "o in middle of line". They are converted to "e" and "o", respectively.

Combining (non-spacing) Characters

ANSEL includes combining characters that modify the following¹ character. In the table below, the Graphic column shows the combining character modifying a dotted circle ◌ (U+25CC).

Please note that per the Unicode Standard, Version 7.0, Chapter 3, D52, "The graphic positioning of a combining character depends on the last preceding base character, unless they are separated by a character that is neither a combining character nor either zero width joiner or zero width non-joiner."

In the table above, the position of the cedilla may be different when it applies to the dotted circle ◌̧ compared to where it appears under a "c" in "ça". In this document, the behavior will vary based on your browser's font choices for "serif" and "sans-serif", and also on your browser's text layout software.

The slash overlay character does not seem to be positioned properly when applied to the dotted circle or the digit zero. I tried multiple fonts and multiple base characters and all combinations produced similar results.