Text Encoding for Developers: ASCII, UTF-8, and Unicode

How computers store text — from ASCII to Unicode to UTF-8. Understand encoding errors, BOM, and why mojibake happens.

The Encoding Problem

Computers store numbers, not letters. Text encoding is the mapping from characters to numbers. When you save a file, your editor encodes each character as bytes. When you open it, the reader decodes those bytes back to characters. If the reader uses a different encoding than the writer, you get garbled text — "mojibake" like "Ã©" instead of "é".

Understanding encoding is not optional for web developers. HTTP headers, HTML meta tags, database collations, API responses, and file I/O all depend on getting encoding right.

ASCII — The Original 128 Characters

ASCII (1963) maps 128 characters to numbers 0-127: English letters, digits, punctuation, and control characters (newline, tab, null). Each character fits in 7 bits. ASCII is the foundation of every modern encoding — the first 128 code points in Unicode are identical to ASCII.

ASCII's limitation is obvious: 128 characters cannot represent Chinese, Arabic, Hindi, emoji, or even accented European characters. Every attempt to extend ASCII (Latin-1, Windows-1252, Shift-JIS) covered only one language group, creating incompatible encodings.

Unicode — One Number Per Character

Unicode (1991) assigns a unique number (code point) to every character in every writing system: over 154,000 characters across 168 scripts. Code points are written as U+0041 (Latin A), U+4E16 (世), U+1F600 (😀).

Unicode is a character set, not an encoding. It says "A is number 65" but does not say how to store 65 as bytes. That is the job of an encoding like UTF-8.

UTF-8 — The Web's Encoding

UTF-8 encodes Unicode code points as 1 to 4 bytes. ASCII characters (U+0000 to U+007F) use 1 byte — identical to ASCII, so every ASCII file is already valid UTF-8. European accented characters use 2 bytes. Chinese, Japanese, and Korean characters use 3 bytes. Emoji use 4 bytes.

UTF-8 is the dominant encoding on the web: over 98% of all websites use it. It is the default in HTML5, JSON (mandated by RFC 8259), Go, Rust, and most modern languages. If you are starting a new project, use UTF-8 everywhere — files, databases, HTTP headers.

Set in HTML: <meta charset="utf-8"> (must be in the first 1024 bytes)
Set in HTTP: Content-Type: text/html; charset=utf-8
Set in databases: CREATE TABLE ... CHARACTER SET utf8mb4 (MySQL) or ENCODING 'UTF8' (PostgreSQL)

Common Encoding Bugs

MySQL utf8 vs utf8mb4 — MySQL's utf8 charset only supports 3-byte UTF-8, which excludes emoji and some CJK characters. Always use utf8mb4.
BOM (Byte Order Mark) — UTF-8 files can start with EF BB BF (the BOM). Some tools add it, some choke on it. PHP will output the BOM before headers, breaking them. Use UTF-8 without BOM for web files.
String length vs byte length — In JavaScript, "café".length is 4 (characters), but UTF-8 byte length is 5 (é is 2 bytes). Emoji are even trickier: "😀".length is 2 in JavaScript (surrogate pair) but 4 bytes in UTF-8.
Double encoding — Encoding already-encoded text creates sequences like Ã© (UTF-8 bytes of é interpreted as Latin-1, then re-encoded as UTF-8). Fix by ensuring consistent encoding at every boundary.

Practical Rules

Use UTF-8 everywhere. Set charset=utf-8 in HTML, HTTP, and database schemas. Never assume string length equals byte length. Test with non-ASCII input (emoji, CJK, Arabic) early. If you see mojibake, check every boundary where text crosses systems — file, database, HTTP, template engine.

Frequently Asked Questions

What is the difference between UTF-8 and UTF-16?

Both encode the same Unicode character set but use different byte strategies. UTF-8 uses 1 byte for ASCII characters (U+0000 to U+007F), 2 bytes for Latin/Greek/Cyrillic, 3 bytes for most CJK characters, and 4 bytes for emoji and rare scripts. UTF-16 uses 2 bytes for characters in the Basic Multilingual Plane (U+0000 to U+FFFF) and 4 bytes for characters above that (including most emoji). UTF-8 is more space-efficient for English-heavy text and is the web standard. UTF-16 is used internally by JavaScript, Java, and Windows APIs.

Why do some files have a BOM (byte order mark) at the beginning?

The BOM (U+FEFF) tells text readers which byte order (endianness) the file uses. In UTF-16, byte order matters because each character is 2+ bytes that can be stored big-endian or little-endian. In UTF-8, byte order is fixed by the encoding rules, so a BOM is technically unnecessary — but Windows Notepad historically added one to distinguish UTF-8 from ANSI. A UTF-8 BOM (EF BB BF) is harmless in most contexts but can break shell scripts, CSV parsers, and JSON files that don't expect leading bytes before content.

Why does string.length in JavaScript sometimes give the wrong count for emoji?

JavaScript strings use UTF-16 internally. Characters outside the Basic Multilingual Plane (like emoji, which start at U+1F600) require two UTF-16 code units called a surrogate pair. The .length property counts code units, not characters, so a single emoji like a flag (which is actually two Unicode code points joined by a zero-width joiner) can report a length of 4 or more. Use Array.from(str).length or the Intl.Segmenter API to count user-perceived characters (grapheme clusters) accurately.