Any opinions expressed here are my own and not necessarily those of my employer (I'm self-employed).

Jan 8, 2012

Introduction to character encoding

Text encoding is a persistent source of pain and problems, especially when you need to communicate textual information across different systems. Every time you read or create an xml-file, a text file, a web page, or an e-mail, the text is encoded in some way. If the encoding is messed up along the way, the receiver will be looking at strange characters instead of the ori�inal t□xt. (ba-da-bing :)

I've been fighting with characters sets on several occasions throughout the years. Just recently, I had a bug in TransformTool related to character encoding and how errors are handled in the .NET framework. While writing about the bug I needed a reference to a basic introduction to character encoding — only to discover that most are very technically focused and dive right into the characters' hex codes. Here, I'll try to fill that gap and explain only the basics. I'll include pointers to more detailed resources in case you decide to dig deeper into the dark world of character encodings.

How encodings work
The Unicode Consortium has a great explanation of how it really works:
Fundamentally, computers just deal with numbers. They store letters and other characters by assigning a number for each one.
The number assigned to a character is called a codepoint. An encoding defines how many codepoints there are, and which abstract letters they represent e.g. "Latin Capital Letter A". Furthermore, an encoding defines how the codepoint can be represented as one or more bytes. We'll use one of the most prominent encodings as our first example: ASCII.

Capital A in the ASCII encoding
Note that an encoding does not determine what a character should look like on your screen, that's taken care of by fonts. The figure shows how two different fonts give two different graphical representations of the A, though it's still the same character.

There, that was the big picture in a few paragraphs! That's how it works! Now we'll go more into detail on how characters are encoded, because that's usually where things go wrong. We'll leave the fonts, if you want to dig further into this see Understanding characters, keystrokes, codepoints and glyphs.

We've seen that ASCII assigns the number 65 to a capital A. But what about the other characters? Here's the uppercase characters in ASCII along with their (decimal) codepoints:

 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90

And here's the lowercase characters and their codepoints:


There you go, that's the english alphabet in both lower- and uppercase. You can have a look at the complete table of printable ASCII characters at Wikipedia where you'll also find numbers, punctuation marks etc. Character encodings are often referred to as code pages or character sets as well.

There are (too) many encodings in common use around the world, each defining their own set of characters with corresponding numbers. Wikipedia lists over 50 common character encodings. The sheer number of encodings is one of the main reasons that things get messy.

How encodings differ
The Unicode Consortium summarizes the problems that arise due to all these different character encodings:
These encoding systems also conflict with one another. That is, two encodings can use the same number for two different characters, or use different numbers for the same character. Any given computer (especially servers) needs to support many different encodings; yet whenever data is passed between different encodings or platforms, that data always runs the risk of corruption.
To show some of the conflicts, we'll discuss two more common encodings, in addition to ASCII: the Latin-1 (ISO-8859-1) and Latin-2 (ISO-8859-2) character sets. Here's how they line up with with ASCII.

  • ASCII is a seven bit encoding. Seven bits lets you count from 0 to 127. Consequently, you can represent 128 different characters.
  • Latin-1 is an eight bit encoding. Eight bits (a byte) lets you count from 0 to 255. You could therefore theoretically represent 256 different characters, but 32 are unused, leaving 224 assigned. Latin-1 was defined to handle western European languages.
  • Latin-2 is also an eight bit encoding, and also has 224 assigned characters. Latin-2 copes with Eastern European languages.
  • Although Latin-1 and Latin-2 contain more characters than ASCII, they are identical to ASCII for the first 128 letters, and are consequently backwards compatible for those letters.
  • Check out the links to have a look at what the tables of characters look like!

The first obvious problem here is that the two Latin encodings define more characters than ASCII do, so they have characters that do not exist in the ASCII-encoding. It's for example impossible for me to represent my name (André) using the ASCII encoding, but it's not a problem with Latin-1 nor Latin-2. The offending character is é, if you haven't already guessed it.

Moving on, the Latin-1 and Latin-2 encodings illustrate the problem of using the same number for two different characters. Here's a comparison for codepoints 192 through 199 for Latin-1 and Latin-2:

Some characters match, but if you look at e.g. character number 197, you see that the same number maps to different characters in the two charsets. Mistakingly reading an e-mail with the Latin-2 encoding instead of the Latin-1 encoding would for example change the Norwegian word FÅRIKÅL to FĹRIKĹL. (Never heard of fårikål? :) As the Ĺ character does not exist in the Norwegian alphabet, this would be gibberish to an average Norwegian.

To summarize, if you write the word FÅRIKÅL to a text file using the Latin-1 encoding, here's how things can go wrong depending on your choice of encoding when reading the file:
  • If you read the file using the ASCII encoding, the byte "11000101" cannot be decoded to a valid codepoint. You might get an error, or an replacement character such as: � or □. Or even worse, you might get an ?. More on that in an upcoming blog post on how .NET handles errors.
  • If you read the file using the Latin-2 encoding,  "11000101" will be decoded to a valid codepoint, which is assigned to the letter Ĺ. FÅRIKÅL then becomes FĹRIKĹL.
These examples show why you have to be careful about what encoding you're using to read and write text, to avoid any loss of data.

To further complicate things, there are encodings that use multiple bytes to store a character. I bet you can imagine that this can open yet another world of problems, since you need to keep track of several bytes. You're right, but it's also the only way to replace all the one-byte encodings which limits a character set to 256 characters.

There must be some kind of way out of here
Unicode comes to the rescue. Quoting the consortium again:
Unicode provides a consistent way of encoding multilingual plain text and brings order to a chaotic state of affairs that has made it difficult to exchange text files internationally.
The Unicode standard defines more than 100 000 characters and their codepoints at the time of writing, but can potentially define more than one million characters. That means that there's no need for several character sets anymore, Unicode can include all characters. The big players in the IT industry work together to develop the standard further, ensuring support across platforms (Microsoft, Apple, Google and more).

There's three Unicode encoding forms, UTF-8, UTF-16. UTF-32. All of these can represent all Unicode characters. The most common encoding on the web is UTF-8, which you've probably come across. The text you're reading now is for example served as UTF-8. UTF-16 is also in widespread use, for example in the .Net framework and the Java runtime environment to represent strings in memory.

UTF-8 uses one, two, three, or four bytes to encode a character. It's backwards compatible with ASCII, which means that all the one byte characters are identical to ASCII. Other characters are stored using two, three or four bytes.

UTF-16 uses two or four bytes to encode a character, while UTF-32 uses four bytes per character. The figure shows how a capital A would be encoded.
Latin Capital Letter A encoded forms
If you want to learn more about Unicode and the different encodings, you should spend some time on Understanding Unicode. There's a lot to wrap your head around if you start digging into the details.

Since you've tagged along this far in this post, here's a fun fact. Unicode defines not just characters but also lots of symbols. The crying smiley depicted in the begining of this post is actually a unicode character. It's called "Face with tears of joy." You'll find it here, along with many others.

I hope this post helped you grasp the overarching logic behind characters and their encoding in computers. If you really want to inflict more pain to the brain, I suggest you spend some time reading the references. You can also play with text encoding in TransformTool, it supports several encodings and can show you the bytes as decimal/hex/binary.

I've highlighted some common problems related to character encoding. When you're building new systems the advice is almost always: "Stick to UTF-8." It's also safe when communicating with legacy systems that use ASCII.

Note however, UTF-8 is NOT compatible if you communicate with systems that use anything other than UTF-8 or ASCII, such as the Latin-(1,2..X) encodings. Then you either have to change the system to use UTF-8, or use the same encoding as the system when reading the data on your side. Knowing just that might help you figure out things a lot faster when things start to break.

Good luck. ☺

PS! If you're a .NET head, stay tuned for an upcoming post on some .NET encoding subtleties. You don't want to miss those.


  1. Nice article, but I think it misses a link to the "classic" blogpost about character encodings by Joel Spolsky (from 2003): http://www.joelonsoftware.com/articles/Unicode.html

  2. You're right, for those who are ready for a more technical article that's a classic blogpost. Thanks for adding it!

  3. Since you can tell what type of byte you're looking at from the first few parts, then even if something gets mangled somewhere, you don't drop the whole series.

  4. If you are a student, you dont have to miss such great educational blogs like this one https://silveressay.com/

  5. This comment has been removed by the author.

  6. Thank you for the interesting news and another interesting story to follow.

    Royal Online

  7. This is an awesome post. Really very informative and creative contents. This concept is a good way to enhance knowledge. I like it and help me to development very well. Thank you for this brief explanation and very nice information. Well, got good knowledge.


  8. لعل من الهام ان تقوم بالتخلص من الاثاث المستعمل لديك حتى لا يتكدس منزلك بالاثاث الغير مستخدم شركة اعالي الخليج افضل شركة شراء اثاث مستعمل بالرياض قادرة على ان تخلصك من اى اثاث مستعمل وتقوم بفحص الاثاث و تقوم باعمال شراء اثاث مستعمل بالرياض يمكنك الاتصال الان على
    ارقام شراء الاثاث المستعمل بالرياض

    شراء اثاث مستعمل

  9. Web Ocean Design is the best IT services provider for complete mobile and web application development. The young development company based in Bihar, India, owned and managed by Vicky who have a good amount of experience in Information Technology, Management and other related fields. We provide technical and creative services ranging from Internet Marketing to Communication maneuver. We are also skilled in website development which includes brand promotion, web designing and software development.

    website design company in patna
    website development company in patna
    website development in patna

    web design company in patna
    web development company in patna
    website design in patna

    website design patna
    seo company in patna
    seo company in bihar

  10. Web Ocean Design is the best IT services provider for complete mobile and web application development. The young development company based in Bihar, India, owned and managed by Vicky who have a good amount of experience in Information Technology, Management and other related fields. We provide technical and creative services ranging from Internet Marketing to Communication maneuver. We are also skilled in website development which includes brand promotion, web designing and software development.

    best seo company in patna
    digital marketing company in patna
    best website design company in patna

    affordable seo service in patna
    website optimization in patna
    educational internet marketing company patna

    social media marketing company patna
    real estate seo company in patna
    ecommerce seo company patna

  11. Not only in Solidworks, but also in other software related assignments, our experts can be extremely helpful. Our experts specialize in 3D mechanical design applications of Solidworks . When it comes to doing an assignment and you get stuck, don’t hesitate to get in touch with Online Assignment Expert for IT assignment help. If you are a university student and looking for assignment writing help then Online Assignment Expert is one stop solution. Our Statistics assignment writing experts in Australia serve the best Statistics assignment help to students in order to solve their queries.

  12. These are really amazing and valuable websites you have share with us. Thanks for the informative post.
    WordPress development company in Chennai

  13. Your blog is quite informative and creative at the same time. These are the two qualities especially help every content to become the most readable of all. However, these qualities are difficult to acquire which is the reason why students opt for assignment help services to get their queries solved within the time specified. My Assignment Services is widely preferred by students if they are looking for assignment Australia queries over the internet. We have more than 2000 professional experts who are available 24x7 to help you with all types of queries in giving you the assignment requirements and strictly adhering to the marking rubrics. My Assignment Services is a professional Australian assignment help service provider who has been expanding its services across the world. You can contact their experts by visiting their website at My Assignment Services if you come across any query related to any type of assignment and any subject.

  14. I would really like to read some personal experiences like the way, you've explained through the above article. I'm glad for your achievements and would probably like to see much more in the near future. Thanks for share.
    Sql server dba online training

  15. Our declaration to providing the best customer experience when it comes to the best custom essay writing service are guided by these principles we entrench daily in our delivery for the best online essay writing service firm.

  16. This helps me a lot. Thank you!
    If you want to change the password of your wifi network but you don't know how, 192.168.l.l has the tutorials you need.

  17. Are you in pressing the need of high quality and Pre Written Research Papers for Sale? Learn about Dissertation Literature Review Help that are congenial to all your needs. Consider hiring quality but also Nursing Paper Writing Service.

  18. Our Professional Medicine Essay Writers work tirelessly to ensure that your Medicine Research Paper Writing is completed within the time frame given to avoid poor scores in your Medical Essay Assignments Writing.

  19. Custom Term Paper Service industry has grown steadily in provision of Legitimate Term Paper Services and high quality Custom Term Paper Writing Services which is preferred by scholars worldwide.

  20. โบนัสเพียบ เล่นง่ายได้เยอะ joker123 ฟรีเครดิต slot online

  21. Best Corporate Video Production Company in Bangalore and top Explainer Video Company in comments , 3d, 2d Animation Video Makers in Chennai
    Awesome article. good read blog. Thanks for sharing

  22. The information you have posted is very useful for online dating. The sites you have referred was good. Thanks for sharing..
    website development United States

  23. Wow, Great information shared. I appreciate the persistence you put into your website and detailed information you provided.
    freelance web developer
    php developers
    Offshore Software Development

  24. Many readers were in search of this information because they want to solve this issue about encoding but now they are satisfied with this knowledge because through this they can solve their problems. Master dissertation writing service.

  25. Every student should get access to our Economics Essay Writing Services because we have professional writers who deliver Economics Dissertation Writing Services as well as offer Affordable academic Help Online that are original and authentic.

  26. Nice Post! Dubbing services should effectively deliver international content in the native languages of the target audience. Dubbing artists should have immense experience in the field to proficiently dub videos by accurately conveying emotions.
    voice over services

  27. Excellent blog with valuable information and just added your blog to my bookmarking sites thank for sharing.
    Data Science Course in Bangalore

  28. Top quality blog with unique content and information shared was valuable looking forward for next updated thank you.
    Ethical Hacking Course in Bangalore


Copyright notice

© André N. Klingsheim and www.dotnetnoob.com, 2009-2018. Unauthorized use and/or duplication of this material without express and written permission from this blog’s author and/or owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to André N. Klingsheim and www.dotnetnoob.com with appropriate and specific direction to the original content.

Read other popular posts