I was a bit confused over why my string values weren’t showing emoji’s in my IDE when hovering over types. For example, take the following:

Well yes – the emoji is a string, but why wasn’t it showing the underlying value? It turns out that I needed to cast the emoji string as itself as well. I went over to the @lewismoten/emoji library and changed the script to cast everything as specific string values.

  /** pie 🥧 */
  pie: "\u{1f967}" as "\u{1f967}",
  /** pig 🐖 */
  pig: "\u{1f416}" as "\u{1f416}",
  /** pig face 🐷 */
  pigFace: "\u{1f437}" as "\u{1f437}",
  /** pig nose 🐽 */
  pigNose: "\u{1f43d}" as "\u{1f43d}",
  /** pile of poo 💩 */
  pileOfPoo: "\u{1f4a9}" as "\u{1f4a9}",

It seems a bit redundant, but it’s purely so that I can use the emoji values as types. After a quick semantic versioning and a publish to update NPM, I found that my types worked well with the emoji afterwards.

Emoji Library

For anyone who isn’t aware, I published a small library some time ago to let me use emoji within my projects. Ideally, I tend to focus on using lower-ascii characters that are printable in my source code. Although I save as UTF-8 formatted files, try to keep unicode out so that they are backwards compatible with ASCII format. Some emoji, especially newer characters, do not appear properly with my fonts. Most often, they appear as a question mark inside of a box.

In some cases – something else shows up. For example, the flag for Sark appears as two letters, “C” and “Q” inside of dotted boxes.

In some cases, my editor is able to display emoji and unicode just fine, but it hi-lights characters that may have been mistakenly used when other characters that look the same are more common to use.

This often drives me to write out the unicode escape sequences for JavaScript. Instead of writing a taco in Unicode as 🌮, I would write \u{1f32e}. That looks like what we call “Magic Numbers”. It’s a value that means nothing to anyone. For clarity, I would create a variable to describe what the value represented.

// Not ASCII compatible
const message = "I would like to eat 🌮";

// Magic Numbers hides context
const message = "I would like to eat \u{1f32e}";

// Named emoji
const taco = "\u{1f32e}";
const message = `I would like to eat ${taco}`;

I prefer to use emoji to help visually identify specific debug information in large swaths of data. I try to find characters related to the data that its associated with, but I found that the specific character doesn’t matter, so long as its different enough from all others to find quickly. I’ve done this for years. The unicode itself doesn’t change, other than new characters are added over time. Most of my time is wasted looking at various online emoji libraries full of advertisements that are somewhat difficult to use. This led me to create a library of Unicode values, as well as a website that allows me to view all of them together or filter through them. Feel free to hop over to the website and search for Emoji.

Emoji Demo

Sometimes I just needed the JavaScript escape codes, or the Unicode itself, so the library often served as a quick way to find and access that information. Other times, I was using the library, and this was a good way to find emoji to use as icons.

From here, you can see that my code samples for tacos can now use the following:

import emoji from '@lewismoten/emoji';
const message = `I would like to eat ${emoji.taco}`;

I no longer need to look up the emoji, work out its unicode escape sequence, and copy it. I just use the library.

The key names for each emoji are camel cased, and based on the Common Locale Data Repository (CLDR) Short names. The names do change over time. Sometimes the original emoji gets a new name, while the old one represents something different.

The current process is to look at the full emoji list on unicode.org and wait for it to load the entire list from 3885 requests. Yes… even with memory cache in my browser, that takes a few minutes to load everything. The library currently has 1,910 emoji. Afterwards, I paste some pure JavaScript code into the console to traverse the DOM and find all of the unicode characters, and the names. Finally, it generates the TypeScript code based on what it found, which I am able to copy to my clipboard.

So the need to add the typed unicode strings today wasn’t that difficult since the process was mostly automated.

Moving forward… I’d like to get the modifiers into the library. This includes skin tones for hands, faces, etc. I’d also like to find a source to grab the data quicker.

Import Raw Data

While writing out my plans, I decided to just go ahead and do it. The first step was to download a file of all known emoji sequences.

https://unicode.org/Public/emoji/16.0/emoji-sequences.txt

Rather than download it with each run, I’m also caching it as it doesn’t change. Parsing the data is easy enough. I remove empty lines and any line starting with an octothorp indicating comments. In addition, I remove data at the end of remaining lines that have a trailing hashtag as well. Then I spit the line with semicolons and trim the content. I’m left with code points, type, and CLDR short name. Well… mostly.

The code points may represent one or a range of characters. In that case, the short name also represents a range, which means it skips some short names if there are more than two emoji in the range.

Take the following for example:

231A..231B; Basic_Emoji;watch..hourglass done # E0.6   [2] (⌚..⌛)

This means we have two emoji – \u{231A} and \u{231b} – and we know that the first is the “watch” and the second is “hourglass done”. That’s good. Now let’s move onto a larger range.

23E9..23EC; Basic_Emoji; fast-forward button..fast down button # E0.6   [4] (⏩..⏬)

We’ve run into a problem. We now have a range that represents four emoji. \u{23e9}⏩, \u{23ea} ⏪, \u{23eb} ⏫, and \u{23ec} ⏬. I can parse out which emoji we have by iterating through 0x23e9 to 0x23ec. Here is some JavaScript that show how I parse the code point ranges.

if (/^[\dA-F]{4,5}\.\.[\dA-F]{4,5}$/.test(text)) {
  const [startCode, endCode] = text.split('..');
  const start = parseInt(startCode, 16);
  const end = parseInt(endCode, 16);
  const emoji = [];
  for (let code = start; code <= end; code++) {
    emoji.push(String.fromCodePoint(code));
  }
  return emoji;
}

Multiple Code Points

Before we jump to how to match up the CDRL short names, let’s talk about the other emoji that are made out of multiple code points. These are often modifiers, variants, and regional variants. Normally, Unicode has this built in for diacritics so that you can add an accent to a proceeding letter, like the letter “o” with various accents ó ò ô. This allows text to still be “almost” readable if your font doesn’t support some of the diacritics, so you just see a letter “o” without the accent, and then an invalid unicode marker after it so you know something is off.

In fact, I take advantage of this by “normalizing” the CDRL short names to remove the additional accent characters when I generate the code to access these emoji to conform to object key names that the lexical tokenization process that JavaScript and TypeScript use without having to resort to accessing them as strings. Notice how the flag for São Tomé & Príncipe has a key without the accents, so that its just flagSaoTomeAndPrincipe.

Without normalization, the emoji would have needed to be accessed as a string value with characters that I can’t type on my keyboard without holding the alt key and pegging the number pad, or looking them up in a character map, or online.

// ASCII - JavaScript & TypeScript lexical tokenization can parse
console.log(emoji.flagSaoTomeAndPrincipe);

// Unicode - JavaScript & TypeScript lexical tokenization can not parse
console.log(emoji.sãoToméPríncipe);
// Bypass as a string
console.log(emoji['sãoToméPríncipe']); 

// Normalize and remove combing marks
const unicode = "São Tomé & Príncipe";
const ascii = unicode
  // Separate accents as separate characters
  .normalize("NFD")
  // Now remove the accents!
  .replace(/[\u0300-\u036f]/g, '');

console.log(ascii);
// Out: Sao Tome & Principe

Emoji also uses the Zero Width Joiner (ZWJ) at \u{200d} to combine emoji. For example, many of the family emoji may have two to four people. Using a zero-width joiner, you can specify what gender and skin tone each family member is.

If a font doesn’t support displaying the combination of the two, you’ll just see the man and boy emoji side by side. Skin tone and other effects are similar. Let’s add a few emoji together – a woman, man, and heart, and provide some skin tones.

• \u{200d} is the zero-width joiner
• \u{1f3fb} is for light skin tone
• \u{1f3fe} is for a medium-dark skin tone
• \u{fe0f} is for variation, often used for indicating pictures

In all, there are 16 variation selectors ranging from 0xFE00 to 0xFE0F. Let’s try them all out to see if there are any differences.

FE00 👩🏻‍❤︀‍👨🏾	FE04 👩🏻‍❤︄‍👨🏾	FE08 👩🏻‍❤︈‍👨🏾	FE0c 👩🏻‍❤︌‍👨🏾
FE01 👩🏻‍❤︁‍👨🏾	FE05 👩🏻‍❤︅‍👨🏾	FE09 👩🏻‍❤︉‍👨🏾	FE0d 👩🏻‍❤︍‍👨🏾
FE02 👩🏻‍❤︂‍👨🏾	FE06 👩🏻‍❤︆‍👨🏾	FE0a 👩🏻‍❤︊‍👨🏾	FE0e 👩🏻‍❤︎‍👨🏾
FE03 👩🏻‍❤︃‍👨🏾	FE07 👩🏻‍❤︇‍👨🏾	FE0b 👩🏻‍❤︋‍👨🏾	FE0f 👩🏻‍❤️‍👨🏾

Nope. Only variant selector 16 has any effect. I’m finding that this is the case with other Emoji as well. Select 16 seems the be the special one that is used for Emoji. I haven’t found any emoji that use the other 15 selectors.

What are the different skin tones? For the longest time, emoji appeared as yellow, similar to the classic yellow smily face before emoji’s existed, and perhaps the influence for the first ascii 🙂 emoji. Now we have variations with five different skin tones in addition to the cartoon yellow colors.

Sometimes I can see a swatch of the skin tones. It doesn’t always work out as expected as it is meant to be applied the the proceeding character. In my web browser, it’s hard to see them if I just past besides text. The swatches show up if I paste them on their own without preceding text.

🏻	🏼	🏽	🏾	🏿
text🏻	text🏼	text🏽	text🏾	text🏿

A little visit from the future, here is how the unicode renders within my IDE for the dark skin tone. Notice that the same emoji is displayed in two different ways in the same dialog. It must be a separate font as it’s not even aware that the glyph exists with the question box.

On my terminal, I do see the swatches of skin tones showing up, but I see individual emoji, rather than the combination there of. Here is a build up to help see what’s going on:

Notice that you don’t see the zero-width joiner, or the variation characters, so it appears as if the same characters are written to the terminal following lines. However, if I select all of the characters in my terminal and paste it into a browser or IDE with a compatible font, I see just one character displaying the man, woman and heart.

Parsing Code Points

Parsing these code points is similar to the range, except that you join all of the code points together. Let’s look at a really big one:

1F3F4 E0067 E0062 E0077 E006C E0073 E007F; RGI_Emoji_Tag_Sequence; flag: Wales  # E5.0   [1] (🏴󠁧󠁢󠁷󠁬󠁳󠁿)

That’s seven code points! However, you resolve the code points and join them all together.

if (/^([\dA-F]{4,5} )*[\dA-F]{4,5}$/.test(text)) {
  const emoji = text.split(' ')
    .map(code => parseInt(code, 16))
    .map(code => String.fromCodePoint(code))
    .join('');
  return [emoji];
}

We now have a list of 2,322 emoji!

Finding Names

We have our emoji. We are missing many names. Next was a look over at the zero width joiner characters.

https://unicode.org/Public/emoji/16.0/emoji-zwj-sequences.txt

This file is much easier to parse as every line is a single character. We also have five categories: Family, Role, Gendered, Hair, Other. The categories are commented, so I needed to capture the category and remember their line numbers, which were then compared to the line number of each emoji being processed to find its category.

Well, that was quick. We’ve got a total of 1,468 emoji that were found. Although we found plenty of short names, It looks like these two lists don’t overlap. We found 2,322 in the other list. That’s 3,790 emoji!

Test

Of all the places, I didn’t think to look at a test file. I found that it had 5,042 emoji. It included the short names, 10 groups, and 100 sub-groups. Groups and SubGroups were setup similar to the categories in the ZWJ file, in that I had to store the line numbers for their location and return the most recent group/sub-group prior to the line being evaluated as an emoji. It also had many values that were both fully-qualified and minimally-qualified, in which an extra code point would be present, but the emoji still looked the same.

Next I verified that all sequences and ZWJ sequences were in the emoji-test file. Last was to start finding and removing duplicate emoji with the fully/minimally qualified status. Just going on duplicate short names alone, I found 1,252 emoji that I can shake out.

Now I needed to evaluate what to drop. All of the duplicates had one fully qualified emoji. In some cases, I had many duplicate values for the same emoji. Let’s evaluate one of them and see what’s going on.

Emoji	Status	Code Points
👁️‍🗨️	Fully Qualified	1F441 FE0F 200D 1F5E8 FE0F
👁️‍🗨	Minimally Qualified	1F441 FE0F 200D 1F5E8
👁‍🗨️	Unqualified	1F441 200D 1F5E8 FE0F
👁‍🗨	Unqualified	1F441 200D 1F5E8
		FE0F – Variation Selector 16
		200D – Zero Width Joiner
👁		1F441 – Eye
🗨		1F5E8 – Speech Bubble

It seems that a fully qualified emoji requires variation selectors. In this case, Variation Selector #16. From an earlier observation, we know that none of the emoji use any of the other variation selectors.

How important is a fully qualified emoji? Normally, you will not see a difference. Sometimes you will. Here is one for the A button (blood type) that I found where the qualifier changed the appearance drastically.

And guess which one my library was using? Unqualified. Which is a bit of a surprise as it was parsed from the unicode website. Ah well, we will get that fixed up just fine.

From here I dropped all duplicates so long as their status wasn’t fully-qualified. This left me with 3,781 fully-qualified emoji, and 9 with a status of “component”. What is a component? Skin tone and hair. You can change the color/style of your emoji hair as well. Unlike skin done, you need to ZWJ to add it.

Skin Tone	Default Hair	Red 1F9B0	Curly 1F9B1	Bald 1F9B2	White 1F9B3
Default Tone	👩	👩‍🦰	👩‍🦱	👩‍🦲	👩‍🦳
1F3FB Light	👩🏻‍	👩🏻‍🦰	👩🏻‍🦱	👩🏻‍🦲	👩🏻‍🦳
1F3FC Medium-Light	👩🏼‍	👩🏼‍🦰	👩🏼‍🦱	👩🏼‍🦲	👩🏼‍🦳
1F3FD Medium	👩🏽‍	👩🏽‍🦰	👩🏽‍🦱	👩🏽‍🦲	👩🏽‍🦳
1F3FE Medium-Dark	👩🏾‍	👩🏾‍🦰	👩🏾‍🦱	👩🏾‍🦲	👩🏾‍🦳
1F3FF Dark	👩🏿‍	👩🏿‍🦰	👩🏿‍🦱	👩🏿‍🦲	👩🏿‍🦳

From what I see, every face emoji can have 30 variations. When you get into the ZWJ sequences with couples or families of four, you can potentially have 810,000 variations for one emoji. I’ll need to consider some kind of logic to help people build-out these types of emoji, rather than hard-code their values.

Now that I have parsed out groups and sub-groups, I’d like to do something with that as well.

Generating Code

Now I had a list of all emoji. For the most part, I was able to copy the code from my scrape.js file and convert it over to TypeScript. I was restricted to pure JavaScript in the past because I was running it in the browsers debug console. Now typescript compiles the code, and node runs it.

I overwrite my emoji.ts and found differences peppered throughout the file when reviewing the diff in GitKraken. Most of the changes were adding on the variant selector to ensure all of the emoji were fully qualified. It was surprising how much it had an effect on the appearance of many of the Emoji that I already had. Instead of black & white emoji, they came to life with color.

Emoji that changed due to being fully qualified included the A, B & O buttons 🅰️🅱️🅾️ for blood type, circled MⓂ️, ferry ⛴️, ice skate ⛸️, information ℹ️, Japanese (congratulations㊗️, monthly amount🈷️, secret㊙️, and service charge🈂️) buttons , last track ⏮️, mountain ⛰️, next track button ⏭️, part alternation mark 〽️, pause ⏸️, P button 🅿️, pick ⛏️, play/pause ⏯️, record ⏺️, rescue worker helmet ⛑️, right arrow curving up⤴️/down ⤵️, shinto shrine ⛩️, skier ⛷️, stop button ⏹️, stopwatch ⏱️, timer clock ⏲️, transgender ⚧️, umbrella on ground ⛱️, wavy dash 〰️, cloud with lightning and rain ⛈️.

Other changes were having the skin tones applied to various emoji – mainly hands, professions, couples. Hair components were applied to the man and woman emoji. The components for skin tones and hair were added as well.

I lost the little indicators that identified new emoji, such as face with bags under eyes and fingerprint. It wasn’t something I had intentionally stuck in there. It’s just part of what was parsed out of the old DOM.

Demo Site

Not only has the library been updated, but the demo site has been updated to allow you to filter by group, sub-group, skin tone, and hair. It’s jumped up to 3,790 emoji to choose from.

Emoji Demo