The Emoji Kitchen mashups might already feel like they've been around since the dawn of time, or the dawn of emoji, anyway, but they are still a recent development, with new and fanciful combinations landing in Gboard regularly. We've all used them, but some of the "how" and "why" behind the fun and fanciful combinations remain (perhaps appropriately) mysterious. For example, did you know that there were five distinct "types" of emoji mashups, with another potentially planned?

Earlier this week, I had the opportunity to talk with Jennifer Daniel — artist, chairperson on the Unicode Consortium's Emoji Subcommittee, head of design for emoji at Google, and arguably head chef in the Emoji Kitchen — and learn a little more about the magic behind the immensely popular stickers.

Edited for brevity, clarity, and to make Ryne’s questions sound more coherent than they actually were.

IMG_20180714_210900_620.max-1000x1000

Jennifer Daniel among her emoji creations. Image via Google.

Android Police: Most of our readers are familiar with Emoji Kitchen and the mashups that it creates between two different emoji. Some of them are cute. Some of them are kind of gross. I know some people see the spider-based combinations as almost graphic and scary. Who came up with the idea of Emoji Kitchen, and how? What’s the history?

Jennifer Daniel: No one owns emoji, Ryne. Emoji are of the people, for the people.

I think the first time I made something about emoji was in 2014. It was an OP-art for the New York Times. It was “The Emojis We Really Need.” And it was very pithy and not real at all. I was like, “I wish emoji existed for those people who complain on airplanes,” and it was a “cactus on an airplane seat” emoji, so it was very silly. I'm sure it's still on there somewhere.

plane

A dish from the ancestral Emoji Kitchen. (Image via The New York Times)

That was sort of it. I continued to play with emoji just as a visual presence that was in my life. And then I had a solo exhibit at an art gallery in San Francisco in 2018, that was just plastered with emoji combinations. And I also letterpressed emoticons — you know, pre-/proto-emoji. There were a number of letterpresses there for that. It didn't really become part of my professional job until around 2018, 2019. I joined the Unicode emoji subcommittee, and I started really understanding it on a level beyond the visual presentation — like, how they're encoded, why they exist, how they're made — and started consulting with more linguists and reading more academic papers about how emoji are understood and how they're used.

All these dots were connecting. I don't know how terribly familiar you are with Unicode, there's no reason to be unless you work on it. But you know, they've only been around for 30 years, and it's a very well established consortium. They're very formal, they digitize established languages, and we don't communicate in such a formal way anymore online, right? Slang and jargon operate at a speed that is unprecedented. And emoji — people wanted emoji to operate at that same speed.

Unicode adds things like “lifebuoy” and “lotus” and “face holding back tears,” which are great. We've been holding back tears since we were cave people, so there's a long history there. But, a confetti ball exploding with spiders — less referenced in literature and pop culture, but it's grounded in something we're familiar with. That surprise effect, high and low. That tension between good and bad. And so Emoji Kitchen is this way of working outside but also within Unicode.

Holding-back-tears-mashup-2

"Face Holding Back Tears" + "Smiling Face with Heart-Eyes."

We have well established visual morphemes that stood the test of time. But also, we want things that are not just that. And early on, with unicode, we had SMS — very slow bandwidth kinds of technology. And now we have everything: we have GIFs, stickers, photos, videos, and emoji — emoji isn't the only visual primitive.

Emoji Kitchen is where a universal character encoding consortium’s responsibility begins and ends, and us as just people of the internet can express ourselves with a specificity that necessitates an octopus holding eight coffees.

When did you start working on Emoji Kitchen specifically for Google? And could you tell me a little bit about the history there — was it something you pitched?

You mean my abstract waxing poetic about the nature of communication was insufficient at explaining this? [Both laugh.] What’s the right way of explaining the — I don’t just work on emoji, I work on a lot of different things. But emoji is the one that captures people's attention the most.

I sit in on these meetings. And, like when I was in journalism, you hear about what everyone's working on. And things stand out. You hear how someone's talking about something and you're like, “I am interested in what you're talking about.” And that I kind of take that same approach to my work [at Google]. It's like, “Okay, you're working on what? That's really interesting.” And I just become a barnacle, I start talking to them, I want to be involved in it, and I start working with an idea. I love that dynamicism in work. I'm not committed to one space, I'm able to do lots of different things. I can work with lots of different people, get a sense of what's important to them, what isn't important to them. Is there an overlap?

Emoji Kitchen was really a byproduct of that process, finding that space in between.

"Does the order of the emoji matter?"

Gboard is such a great collaborator. They have generously given a surface that has so many clicks, which is just your emoji keyboard, and it feels like a natural evolution of that emoji. The experience informs the content, the content informs the experience — which sounds like a cliché, but it really does, once you have a sense of what the content could be.

There was a period of time when I was working on it when I was like, “Oh! Does the order of the emoji matter? If you send ‘heart eyes’ and ‘loudly crying,’ does it change if you do ‘loudly crying’ and then ‘heart eyes?’” Then someone slowly walked me off the cliff of that massive fail. No, sequencing does not need to matter. There's no one right way of doing anything, there's a million ways to do the same thing.

That leads into the next question here, especially the idea of the order of the emoji potentially mattering. Because there's a sense of discovery when you use Emoji Kitchen for the first time. It's not well documented — not in a way that makes it awkward to use, but just very emergent. I don't know if it always feels natural. But once you figure it out, you've stumbled upon some sort of secret. Each one is something that's new to be found; some combinations work, some don't.

Cat Owl emoji mashup

"Cat" + "Owl"

So what is it like creating the mashups? And, I know Google loves to automate solutions for things. Is there any automated component to how you guys think of emoji or is it like a writers room and just, “What if we combine this with this and: ‘Octopus with coffee mugs?’” What’s the process?

Ryne, how much of the magician's trick do you really want to know? I'll tell you a little bit, though.

2018 was a remarkable year for me just in terms of shifting how I thought about approaching illustration at a company like Google because I don't have a tech background at all. I came in from a newsroom. I was like, “what's a PM?” I called my PM a PGM, which really annoyed him, you know, it's not what he did. So I had to learn a lot about Google culture while simultaneously making product. I was reading a book called The Visual Language of Comics by Neil Cohn. He's this fantastic comics theorist meets neuroscientist, and he scans people's brains when they read comics. And he describes something that I have observed both as a maker of imagery and as a consumer of it.

"...how much of the magician's trick do you really want to know?"

There are innate structures and rules about creating work that is meant to communicate. If you put an image — let's say, a heart above someone's head — it's interpreted differently than when it's on someone's eyes.

You can replace that with anything, gears above someone's head versus gears in their eyes, or a lightbulb. And once I started understanding that there are these patterns (you might even call them templates) that people use when they make comics, you can naturally extend that to anything. AR face masks, emoji, stickers — you could do it with anything that's combining one thing with another, be it reality, in photography, or illustrative in emoji.

When you look at Emoji Kitchen, it definitely is artisanal. But It's grounded in an academic understanding of how we visually communicate. There are an “n” number of Emoji Kitchen you look at you're like, “How would I use this?” But it doesn't alienate you. It presents a challenge, like “How would I use this? A hot dog in a coffee? Well, clearly, this is for two dogs who are meeting for coffee later today.” You are able to operate without the bounds of reality. Instead of being grounded in how it looks, it's about how you feel, which is so much more powerful and liberating. The process is really grounded in an understanding of those kinds of things.

Hot dog coffee mashup (2)

"Hot Dog" + "Coffee"

There are five kinds of… combinations:

  1. Color modifiers. So if you combine an emoji with any of the colored hearts, it changes the color. So you can make a black lipstick if you’re feeling goth (“black heart” plus “lipstick”), there's lots of examples of color.
  2. Another type is “fusion,” which is when, you know, you combine a face with a face and this creates a whole new expression based on mashing two emoji together.
  3. Another is what I would call “juxtapositioning.” It's when you combine an object, fruit, or something that's inanimate with the faces. So now you have “pineapple nerd” based on combining “pineapple” with “nerd face.”
  4. The fourth one is the “double taps,” which are emphasis. So if you tap “three heart face” and “three heart face,” you get a face just engulfed in hearts.
  5. The fifth one is conceptual. This is where you find things like "owl" plus "snowflake" equals "snow owl," or "birthday cake" plus "blue heart" gets you this super ornate cake that is blue. And it looks delicious. And now I want cake.

[Editor's note: Originally we discussed six in our interview, but only five were described in detail. In a follow-up we were told that a new type may be in the works, though it isn't here yet.]

These all really drive the content. I think about it similarly to how I work on the emoji subcommittee, which is thinking about how they're encoded. And, not that emoji kitchen is a font, but if it were, would you understand it? Would you understand “owl” + ”snowflake” is a snow owl? Perhaps if there was some text next to it, but it's grounded in something that's meaningful, represented in its rebus form.

But they all fall within those categories of different types.

Do you think there's a limit to the number of mashups you can do, or is this functionally infinite? Do you see no end to the emoji with those that are added by the consortium every year? I know Android picks them up pretty regularly. And obviously, from your position in the consortium, I'm sure you can load in enough to make sure you've got good combinations for Emoji Kitchen if you need. But do you ever think it will be complete, or is this an ongoing effort?

Oh, when is a garden done growing, Ryne?

"What is the spectrum of realism to abstraction?"

Honestly, the “threshold question” is something I've gotten since I started here. Because that makes sense, it's a very engineering-driven question. And it's one that I am challenging a bit with Emoji Kitchen, but also in an attempt to understand it. How much is too much? How much is too little? What is the spectrum of realism to abstraction? Where do we fall? And where are the opportunities for more scaling? This project is explicitly a way for me to understand that.

On the emoji side, on the Unicode formal side, it's like, “Can we encode every concept in the world?” No. “Are we trying to?” Maybe. “Should we?” Probably not. So then, where do you draw those lines? And Emoji Kitchen is helping us understand that.

I love it, you know. I love that I can straddle both a very formal, methodical, open standards body and an iterative project that is grounded in the fluid nature of how we communicate, which is fleeting and infinite. It’s kind of like the beauty of written language. It's based on like a finite number of glyphs that you can then do seemingly infinite things with and what is that? What is that? [Long pause.]

Are you asking me?

[Both laugh]. It’s rhetorical! Tell me, what is the answer? People are always like — I was just talking about this earlier today, it happens every year — “Why isn’t there a symbol for sarcasm?” There’s always a demand for clarity in explaining how you feel, who you are, and what you want. And Emoji Kitchen operates in all these spaces.

Pineapple Nerd mashup

"Nerd Face" + "Pineapple"

This is maybe a minor digression, but one of the things I thought would be cool is if there was some sort of automated component. Because, as far as our readers and a lot of people are concerned, it’s magic. How this happens is not immediately understood by the user — whether each one was drawn by hand, or if there's an automated aspect to any of them. I also thought it could be interesting if you could do this sort of like “emoji painting,” if you could just string together emoji and have some sort of system that understands their meaning construct an image, an expression of the emoji string into a graphic, like the Emoji Kitchen mashups.

So to that end, is there any chance of this expanding beyond just two emoji in the future?

Oh, there's a lot to that question — a big lead up. There's lots of interesting kinds of observations there.

I guess the way to think about Emoji Kitchen is again, following how slang and jargon work. So sometimes it's because two words are squished together like “duckface,” or it's just a blend of two words like “brunch” or “motel.” And generally it's two concepts that seem unrelated. Then when put together, their connection becomes immediately obvious. Like, you don't have to explain what “duckface” is, you know what it means immediately. And so Emoji Kitchen [follows in those footsteps], it's combining two concepts together to create something meaningful.

Biting lip blob mashup

"Biting Lip" + "Magic Wand"

You know the joke about German, “I'm sure there's a German word for that!” I'm sure there's an Emoji Kitchen for that, if we started expanding and scaling in that way. But I think if we look at just how words are combined and people's expectations of how emoji should operate, it generally — and this could be a bit of an overstatement — it generally is in this manner.

That being said, there are lots of different ways people use emoji. They multiply the emoji by tapping on them more than once to denote emphasis, which is why there's a whole sub-taxonomy of Emoji Kitchen that is emphasis. What would triple heart face look like if you hit it five times? Would that look different than two times? That's like a really awesome question to try to design out. And as we look at how people use emoji, then this is where the experience informs the content. So if you see people using emoji, like triple engrams or trigrams of emoji, then we would, but I think the more unique combinations tend to be two.

That makes sense — if the intention is communication in a sort of word form but pictorially, then a straight portmanteau as opposed to, like, a long conglomerated string of German.

What were the biggest challenges that you or the team had to overcome when working on Emoji Kitchen?

I would have to say, it is the experience — using it. We are operating in a space that has remained unchanged for a long time. Even our keyboards look like typewriters. Why? I do not know. And our emoji palettes are the same as they were when there were only 700 emoji and now there’s 3000, almost 4000. And so how do you evolve that experience?

"...there's an intentional amount of friction in it, to make it feel deserved."

You can evolve it by challenging it a little bit, by fundamentally changing what it generates. I joke, “When is a garden done growing?” But that's the project, right? It really is about figuring out: How do we evolve the experience? How do we make this feel just as unconscious in terms of communication — not force people to be consciously thinking about how they feel. Like, I'm talking to you right now, and I'm clearly excited, so I'm nodding my head a lot, and my enthusiasm is going up and down. You can't reproduce that digitally, but there are mechanisms that make a connection there. I think that, to me, is an interesting problem — an interesting opportunity for Emoji Kitchen that is worth exploring.

And to your point, there's an intentional amount of friction in it, to make it feel deserved. I think that that is a non-trivial part of the experience. I'm smiling because I'm thinking about — even as someone who makes it, I'm like, “Oh, I forgot about that one.” It's fun.

Is there a specific emoji combination that was really hard for you guys to work on compared to another? I brought up earlier, some more spider sensitive people seeing or using the spider variations — is there one that gave you trouble?

I'm thinking about “spider” and… maybe they are so troubling because I am — I also have arachnophobia, and I drew them from the perspective of someone who is deeply scared of spiders. So they are very scary, probably, to someone — they are rooted …

Your own unconscious fear comes through in the art?

Definitely. But, he's a smiley little spider right? He's a friendly little guy, except when he has 800 babies popping out of a confetti ball.

Spider Confetti ball mashup

"Spider" + "Confetti Ball."

There's different ways to think about it. Artistically, there are ones that are definitely more challenging than others. I loooove (I think they're so underrated) the “wood” combinations. It turns the emoji like they’ve been whittled out of wood. Like if our ancestors had emoji, what would they look like? And it is a complete transformation of the emoji. So you get like, whittled goat — what does a wood diamond look like? Those are ones where we take more time. They are really like a challenge to ourselves. How far can we push the aesthetic and understand more about the aesthetic having meaning?

Conceptually, I think the ones that are most challenging are ones that we just didn't launch with.

Like ideas you had that just never realized?

Oh, yeah. Can you imagine if we shipped everything we thought of? That would be terrible. That would not work. There’s a zillion things on the floor. And that's part of the process — sketching, figuring out, “does this work?”

What's your favorite Emoji Kitchen mashup? If you had to choose one, just one.

You know, I asked some folks I work with what their favorites were. An engineer I work with, his favorite is “octopus with lots of coffee” — loves that one. He also wanted me to convey, he loves the animals, especially [inaudible]. Our researcher leans a lot on “hot dog turtle,” for no reason. Just hot dog turtle. But he's been using “lip biting” plus “zany face” whenever he's coding, because it's just the horror, the horror of it all.

I really am one of those people who — one week I'm fully “snail.” Nothing can shake me, I’m snail. Next week I’m cactus. For a day, maybe, I'm “ghost.” Who I'm talking to will largely depend on what I'm using. But really, they're all my children. I lean on a lot of different ones.

"Turtle definitely is our mascot"

I will say, “Yeah, this is it. This is it. This is the one I love. This is gonna be it.” And then the next week, I've forgotten all about “snail,” poor little snail.

A mother can't play favorites among her children?

I suppose not. [Laughs.] But I do love seeing which ones people use. It makes me so happy that they saw themselves in a cat dressed up as… there's lots of versions of the cat dressed up as things.

So if you like seeing people use them, which one does the team use the most often? Do you communicate in emoji secretly — when your project manager asks you the status of something, do you just respond in emoji?

There's definitely shorthand, a lot of “reacts” on this team.

I will say — I think, due to the diversity of the content — you don't have to rely on the same old one all the time. I asked the group earlier this morning “Is there one that we use a lot?” There's something really neat that you're not using the same old tears of joy every time. It makes that moment of communication more intimate — now, I'm projecting onto it a little bit here — but you don't have to use the same ones over and over again.

Octopus Coffee mashup

"Octopus" + "Coffee"

That being said, they become inside jokes. Then you develop that insider language, and you know when someone is dropping whatever emoji, this is how they feel about it, because that one time you they did it by accident — or whatever the lore of that inside joke is.

Turtle definitely is our mascot. But everyone has preferences. Do you use one in particular?

I think we have a lot of the new blobmoji in our Slack — that’s a requirement, almost.

They live on, they have been found and resurrected from their early demise. Bringing the blobs back has been super fun. “What would a lip-biting blob look like? Okay, we can do that.” Do you use it mostly as reactions?

Mostly reaction-based now. Years ago, had we had the option, we might have just spammed them. It would have been a Twitch chat of blobmoji.

Yes! Awesome.

One last question: If you could add a new emoji to every device tomorrow, what would it be?

Um… We haven’t launched it yet.

Oh, but it’s coming? So your favorite, the one you couldn’t choose yet, is on its way?

Yeah. After you’ve made over 20,000 combinations, it’s hard to say which one you’re most excited about. But I would say yeah, I’m really excited about some new ones I’m working on.