The Not Too Late Show with Elmo Season 3 is yet to be announced by HBO Max

Elmo isn't shy about an important part of his bedtime routine: potty time! After he challenges Jason Sudeikis to a round of the "Silly Dance Game," Ciara stops by to share an inspiring musical message. Meanwhile, Grover shakes things up as he plays usher.

The Count helps Elmo keep an eye on the clock in preparation for jammie time. Elmo welcomes Olivia Wilde for a game of "Teddy Bear Dress Up." Musical guest Sara Bareilles brings out two of her besties -- plus an extra special Sesame Street friend -- to help her perform her very own version of "Twinkle Twinkle Little Star."

It's time for Elmo's show, but his first guest, Grover is nowhere to be found! Not to worry, Andy Cohen steps in to do his best impression of the MIA monster, while Josh Groban takes Oscar's lead in singing a "trashy" refrain.

Elmo brings in "monster makeover specialist" Jonathan Van Ness to unleash the gorgeous monster inside Sesame Streeter Alan, before Elmo and Cookie unveil hair and makeup transformations of their own. Mykal-Michelle Harris gives hosting a whirl and faces off with Elmo in a "No Laugh Challenge," and H.E.R. performs a song about her beautiful, versatile hair.

Elmo does some hard-hitting journalism to learn about Hoda Kotb's morning routine before challenging her to a game of "Hot Potato." Ben Platt treats Cookie to a song about his very favorite thing. Grover offers his expertise on how to pack Elmo's backpack for school.

Elmo finds out what animal John Oliver would be, and the two compete in the world's slowest and most dramatic race. Sofia Carson joins the slow-mo fun with a lovely lullaby, and Kwame Alexander recites an impromptu poem about - who else - Elmo!

Elmo and Cookie Monster talk weird dreams before Blake Lively returns to Sesame Street for a game of "Cookie Monster Says." Dan + Shay perform their own rendition of "I Don't Want to Live on the Moon." Oscar the Grouch prepares to finally get his chance in the spotlight for "Trash Talk."

Elmo faces off in a nursery rhyme rap battle with Miles Brown, and the Joyous String Ensemble drop by, in their PJs, to play a medley of familiar tunes. Cookie Monster babysits his little niece, who runs loose around the studio before her bedtime.

Trainer Nature Nick brings some animal friends on-set to help teach Elmo about nocturnal creatures. Jordin Sparks sings one of her favorite childhood songs and shares the thing she loves most about nighttime. Correspondent Grover takes to Sesame Street to investigate whether or not the sun should be in bed. Correspondent Grover takes to Sesame Street to investigate whether or not it's night.

Batman is in the house to play a game of Beach Ball Ballyhoo with Elmo. In search of the perfect bedtime story, Elmo welcomes acapella group Pentatonix, who performs a mashup of lullabies. Rosita looks for three sheep so she can tell her favorite sleepy time tale.

Elmo looks inside some awesome goody bags before challenging guest John Mulaney to a nail-biting tricycle race and duetting with Lil Nas X. Ernie leads a baaa-ckstage tour right in the middle of the show, and Cookie Monster gets a special visitor.

The Jonas Brothers drop by Elmo's show to compete in a Silly Freeze Dance contest and share a song about brushing those pearly whites. Abby Cadabby's magical fix causes havoc in the control room for Bert and Ernie.

Before bedtime, Elmo takes on the extra special role of talk show host! And for his very first episode, Elmo welcomes guests Jimmy Fallon and Kacey Musgraves, who sings a Sesame Street classic.