Former US Open champion Gauff blew a healthy advantage in the second set as Kostyuk sealed a 6-2, 7-5 victory and a third-round meeting with the winner of an all-Polish tie between Magda Linette and Magdalena Frech.
"Coco is an incredible fighter," Kostyuk said. "We always have incredible matches. I tried to stick to what I had to do."
Gauff, playing for the first time since January 21st's quarter-final loss to Paula Badosa at the Australian Open, served seven double faults in an erratic display.
The American made a terrible start, slipping 4-1 behind after dropping her serve twice.
Gauff was broken again in the eighth game as Kostyuk secured the first set in just half an hour.
The world number three hit back with a break en route to a 4-2 lead in the second set as she started to find her rhythm.
But Kostyuk battled back, taking advantage of an error-strewn Gauff performance which ended when she served two double faults to be broken for the fifth time.
"The double faults made it easier for me... I'm very happy that it ended in two sets," Kostyuk told BeIN Sports.
Sabalenka & Badosa stunned
Later in the day, world number one Sabalenka was sent tumbling out of the tournament by in-form Russian Ekaterina Alexandrova.
The world number 26 beat the Belarusian 3-6, 6-3, 7-6 in a tense encounter lasting over two and a half hours.
This was Sabalenka's first match since losing the Australian Open final in three sets to Madison Keys in January.
Alexandrova has turned up in Doha in good form after claiming her fifth WTA title in Linz last week.
Alexandrova picked herself up after going one set down to level in the second - but not before an unscheduled appearance by an inquisitive four-legged feline spectator.
With the score at 2-2, 40-30 to Sabalenka, a cat nonchalantly took centre stage, inspecting proceedings courtside before sauntering off via the player's entrance.
Alexandrova shrugged off the interruption to level the match, then held her nerve to force the deciding set to a tie-break which she took 7-5.
Sabalenka was back competing at the Qatar Open for the first time since 2022, having won the 2020 title in what is the first WTA 1000 event of the season.
Alexandrova's reward was a last-16 date with Belgian Elise Mertens.
Ninth seed Badosa was also sent packing on Tuesday when she lost to Amanda Anisimova 6-4, 6-3 in the final match of the day.
Rybakina & Paolini progress
Earlier in the day, Rybakina, who lost to eventual winner Belinda Bencic in the Abu Dhabi semi-finals last week, saw off Peyton Stearns in her opening match.
The fifth seed cruised to a 6-2, 6-4 win despite a late wobble to set up a third-round tie against either Russian teenager Mirra Andreeva or Slovakia's Rebecca Sramkova.
Rybakina led 5-0 in the second set but missed three match points to take it 6-0 and then briefly fell apart as Stearns dragged the set back on serve.
The former Wimbledon champion belatedly got over the line on her fifth match point.
Chinese seventh seed Zheng Qinwen joined Gauff and Sabalenka in crashing out, though, beaten 6-4, 6-2 by Ons Jabeur.
Three-time Grand Slam finalist Jabeur claimed her first win against a top-10 opponent since last year's Madrid Open after struggling with injury.
The Tunisian will next face 2020 Australian Open champion Sofia Kenin.
Jasmine Paolini, last year's French Open and Wimbledon runner-up, also made a winning start to her campaign with a 6-3, 6-4 victory against Caroline Garcia in the second round.
American sixth seed Jessica Pegula negotiated a tricky second-round match, battling past Elina Svitolina 6-3, 7-6.
Czech youngster Linda Noskova earned a last-16 tie against three-time defending champion Iga Swiatek with a 6-2, 6-3 win over Kazakhstan's Yulia Putintseva.