After losing a close first set in a tiebreak, Tsitsipas rallied to win the second and level the match, but appeared to have an abdominal issue late on before slumping to defeat in three hours and 54 minutes.
Tsitsipas, a two-time Grand Slam finalist, has a poor record at Flushing Meadows and has never made it past the third round of the U.S. Open.
He also struggled in the build-up to the tournament, winning just one of his three matches at the Canadian Open and Cincinnati Masters.
For Kokkinakis, it was a first win at the New York Grand Slam since 2019.
"I had a five-set battle with him a couple of years ago at the Australian Open and I was expecting a war," Kokkinakis said in his on-court interview.
"It was physical out there and I have a tendency to sometimes drag these matches on and have some real long battles, but I just tried to stay focused, stay the course and just tried to play every point."
He next faces either Nuno Borges or Federico Coria.