The 20-year-old Czech saw off the 11th-seeded Russian 6-3, 7-6 (8/6), 4-6, 2-6, 6-3 on Court Suzanne Lenglen to set up a meeting with either Joao Fonseca or Casper Ruud.
Mensik, who won the Miami Open last season, posted his previous best performance at a major in Melbourne earlier this year but surpassed that fourth-round run with his victory at the same stage over Rublev in the French capital.
"It was super difficult for me. Even if I was 2-0 up, I felt in some moments that I'm actually losing," Mensik said.
"I'm just happy that in the fifth set I came back and took the momentum back on my side and basically finished the match how I finished it."
Rublev took an early 3-0 lead in the first set, before Mensik responded by winning six games in a row to claim the opener.
Mensik had a chance to serve out the second set at 5-4 but was broken by his 13th-ranked opponent, before clinching the tie-break at the second time of asking with a backhand winner that wrong-footed Rublev.
Rublev riposted by breaking Mensik in the very first game of the third set before serving out to force a fourth frame.
Both players' serves went somewhat haywire as three consecutive breaks opened the fourth set, until Rublev held and broke again to get an insurmountable 4-1 lead.
But Mensik found his range again on serve and forced the crucial break against Rublev in the eighth game of the final set, before firing a huge ace on his first match point to seal victory after three-and-three-quarter hours on court.