All Blacks scrum-half Cam Roigard and winger Will Jordan both grabbed a brace of tries, while Sam Lakai also crossed for the hosts in front of a packed house at Christchurch's new Te Kaha stadium.
Jordan's second try gave the All Blacks a nine-point cushion 10 minutes from time but France stormed back through fly-half Matthieu Jalibert who burst over the line in the 77th minute.
Antoine Hastoy's conversion trimmed the deficit to two points to set up a grandstand finish but the All Blacks held on to deny France a first win in New Zealand since 2009.
"The French, they're very unpredictable. They love to play what's in front of them," said Roigard.
"Credit to the boys for hanging in there."
Six Nations champions France were without captain Antoine Dupont and other first-choice players rested after playing in the Top 14 final.
But their replacements performed well in a match played at breakneck pace.
Winger Theo Atissogbe and Hastoy scored second-half tries to keep Les Bleus in the game after the hosts threatened to run away with it.
Rennie's first Test in charge of the All Blacks started shakily, with an energised France scoring their first try through winger Damien Penaud barely a minute after kick-off.
Home fly-half Ruben Love's debut in the number 10 shirt also got off on the wrong foot, with a yellow card for high contact on full-back Max Spring in the build-up to Penaud's try.
The hosts shrugged off the setbacks, though, with recently confirmed captain Ardie Savea lighting the fuse.
Savea won a turnover in scoring territory and Jordie Barrett heaved a long ball to Jordan who dived over at the right corner in the eighth minute.
Twelve minutes later, the All Blacks had their second try with a raid down the other flank, with Peter Lakai crossing after a neat one-two with winger Caleb Clarke.
France scrum-half Maxime Lucu nailed two penalties to keep Les Bleus in touch but the All Blacks finished the half ascendant with a 19-13 lead thanks to set piece pressure.
Hooker Codie Taylor burst away from a scrum to be brought down two metres from the line but Roigard arrived quickly in support and ran over unopposed.
The try-fest continued after the break, with Hastoy coming off the bench and touching down at the right corner to complete a sparkling team try straight after the restart.
New Zealand hit back through Roigard, who grabbed his second try after being released by a Jordie Barrett off-load.
Atissogbe snatched back the lead for the French near the hour-mark with a try near the right corner.
But Love's 65th-minute penalty and Jordan's second try gave the All Blacks just enough breathing room to close out a classic despite Jalibert's late flourish.