Ipswich had threatened to provide the latest cup upset when George Hirst slotted in the openener shortly after half-time, but Ryan Yates netted near the 70-minute mark to bring the tie back to parity.
Neither could find a winner in either the 90 minutes or in extra-time thereafter, prompting penalties at the City Ground.
High-quality penalties were traded by the first four to step up from each team, but after Callum Hudson-Odoi scored Forest's fifth, Jack Taylor's effort was palmed away by the diving Matz Sels to spark wild celebrations from the home team and their support.
Forest will play Brighton and Hove Albion at the Amex in the next round, after the coastal side beat Newcastle on Sunday.
FA Cup quarter-final draw in full:
Fulham v Crystal Palace
Preston North End v Aston Villa
Bournemouth v Manchester City
Brighton v Nottingham Forest