The shootout swung in Switzerland's favour when their goalkeeper Gregor Kobel saved Cucho Hernandez's penalty after Davinson Sanchez had earlier hit the crossbar for Colombia.
Substitute Ruben Vargas then converted the winning spot kick for Switzerland to beat the South Americans 4-3 following a game that finished 0-0 after 120 minutes.
He was one of several substitutions, including that of forward Cedric Itten a few minutes earlier and Zeki Amdouni a little later, that refreshed the Swiss frontline after the 87th minute.
"It was difficult in the beginning, but we stayed in it and showed great mentality," Yakin said. "Then I brought on the right players at the right moment."
Vargas, who came on in second-half stoppage time, kept his cool a little later to send his team through to their first World Cup quarter-final since 1954. Earlier, fellow substitute Amdouni had calmly converted his penalty.
"It was just amazing to see him smash that (Vargas penalty) in. I was so relieved and happy," Yakin said.
"We are getting into the flow and have to slowly process what has happened. It was an unbelievable victory. We had to stay very patient, but we put in a top performance today, so I am very happy with my players."
Vargas, whose team had lost their previous shootout at the Euro 2024 quarter-finals to England, felt no nerves, he said.
"I had to believe in myself. I felt very safe and very secure," Vargas said. "It was amazing I could score the decisive penalty."
Switzerland will next face Argentina, who staged a thrilling comeback to beat Egypt 3-2 earlier on Tuesday.
FIFA World Cup 2026
The 2026 World Cup is taking place from June 11th to July 19th in the United States, Canada and Mexico. The tournament features 48 national teams and is played at 16 modern stadiums.
Match schedule and times | Group standings and knockout bracket | How to watch the World Cup | World Cup Format | Past winners of the tournament