Infantino has been the butt of scathing criticism ever since launching, and then withdrawing within days, a plan to bring in private investment into FIFA competitions, including the World Cups.
FIFA had hit back on Saturday at "a concerted and ongoing effort by some to undermine" football's governing body and Infantino.
However, Monday's letter from the governing bodies of European football (UEFA), North and Central America (CONCACAF) and Asia (AFC), showed his critics were unbowed.
"Leadership in football is not a possession. It is not about holding - or demanding - power to be held," they wrote.
"When trust is broken through deception, when an individual places himself above the collective that entrusted him with authority, that duty has been abandoned," the confederations added.