Power assist brake booster works by outside air pressure.
The engine creates a low pressure, via the intake vacuum, in 3/4 the power assist tank.
When you push on the brake pedal the atmosphere valve inside the cab opens and lets outside air pressure into the other 1/4 of the tank, because of the lower pressure the outside air pressure assists you in pressing down the pedal.
On the engine side is the vacuum line and Check valve .
The check valve sounds like the problem or its gasket, could be the diaphragm inside the booster, but they usually last a long while.
The check valve only allows lower pressure, so at idle the power booster is "charged", a "charge" will usually last for 3 or 4 uses of the brake pedal with engine off.
As brakes are used with engine running at higher RPMs the booster gets slightly higher pressure on the low pressure side, some may notice that brakes feel stiffer, but as soon as throttle plate is closed, coasting or idling, the booster is "charged" again.
So what your symptom reads like, is that you are losing low pressure pretty fast, pedal gets stiffer and the push back is when pressure is equal, you are feeling all the springs in the system, in the master, in the booster, as there is no more assist.