How many miles on the '04?
There should be no vacuum to the EGR valve at idle, so............
When you pulled off that vacuum line two things happened.
1. If there was enough vacuum on the hose then EGR valve was open and exhaust gas coming in at idle, and above, this was causing the rough running.
2. Now open vacuum hose added more air to intake allowing better fuel/air mix that was too rich.
Next time try putting your finger over that vacuum hose to see if #2 is true.
The EGR valve gets it's vacuum thru the DPFE sensor, it will have two vacuum lines and an electrical connector.
Computer opens vacuum to EGR valve via the DPFE sensor only when engine is under load, cylinders heat up under load, the exhaust gas cools them down, as odd as that sounds, lol.
DPFE sensor can be tested but if it is passing vacuum to EGR at idle then it is probably bad, but volt/ohm meter can be used to test it.
Normally you would get a code if DPFE sensor had failed, but if the valve inside it has failed the computer wouldn't know that