What you describe is a typical vacuum leak.
I'd start off by cleaning the PCV with some carb cleaner (or replace it)
If it's not better, let it idle and start looking at the brake booster lines. I'd look at the areas that larger leaks could occur. Brake lines, gaskets, IAC, or maybe the throttle body plate isn't closing all the way?