You just have to actually turn the XM unit off and wait 30 seconds. I have the direct FM adaptor too and that's the way it works.
Fooled me for awhile also.
The adaptor works by sensing power to the XM antenna and switching off the AM/FM antenna and connecting the radio to the XM modulator signal. To get it to switch back, you have to turn off the XM radio and wait for the caps in the system to discharge, then your AM/FM antenna kicks back in.