I would like to recover the bits of the digital data from a
> conversation between two modems.
Hmmn.
Why do you think this is virtually impossible for a standard modem
> connection with an analog-side speed above 2400 BPS?
Because each end sends data using the full spectrum available, and uses echo cancelling to subtract off its own signal and recover the signal from the other end. If you don't already know what one end is sending, you can't do that.
R's,
John