Old, slow (< 9600 bps) modems used relatively simple FSK and PSK modulation. The classic Bell 212A (long the standard 1200 bps dialup modem), for example, used 4-ary differential PSK.
New modems use trellis-coded modulation with multi-dimensional signal constellations. Analyzing them is not trivial (graduate-level signal processing). Decoding them isn't either.
Exactly what are you trying to accomplish?
Laura Halliday VE7LDH "Que les nuages soient notre Grid: CN89mg pied a terre..." ICBM: 49 16.05 N 122 56.92 W - Hospital/Shafte