It might be, or it could be something at your end. Might you have two phones or ata's behind one NAT box? If so, things might be a bit crunchy. Incoming packets for port 5060 (sip) and/or 5004 (rtp) could be going to the wrong phone/ATA when an externally initiated connection was starting up.
The only way to really tell what is going on is slap a tcpdump on the external line between the modem and nat-box and watch the transactions. From the SIP headers, it should be very obvious if your side is blowing off the incoming connections, or if the problem is further upstream.
Having just gone through this with two of my voip/pstn gatewaying providers, I can say that problems like this do exist. In my case, the a lack of any SIP packet coming in around the time the remote end got a fast busy is pretty strong proof that something upstream is either overloaded or misconfigured.
-wolfgang