What happens to any wired computers connected to this router? How is the speed of the network between your computers? Are you running both computers at the same time? Try using only one to see if it still happens. Could it be your ISP provider or your modem?
Hmmm 54kbps??? Is this a dial up connection?