Exposure to the modest RF energy from a cell phone is highly unlikely, in my view, to cause reduced sperm count. Even if you place the cell phone at your belt line and use a headset, there is insufficient energy to cause temperature elevation sufficient to impact male fertility.
A more likely explanation is that people typically use cellphones while sedentary. More cell phone use may equate with a more sedentary lifestyle, and poor fitness leads to decreased fertility. [Magnusdottir, et al., Persistent organochlorines, sedentary occupation, obesity and human male subfertility, Human Reproduction
2005 20(1):208-215]. Sedentary lifestyles also lead to many other factors that can result in decreased fertility.
Bob Weller IEEE Committee on Man and Radiation (COMAR) International Committee on Electromagnetic Safety, SC4 (RF)