Because the problem is *distraction* and not just fumbling with a handset. Despite what people think the ability to multtask such as drive a car and talk on the phone the person talking on the phone often is giving more priority to the conversation that's being had between the driver and the person at the other end of the telephone conversation than to driving.
I'm no longer a car driver for the most part, but I'm amazed at the number of people who don't have any sense about them what's happening and almost mow down people who are chatting on their phones oblivious to what's happening around them.