I finally got around to reading the "Searching the Web by Voice" paper that describes the techniques behind the now defunct Google Voice Search.
It's an interesting idea. Apparently, when this demo was active, you could call a phone number, speak the Google Search you want performed, and then hear some summary of the search results, all over your phone.
But, the paper makes it clear that this is very hard to do. They have to understand an unrestricted vocabulary, cut through noise and accents, and do it all in real-time. The paper reports that their accuracy -- accurate transcriptions of what was said -- was below 50%. Improving accuracy was blocked by the fact that more complicated models took too much time for real-time responses.
Sounds like a fun project. Perhaps an insurmountable challenge, but that shouldn't mean it isn't worth pursuing.