Yes and no.
Let’s separate it into two questions:
1) Can agencies listen into your conversations?
Answer: absolutely yes. This is taking place at the various switching and signal processing facilities that the NSA forced the different phone companies to build about 10 years ago. So when your Verizon cell phone connects, it is actually routing your call to Verizon large electronic switchboard, which (a) sends the call to your destination, but at the same time (b) stores your conversation. My understanding is that not all calls are listened to by humans all the time, but that there are some “keyword” scanners that examine all words trying see if you are triggering a suspicious one. (Like: “blow up” or “Malaysia”)
2) Can apps listen in on your conversations?
Answer: probably not, yet. As good as smartphones are, they don’t have the capacity to do real time voice analysis themselves in real time. That’s why Google voice commands have to be sent to California for processing – the computers are better/faster there. They’ll get there, of course, but voice processing is fairly computationally intensive.
By the way, texting and email is much easier for the eavesdroppers to analyze, because it’s already in text. No voice processing is required. So it’s safe to assume that every text message and every email is being read, in very close to real time. Voiced, not so easy.