I think a lot depends on whether or not your current conversation style is adversely affecting your ability to communicate with one other.
If you're both still able to convey and exchange ideas productively, then perhaps there's no problem. I think you can gauge from the person's facial reactions and tone of voice whether or not they find your interjections frustrating.
I'm actually the type of person who will stop in mid-sentence when people interject with a comment because by nature, I'm more of a listener than a talker. It's also a by-product of my upbringing -- as a child, I was taught not to interrupt because it was considered "disrespectful".
Now, when something really matters to me and I don't want to lose my train of thought, I will on occasion hold up my hands and ask the other person to let me finish first. Perhaps you can ask your friend to do the same if they do not want to be interrupted.