It depends. Not really with friends, if you are feeling spontaneous, but you need to expect that a lot of people will be busy.
If one person (a date or a friend) does it to you regularly, you might suspect that you are a fall back person for them. In that case, I would either say something like, “Sorry, I’m busy. You’ll probably need to call earlier in the week if you want to get together” or just keep saying no.