This is only my point of view, but based on your feedback, your clients are obviously very happy with your service. They like what they hear. Doubling your price to $10 is not going to change what they hear. And when I say hear, I mean the quality of your voice and also the quality of the audio. They’re happy.
Therefore if the only thing you change about your gig is the price (and you leave your description and samples exactly the same as now), then logically your clients old and new are still going to like what they hear.
As much as it pains me to say this @cyaxrex is also correct when he suggests charging $20 for the same service you’re offering now, and not taking baby steps to get there. Even if the number of clients halve, you will still make double. There is a massive logic and reality in what he says.
However I do recognise that from your point of view at the moment doubling your rate may feel like a giant step and you’re obviously worried that client expectation will change. Based on my experience though, it won’t. If you start charging $50 / $75 / $100 then yes of course it will - that’s a different league of voice over. But the expectations of a $5 gig vs a $10 are the same. Double your price today.
By increasing your gig price to $10 / $15/ $20 you will lose the bottom end of the scammer / really bad buyer territory, you’ll probably see a small fall in sales (please don’t worry about that), but you’ll make more money and feel more satisfied.
When I sold voice overs I included a line in the gig that went something like “Please listen to the sample audio provided and ensure that you are happy with the quality of the audio recording. You can expect a very similar standard of audio quality for your own voice over”. That line was more for my benefit rather than the seller’s.
***** I’ve come back to add another quick thought…
A good compromise for where you’re at right now in your mindset vs your fears might be to double your price to $10 but to reduce your word count from 350 to 250 words per gig. There’s nothing stopping you doing that for a month or two, see how it goes and then maybe asking another $5 or $10 on top later on.