Codes of ethics and conduct and discrimination don’t really apply to this situation. The OP asked if people would continue working with a buyer who gave them a lower than 5 star rating.
Like others have said, it’s entirely the buyer’s prerogative to leave an honest review. But it’s also the seller’s prerogative to work with whomever they wish to work with, or not, as the case may be. Nowhere in the terms of service does it say that by joining Fiverr, I agree to work with anyone and everyone, no matter what. We’re freelancers here, we have choice. I maintain the right to decline work if I wish to - I can’t be forced to do something I don’t want to do.
Granted, if you start writing things like “I’m not going to work with you because…” - then you might be leaving yourself open to a warning about discrimination. But you don’t need to. A polite ‘no thank you’ and the use of the Block button is enough.
To the OP - if a buyer consistently gave us less than 5 stars, I’d probably question why. If I know it’s something we’ve done, I’d try and fix it. If it’s them (“oh, it’s not personal, I just never like leaving a 5 star review” etc) - then yeah, I’d probably decline working with them in the future. Depends on the value and frequency of their orders. An occasional 4.7 in exchange for an absolute whale of an order? I can probably live with that. Regular 4.7s for $20 a piece? No thank you.