No, not at all, why would the customer pay, or think they pay, 40% over?
They pay your price - which you set, and of course you have to include your costs which are the 20% Fiverr takes from you in this case.
See it like this - if you sell something/somewhere else, online or offline, it’s the same, you need to set your prices so that they cover your own costs plus some profit.
If you own a website and sell from there instead of from Fiverr, you need to calculate the costs to run that website plus marketing, etc., into your prices. If you pay rent for your brick-and-mortar store, of course you need to calculate the rent, the utilities, the trade tax, whatever you need to pay yourself to be able to sell, plus make a profit.
Your buyers in most cases are business people themselves, they understand that it costs money to earn money and that you need to make your prices so that you not just cover your own costs but earn something so you can live as well.
You set your price, your buyer agrees, and then they expect to pay that agreed-on price, plus Fiverr’s buyer fee, that’s it. How much you take, or how long you need to fulfill your orders, all that is completely your own business. You don’t have to justify yourself for the price you take, just as your buyers don’t have to justify themselves for whatever they take for whatever they do with your delivery.
It’s just business.
If you can find customers for the price you need to take, all is fine, if not, you might have to lower your costs or add something that has more value to your customers so they won’t mind the price, or advertise more to find the right kind of buyer who will value your work enough to pay the price, etc., but that leads too far. 
Just make your prices as you need them to be and tell your customer the end price from your side, so they’ll see that price, plus Fiverr’s buyer fee on the offer and their bill, and all’s good.
The problem usually is more that a buyer doesn’t like to see an additional “surprise item” on the offer, like an added 20% seller fee where they only expected 5% buyer fee, just keep in mind that you negotiate your end price (including the 20%).
See, when a buyer directly orders your gigs, they only pay what you set your gigs at + 5% buyer fee too. No 20% there anywhere, apart from that they’ll be deducted from your revenue by Fiverr, which means you need to include them in your prices.