The buyer does not deserve a refund in this case. You did the work, and the order has been marked as complete. YOU deserve to get paid for the work that you do.
In addition, Fiverr specifically states in their Terms of Service that buyers are not entitled to refunds if the work was completed according to the terms of the order/gig. If you did a great job, and honored all terms of the order, then I think you should keep what you have earned. The buyer does not deserve to have his money back – after the order is marked as complete – just because he didn’t respond sooner, or ask for modifications.
Stand your ground. Get paid for the work that you do.