Hutchie were left with a mountain to climb after they conceded just three minutes in when Dunbar striker Jamie Devlin capitalised on a defensive mishap to fire in to an unguarded net.
Sandy Cunningham pulled the hosts level from the penalty spot after Dunbar goalkeeper Daniel Laing barged over Errol Douglas leaving referee Chris Gentles with a simple decision to make.
Gentles then showed Devlin a red card on the stroke of half-time for an apparent stamp, and the hosts came out of the interval buoyed by their man advantage with Cunningham striking again two minutes after the restart with what proved to be the winner.
Lothian boss Harding said his side didn’t settle until going behind.
Hearts exit on the cards for Josh Ginnelly
Hibs 'acting like they'd won a trophy' highlights Edinburgh gulf, insists Hearts defender
Former Hibs goalkeeper Matt Macey struggles on full debut as Luton Town crash out of Carabao Cup
Hearts confirm proposed kick-off times and alternative venue for Europa League play-offs
Steve Kean reaction as Hibs B bow out of SPFL Trust Trophy and Paul Hanlon gets minutes under his belt
“With it being the Scottish Cup there was a wee bit nerves about them, but once they scored the goal, amazingly, we seemed to relax a bit. We started to get on the ball and dominate the first half, they got a man sent off on 45 minutes, but we dominated without creating a lot of chance.
“Dunbar getting a man sent-off right on half-time it completely changed my team talk. We’ve played three times this season with ten men and twice we have came from behind to win the game, so I just reminded them how dangerous teams can be with ten men.
“In cup football you can play the best game of your life and still walk away getting beat, I don’t think we were poor, but I don’t think we were great, but all that matters is we are in the next round.
“It’s a strange one because playing against a League Two team is a great chance for the guys to go and enjoy themselves against a side two divisions above, but our home pitch is their home pitch so it’s just going to be another home game for them. That advantage is out of the window. On paper, they should win the game, but anything can happen in cup football. The incentive is to potentially set ourselves up to play against Rangers, Hearts, Hibs or Celtic.”