But the ex-Scotland international, who also had an unsuccessful spell at Easter Road earlier in this career, insists it won't faze him at all as he tries to mastermind a Scottish Cup upset.
Hartley hopes to see Shaun Maloney, an old team-mate from his Celtic days, succeed as the new Hibs head coach, but only after he's survived through the ignominy of a shock cup exit.
"Do I expect to get stick? One hundred per cent," Hartley told the Scottish Sun. "Does it bother me? Not in the slightest!
"It's always good going up against boys you played with. So many of my team-mates have moved into coaching and I like to see them doing well.
"I'm sure Shaun will do a good job at Hibs, but the fact he's a former team-mate won't stop me wanting to get a result.