The defender was booked in the aftermath of Kyogo Furuhashi’s 72nd-minute winner after confronting the whistler to complain about the nature of the goal and felt his side should have had a chance to level in injury time when Ryan Porteous was pushed by Hoops defender Carl Starfelt in the area.
"We are really annoyed about the second goal,” McGinn said afterwards. “It’s not a foul and when it is, the fourth official is holding the board so we just assumed it’s a sub.
"Jamie Murphy could have stood over the ball but we just assumed it’s a sub. The referee has no control over the situation and just lets it happen, and the goal goes in.
"You can see me running to him straight away and I get myself booked. It’s inept.
Ken Buchanan: Statue of Scottish boxing legend unveiled on Edinburghs’s Leith Walk
Robbie Neilson reveals Hearts' chances of a new signing for Zurich tie - plus Alex Cochrane and Craig Halkett injury update
Hibs debrief: Alarm bells in the stand (and defence); Tavares decision; ref's big calls
Former Hearts striker steps out of 'comfort zone' with move to Turkish top flight
How the Hearts players rated in the 4-1 victory over Dundee United
"Our kitman has handed the board to the fourth official. I'm almost certain that's the case and we see the numbers so we just assume, Jamie steps back from stopping and we all assume.'Even 90 seconds later, Celtic are going to make a sub and there’s a throw-in and the ref whistles it back for the sub to happen. So he kind of knows. I ran to him and asked him what was going on. There is no way we'd have got that the other way. No way.
"It's anger, but Celtic had three or four moments, Tom Rogic is an unbelievable football player and there were times where he ran the show.
"The first goal against us was far too simple. Kyogo was the difference today. Celtic had some great attacking moves where they cut us open and defended really well but it's just a simple ball over the top and then scoring at the near post.”
McGinn also believes Hibs should have had a late penalty but conceded it was hard for the officials to see in real-time.
"It all happens at 100 miles per hour so it's hard for him as there's so much going on. We had a chance right before that so there's a lot going on.”
McGinn was brought to the club by Jack Ross, who was relieved of his duties earlier this month, and the former St Mirren man admits the players feel guilty for the departure of Ross and No.2 John Potter.
"They were very popular and we feel we are to blame because of our run in the league,” he said.
“There's a bit of luck as well, as there were games where we should have won. The club has decided to go in a different direction and we just need to get going and once the new manager comes in we will all have a fresh start.
"It's been really tough. It was a real shock, as it was probably the first bad run since they arrived. Dave [Gray] has been really good at getting us going and keeping us going. He's definitely got a bright future in the game off the field.”