Alan Trouten’s penalty deep in extra-time took the part-time Clackmannanshire outfit through with a surprise result at the Indodrill Stadium.
Hearts enjoyed large spells of possession but could not find a way past Neil Parry in the home goal.
Neilson felt the penalty – awarded for a foul by Jamie Walker on Alloa substitute Robert Thomson – was the wrong call by referee Gavin Duncan.
“We dominated the game, we should have won it by two or three goals at least, but we didn’t take the chances and got done at the end by a bad decision,” said Neilson.
‘I didn’t think there was contact at the time, I've seen the replay and there’s none at all. You'd hope you'd get a referee who’d see that but it ends up being the pivotal moment in the game.
“The most frustrating thing is we shouldn’t have been in that position in the first place. We dominated the play.
“I always felt we could get a goal the longer it went, and we should have scored, the amount of chances we had, but that’s why we didn’t progress.
“We have to go away and look at the final moment.”
The referee issued Neilson a yellow card for his protests at the end of the match. “I spoke to the ref but at the end of the fay he’s made a bad decision,” added the manager.
“I got a yellow card for trying to speak to him. Some refs you can speak to, he’s obviously one you can’t. I didn’t swear at him or anything but some refs just won’t take people speaking to them
“When he looks back he’ll realise he’s made a bad decision and hopefully he learns from it.”
Peter Grant, the Alloa manager, was understandably delighted as his club celebrated their first competitive win against Hearts. “We thoroughly deserved it over the piece,” he said.
“People say we don’t score goals but we done that and kept a clean sheet so what more can I ask for?”