Former Falkirk and Inverness striker Austin had given the Fife men a deserved first-half lead, heading back across goal and into the far corner from a Chris Dodd cross in the 15th minute.
However, Strollers fought back and levelled from the penalty spot just before the hour mark through Alieau Faye, Ryan Shanley having gone down under the challenge from Arnault Bembo.
The hosts looked the more likely to go on and win the tie in 90 minutes, but it was Barry Ferguson’s side who sealed their place in the fourth round when Austin pounced right at the death of extra time.
Strollers boss Gary Jardine said: “I thought Kelty were the better side in the first half. We changed the shape and asked for a response in the second half and I thought we were excellent and were unfortunate not to win in 90 minutes.
“My players gave me everything they had and deserved more for their efforts. Football can be cruel at times.”
Elsewhere, Spartans also exited the competition after a narrow 3-2 defeat to Stirling University at Ainslie Park.
Scott Maxwell struck early to give Dougie Samuel’s outfit the lead, but the two-time Lowland League champions were pegged back and trailed 2-1 before Mikey Allan levelled the tie up. However Jason Jarvis’ superb free-kick in the dying embers of the match won it for the students.
Current Lowland League leaders Bonnyrigg Rose were comfortable 3-0 winners over visitors Threave Rovers at New Dundas Park, Neil Martyniuk, Scott Gray and George Hunter all on target for Robbie Horn’s side.
East of Scotland Premier Division club Penicuik were knocked out 3-2 by Hill of Beath, Regan Lumsden with the winner after the visitors had raced into a 3-0 lead.
Meanwhile, Tranent registered an impressive 3-0 win at Upper Annandale to seal their place in the next round.
Mark McGovern opened the scoring in the Borders before former Livingston defender Shaun Rutherford doubled his side’s advantage. Bob Berry capped off a fine afternoon for Calvin Shand’s men with a third.
Premier Division: Sauchie 1-1 Whitehill Welfare.
Conference A: Lothian Thistle HV 2-1 Haddington Athletic, St Andrews United 8-0 Peebles Rovers, Leith Athletic 1-1 Burntisland Shipyard.
Conference B: Arniston Rangers 1-5 Tynecastle, Coldstream 3-4 Edinburgh United, Stirling University (EOSFL) 5-2 Hawick RA, Tweedmouth Rangers 0-2 Dalkeith Thistle.
South Challenge Cup third round): Upper Annandale 0-3 Tranent, Blackburn United 1-2 East Stirlingshire, Camelon 4-1 Heriot-Watt Uni, Craigroyston 0-6 Preston Athletic, Dunbar 1-2 Broxburn, Dunipace 3-1 Glenrothes , Easthouses Lily 0-4 East Kilbride, Penicuik Athletic 2-3 Hill of Beath, Bonnyrigg Rose 3-0 Threave Rovers, BSC Glasgow 3-1 Gala Fairydean Rovers, Caledonian Braves 5-3 Musselburgh Athletic, Civil Service Strollers 1-2 Kelty Hearts, Cumbernauld Colts 1-0 Jeanfield Swifts, Heston Rovers 0-6 Linlithgow Rose, Nithsdale Wanderers 1-4 Bonnyton Thistle, Spartans 2-3 Stirling University.