Harvey: The negativity at Berwick Rangers is unbelievable
The former Penicuik Athletic boss was critical of the "abuse" he received from the Berwick fans and the general atmosphere around the club.
He urged the supporters to back him for the last three crucial games as they look to avoid a relegation play-off.
Harvey said: "The abuse I'm getting as the manager of Berwick, I'm a young manager who has earned the right to be at this club in my opinion because of what I have done and what I can do.
"I'm here to do what I can with the best of my ability for Berwick Rangers and the supporters are getting on my back.
"Until someone upstairs tells me that I'm not doing this job, I'll be working off my proverbials and I'll be spending more time away from my family - just like the last manager did.
"He gave everything for the club and had to end up walking away because ... the negativity around here is unbelievable, that's maybe a bit harsh but it is the reality.
"The club is in a situation where it can't compete financially with other clubs. You have got a manager who is determined and working hard and looking at all avenues next season trying to improve it.
"I can't stop crosses, I can't chuck myself in front of crosses and it's my fault."
He added: "We're going try our best in training as we do every week and go into Clyde with a bit of optimism.
"That's exactly what I'm going to do until someone tells me that I'm not the man for the job but good luck to the next guy because it is a very difficult job and you won't get somebody that works as hard as I do or my coaching staff.
"You can tell I'm annoyed and not just with the players."