But he will remain at Tynecastle until the end of the season when his contract expires.
He will then sever links with the club, stepping down from the board.
The club confirmed the news in a statement issued just after 4pm on Thursday.
Hearts have won just one league match all season and the 1-0 defeat by St Johnstone on Wednesday night saw them slide to joint bottom of the Premiership.
The Tynecastle side are due to play Rangers in the Betfred Cup semi-final at Hampden on Sunday.
Austin MacPhee, Levein's assistant, will take charge of the side.
The Hearts statement said: "The Board of Directors of Heart of Midlothian Football Club has today taken the decision to relieve Craig Levein of his first team managerial duties.
"The Board would like to thank Craig for all of his efforts as first team manager. Craig has worked tirelessly and selflessly, around the clock, to try to achieve success, despite the many challenges with which he has been faced.
"Craig will not return to his previous Director of Football role and, that being so, will step down from the Board. He will, however, see out his contract, which runs until the end of the season, working closely with the Executive Management Team in continuing to develop and improve the structure of the backroom and youth operations within the development.
"We are confident that his knowledge and experience will contribute massively to ensuring that, come the end of the season, he will leave the football department in a healthy and thriving state.
"Effective immediately, Austin MacPhee will be in charge of first team affairs, on an interim basis, while the Board begins the process of recruiting a new manager. The new manager, and indeed the interim manager, will report directly to the Chief Executive."