The verdict from Chief Justice James Allsop came following a unanimous decision from the three judges hearing the case at the Federal Court of Australia on Sunday.
The decision means nine-time champion Djokovic will not be defending his title at the Australian Open, which starts on Monday, and is banned from Australia for three years - although that can be waived.
Djokovic had the option of trying to take the legal fight further but said in a statement he would cooperate with the deportation despite being "extremely disappointed" by the decision.
Having already spent five nights in the Park detention hotel, he wasted no time getting out of the country and was pictured accompanied by Australian Border Force officials at Melbourne's Tullamarine Airport before boarding an Emirates flight to Dubai that left at 10.30pm.
He said: "I would like to make a brief statement to address the outcomes of today's court hearing. I will now be taking some time to rest and to recuperate, before making any further comments beyond this.
"I am extremely disappointed with the ruling to dismiss my application for judicial review of the minister's decision to cancel my visa, which means I cannot stay in Australia and participate in the Australian Open.
"I respect the court's ruling and I'll cooperate with the relevant authorities in relation to my departure from the country.
"I am uncomfortable that the focus of the past weeks has been on me and I hope that we can all now focus on the game and tournament I love. I would like to wish the players, tournament officials, staff, volunteers and fans all the best for the tournament.
"Finally, I would like to thank my family, friends, team, supporters, fans and my fellow Serbians for your continued support. You have all been a great source of strength to me."
The three judges deliberated for a little over two hours before Chief Justice Allsop delivered the verdict just before 6pm in Melbourne having heard submissions from both parties earlier in the day.
Immigration Minister Alex Hawke re-cancelled Djokovic's visa on Friday using personal powers after the world number one had won an appeal against the initial decision to bar him from the country when he arrived last week.
Djokovic had been due to face compatriot Miomir Kecmanovic on Monday but has been replaced in the draw by Italian lucky loser Salvatore Caruso.
Hawke's decision on Friday was unexpectedly based not on the validity or otherwise of Djokovic's exemption from Covid-19 vaccination, which was the reason for the initial cancellation, but on the notion his presence in the country could stoke anti-vaccination sentiment, making him a danger to public health, as well as civil unrest.
To succeed in an appeal, Djokovic's legal team had to prove that Hawke had either acted outside his powers or that his decision was irrational, and Chief Justice Allsop made a point of stressing the judges were not able to assess the merits of the case.
Nick Wood, acting for the Serbian, focused on three aspects - that there was no evidence his presence would stoke anti-vaccination sentiment, that evidence was also lacking for the idea Djokovic opposes vaccination and that Hawke had not considered whether deporting the nine-time Australian Open champion would lead to increased support for the anti-vaccination cause.
Wood said: "Not a single line of evidence in the material provided any specific or logical foundation whatsoever that the mere presence of Mr Djokovic in Australia in itself may somehow foster anti-vaccination sentiment."
Hawke was represented by lawyer Stephen Lloyd, who spent his submission laying out the reasons why the minister's decision was rational.
On the issue of whether it was fair to present Djokovic as taking an anti-vaccination stance, Lloyd said: "His ongoing non-vaccination status is open to infer that a person in the applicant's position could have been vaccinated if he wanted to be.
"Even before vaccines were available he was against it - his prima facie position was to be against them."
He made reference to anti-vaccination groups "treating the applicant as a hero" as he moved on to Hawke's central claim of Djokovic's presence having the potential to negatively impact public health.
Lloyd said: "He's a high-profile person who is in many respects a role model for many people. His presence in Australia would present more strongly to Australians his anti-vaccination views.
"People use high-level athletes to promote ideas and causes all the time. His connection to a cause, whether he wants it or not, is still present."
The case has led the news agenda around the world and caused a political storm in both Australia and Serbia.
Hawke and Prime Minister Scott Morrison both greeted the verdict with satisfaction, with Morrison saying: "I welcome the decision to keep our borders strong and keep Australians safe.
"Strong borders are fundamental to the Australian way of life, as is the rule of law. Our government has always understood this and has been prepared to take the decisions and actions necessary to protect the integrity of our borders."
The reaction was very different in Serbia, where President Aleksandar Vucic said in a statement reported by Novosti: "I talked to Novak a while ago and I encouraged him and I told him that I can't wait for him to come to Serbia and return to his country, and to be where he is always welcome.
"They think that they humiliated Djokovic, but they humiliated themselves, and he can return to his country and look everyone in the eye with his head held high."
The saga leaves many questions for Tennis Australia, which pushed for exemptions to be available for players in Djokovic's position despite widespread public opposition.
In a brief statement the organisation said: "Tennis Australia respects the decision of the Federal Court. We look forward to a competitive and exciting Australian Open 2022 and wish all players the best of luck."
The ATP was stronger, describing what has played out over the last week and a half as a "deeply regrettable series of events" and said Djokovic's absence was "a loss for the game".
"We know how turbulent the recent days have been for Novak and how much he wanted to defend his title in Melbourne," said the governing body.
"We wish him well and look forward to seeing him back on court soon. ATP continues to strongly recommend vaccination to all players."
It remains to be seen where Djokovic will play next, with Australia far from the only country where he is likely to experience travel issues if he continues to refuse the vaccine.