Paramedics rushed him to the Royal Infirmary of Edinburgh after he was spotted near to the Royal Yacht Britannia, in Leith, on Wednesday at around 10pm.
He later died and a spokesman for Police Scotland said inquiries in to the circumstances of his death were ongoing.
He said: “Police in Edinburgh responded to Ocean Terminal at around 10pm on Wednesday, July 12 after an elderly man was seen to enter the water near to the Royal Yacht Britannia.
“The male was removed from the water and taken to the Royal Infirmary of Edinburgh where he sadly passed away. Inquiries into the full circumstances surrounding this incident are ongoing.”