The Scottish Labour leader also gave his full backing to implement national legislation around anti-abortion buffer zones as he said any delay to this is “unnecessary”.
Set against the backdrop of the Riverside Museum, Mr Sarwar led a Labour campaign event in Glasgow on Monday morning ahead of the local council elections on Thursday.
His comments come with misogynistic and sexist claims rife in Westminster, with Conservative MP Neil Parish recently standing down after he admitted to watching porn twice in the Commons.
A UK Government minister has since rejected calls for an all-women Tory shortlist for the by-election to replace the disgraced MP.
Asked whether his party has an issue with misogyny, Mr Sarwar, who is currently celebrating Eid, said: “You’ve seen individual cases over the last few weeks of misogyny and sexism and they are focused on the Conservative party.
"But when I talk about, for example, Islamophobia, or I talk about anti-Semitism, racism or homophobia or of sexism, we can’t pretend they are limited to one political ideology or one political party.
"I don’t think there is any organisation or any institution anywhere in the UK that is immune to prejudice and hate and I think we all need to redouble our efforts to make sure we have a society which is fair for all.”
Asked if there were specific solutions to misogyny in Labour’s local manifestos, Mr Sarwar said he was “resolute” that all forms of prejudice, including misogyny, had to be “rooted out” of society.
Mr Sarwar said: "Far too often we see people talking about women having to think and change their behaviour and, frankly, no, it’s men who need to change their behaviour. It’s men who need to change the culture in order for us to tackle misogyny and sexism.
“We have a duty to making sure we are addressing that and confronting it.”
Mr Sarwar also addressed concerns around anti-abortion protesters as areas such as Glasgow experience a rise in protests outside hospitals and clinics.
Last month more than 100 anti-abortion activists protested outside the Queen Elizabeth University Hospital.
Nicola Sturgeon said last week there were “some complex legal issues involved” in implementing the 150m buffer zones that campaign group Back Off Scotland is calling for.
Ms Sturgeon said any laws which are passed need to “be compliant” with other areas such as the European Convention of Human Rights Act.
A draft member’s bill on buffer zones brought forward by Green MSP Gillian Mackay is expected to be published by mid this month. However, this could take over a year to implement
Labour MSP Monica Lennon has called for the Scottish Government to “urgently” introduce national legislation on implementing buffer zones around healthcare facilities so that women can access healthcare “without harassment”.
Backing Ms Lennon's calls, Mr Sarwar said: “Any delay is unnecessary. I think we’ve got broad political support, I think we have broad public support. The delay is now one for the Government and the minister and I think we should move quickly now to make this a reality.”