The warning came as transport secretary Michael Matheson confirmed the public sector could bid for the next franchise from 2025, and that CalMac owner David MacBrayne was interested in bidding.
The Scottish Government-owned firm told MSPs in May it would consider a bid, while ministers have had the powers to permit a public sector bid since 2016.
Mr Matheson said today: "David MacBrayne Limited has confirmed its interest in exploring the possibility of a bid for a Scottish rail franchise contract."
ScotRail is the Scottish Government's biggest single contract, worth about Â£200 million a year.
However, the franchise is very tightly specified, with current operator Abellio bound by a detailed contract including timetables and fares.
This is likely to continue if a public sector bid won the competition for the next franchise.
Scottish Conservatives transport spokesman Jamie Greene said: “A public sector operator taking control of the ScotRail franchise shifts huge risk and potentially millions of pounds onto the shoulders of the Scottish taxpayer without any guarantee of an improvement in the quality of service offered.
“The Scottish Government must ensure any potential additional cost to the Scottish taxpayer, including that of putting together a bid, must be clearly outlined to the public before any bidding takes place.
“Passengers will rightly have reservations over David MacBrayne Ltd running the railways given the chronic state of publicly-run ferry services across the country.
"Costs are skyrocketing, the fleet is ageing and services are either frequently disrupted or cancelled altogether.
"It’s important that our railways don’t suffer the same fate as our ferries."
However, Scottish Labour called for full nationalisation rather than a public sector franchise.
Transport spokesman Colin Smyth said: “Instead of trying to save the current franchise until 2025 and dragging their heels over a public sector bid, the SNP government should use the growing evidence of poor performance to end the contract sooner rather than later.
“When the SNP handed the franchise to run Scotland’s railways to Abellio, they said it would be a 'world leading' deal, but the reality for Scotland’s hard-pressed passengers has been anything but with hikes in fares, overcrowding, delays and stop skipping.
“Under Labour’s plans, franchising would be scrapped and our railways would be brought under full public ownership, so we have a railway system for the many, not the few."
Rail union TSSA called for a public body should be directly appointed instead to run ScotRail.
General secretary Manuel Cortes said, “The franchise process means passengers pay more and get less.”
A spokesman for David MacBrayne said: “The Scottish Government has indicated its desire that a public sector bid is included when the franchise goes out to tender, and we have indicated that we would be open to a discussion, when the time is right, on the shape of a future rail franchise contract and what this might entail.
"There have been no further discussions in this regard, and with the current ScotRail franchise due to be in place until 2025, we do not expect these to take place for some time.”Cabinet
Mr Matheson said: "Our consistent view over many years is that there ought to be a level playing field between the private and public sector in bidding for rail franchises.
"We secured the right for a public sector operator to bid for a rail franchise in Scotland, after being denied by previous UK governments.
"Identifying a public sector body to make a bid has not been easy.
"It was vital to consider a range of scenarios as we want to strengthen competition and achieve best value for the public purse, while delivering continuous service improvements.
"It is important for any public sector bidder to have the appetite to bid; the capability and capacity to see the bid through; and the ability and resources to make the franchise a success."
The Scottish Government's Transport Scotland agency, which manages the franchise, added: "Future Scottish rail franchise competitions will continue to be fair and transparent, giving equal treatment to all bidders whether from the public or private sectors."