Port Erin Commissioners have applied to have the Cosy Nook cafe building stripped of its protected status.
The building was placed on the buildings register just a few months ago, last April.
That means the site is now officially recognised as having historical significance and is legally protected from demolition.
The move was welcomed by heritage groups on the island and the government said it would preserve the site for future generations.
However, in a recent planning application town commissioners say the building was registered without a full inspection.
They claim the building is in a poor state and has no protection from an "increased risk of flooding".