Port Erin Commissioners have applied to have the Cosy Nook cafe building stripped of its protected status.
The building was placed on the buildings register just a few months ago, last April.
That means the site is now officially recognised as having historical significance and is legally protected from demolition.
The move was welcomed by heritage groups on the island and the government said it would preserve the site for future generations.
However, in a recent planning application town commissioners say the building was registered without a full inspection.
They claim the building is in a poor state and has no protection from an "increased risk of flooding".
Emergency services dealing with ‘unknown substance’ on beach
Archallagan hosts orienteering race
Your guide to Boxing Day traditions
Have you taken the binman's tip this Xmas?
Ramsey Domino's delayed until next year
No timeline for rights of way at nature trails
Manx driving licence scam on Facebook
Government reminder to clear MiCard balances soon