A row over the use of a protected woodland as a private garden is being taken to the government's Planning Inspectorate.