If you do not co-operate in trying to arrange a visit, HMRC have the power to arrange an unannounced visit, so this may only defer the problem; not eliminate it.
Whether you accept their proposed arrangement for their visit or they turn up unannounced, they cannot force entry on to your premises. So you are entitled to refuse entry, although the inspector will try to establish why and they will try to re-arrange another appointment with you.
In either case, you are entitled to suggest an alternative venue, such as your accountant’s or a HMRC office.
During the course of an inspection, you are also entitled to ask them to leave, but again, they will note down the circumstances and ask you for a reason, and will want to arrange another time for them to complete their inspection.
Note, if you never used your home for business purposes, they would have had no right of entry whatsoever to your home- unless you offered that venue to them.
Your local TaxAssist Accountant can help you to prepare for the visit and attend, so please feel free to contact them if this would interest you.