According to the department, thousands of people still have late-filing penalties of £1,300 or more because they did not complete their Self Assessment form on time.
Letters are now being sent to these people as HMRC looks to recoup the amount of tax it should have received, with debt collectors seen as a last resort.
A spokesman from HMRC is quoted by the BBC as saying: "These non-filers have ignored numerous communications from HMRC, dating back to April 2011, including flyers, reminder letters, penalty notices and warning letters.
"A customer can still phone us if they think they should not be in self-assessment, and will be taken through a number of questions to indicate if they should be in self-assessment or not."
It comes after HMRC urged taxpayers to apply for online Self Assessment by HMRC for the 2011-12 tax year, with the deadline for online filing on January 31st.
Posted by Thomas Fletcher