Yes, third-parties may attack you legally on the grounds of non-use. Avoid this by using your registered trademark within 3 years. Take note also that formal application for cancellation of trademark due to non-use cannot be filed until 5 years have already passed from registration date.