Firstly, start by visiting Open Site explorer, and search your websites backlinks manually. If the site that links to you looks dodgy, you can usually tell by just looking at it. If it is off topic, zero-page rank, or looks bad it will give you negligible positive at best. The thing is, that most of the time, the site linking to you might not be doing it for legitimate reasons, and so it will really be hard to get them to remove these links.
There are paid backlink finders such as Link Dtox. This tool lets you do an in-depth link audit of your website, although it is not free, it is one of the best out there. You could also use Google Webmaster “Links to your Website”, and the Google Disavow tool.
This tool allows you to have a look at all websites that link to your website, at domain level, and page level. Although fairly accurate, you need to manually review all backlinks to decide which appear to be beneficial, and which are harmful to your website. Once you have selected all unhealthy domains that link to your website unnaturally, you will then need to tell Google that you do not wish to associate your domain with these.
You would then need to build a txt file containing all harmful domains, and submit this to Google via the Google Disavow tool. This is an advanced SEO tool, and one should take precaution when removing any links to a domain. This could either positively or, if done incorrectly, negatively affect the organic performance of your website.
Sometime, these links are built unnaturally by the SEO specialist that you have hired, in the attempt of gaining authority on SERPs quickly, which violate Googles quality guidelines. Removing spammy, or low-quality links is the first step you should take when optimising your website.