I have some annoying plugins that I wanted to remove from Safari but under Preferences > Security > Plugins you can only manage the options for each plugin, you are not able to remove them from this windows. Do the following to remove Safari plugins:
1. In Safari open Help > Installed Plugins.
2. Look for the plugin you want to remove, at the end should be something like “from file “xxxxx.plugin”, thats the name of the file you want to delete.
3. Go to finder and click on Go > Go to folder or shift+cmd+g, enter /Library/Internet Plug-Ins
4. Look for the file of the plugin you want to remove and delete it, thats it, you just removed the plugin from Safari.