This sounds a lot like my experience!

I've also found that I can sometimes de-smell plastic bickets with vinegar. I also tape (relatively fresh) bay leaves to the underside of the bucket lids to discourage moths. I haven't noticed any transfer of smell/taste.