There is a reason why smells are so closely related to memories. In the brain, the part that is for smell recognition is right near the hippocampus, which creates and stores memories. So the fact that they are in such close vicinity to each other creates these "smell memories" as you call them. It really is an amazing thing
But sometimes it makes you want to cry. Especially when every time you turn around something you smell reminds you of someone you don't want to be reminded of anymore
