I get this! For me it usually sounds like someone stomping on the floor upstairs...but I live on the top floor. Pretty much always falling asleep for a nap, rarely at bedtime.
A funny anecdote: one time it sounded like someone let out a huge fart right next to my head. The big BRRRAAAAPPPP made me laugh so hard, I completely lost my desire to nap