Memes

Or you shedding some of your microbiome's bacteria and fungi into the environment and whoops: they outcompeted something "local" and now whole species change.

I honestly don't think there'd be any way to avoid doing something that could possibly change the future in a dramatic way, because that far back incredibly minute changes could possibly lead to huge differences (because chaos theory), to the level of "a butterfly didn't flap its wings because I accidentally squashed it with my time machine, and now humanity never happened. Oops." But any change that means you didn't ever go on your trip means you have some sort of paradox on your hands, and then it becomes a question of how timelines work