If you're passing them off as just regular people nbd in that setting then yeah that'd be inaccurate.

Then again, plopping in random white people into an Ancient Chinese setting would be pretty inaccurate too, even though there might've been "some non zero number" of whites over there at the time. Or in a random crowd shot of Nazi soldiers you plop in a few black soldiers. Certainly existed, but while funny it does make it seem inaccurate (and silly imo).

"Most historical settings"

Roman sure, especially as you get closer to Africa but nonzero elsewhere also

Middle ages, mediæval and renaissance almost certainly limited to higher nobility households either as nobles or "interesting" servants or major trading ports, especially closer to Africa.

The chances of a mediæval serf in a germanic country not looking northern Europe, or Mediterranean at a huge stretch, are functionally zero though, as anyone who came with the Romans will have been long dead with their genetics widely dispersed, and anyone who came over recently would likely be in an urban area, with marriage or higher level employment being their only chance to end up in a rural area.

I'm saying that depicting black people as a normal feature of Medieval Europe would be huge stretch. Whether that should stop people from doing so, I don't really care about that. Accuracy isn't exactly the only thing to consider in such situations.