You can't really enforce it, but in smaller community sizes self-enforcement can occur through a community culture that's self-reinforcing. This is typically done through repeated reminders and a constant back and forth between community management and the membership, often through a regular post of sorts.
There is a tipping point in size where that stops working too though but it's somewhere in the 40k-100k users range.
In my experience the only thing that removing downvotes changed in sorting was that it prevents users from suppressing the extremely shit posts. The sorting of the middle and high end of both comments and posts ends up almost identical.
The worst posts that would have received a tonne of downvotes are also almost always something that moderation would agree with removing entirely.
This tradeoff comes with the benefits of eliminating all the psychological issues that occur.
BUT, Lemmy can't go down this road imo as it would no longer be a reddit clone if it didn't have the downvote. My general opinion is that the downvote hurts conversation, creates circlejerks, hurts the way people see one another, and creates a mindset of writing a comment for the audience to "win" the interaction instead of writing a comment for the recipient. But the decisions about Lemmy should be about what's best for Lemmy's future as a tool though which is probably best to maintain the "reddit clone" selling point for the timebeing.