This scheme isn't that different from Medieval Icelandic society, which some authors have called anarchy -- and which I think could accurately be described as proto-anarchy. Medieval Icelandic courts had no judges, only juries. Serious crimes were punished with exile. The most serious crimes were punished with redistribution of their possessions and home by the clan council and permanent exile. If they hadn't left the Island before the end of a grace period, they existed outside the protection of the social contract, and this is where the word 'útlagr' or 'outlaw' comes from.

I don't know how good the scholarship on this is, I just found it from a cursory search:

Ordered Anarchy: Evolution of the Decentralized Legal Order in the Icelandic Commonwealth

The Enforcement of Law and Penalties for Defection

All penalty was either in the form of restitution or fines.21 Restitution, or Útlegð, was used for lesser offenses, while fines were demanded in more serious cases. Fines, or Sekt, were either sentences of lesser outlawry, Fjörbaugsgarður, or greater outlawry, Skóggangur. A person sentenced to lesser outlawry was required to leave the country, the protection of "Our Law," for three years. Someone sentenced to greater outlawry was to leave the country permanently, and could be rightfully killed after three months. Both types of outlaws lost their property, which was distributed by the Féránsdómur. Only the guilty person's property, not that of his wife or other family members, could be confiscated, as long as the family member could show legitimate ownership.

Enforcement of judgments was private, in that the victim was responsible for enforcing a judgement in his favour. In most cases the law specified when payment of a judgement should take place, and failure to pay on time was itself a criminal offense. To make the system more effective, the payment of a judgment usually required witnesses or consultation with the aggressor's chieftain, and, in addition, the victim could sell his judgement to someone stronger than himself.

If the property of the outlaw was valued at more than the victim had a right to, complicated rules governed the distribution of what remained. The distribution of the remainder was so arranged as to provide incentives, usually monetary incentives, for others in society to see that the enforcement of the judgement was carried out. It seems that the Icelanders were keen not only on compromise in major disputes on what the law was, or should be, but in disputes between individuals compromise was also common. According to MILLER [1984:99], despite having "had a complex court structure, most disputes did not lead to adjudicated outcomes." In customary legal systems, this preference for compromise is widespread.

I don't advocate for a return to medieval practices, but it should be noted that the system was both relatively efficacious and much less cruel than the systems of justice popular on the European continent at the time.

This reminds me of the online game Tibia. If you attacked or killed another player, you got a skull over your character and if someone attacked/killed you while you had it, they wouldn't get a skull. That was it. It worked pretty well for as simple as it was, acting basically as a both warning and permission. Granted, guilt was determined by the all-knowing, impartial, moderation of the game engine, but it stuck with me. I don't know if I thought much about how similar it is to the old outlaw frameworks.

I suppose it also had the problem where if the outlaw is dangerous enough or has enough dangerous friends, the community's monopoly on violence falls into question. (We sometimes saw groups of high level players with red skulls walking right into town). It also depends a bit on society having people who want to personally do something to the outlaw.