However, as I understand it, at times the only real way to protect the community in the society we currently live in seems to be talking with the pigs.

I think this is true in some situations and imo its okay to call the cops in a situation that not only endangers the community but also oneself. Like people trying to physically harm you and you don't have any safer alternatives to protect yourself. Sadly many people are not part of communities that enable them to fully skip every interaction with cops.

To your serial killer example: If in your hypothetical situation there is a key benefit in turning the evidence over to the police in comparison to just publishing it so everyone(including the police)/only your local community can access the evidence, I dont see why not. Its also not something I would call "snitching".

In an anarchistic society the serial killer could be sent to the psych ward and dealt with humanely.

Also I just want to mention, I think this is not something with a consensus in anarchist circles.

I have had many encounters with cops, and I decide about the extent of cooperating with them on a case-by-case basis.

• landlord is illegally evicting my mother's neighbour --> I call cops and they prove how useless they are at prevention, but the matter goes to court and the landlord gets convicted later, and it was my only time to testify in court
• cops accuse me of ignoring their lawful order --> sorry, I was listening to music, didn't hear nothing, no comment, no comment, I admit nothing (nothing came of it)
• cops intrude into the squat yard without knowing it's a squat --> "the ~~droids~~ drunken people you look for are elsewhere" --> the cops go elsewhere
• cops raid the squat --> refuse to provide documents until threatened, require cops to provide their own ID, contest every statement and discuss the matter publicly in media
• cops try to steal equipment during a demonstration --> pull the equipment back and yell to them a description of what it is (I assume they thought I was planting a bomb instead of packing up)
• cops want to interview me about illegal demonstrations --> I politely tell them to fuck off, then call back and volunteer for the interview to convey the opinion of other anarchists :D
• a new squat is being established --> establish a security perimeter that is watched with attention and never let cops close
• an attempted squat gets burglarized and set on fire -> inform the fire brigade that a bottle of propane could be present (fortunately it was stolen), the fire brigade had better things to do than involve cops
• a new squat gets burglarized --> pepper spray the burglar and take their tools, without involving cops
• a new squat gets burglarized, episode N --> threaten the burglars and take their tools, without involving cops
• cops try to fall into a hole in ground during a stupid training excercise --> tell the cops not to go there, as they might fall in (leave untold: it would be a major embarrasment for squatters to rescue them)
• the squat is suddenly in the security perimeter of a NATO summit --> find some military lurking in the yard and invite them into the squat so they could be reasonably certain we don't have anything that shoots down planes :P (runway was about 250 m away)
• a drunken person tries to SWAT me at a street party --> fully explain the situation to the SWAT team and later participate in amateur theatre with cops to get the drunken person safely removed from the police station :o
• one drunken neighbour hits their spouse and when I forbid, hits me --> seeing that the neigbour has already paid for his deed since pepper was 100% effective and he'll feel extremely bad for many hours, I did not file a complaint to cops, although they were called and showed up
• after two geniuses tried to steal my car, but fled after a warning shot --> I did not involve cops
• after some person attacked his partner and hit her on street --> I pepper sprayed him, and since he took out a knife and attempted to come at me (I evaded, no harm occurred to me), I did call cops and make a complaint, as did the woman he had hit
• cops call me about one neigbour's car --> I don't remember anything (I did actually remember, but wasn't in a mood for helping them repress a neighbour)
• my car gets burglarized --> I ask the cops for info, they have none, I don't involve them beyond that

...etc.

In an anarchistic society the serial killer could be sent to the psych ward and dealt with humanely.

I suspect, in an anarchistic society, serial killers would be killed in turn by the victims' friends and relatives, and the rest of society would shrug and say "murdering people is wrong but in this case we can't really blame them".

Anarchists aren't necessarily pacifists, after all.

Really, if you ask yourself "what would happen to someone in an anarchist society who killed a serial killer/rapist/molester/etc etc in revenge" and the answer is "little or nothing" you probably have your answer to how that society would handle serial killers, rapists, molesters, etc.

also, what kind of an "anarchist society"?

put 10 anarchists together and you would have 10 different visions of an "anarchist society"

What makes you think that a "psych ward" would be an acceptable solution in an anarchist society?

confinement would still be involuntary and an imposition.

you already have your response, if you call the police "pigs" and alerting "snitching".

Remember kids, if you see someone brutally murder an innocent person, no you didn't.