cybersecurity

3249 readers

4 users here now

An umbrella community for all things cybersecurity / infosec. News, research, questions, are all welcome!

Community Rules

Be kind
Limit promotional activities
Non-cybersecurity posts should be redirected to other communities within infosec.pub.

Enjoy!

founded 1 year ago

MODERATORS

shellsharks@infosec.pub

tweedge@infosec.pub

What are You Working on Wednesday (infosec.pub)

submitted 7 months ago by shellsharks@infosec.pub to c/cybersecurity@infosec.pub

7 comments fedilink hide all child comments

Weekly thread to discuss whatever you’re working on, big or small, at work or in your free time.

you are viewing a single comment's thread
view the rest of the comments

[–] dubbel@discuss.tchncs.de 4 points 7 months ago (1 children)

Private project, not really security related: Crawling robots.txts to gather some statistics on which bots people are most often excluding - weirdly I couldn't find any recent/regularly updated stats on this.

[–] PaddleMaster@beehaw.org 1 points 7 months ago (1 children)

That’s a neat project. Are you looking for trends, or something specific?

[–] dubbel@discuss.tchncs.de 3 points 7 months ago (1 children)

It started with a popular mastodon posts on how to block openai crawlers I think, and I'd like to know whether people are actually implementing it.

[–] PaddleMaster@beehaw.org 1 points 7 months ago

That’s neat. I’m curious about this now. With “normal” search engines that have generally gone to shit, AI chat bots are on trend to give better results. If the robots.txt file is blocked from OpenAI, can I assume it hits other chatbots? And would that extend to Google/bing?