this post was submitted on 22 Aug 2024
3 points (80.0% liked)

Pulse of Truth

493 readers
67 users here now

Cyber Security news and links to cyber security stories that could make you go hmmm. The content is exactly as it is consumed through RSS feeds and wont be edited (except for the occasional encoding errors).

This community is automagically fed by an instance of Dittybopper.

founded 1 year ago
MODERATORS
 

For a whille now, I have seen scans that contain the pattern "%%target%%" in the URL. For example, today this particular URL is popular:

top 4 comments
sorted by: hot top controversial new old
[–] drkt@lemmy.dbzer0.com 1 points 2 months ago (1 children)

I have a robots.txt which does nothing but exclude "GPTBot"

I was coincidentally looking at my logs for unrelated reasons and caught it reading my robots.txt... and then promptly ignored it and scraped my whole site. Like yeah okay cool man

[–] aviation_hydrated@infosec.pub 1 points 2 months ago (2 children)

How does Reddit block users? Just by headers and IP addresses? Could the same be done once the GPTbot headers are known?

[–] drkt@lemmy.dbzer0.com 4 points 2 months ago* (last edited 2 months ago)

Every bit of information being sent to your web server can be spoofed. There is nothing you can do about this unless you're willing to exclude an increasing percentage of real users.

My webserver is constantly barraged by crawlers and bots because I have zero defenses. I've considered intercepting the obvious ones, like the ones targeting wordpress plugins. I don't use wordpress. I could serve them a 200 instead of a 404 and hopefully waste a real humans time if they check the hits manually.

[–] GBU_28@lemm.ee 1 points 2 months ago

Header spoofing is scraping 101