After South Korean Portal Daum Announced Blocking of AI Crawlers, Subdomains Hosting News Remain Fully Open
Lee Sung-kyu, co-founder of Bluedot Intelligence, found through HTTP testing and robots.txt checks that while Daum set AI crawler blocking rules for the main domains www.daum.net and daum.net on October 7, the subdomains actually hosting news articles, v.daum.net and news.daum.net, have no robots.txt files and do not block crawlers like GPTBot, ClaudeBot, or PerplexityBot at the HTTP level, leaving news content freely accessible to AI search engines.
ImportanceMaterialEvidenceE3 inspectableWrite-upStandard
The South Korean portal Daum updated its robots.txt rules on October 7, announcing a block on AI crawler access to news content. Lee Sung-kyu, Co-founder and CEO of Bluedot Intelligence, subsequently conducted two tests, concluding that the block only covered the main domain, while the actual paths hosting news remained fully open.
The key lies in the scope rules of robots.txt. According to RFC 9309 published by the Internet Engineering Task Force (IETF), robots.txt applies per host rather than per domain. Although www.daum.net ↗, v.daum.net, and news.daum.net share the top-level domain daum.net, they are distinct hosts. Lee Sung-kyu found that no robots.txt files exist on v.daum.net and news.daum.net. Under RFC 9309, when a crawler requests robots.txt and receives a 404 response, all resources on that server are considered accessible. He further directly requested news pages using six mainstream AI crawler User-Agent identifiers, including GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot; all returned HTTP 200 status codes, indicating no interception at the HTTP level.
This means that South Korean media outlets contributing content to Daum should continuously demand that Daum deploy separate robots.txt files on v.daum.net and news.daum.net to block AI crawlers. Lee Sung-kyu pointed out that if these requirements are not met, measures taken by media outlets to independently block AI learning bots on their own websites will have no practical effect, as AI search companies will still be able to freely use their news content without permission.