Your robots.txt probably does not block the AI crawler you think it blocks

Recent reports highlight a critical robots.txt vulnerability where naming a specific bot, such as GPTBot, disables wildcard rules under RFC 9309, potentially exposing restricted directories. Meanwhile, discussions regarding AI slop in software development suggest that AI-generated landing pages and UI patterns are less problematic than perceived. Since most customers are not developers, they typically remain unaware of tools like Claude or Cursor, prioritizing whether the final product effectively solves their specific problems over the development method.