mirror of
https://github.com/cloudflare/cloudflare-docs.git
synced 2026-01-11 20:06:58 +00:00
07b6134588
* Update robots.txt * Apply suggestion from @GregBrimble Co-authored-by: Greg Brimble <gbrimble@cloudflare.com> * move below user agent --------- Co-authored-by: Greg Brimble <gbrimble@cloudflare.com>
40 lines
1.4 KiB
Text
40 lines
1.4 KiB
Text
# As a condition of accessing this website, you agree to
|
|
# abide by the following content signals:
|
|
|
|
# (a) If a content-signal = yes, you may collect content
|
|
# for the corresponding use.
|
|
# (b) If a content-signal = no, you may not collect content
|
|
# for the corresponding use.
|
|
# (c) If the website operator does not include a content
|
|
# signal for a corresponding use, the website operator
|
|
# neither grants nor restricts permission via content signal
|
|
# with respect to the corresponding use.
|
|
|
|
# The content signals and their meanings are:
|
|
|
|
# search: building a search index and providing search
|
|
# results (e.g., returning hyperlinks and short excerpts
|
|
# from your website's contents). Search does not include
|
|
# providing AI-generated search summaries.
|
|
# ai-input: inputting content into one or more AI models
|
|
# (e.g., retrieval augmented generation, grounding, or other
|
|
# real-time taking of content for generative AI search
|
|
# answers).
|
|
# ai-train: training or fine-tuning AI models.
|
|
|
|
# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS
|
|
# RESERVATIONS OF RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN
|
|
# UNION DIRECTIVE 2019/790 ON COPYRIGHT AND RELATED RIGHTS
|
|
# IN THE DIGITAL SINGLE MARKET.
|
|
|
|
User-agent: *
|
|
Content-Signal: ai-train=yes, search=yes, ai-input=yes
|
|
|
|
Allow: /
|
|
Disallow: /client-ip-geolocation
|
|
Disallow: /plans/
|
|
Disallow: /constellation
|
|
Disallow: /cdn-cgi/
|
|
Disallow: /email-security/
|
|
|
|
Sitemap: https://developers.cloudflare.com/sitemap-index.xml
|