@@ -113,32 +113,34 @@ of the information availble through our API.
113
113
If the index does not have the information you need, we're also happy to
114
114
discuss solutions to your needs that don't require you to crawl the registry.
115
115
You can email us at <
a href =" mailto:[email protected] " >
[email protected] </
a >.
116
+ </p >
116
117
118
+ <p >
117
119
We allow our API and website to be crawled by commercial crawlers such as
118
120
GoogleBot. At our discretion, we may choose to allow access to experimental
119
121
crawlers, as long as they limit their request rate to 1 request per second or
120
122
less.
123
+ </p >
121
124
125
+ <p >
122
126
We also require all crawlers to provide a user-agent header that allows us to
123
127
uniquely identify your bot. This allows us to more accurately monitor any
124
128
impact your bot may have on our service. Providing a user agent that only
125
- identifies your HTTP client library (such as "request/0.9.1") increases the
129
+ identifies your HTTP client library (such as "< code > request/0.9.1</ code > ") increases the
126
130
likelihood that we will block your traffic.
127
131
128
132
It is recommended, but not required, to include contact information in your user
129
133
agent. This allows us to contact you if we would like a change in your bot's
130
134
behavior without having to block your traffic.
135
+ </p >
131
136
132
- Bad:
133
- User-Agent: reqwest/0.9.1
134
-
135
- Better:
136
- User-Agent: my_bot
137
-
138
- Best:
139
- User-Agent: my_bot (my_bot.com/info)
140
- User-Agent: my_bot (help@my_bot.com)
137
+ <p >
138
+ Bad: "<code >User-Agent: reqwest/0.9.1</code >"<br >
139
+ Better: "<code >User-Agent: my_bot</code >"<br >
140
+ Best: "<code >User-Agent: my_bot (my_bot.com/info)</code >" or "<code >User-Agent: my_bot (help@my_bot.com)</code >"
141
+ </p >
141
142
143
+ <p >
142
144
We reserve the right to block traffic from any bot that we determine to be in
143
145
violation of this policy or causing an impact on the integrity of our service.
144
146
</p >
0 commit comments