Skip to content

Commit d9122da

Browse files
Merge #1581
1581: Make crawler policy more readable r=sgrif a=IsaacWoods I found the wall of text about the crawler policy less than readable, so I added some paragraph breaks and `<code>` tags in an attempt to make it slightly more readable. This doesn't change any of the text of the policy. Co-authored-by: Isaac Woods <[email protected]>
2 parents f7fbc01 + 45e3578 commit d9122da

File tree

1 file changed

+12
-10
lines changed

1 file changed

+12
-10
lines changed

app/templates/policies.hbs

+12-10
Original file line numberDiff line numberDiff line change
@@ -113,32 +113,34 @@ of the information availble through our API.
113113
If the index does not have the information you need, we're also happy to
114114
discuss solutions to your needs that don't require you to crawl the registry.
115115
You can email us at <a href="mailto:[email protected]">[email protected]</a>.
116+
</p>
116117

118+
<p>
117119
We allow our API and website to be crawled by commercial crawlers such as
118120
GoogleBot. At our discretion, we may choose to allow access to experimental
119121
crawlers, as long as they limit their request rate to 1 request per second or
120122
less.
123+
</p>
121124

125+
<p>
122126
We also require all crawlers to provide a user-agent header that allows us to
123127
uniquely identify your bot. This allows us to more accurately monitor any
124128
impact your bot may have on our service. Providing a user agent that only
125-
identifies your HTTP client library (such as "request/0.9.1") increases the
129+
identifies your HTTP client library (such as "<code>request/0.9.1</code>") increases the
126130
likelihood that we will block your traffic.
127131

128132
It is recommended, but not required, to include contact information in your user
129133
agent. This allows us to contact you if we would like a change in your bot's
130134
behavior without having to block your traffic.
135+
</p>
131136

132-
Bad:
133-
User-Agent: reqwest/0.9.1
134-
135-
Better:
136-
User-Agent: my_bot
137-
138-
Best:
139-
User-Agent: my_bot (my_bot.com/info)
140-
User-Agent: my_bot (help@my_bot.com)
137+
<p>
138+
Bad: "<code>User-Agent: reqwest/0.9.1</code>"<br>
139+
Better: "<code>User-Agent: my_bot</code>"<br>
140+
Best: "<code>User-Agent: my_bot (my_bot.com/info)</code>" or "<code>User-Agent: my_bot (help@my_bot.com)</code>"
141+
</p>
141142

143+
<p>
142144
We reserve the right to block traffic from any bot that we determine to be in
143145
violation of this policy or causing an impact on the integrity of our service.
144146
</p>

0 commit comments

Comments
 (0)