47:
BotSeer had indexed and analyzed 2.2 million robots.txt files obtained from 13.2 million websites, as well as a large Web server log of real-world robot behavior and related analysis. BotSeer's goals were to assist researchers, webmasters, web crawler developers and others with web robots related
35:
BotSeer served as a resource for studying the regulation and behavior of Web robots as well as information about the creation of effective robots.txt files and crawler implementations. It was publicly available on the World Wide Web at the
College of Information Sciences and Technology at the
43:
BotSeer provided services including robots.txt searching, robot bias analysis, and robot-generated log analysis. The prototype of BotSeer also allowed users to search 6,000 documentation files and source codes from 18 open source crawler projects.
48:
research and information needs. However, some people received BotSeer negatively, arguing that it contradicted the purpose of the robots.txt convention.
289:
183:
299:
86:
225:
204:
146:
164:
37:
69:
294:
20:
261:
23:
deployment and adherence. It was created and designed by Yang Sun, Isaac G. Councill, Ziming Zhuang and
180:
52:
90:
236:
119:
19:
was a Web-based information system and search tool used for research on Web robots and trends in
212:
240:
187:
168:
150:
143:
283:
56:
192:
Proceedings of IEEE/WIC/ACM International
Conference on Web Intelligence (WI 2007)
161:
24:
205:"Zoom Web Media Offers Affordable Web Design, Development and SEO Services"
265:
27:. BotSeer was in operation from 2007 to 2010, approximately.
244:
123:
262:"BotSeer: Robots.txt and Web Crawler Search Engine"
226:BotSeer? - SEO Best Practices Search Engine Forums
181:Determining Bias to Search Engines from Robots.txt
89:. Network World. November 15, 2007. Archived from
55:to test the ethics, performance and behavior of
72:. Newsvine. Associated Press. November 28, 2007
243: (archived December 22, 2008) (instead of
179:Yang Sun, Z. Zhuang, I. Councill, C.L. Giles,
8:
237:Web Robot Behavior and Performance Test
111:
70:"Webmasters May Shape Search Results"
7:
14:
290:Defunct internet search engines
87:"Google favored by Web admins"
51:BotSeer had also had set up a
1:
300:Pennsylvania State University
38:Pennsylvania State University
316:
167:December 28, 2007, at the
21:Robot Exclusion Protocol
245:unrelated current site
149:May 17, 2014, at the
209:www.zoomwebmedia.com
93:on December 18, 2007
186:2015-04-02 at the
144:Isaac G. Councill
307:
295:Online databases
276:
274:
273:
264:. Archived from
248:
234:
228:
223:
217:
216:
211:. Archived from
201:
195:
194:, 149-155, 2007.
177:
171:
159:
153:
141:
135:
134:
132:
131:
122:. Archived from
116:
101:
99:
98:
80:
78:
77:
315:
314:
310:
309:
308:
306:
305:
304:
280:
279:
271:
269:
260:
257:
252:
251:
241:Wayback Machine
235:
231:
224:
220:
203:
202:
198:
188:Wayback Machine
178:
174:
169:Wayback Machine
160:
156:
151:Wayback Machine
142:
138:
129:
127:
118:
117:
113:
108:
96:
94:
85:
75:
73:
68:
65:
33:
12:
11:
5:
313:
311:
303:
302:
297:
292:
282:
281:
278:
277:
256:
255:External links
253:
250:
249:
229:
218:
215:on 2012-11-30.
196:
172:
154:
136:
110:
109:
107:
104:
103:
102:
82:
81:
64:
61:
32:
29:
13:
10:
9:
6:
4:
3:
2:
312:
301:
298:
296:
293:
291:
288:
287:
285:
268:on 2010-02-08
267:
263:
259:
258:
254:
246:
242:
238:
233:
230:
227:
222:
219:
214:
210:
206:
200:
197:
193:
189:
185:
182:
176:
173:
170:
166:
163:
162:Ziming Zhuang
158:
155:
152:
148:
145:
140:
137:
126:on 2014-01-04
125:
121:
115:
112:
105:
92:
88:
84:
83:
71:
67:
66:
62:
60:
58:
54:
49:
45:
41:
39:
30:
28:
26:
22:
18:
270:. Retrieved
266:the original
232:
221:
213:the original
208:
199:
191:
175:
157:
139:
128:. Retrieved
124:the original
114:
95:. Retrieved
91:the original
74:. Retrieved
57:web crawlers
50:
46:
42:
34:
25:C. Lee Giles
16:
15:
284:Categories
272:2011-12-11
130:2019-06-13
120:"Yang Sun"
97:2007-12-19
76:2011-12-11
63:References
184:Archived
165:Archived
147:Archived
53:honeypot
239:at the
31:History
17:BotSeer
106:Notes
286::
207:.
190:,
59:.
40:.
275:.
247:)
133:.
100:.
79:.
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.