unforum

[Kender]

IRC bot that monitors in-game sites for changes?

[Guest]

That's an interesting idea. I can see how it would be useful to players -- to be notified of changes as they happen. However, I can see a couple potential downsides to this as well. I don't mean to disuade you from writing one, but I think these issues are important.

First off, it would generate artifical traffic for the webmaster. If linked to a page with limited bandwidth -- a Geocities page, for example--there is a possiblilty that such a bot would cause the page to reach it's allotment and locked much quicker. If a game relies on ad-generated revenue, you'd be hurting the PMs by reducing their hits.

Secondly, it would deter players from checking a page more than once. Sometimes subtle details aren't noticed on the first visit, and it's on subsequent visits that clues are more likely to be spotted. If a player only visits a page once then they're less likely to catch these.

Finally, it would be easy for a snarky PM to rig a page to add random content each time a page is loaded, to generate more traffic. Even unsnarky PMs might have pages with revolving content. While I'm sure your bot would be able to have such pages marked and removed, it would still be an inconvenience and might train players to ignore the bot.

[Ozy_y2k]

Some of us "snarky" PMs have done EXACTLY that in the past; made minor, trivial modifications to pages (such as messing with the spacing in a text) so as to discourage exactly those sorts of bots in the past. Hee hee.

[jamesi]

[ustice]

Personally, I don't think that either of these are problems, really.

If a PM does not wants their pages added, there could be a way to opt out, or as you have said, they could be "snarky."

As for traffc, it wouldn't have to check a site per PERSON, but instead only one time in say 10 minutes. Now, this may seem that it is generating a lot of traffic, but there will be a savings from all of the people that will not have to check the page every 5 minutes despirately looking for new content.

Great idea.

[edit]
I posted this on metaunlimited.com in responce, but I figured that I would post it here too. Yay, redundancy.

While I will be happy with any sort of features that you have in it, these are my ideas on how this could be accomplished:

There should be a database (simple) that will hold user accounts (any ways that this could be linked to a IU forum account???)
- table USERS with username*, password, contact info, other user data/settings
- table SITES with watched sites, a hash of the data
- table SUBSCRIPTIONS with a connection linking a username to a website (to form a many to many relationship)

Every 10 minutes or so (maybe as much as 30 minutes to keep from causing too much traffic on the sites), the SITES table is queried to find all of the sites that all of the users want to check. the corisponding files of the sites are then checked for differences and then cached (note that this could be done with a hash instead of the whole site).

When differences in the sites are detected, the SUBSCRIPTIONS table is queried for those sites, to see what users have subscribed to the sites that have changed, and then the users are contacted (however that is done) with the changes.

A website could be updated with all of the sites that are monitored, and when they were last updated.

Advanced options would be needed as well to prevent abuse (you know it will happen)

There should be the ability to block certain sites to be added to the list, or to only be added through administrator approval. There needs to be a user page, where a user can log in and access the pages that they are aubscribed to and to change their contact method. Also, there should be an administrator page that will allow the admin to make direct changes as needed.

This of course is not a trivial task, and I know that if I did it alone would take several months (given my current schedule) and would require a server that this could run on. You may be thinking on a whole different line, but this is what I was thinking when you mentioned it.

Once again, I will be happy to help as I can (including staying the heck out of it).

In any case, there's my buck fiddy.
[/edit]

[jchillerup]

Or easier, the bot could state in the HTTP header that it was agent (browser) "ARGCrawler" instead of "Mozilla" or whatever.
Then in the .htaccess (ok, this is gonna be a bit technical), you could state that all queries from ARGCrawler should be denied.

Do y'all understand me?

Edit for you that are lost...
http://en.wikipedia.org/wiki/.htaccess
http://en.wikipedia.org/wiki/User-Agent

[bagsbee]

[Tien_Le]

to bot or not to bot

[AnthraX101]

I really don't like this idea, but if you do decide to go ahead with it, you should make it easy to block at the server. Something like a modified user-agent header, or a well known IP is a must. It is also a little harder to write a crawling program then most people think. It's easy to get stuck in dynamic scripts. It should also respect robots.txt.

AnthraX101

[Nik_Doof]

i second that anthrax, while it is essentially a bad idea^tm if you do it then please make it stoppable so PMs can keep some secrets on the site.

Most probably you would use wget for the crawling script, with does abide to robots.txt files

[ustice]

I disagree wtih Tien_Le and bagsbee on this. I personally find it annoying to keep checking sites over and over only to get fatigued, and not want to do it anymore. What I want is content, and updates. One thing I do agree with them (you?) on is it being more of a personal decision. I know that some enjoy looking. I can't immagine that the crawler will be perfect, and we will still have to check some pages on our own ANYWAY. it will just make life a litte easier.

As for robots.txt, that is a must. Always best to respect the wishes of the site's admin.

horray for tools!

[Varin]

I'll agree with Tien_Le and bagsbee. I like to look at the in-game sites and look for updates. I don't think I'd ever use a tool like this because I don't intricately search each and every site for every minor update anyway. But personally, I don't really care if the PMs update the site to correct a typo. All I care about are the major updates which are really easy to spot with a quick visit to the site. I'm not really a big F5-hitting stalker of sites. I think it's just a matter of play type and opinion.

[jchillerup]

[Munchkin]

Maybe this kind of ruins the point. But something that recorded all the changes and then posted them somewhere that people could view might be useful - that way you could check (with a good degree of certainty) when the changes were made. Also, you wouldn't have to look unless you wanted to. I personally would rather be told about an update by a person than by an automated bot - but it would make sure nothing was missed.

[Cortana]

Make it compliant with robots.txt