InternetArchiveBot setup and requirements

[daw5423]

I am trying to test InternetArchiveBot for use in a private wiki. Is the bot hardcoded for Wikipedia only? How do I specify the wiki address, setup the bot account on the wiki, etc.?

I have read the documentation page at https://meta.wikimedia.org/wiki/InternetArchiveBot. Is this the only documentation page? It would be great to write a requirements section. Like does it require a MySQL database, Apache web server, and what else?

[cyberpower678]

The bot turns out to have multiple little bugs that makes setup difficult. A new docker compose file will be made available soon to allow for quick spin up and installation of the bot.

[daw5423]

Thank you. A Docker compose file would be interesting. Is it better to continue discussion in this GitHub thread instead of the wiki talk page, where the discussion gets burried in the archives?

I can help with the setup documentation if necessary once I have it running. I was planning to have a guide for private wiki use because there are US departments that are interested in setting up the bot for their wikis.

[cyberpower678]

Probably best. I’m honored that there are US departments interested in the bot.

[thomas-oneill]

Thank you! I would love for DOE to be able to use such an incredibly useful Bot and other agencies have also expressed a lot of interest in adding it to their wikis.

[cyberpower678]

Thank you! I would love for DOE to be able to use such an incredibly useful Bot and other agencies have also expressed a lot of interest in adding it to their wikis.

Just out of curiosity, are we talk Department of Education or Energy? I'm genuinely interested and would like to know more about this.

[thomas-oneill]

Hi @cyberpower678! I work for the Department of Energy and oversee our internal wiki that just celebrated it's 10th birthday (https://www.youtube.com/watch?v=BcjOSp3XR6A). @daw5423 is one of our interns who has been investigating adding various bots, including the InternetArchiveBot, to our private wiki. Other in the Federal Wiki Community of Practice in which I help run are also interested in your bot. Many of us were part of the recent EMWCon (https://www.mediawiki.org/wiki/EMWCon_Spring_2020) that was organized by Cindy Cicalese, the Product Manager for the MediaWiki Platform team. Mark and Jefferson from the Internet Archive have also tried to connect us. Does that help?

[daw5423]

I was looking at the recent commit c631068 "Add docker file". Is there supposed to be a Docker compose file? I thought the compose file is in YAML format.

[cyberpower678]

Sorry for the long wait. IABot no longer lives here now.

[cyberpower678]

You can now clone https://github.com/internetarchive/internetarchivebot.git which is where IABot will live going forward.

Once cloned run a composer install, and then rename the deadlink.config.docker.inc.php file to deadlink.config.local.inc.php and change all of the values you need, not touching any of the preconfigured values in the file. You will know it when you read the comments. :-)

Finally spin up the docker image and give mariadb a few minutes to do it's first time setup, and it should just show up under http://localhost:8080/

[thomas-oneill]

Thank you!!!!

[cyberpower678]

Does it work? Do you need help?

[daw5423]

I have the server running in Docker. For the second page of the setup, what do I put for:

• i18n source URL:
• i18n source name:

After setup and login, it wants to define the archive templates. I imported the webarchive template from Wikipedia. How do I define it?

{{webarchive |url=https://web.archive.org/web/20160801000000/http://example.com |date=1 August 2016}}

• Template name:
• Template behavior:
  • Append template to URL
  • Replace original URL with template
• This is the name of the template being edited.:

[cyberpower678]

For the i18n source URL, it would be whatever is the meta of your wiki farm. If you don't have a wiki farm, then it would be the same URL as the wiki itself. For the name, just give it a one word name.

[cyberpower678]

Here's enwiki for example:

[cyberpower678]

For the web archive template, the syntax you want to use to define it is:

url|1={archiveurl}|date|2={archivetimestamp:automatic}|title={title}

[daw5423]

I am on the define user groups page.

I created a group by specifying only the group name and group class of default, but it says this:

"Success: Successfully saved group settings, but one or more defined groups is dependent on a non-existent group. You will not be able to leave this page until the groups are correctly defined."

How do I define the group?

[cyberpower678]

If you defined a group to inherit another group’s permissions, and it doesn’t exist, the interface can not function correctly until the inherited group actually exists.

[daw5423]

I left the other fields blank. At first, I checked the checkboxes and submitted. That's when the dependent message showed up. I edited the group and deselected the checkboxes, but it still shows the message. There's no way to delete a group and start over in the interface?

EDIT: I deleted the rows "interface-usergroups" from the database and created 1 group with only group name specified. All other fields blank. It still shows the dependent message.

[cyberpower678]

An empty group does nothing.

Here's an example of what's set up on Wikipedia. This is for the root group. As you can see roots have the following flags set on anyone that is a member of root. You can also see that it inherits the flags of the admin and bot groups, so those groups must exist as well.

As seen, this is the bot group which only grants one flag, plus all of the flags belonging to the user group.

And here is the admin group. You can see what it depends on.
In the end, if you create a group dependent on another group, be it inheritance of flags, or just being able to assign or revoke the group from other users, that group needs to be subsequently defined. In the end, regardless of how you distribute permissions across groups, all permissions should be set. Creating a permissions group with no permissions, effectively renders the interface non-functional.
Here are more screenshots.

Autogranting criteria's can also be set to automatically grant the permission to users:

This above means the admin group is granted if the user is a member of sysop group on Wikipedia, has at least 6,000 edits, and registered more than 6 months ago.

This is what is set for the root group. This means the permission isn't automatically granted at all, and must be assigned manually.

[cyberpower678]

Did it work?

[daw5423]

Yes, the group explanation and screenshots were helpful. I have completed the setup, and I am now testing a couple pages.

One problem I would like to address is that the web server in Docker was taking up to 1 minute to load one PHP page. After some testing, I found out that the xdebug was causing it. It was trying to connect to a non-existent client on each PHP request. I wasn't using xdebug, so after I disabled remote autostart, pages load normally now.

xdebug.remote_autostart = 0

enabling xdebug remote debug makes apache server very slow:
https://stackoverflow.com/questions/17736304/enabling-xdebug-remote-debug-makes-apache-server-very-slow/19273025#19273025

If not using xdebug, this might be something important to address in the README.

When I tried analyzing one page, there was a bug (fatal error) causing it to generate a blank page after analyzing for 30 seconds. I opened the page source and found:

<!-- Missing locale for "en" -->
<!-- No fallback function found, application will use system default -->
<!-- Missing locale for "en" -->
<!-- No fallback function found, application will attempt to use "en" -->
<!-- Missing locale for "en" -->
<!-- No fallback function found, application will use system default -->
<!--
Logging on as InternetArchiveBot...Success!!

The paywall table exists

The global external links table exists

The demo external links table exists

The archive cache table exists

A log table exists

The edit error log table exists

Analyzing Test (47)...
Fetching all external links...
ERROR: Wiki database usage failed.  Defaulting to API Binary search...
Delaying one or more links!
http://example.com/ is ALIVE!
Running a full check on:
	http://f.com/
http://f.com/ is DEAD!
Running a full check on:
	http://f.com/
<br />
<font size='1'><table class='xdebug-error xe-fatal-error' dir='ltr' border='1' cellspacing='0' cellpadding='1'>
<tr><th align='left' bgcolor='#f57900' colspan="5"><span style='background-color: #cc0000; color: #fce94f; font-size: x-large;'>( ! )</span> Fatal error: Maximum execution time of 30 seconds exceeded in /vendor/wikimedia/deadlinkchecker/src/CheckIfDead.php on line <i>388</i></th></tr>
<tr><th align='left' bgcolor='#e9b96e' colspan='5'>Call Stack</th></tr>
<tr><th align='center' bgcolor='#eeeeec'>#</th><th align='left' bgcolor='#eeeeec'>Time</th><th align='left' bgcolor='#eeeeec'>Memory</th><th align='left' bgcolor='#eeeeec'>Function</th><th align='left' bgcolor='#eeeeec'>Location</th></tr>
<tr><td bgcolor='#eeeeec' align='center'>1</td><td bgcolor='#eeeeec' align='center'>0.0006</td><td bgcolor='#eeeeec' align='right'>478520</td><td bgcolor='#eeeeec'>{main}(  )</td><td title='/var/www/html/index.php' bgcolor='#eeeeec'>.../index.php<b>:</b>0</td></tr>
<tr><td bgcolor='#eeeeec' align='center'>2</td><td bgcolor='#eeeeec' align='center'>0.0731</td><td bgcolor='#eeeeec' align='right'>5650568</td><td bgcolor='#eeeeec'>analyzePage(  )</td><td title='/var/www/html/index.php' bgcolor='#eeeeec'>.../index.php<b>:</b>180</td></tr>
<tr><td bgcolor='#eeeeec' align='center'>3</td><td bgcolor='#eeeeec' align='center'>0.6443</td><td bgcolor='#eeeeec' align='right'>5758624</td><td bgcolor='#eeeeec'>Parser->analyzePage(  )</td><td title='/var/www/html/Includes/actionfunctions.php' bgcolor='#eeeeec'>.../actionfunctions.php<b>:</b>2039</td></tr>
<tr><td bgcolor='#eeeeec' align='center'>4</td><td bgcolor='#eeeeec' align='center'>0.6443</td><td bgcolor='#eeeeec' align='right'>5758880</td><td bgcolor='#eeeeec'>Parser->getExternalLinks(  )</td><td title='/var/www/Core/parse.php' bgcolor='#eeeeec'>.../parse.php<b>:</b>190</td></tr>
<tr><td bgcolor='#eeeeec' align='center'>5</td><td bgcolor='#eeeeec' align='center'>0.7609</td><td bgcolor='#eeeeec' align='right'>5842256</td><td bgcolor='#eeeeec'>Parser->updateLinkInfo(  )</td><td title='/var/www/Core/parse.php' bgcolor='#eeeeec'>.../parse.php<b>:</b>1085</td></tr>
<tr><td bgcolor='#eeeeec' align='center'>6</td><td bgcolor='#eeeeec' align='center'>0.7609</td><td bgcolor='#eeeeec' align='right'>5842632</td><td bgcolor='#eeeeec'>Wikimedia\DeadlinkChecker\CheckIfDead->areLinksDead(  )</td><td title='/var/www/Core/parse.php' bgcolor='#eeeeec'>.../parse.php<b>:</b>3306</td></tr>
<tr><td bgcolor='#eeeeec' align='center'>7</td><td bgcolor='#eeeeec' align='center'>46.8918</td><td bgcolor='#eeeeec' align='right'>5850744</td><td bgcolor='#eeeeec'>Wikimedia\DeadlinkChecker\CheckIfDead->performFullRequest(  )</td><td title='/vendor/wikimedia/deadlinkchecker/src/CheckIfDead.php' bgcolor='#eeeeec'>.../CheckIfDead.php<b>:</b>315</td></tr>
</table></font>

The comment is hiding the HTML. That is why there is a blank page.

[cyberpower678]

Try adding ini_set( 'max_execution_time', 300 ); to the init.php file. Best to put it after the line that sets max memory usage.

[cyberpower678]

I take it that fixed your problem and you are up and running?

[daw5423]

Yes, that max_execution_time worked. After analysis, the run duration showed 30-60 seconds.

Since then, I have written a guide. There might be things that could be added or changed. You can see here:
https://github.com/PowerpediaInterns/mediawiki-docs-InternetArchiveBot/wiki

I have also written a Bash script to automate the install using the Docker method:
https://github.com/PowerpediaInterns/mediawiki-docs-InternetArchiveBot/blob/master/Scripts/install-docker.sh