Χρήστης:AtouBot/getrcs.sh/docs

README.txt

Last updated: 28 January 2008

***What is this?  

This is a small collection of scripts that will go get pages changed, added, or deleted 
to a wikiproject and update the full xml dump accordingly, to produce a current snapshot.

***System requirements:

linux (untested on other unix variants)
bash, awk, sed, grep and all the usual goodies
perl
curl

***How to install:

See the file INSTALL for details.

**How to run:

cd into the directory where this package was unpacked. 

If this is the first time running it, you will want a full copy of the XML dump 
for your project to start with.  See 
http://download.wikimedia.org/backup-index.html
for these dumps.  You will want the one that has "All pages, current versions only", including 
discussion pages. (pages-meta-current.*.xml)

*Copy* this file into last_full.xml (the file will be overwritten later with the new
snapshot).  If you edited config.txt to change the values for snapshot or snapshotdir,
move the file accordingly.
Look at the date of the xml dump; you are going to want to get everything from that date
through today.  So for example if your dump is dated Jan 13 and today is Jan 18, you will
want to get 6 days of data.  There may be some overlap with existing content; this is
ok. Overlap ensures that you don't miss some changes.

For the full run, type
./getrcs.sh today today-numdays

In our example, we would have 
./getrcs.sh today today-6

Now wait for a while.  The script will update you as it goes along.  It fetches
the (relevant part of the) rc logs, the move logs, the import logs, the upload logs, and
the delete logs.  It then retrieves all pages called for by those logs (except for the 
deletes :-) )  Finally, it merges these pages into the last full dump and produces
a new current snapshot., which will be found in last_full.xml

The next time you run this script, look at the date you generated the last_full.xml
file, and make the same calculation for the number of days. 

If you are running this script on more than one wiki for example, you can use multiple
configuration files and give the command 
./getrcs.sh today today-6 my-config-file

You can also specify absolute timestamps, a base date that depends on the last run time, 
or hour increments instead of days. Type 
./getrcs.sh
for more information. 

**Other info

Temporary files live in ./tmp, and you can remove them when the run is finished.
They will be removed by the script itself on the next run, all except for the
files with the extension raw or raw.save; these you will want to clear out yourself
once in awhile, as they are kept in case debugging is needed.

This script is meant to be run no more than once a day, so the timestamps it
puts in filenames are the date.  If you need to run it more often because
you are on a very active project, change the file naming convention by editing
the variable "ext" near the top of the file.

**Still more info (What are all these scripts?)

sort.pl and uniq.pl             tiny scripts that replace sort and uniq on linux 
                                cause they are busted for some utf8 characters in my 
                                locale, as I found out the hard way. 
merge-pages-main-and-export.pl  grabs the pages we exported from the live dbn
                                and folds them into the last full xml file.
merge-deletes.pl                deletes selected pages based on the retrieved
                                portion of the delete log.
do-links.sh                     makes symlinks to the getrcs.sh script in case 
                                you want to run certain phases of it in 
                                isolation.
symlinks (getchanges.sh etc)    allows you to run each phase of the script
                                separately; see the INSTALL file for more info.  

**A teeny bit more info

By default the script sleeps 5 seconds between requesting page exports, which are done 
in batches of 500 pages each, and 2 seconds between requests of log portions, which are 
500 lines each.

See TODO for things that probably should be... done.

**Copyright

This little mess is released for use under the GPL v3 or later, as well as under the GFDL 1.2 
or later; the reader may choose which one to use.

Copyright (C) Ariel T. Glenn 2008 (and all other editors of this page; please see
the history page for details).  Please improve it and share!  

INSTALL.txt

Last updated: 20 January 2008

***System requirements:

linux (untested on other unix variants)
bash, awk, sed, grep and all the usual goodies
perl
curl

***How to install:

Untar the file into a convenient location.

Edit the file config.txt and change the line 
wiki="en.wiktionary.org"
to contain the name of your project.

Change the line 
expurl='Special:Export'
to contain the name of your Special:Export page.

You can change the number of seconds between requests for pieces of
the various logs by editing the line
logsecs=2

You can change the number of seconds between requests for page
exports (500 pages each export) by editing the line
pagesecs=5

You can also change the temporary work directory by editing the line
tmp="tmp"

You can change the directory and the name where the last snapshot 
be found, by editing the lines
snapshot="last_full.xml"
and
snapshotdir="./"

You can change the name of the file where we put the startdate for
this run by editing the line
lastrun="lastrun"

If you want to run the script on more than one wiki for example, you
can create multiple configuration files, one for each.


After that, run the command
./do-links.sh

This will create sym links to other names you can use for invoking the script 
one piece at a time.  

**Note

You can run pieces of this script at a time. If you do a directory listing,
you can see that there are several symlinks; each of these is to a name that, 
if used to invoke the script, will run that phase.

Generally, if you are going to do that, you should pass the same number of days
to each phase that you run separately, with the exception of the getpages and domerge
phases, where the number of days isn't actually used :-)

The phases are:

(1)
getchanges.sh  
getmoves.sh  
getimports.sh
getuploads.sh
getdeletes.sh  

(2)
getpages.sh  

(3)
domerges.sh  

The scripts in phase 1 should be run before phase 2 which should be run before
phase 3. 

The scripts in phase 1 retrieve the appropriate part of the specified log.
Titles of pages to retrieve or to delete are generated from these lists.

The script in phase 2 retrieves all pages (except for the deletes) that we put
together in phase 1.

THe script in phase 3 merges these new pages into the old full dump and 
deletes any that need to be removed, checking timestamps to see which 
version is most current or whether the deletion was before a recreation
of the page (for example).

Why would you want to do this?  Maybe you are debugging :-/
But, more likely, you may want to get all the log updates once a day but only 
build a snapshot once a week. (Depending on how large your project is, 
building the whole snapshot could take a long time.)
You would have to do some manual catting of files together for this
but it might be useful.