Sooner or later, when you are processing, importing and ingesting data from large suppliers such as Amazon, iTunes Music Store and Virgin Megastore, to name but 3, you are likely to hit a scenario where you are need to handle, process, import and output XML files that are serveral hundred megabytes or even gigabytes in size. In the course of the past year, doing just that, I have come up with some of my own tips and tools for actually processing the suckers, and here I try to document some of these for the benefit of others.
To start you have to retreive the XML from the supplier's server, this tends to range in process from FTP access to using WGET commands. I personally use an uatomated shell script for ftp'ing across and grabbing the file and then unzipping it with a simple unzip / gunzip command where necessary:
#!/bin/bash
# nav to dir
cd /path/to/repository
# set vars
HOST='ftp.domain.com'
USER='username'
PASSWD='pass'
FILE='filename.xml.gz'
#clean up dirs and remove any XML
rm -f *xml*
rm -f splits/*
#login to ftp and channel commands
ftp -n -v $HOST << EOT
ascii
user $USER $PASSWD
prompt
cd /
mget $FILE
bye
bye
EOT
#decompress file
gunzip $FILE
After preparing the XML you are ready to start processing it. PHP has horrendous memory usage skills and therefore it is hardly the best idea to try and read the file into memory and parse it using the XML tree-based parsers that are built in. To avoid having to switch to using Perl stream-based parsers, I devised an ingenious means by which to split the XML file into variable sized chunks, thus making it much easier to both process the XML with PHP but also to stop and start the process.
The process of chunking the file essentially take advantage of the line breaks int he XML and reads through the XML line by line. It is then programmed to perform certain actions as it reaches certain parts of the XML. For one it increments a counter as it reaches the end of each record, so that when it hits a designated number it can close the current chunk file and begin a new one. The end result of the process is that you will end up with a decent number of bite-sized xml files that are well-formed and readable by any parser.
To start with just initialise all the required variables:
<?php //initialize vars $begin=time(); // script start time $start = time(); // last gate time $interval=time(); // current gate time $minutes=1; // intervals for gates $filenum = 1; // start chunk file number at 1 $recordnum = 1; // start at record 1 //file settings $basefilename = ""; // the base file name for the chunks $xmlfile = ""; // the xml file name to be processed $xmldatadelimiter = ""; // core data delimiter $xmlitemdelimiter = ""; // record delimiter $chunksize = ""; // number of records in each chunk file $xmlstring ="<?xml version="1.0" encoding="UTF-8">\n"; $xmlstring.="<$xmldatadelimiter\n>"; // xmlchunk file header //dirs and files $dir = "/path/to/directory"; // path to where splits will be stored $exportfile = "$dir"."/splits/$basefilename-$filenum.xml";
Once this is done you are ready to start processing the initial XML file:
//start processing
echo "Processing (".$dir."/$xmlfile)\n";
$handle = @fopen($dir."/$xmlfile","r");
if ($handle) {
while (!feof($handle)) {
$buffer = fgets($handle, 4096);
This essentially loops through the file line by line. This means that you need to ensure that the XML file has line breaks at the end of each element. If it does not, the entire XML file will be read into memory by PHP!
No that you have the current line held in $buffer, you can start processing the contents of the line and act accordingly in any or all of 4 possible ways:
And so each of these actions can be acheived as follows:
// if item delimiter reached
// increment record number iterator
if (eregi("",$buffer)==true) {
$recordnum++;
}
//write line to chunk file
error_log("$buffer",3,$exportfile);
// if chunk limit reached then start to
// close the file with well formed xml
if ($recordnum>chunksize) {
// post feed end tag
error_log("",3,$exportfile);
// and increment file number to start new log file chunk
//reset record counter number for new chunk file
$recordnum=0;
$filenum++;
//update export file name
$exportfile = "$dir"."/splits/$basefilename-$filenum.xml";
//echo status report to STDOUT
echo"Segment $filenum. Record ".($chunksize*$filenum).".\n";
// write new chunk xml file header
error_log($xmlstring,3,$exportfile);
}
After this just run through some internal script time logging and script killing code to (1) keep track of the time it takes and (2) to kill it in event of the script running out of control.
//put in a catch so that script doesn't run riot and
//will die after X number of cycles
if ($filenum>5000) {
die();
}
if (($interval-$start)>60) {
$minutes++;
echo $minutes." Minutes so far.\n";
$start=time();
} else {
$interval = time();
}
Then just close the file handle for the original XML file, and then as part of the initial check to see if the XML file exists, add an error message to the else clause:
}
fclose($handle);
} else {
echo"Unable to open file! (".$dir."$xmlfile")\n";
}
To finish up the process just echo out some time based stats:
$procend = time();
echo "\n####\n";
echo "Split Complete (".floor((($procend-$begin)/60))." Minutes)\n";
?>
And so in turn, once you have these files, you can use a PHP script to read the contents of the directory containing the split files and then read then into whatever parser you chouse, be it something like simpleXML or a custom php XML handler.
Find out more over at the PHP documentation site