In writing “quad-quad”, which is a set of four 4-letter speak-able words that can be used as a user-friendly “bookmark” into easily finding a record, I was writing a “quick” program to extract the contents of wikidatawiki-20220820-pages-articles-multistream.xml (a wikipedia dump) and came into this large delay in the following loop:
$alphas = 'qwertyuiopasdfghjklzxcvbnm ';
$newline = '';
for ($x = 0; $x < strlen($line); $x++) {
$c = substr($line, $x, 1);
if (strpos($alphas, $c) !== false) {
$newline = $newline . $c;
else {
$newline = $newline . ' ';
}
}
The loops main purpose is to sanitize any non-letter data by replacing unknown characters with a space for later processing. The end result would be words that I could filter down to 4-character words and tally them up.
When the program read a line around 1mb in length it would “hang” for a bit as it chewed through the data. In a nutshell 25,100,655 bytes of data would take 24m36s. It was time to optimize.
Replacing the previous with the following regex performance was increased immensely.
$newline = preg_replace('/[^a-z]/', ' ', $line);The same amount of data took 1.892s.
Lesson: If you don’t know regexes, learn regexes.