by Juho » Sun Dec 09, 2012 3:48 pm
It seems that the actual crawling takes much longer than calculated from the interval: about one third to double time, depending about the interval. I just have to allocate more time or to shorten the interval. I did split the crawling into 15 keyword sections per cron job execution. I also noticed, that the splitting doesn't work if the previous cron job hasn't finished before new one executes. If the previous crob job hasn't finished, the new execution starts crawling the same keywords as the unfinished.