Quick tip today: Run your WordPress XML sitemap creation via a cron job when you’re doing any serious auto-blogging.
Now, kinda seems like a no-brainer here, but I wasn’t thinking at all and set up a pretty large auto-blog about a month ago. I tossed in my normal plugins, and let it run. And then I noticed my server resources climbing.
I did a few other site launches at the same time, and assumed it was just the increased traffic and auto-posting – so I deployed another node of resources and didn’t think anything of it. Then, and I’m sorry for the visual, I was in the shower the other morning and I thought “You idiot… you’re rebuilding your sitemap on every post aren’t you? Thats’s like 850 times a day, moron”.
Sure enough… I had the XML Sitemaps plugin for WordPress set to rebuild on every new post. And my auto-blog has made about 32,000 posts in 5 weeks, using about 30 mb of memory each time and about 45 seconds of CPU processing. So… you can do the math on that. Here’s a clue – It’s a lot of freaking server resources.
Thankfully – the solution is really simple. I did two things:
1) Switched from auto building the sitemap with each post to running on a cron job.
2) Limited the pages in the sitemap to 750 – since it’s building and pinging every hour, 750 is plenty.
And now… the results.
See that big dip on Saturday? That’s when I made the switch.
Now, doing the cron job is easy in cPanel (here’s a tutorial on creating cron jobs). I just selected “Once an Hour” under Common Settings” and then ran the following command:
lynx -dump “URL Provided by XML Sitemap Plugin Here”
And that was that. Each hour the sitemap gets rebuilt, my indexing is still solid, and my poor VPS is under much less strain.
So, guess this means I need to build more sites.