Cron Your XML Sitemaps

17Apr by Ian

Quick tip today: Run your WordPress XML sitemap creation via a cron job when you’re doing any serious auto-blogging.

Now, kinda seems like a no-brainer here, but I wasn’t thinking at all and set up a pretty large auto-blog about a month ago. I tossed in my normal plugins, and let it run. And then I noticed my server resources climbing.

I did a few other site launches at the same time, and assumed it was just the increased traffic and auto-posting – so I deployed another node of resources and didn’t think anything of it. Then, and I’m sorry for the visual, I was in the shower the other morning and I thought “You idiot… you’re rebuilding your sitemap on every post aren’t you? Thats’s like 850 times a day, moron”.

Sure enough… I had the XML Sitemaps plugin for WordPress set to rebuild on every new post. And my auto-blog has made about 32,000 posts in 5 weeks, using about 30 mb of memory each time and about 45 seconds of CPU processing. So… you can do the math on that. Here’s a clue – It’s a lot of freaking server resources.

Thankfully – the solution is really simple. I did two things:

1) Switched from auto building the sitemap with each post to running on a cron job.
2) Limited the pages in the sitemap to 750 – since it’s building and pinging every hour, 750 is plenty.

And now… the results.

See that big dip on Saturday? That’s when I made the switch.

Now, doing the cron job is easy in cPanel (here’s a tutorial on creating cron jobs). I just selected “Once an Hour” under Common Settings” and then ran the following command:

lynx -dump “URL Provided by XML Sitemap Plugin Here”

And that was that. Each hour the sitemap gets rebuilt, my indexing is still solid, and my poor VPS is under much less strain.

So, guess this means I need to build more sites.

This entry was posted in Hosting. Bookmark the permalink.
  • Michael

    Gracias for the tip, when I get there it will come in handy. :)

  • Gomez

    Cron job???

  • halo18

    @Gomez – http://en.wikipedia.org/wiki/Cron

    Basically, a cron job let’s you schedule an action. WordPress has a built in cron, but it requires a page view to run so the timing isn’t very exact. Especially if you’re not doing much traffic yet.

    A lot of hosting accounts come with cPanel, which make setting up cron jobs really simple.

  • Ashlee Northey

    Thanks for the info. I’ve found large differences between some VPS hosts, mainly in their firewalls and options (EuroVPS being very forgetable). I used to be with Fasthosts which was super for a bit but then the performance failed – it locked up 2 times in 6 weeks and once required a hardware change to sort out the hard drive. Their storage had some faults. I tried Dediserve and since joining they’ve been fine!