Travian WorldMap Import Part 2 – Parallelism

Processing the Travian WorldMap INSERT statements so they can be bulk inserted into MS SQL requires processing on each line of the map file:-

1. Remove everything except the actual village data
2. Split the data into its component parts and remove/adhere to quotes

Once I’d found the fastest way to do this (string.remove) and a custom split routine (one that doesn’t split data inside quotes), it took around 450ms to complete the file. With the file import is now down to 40ms that’s roughly 500ms; or half a second per server, that’s 2m 30s total time spent not doing much.

One of the reasons I wanted to use VS 2010 is the new Parallel library features- It was relatively easy to implement and reduced the time to 160ms. So with a simple change I’ve got the time down from 2m 30s to 1m - that’s a 2.5 times faster.

Once I move to a larger development box (16 cores) the performance gains will increase automagically by many factors – excellent, self scaling code!