I've moved this blog to http://ukinbox.spaces.live.com/ - the simple reason being that I have more control over the Blog and layout on live.com; and I'm allowed far more screen real-estate :)
Pop over and get the latest post on Travian iMailer there. See you shortly
Wednesday 20 May 2009
Travian WorldMap Import Part 2 – Parallelism
Processing the Travian WorldMap INSERT statements so they can be bulk inserted into MS SQL requires processing on each line of the map file:-
- Remove everything except the actual village data
- Split the data into its component parts and remove/adhere to quotes
Once I’d found the fastest way to do this (string.remove) and a custom split routine (one that doesn’t split data inside quotes), it took around 450ms to complete the file. With the file import is now down to 40ms that’s roughly 500ms; or half a second per server, that’s 2m 30s total time spent not doing much.
One of the reasons I wanted to use VS 2010 is the new Parallel library features- It was relatively easy to implement and reduced the time to 160ms. So with a simple change I’ve got the time down from 2m 30s to 1m - that’s a 2.5 times faster.
Once I move to a larger development box (16 cores) the performance gains will increase automagically by many factors – excellent, self scaling code!
Travian WorldMap Import Part 1
As previously mentioned, performance for Travian iMailer version 2 is paramount (+new features :))
Importing the Travian WorldMap file and processing means getting the file off the disk and into memory in a format that’s suitable for editing – unfortunately, the Travian servers perform an SQL dump and this uses a none standard line delimiter (none standard in the Windows world anyway), the LineFeed (LF), Windows uses LineFeedCarriageReturn (LFCR)
Loading the file into memory and splitting it by LF is very simple, but from a performance perspective, is there any different methodologies for this? well, yes.
In Visual Studio 2010 there are several methods. In testing I used a 3MB WorldMap file and preloaded it into cache so disk performance was removed from the calculations:-
- Slowest performance:- Using the built in Filesystem.ReadAllText (81ms)
- Almost as slow:- Using the built in Filesystem.ReadAllBytes and splitting it by LF (79ms)
- Fastest, specifically, twice as fast – Using the IO.StreamReader (40ms)
Does 40ms really matter; Yes, 302 servers = 12 seconds additional time; not major, but this is just one simple routine (read the file into memory for processing) and there are a lot of routines.
SQL Import Part 2
Travian provides WorldMap data as a SQL dump; this is very simple and consists of rows of INSERT statements – At first glance this looks great and we now have many Travian Utilities using it.
When we import this into SQL the obvious starting point is to pass the INSERT statements to SQL and let it ‘do it’s thing’ – expect there are a few issues:-
- Travian don’t use Microsoft SQL and part of the INSERT statement is incompatible (quote around table name)
- Every INSERT statement for every Travian server tries to INSERT into the same table
- Multiple INSERT statements are not efficient to import
So let’s take each of these issues:-
1. Simple, modify the INSERT statement. This means we now have to read all the INSERT statements, modify them, then pass them on.
2. Simple. As we have to modify every INSERT statement anyway for compatibility, this is a very fast in-memory operation to correct
2. For most Travian Utilities this is probably not a major issue, but for Travian iMailer this is a BIG issue and would mean our import of all 300+ servers, every day, could take hours. Instead we read all the INSERT statements, process them and bulk import into SQL
Tuesday 19 May 2009
SQL Import
I just thought I’d write about importing the Travian game world data into SQL.
After optimising the code and using SQL Server, quick calculations show that, using a single thread we can download the Travian world data, process it, and import it into SQL in around 5 seconds – significantly faster than currently.
Moving to parallel development
Ouch! Visual Studio 2010 Beta 1 is out today and I’ll be downloading it later and looking forward to all the new features, especially the new parallel tasks – Why Ouch; I’ve worked with parallel programming before and I know how hard debugging it is and I have a not so simple task of converting all our code libraries while getting to grips with a new database format.
It’s all about performance
One of the big areas I’m looking at for v2 is performance. Currently we dynamically build a list of active Travian servers and monitor them until Travian upload the new WorldMap data; then the race is on, we have to:-
- Download over 300 global servers data
- Import data on every player, alliance, village, etc.
- Compute game world delta statistics (alliance changes etc.)
- Send out personalised emails to our subscribers
So, that doesn’t too difficult you say, but the advantage that Travian iMailer provides is that first chance hit on a new farm and claiming ownership :) So we need to get those emails out as soon as possible.
Performance is one of the main areas I’ll be looking at and as we move over from a custom database to Microsoft SQL Server I need to squeeze out everything we can and move to a massively parallel service.
Relay
Travian iMailer V1
iMailer version 1 has been up and running for a while now and the team are very pleased with the reception it’s received. We get a lot hits and refusing all offers of advertising :)
So who am I – I’m the senior developers with overall responsibility for iMailer version 2.
Rather than create a post on the TravianiMailer website I thought I’d take the opportunity to blog here about version 2 and general things Travian. I also play the game and currently reside on s1.travian.co.uk – feel free to post me a message here or in-game. I’ve been playing the game for several years and been through many World Wonders and so when I was asked to put my developer skills to some use, I jumped at the chance.
Subscribe to:
Posts (Atom)