Quake III Arena Demo File Specifications

File formats: *.dm_66, *.dm_67, *.dm_68

 

Document Version 4

 

 

1. About

2. Sizebuf_t structure

3. Demofile format

4. MSG_ReadBits and MSG_WriteBits

5. Common MSG_* functions

6. Delta encoding of game structures

7. Huffman encoding

8. Server to Client commands

9. Operations with gameState_t

10. Parsing whole demo file

11. Thanks to

 

 

1. About

 

This document contains complete Quake III Arena Demo File specifications, useful for programmers working in C or C++  developing demo players/editors. It introduces demo message encoding/decoding routines that allow parsing and writing Quake III Arena Demo Files.

 

Note this code doesn’t provide full Quake III Arena network protocol support, as it uses much more encryption. All network-only stuff was removed from this document.

 

This code is redistributed under the terms of GNU General Public License:

 

/*

Quake III *.dm_6? Demo Specifications

 

Copyright (C) 2003 Andrey '[SkulleR]' Nazarov

Based on Argus and Quake II source code

Also contains some stuff from Q3A SDK

 

Argus is Copyright (C) 2000 Martin Otten

Quake II and Quake III are Copyright (C) 1997-2001 ID Software, Inc

 

This program is free software; you can redistribute it and/or

modify it under the terms of the GNU General Public License

as published by the Free Software Foundation; either version 2

of the License, or (at your option) any later version.

 

This program is distributed in the hope that it will be useful,

but WITHOUT ANY WARRANTY; without even the implied warranty of

MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. 

 

See the GNU General Public License for more details.

 

You should have received a copy of the GNU General Public License

along with this program; if not, write to the Free Software

Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.

*/

 

You can also download this example application (source code + binary):

http://skuller-vidnoe.narod.ru/downloads/dm_68.zip

It contains all source code present in this document and all necessary headers. The only thing this app does is dumping obituaries and some other info from a *.dm_68 file.

 

2. Sizebuf_t structure

 

All MSG_* functions operate with the sizebuf_t structure, which holds all necessary information about current buffer state and is used both when reading and writing bits to the buffer. Normally nothing outside MSG_* functions should modify this structure.

 

typedef struct sizebuf_s {

       qboolean     allowoverflow;      // if false, do a Com_Error

       qboolean     overflowed;        // set to true if the buffer size failed

       qboolean     uncompressed;        // don't do Huffman encoding, write raw bytes

       byte         *data;       // pointer to message buffer, set by MSG_Init

       int          maxsize;     // size in bytes of message buffer, set by MSG_Init

       int          cursize;     // number of bytes written to the buffer, set by MSG_WriteBits

       int          readcount;   // number of bytes read from the buffer, set by MSG_ReadBits

       int          bit;         // number of bits written to or read from the buffer

} sizebuf_t;

 

 

 

3. Demofile format

 

Actually Quake III Demo File is a version of server-to-client communication protocol, written to a disk in the following format:

<demo block> <...> <demo block> <end block>

 

Demo block structure:

Size, bytes

Description

4

Message sequence, <seq>

4

Message length, <msglen>

<msglen>

Message data

 

            Demo is parsed until <seq> and <msglen> are not equal to -1 (end block), which indicates demofile EOF. <msglen> should be always less than MAX_MSGLEN.

 

#define      MAX_MSGLEN          0x4000 // max length of demo message

 

Sample routine for demo file parsing:

 

FILE *demofile;

int demoMessageSequence; // used for delta decoding

 

/*

=====================

Parse_NextDemoMessage

 

  Read next message from demo file and parse it

  Return qfalse if demo EOF reached or error occured

=====================

*/

qboolean Parse_NextDemoMessage( void ) {

       sizebuf_t    msg;

       byte         buffer[MAX_MSGLEN];

       int                 len;

       int                 seq;

 

       if( fread( &seq, 1, 4, demofile ) != 4 ) {

             Com_Printf( "Demo file was truncated\n" );

             return qfalse;

       }

      

       if( fread( &len, 1, 4, demofile ) != 4 ) {

             Com_Printf( "Demo file was truncated\n" );

             return qfalse;

       }

 

       if( seq == -1 || len == -1 ) {

             return qfalse; // demo EOF reached

       }

 

       MSG_Init( &msg, buffer, sizeof( buffer ) );

 

       demoMessageSequence = LittleLong( seq );

       msg.cursize = LittleLong( len );

 

       if( msg.cursize <= 0 || msg.cursize >= msg.maxsize ) {

             Com_Error( ERR_DROP, "Illegal demo message length" );

       }

 

       if( fread( msg.data, 1, msg.cursize, demofile ) != msg.cursize ) {

             Com_Printf( "Demo file was truncated\n" );

             return qfalse;

       }

 

       Parse_DemoMessage( &msg ); // parse the message

 

       return qtrue;

}

 

 

4. MSG_ReadBits and MSG_WriteBits

 

            The basic functions for performing i/o operations on message data are MSG_WriteBits and MSG_ReadBits, which do Huffman encoding/decoding of the data bitstream using Huff_* code. If uncompressed flag is set on the sizebuf, then raw bytes are used, i.e. bits value is rounded to a full byte (q2-style). Currently this is not used when working with demo files.

 

/*

============

MSG_WriteBits

============

*/

void MSG_WriteBits( sizebuf_t *msg, int value, int bits ) {

       int                 remaining;

       int                 i;

       byte                *buf;

 

       if( msg->maxsize - msg->cursize < 4 ) {

             msg->overflowed = qtrue;

             return;

       }

 

       if( !bits || bits < -31 || bits > 32 ) {

             Com_Error( ERR_DROP, "MSG_WriteBits: bad bits %i", bits );

       }

 

       if( bits < 0 ) {

             bits = -bits;

       }

 

       if( msg->uncompressed ) {

             if( bits <= 8 ) {

                    buf = MSG_GetSpace( msg, 1 );

                    buf[0] = value;           

             } else if( bits <= 16 ) {

                    buf = MSG_GetSpace( msg, 2 );

                    buf[0] = value & 0xFF;

                    buf[1] = value >> 8;

             } else if( bits <= 32 ) {

                    buf = MSG_GetSpace( msg, 4 );

                    buf[0] = value & 0xFF;

                    buf[1] = (value >> 8) & 0xFF;

                    buf[2] = (value >> 16) & 0xFF;

                    buf[3] = value >> 24;

             }

             return;

       }

 

       value &= 0xFFFFFFFFU >> (32 - bits);

       remaining = bits & 7;

 

       for( i=0 ; i<remaining ; i++ ) {

             if( !(msg->bit & 7) ) {

                    msg->data[msg->bit >> 3] = 0;

             }

             msg->data[msg->bit >> 3] |= (value & 1) << (msg->bit & 7);

             msg->bit++;

             value >>= 1;

}

       bits -= remaining;

 

       if( bits > 0 ) {

             for( i=0 ; i<(bits+7)>>3 ; i++ ) {

                    Huff_EmitByte( value & 255, msg->data, &msg->bit );

                    value >>= 8;

             }

       }

 

       msg->cursize = (msg->bit >> 3) + 1;

}

 

 

/*

============

MSG_ReadBits

============

*/

int MSG_ReadBits( sizebuf_t *msg, int bits ) {

       int i;

       int val;

       int bitmask = 0;

       int remaining;

       qboolean extend = qfalse;

 

       if( !bits || bits < -31 || bits > 32 ) {

             Com_Error( ERR_DROP, "MSG_ReadBits: bad bits %i", bits );

       }

 

       if( bits < 0 ) {

             bits = -bits;

             extend = qtrue;

       }

 

       if( msg->uncompressed ) {

             if( bits <= 8 ) {

                    bitmask = (unsigned char)msg->data[msg->readcount];

                    msg->readcount++;

                    msg->bit += 8;

             } else if( bits <= 16 ) {

                    bitmask = (unsigned short)(msg->data[msg->readcount]

                           + (msg->data[msg->readcount+1] << 8));

                    msg->readcount += 2;

                    msg->bit += 16;

             } else if( bits <= 32 ) {

                    bitmask = msg->data[msg->readcount]

                           + (msg->data[msg->readcount+1] << 8)

                           + (msg->data[msg->readcount+2] << 16)

                           + (msg->data[msg->readcount+3] << 24);

                    msg->readcount += 4;

                    msg->bit += 32;

             }

       } else {

             remaining = bits & 7;

 

             for( i=0 ; i<remaining ; i++ ) {

                    val = msg->data[msg->bit >> 3] >> (msg->bit & 7);

                    msg->bit++;

                    bitmask |= (val & 1) << i;

             }

            

             for( i=0 ; i<bits-remaining ; i+=8 ) {

                    val = Huff_GetByteEx( msg->data, &msg->bit );

                    bitmask |= val << (i + remaining);

             }

      

             msg->readcount = (msg->bit >> 3) + 1;

       }

 

       if( extend ) {

             if( bitmask & (1 << (bits - 1)) ) {

                    bitmask |= ~((1 << bits) - 1);

             }

       }

 

       return bitmask;

}

 

There is a wrapper interface for writing/reading bytes, shorts and ints. It is implemented as macros and inline functions.

 

#define      MSG_WriteByte(msg,c)                    MSG_WriteBits(msg,c,8)

#define      MSG_WriteShort(msg,c)                   MSG_WriteBits(msg,c,16)

#define      MSG_WriteSignedShort(msg,c)             MSG_WriteBits(msg,c,-16)

#define      MSG_WriteLong(msg,c)                    MSG_WriteBits(msg,c,32)

 

static ID_INLINE int MSG_ReadByte( sizebuf_t *msg ) {

       int c = MSG_ReadBits( msg, 8 ) & 0xFF;

 

       if( msg->readcount > msg->cursize ) {

             return -1;

       }

 

       return c;

}

 

static ID_INLINE int MSG_ReadShort( sizebuf_t *msg ) {

       int c = MSG_ReadBits( msg, 16 );

 

       if( msg->readcount > msg->cursize ) {

             return -1;

       }

 

       return c;

}

 

static ID_INLINE int MSG_ReadSignedShort( sizebuf_t *msg ) {

       int c = MSG_ReadBits( msg, -16 );

 

<p class=code style='border:none;mso-border-alt:solid