Meandering thoughts of an Addicted Games Programmer

What is an "Engineer Architect"

2011-07-22T11:14:00.001-05:00

A question that i get asked rather a lot is "What is an Engineer Architect"

en·gi·neer
noun /ˌenjəˈni(ə)r/
A person who designs, builds, or maintains engines, machines, or public works

ar·chi·tect
noun /ˈärkiˌtekt/
((computer science) the structure and organization of a computer's hardware or system software) "the architecture of a computer's system software"

This basically translates to the same thing all engineers do:

Analyze the problem
Formulate a possible set of solutions
Analyze the solutions
Decide what to implement
Implement
Test/bug fix
Profile/sanitize
Repeat until satisfied

however with one important difference. My role involves the entire program, how its parts interact and how they are designed, what their dependencies are and how they affect the final output and the performance of that output. Combine this with data parallel infrastructure, task parallel objects, multiple platforms, myriad hardware limitations and a large programming team ... under many circumstances one might imagine something akin to

and you often wouldn't be too far from the truth on a daily basis. Long term however we (the team) have a plan and a set of goals. Over time we re-assess based on those we hit and those we don't, new requirements and ultimately how the game is progressing and where it needs to go. The role I play in this is simply to ensure that under the stress and strain of day to day development, we're still aiming in the right direction as a whole. That decisions take into account as much of the big picture as possible and weigh that against the immediate requirements of the current goal. There are several people in similar roles at Bungie and our interactions provide a simple but effective method of applying "Checks & Balances" to the progress we make; we each bring our own flavor to the table.

Day to day this involves tasks such as

Advising/Teaching on how to handle concurrency
Long term interface design
Short term prototyping/bug fix hacks
Discussing/Advising on future platforms
Optimizing programmer iteration, debugging & general workflow
Managing external teams
Auditioning Middleware
designing/writing/debugging infrastructure systems

Technically i'm a member of Bungies "Infrastructure" team. This means that if its something "unexciting", "behind the scenes" or seemingly doesn't affect the final game at all... we handle it. This involves systems like

Memory Allocation
File System
Network Transport
Crash Handling, Minidumps
Debugger Plugins
Multi-threading Infrastructure & Architecture
Asset import/baking
Math Library
Schematization/Reflection
Audio Engine
Container Classes
Profiler Infrastructure
Compiler Configuration
Low & High level Optimization
Build systems
Flux Capacitor Maintenance
- you still reading?

as i said... the stuff most most programmers find tiresome and boring. Our team love this stuff and we're good at it.

So there you have it, a much better idea of what i do @ Bungie and hopefully a guide to those in school who might want to progress towards a similar role (or avoid it).

Writing a "Pre-Main" function - Forcing Global Initialization order within VC++

2011-07-07T01:26:00.000-05:00

One rule of C/C++ programming that bites everyone eventually is that initialization order of global variables across compilation units is not guaranteed. I've seen programs with global variable dependencies run fine for years then suddenly develop "issues" resulting in a hard lock before main(..) is hit. I've seen other programs have dependency issues but not actually crash resulting in subtle bugs that never seem to negatively affect run-time... until they do, usually on the day before gold master when the lead programmer is in Peru.

So here's the situation, you want to create objects "a" through "g" in various files who register themselves with a manager into a list on construction... a relatively common setup and relatively simple within a single compilation module; for VC++ define them from bottom to top manager first and it will work. However across compilation modules things are not so simple and more often than not the manager will construct itself at a very inopportune time; likely after some objects have registered themselves and before all objects have.

Without simply knowing that the manager will initialize first there really isn't a "Good" solution tho there are several lazy initialization techniques.

So here's where things get "fun" and by "fun" i mean, people look at the code and get confused looks on their face.

Put the following code into a cpp file on its own and add it to your project of choice.

#pragma init_seg( ".CRT$XCB" )

class c_blog_first_class_construction
{
public:
    c_blog_first_class_construction() 
    {
        printf("first class construction\n");
    }

    ~c_blog_first_class_construction() {};
};

static c_blog_first_class_construction blog_first_class_construction;

the constructor for c_blog_first_class_construction will be called before any other constructor in your code (assuming you have nothing else like this in there).

NOTE: Expect C4075: initializers put in unrecognized initialization area, disable it if you feel the need.

the key to the code is

#pragma init_seg(...)

for more info on what this does see kb104248

The crux of it however is we are naming this compilation unit with a specific identifier, when the linker reads various .CRT groups, it combines them in one section and orders them alphabetically. This means that the user-defined global initializers (which the Visual C++ compiler puts in .CRT$XCU) will always come after CRT$XCA and before .CRT$XCZ.

So we are depending linker to do the right thing and insert our section into the CRT sorted sections (verified in VS2010 as of 7/6/2011 for x86,x64,x360). It works and its very useful.

Now if you want to get really fancy... make the constructor for c_blog_first_class_construction call a function that initializes your "Pre construction" requirements engine wide (usually memory management, assert systems).

For other platforms (GCC/SNC are the only others i use) there are other options such as "init_priority" which is a per instance attribute. These can be used to achieve the same result but on a more granular level.

It should be noted that this is not a method that one should use all over a codebase, personally i only ever use it on one compilation module and it is always VERY well commented.

enjoy.

Becoming a Console Programmer : Extending The Watch Window (debugger addin)

2011-05-22T09:27:00.001-05:00

So previously we established how to use autoexp.dat for simple format changes and how to use a DLL to achieve those changes. There are major limitations to the DLL approach however in that we cannot access global symbols and thus any system that relies on global state for debug (as many do) cannot be debugged.

Here I will discuss how to create a VS2010 plugin using C# (plugins also work for VS2008 however not using C#).

The Development Tools Environment framework was setup around a decade ago to allow Visual Studio extensibility. I would recommend that any programmers read through what is possible using DTE however i'm not going to invest much time into that here; I will simply use one aspect of it to achieve our goal: global symbol access.

VS2010 has a great Wizard to help us here, Select File->New->Project then on the left hand side go to Other Project Types->Extensibility->Visual Studio Addin, click next and then choose option (default) Create an Add-in using Visual C#, leave application host as default, then give your addin a recognizable name. Set your addin to load when host application starts. The rest is default.

This provides us with a very simple addin with a LOT of C# goodness and very little code. I should note that when i wrote this project it was my first foray into C# and thus my knowledge in this area is shady at best, i had lots of help from experts local to me in setting this up.

If you read through the code you will find some relatively basic functionality. Your new addin has interfaces to intercept callbacks for various VS2010 events, OnConnection is the only default interface i used during which i installed new event handlers.

So lets extend the default a little:

Add a member to the class of type "DebuggerEvents"
instantiate it to "_applicationObject.Events.DebuggerEvents"
setup handles for:

OnEnterBreakMode
OnEnterRunMode
OnEnterDesignMode

VS2010 should auto-complete and auto generate these for you but the basics should look like :

_debugger_events.OnEnterBreakMode += new _dispDebuggerEvents_OnEnterBreakModeEventHandler(DebuggerEvents_OnEnterBreakMode);
_debugger_events.OnEnterRunMode += new _dispDebuggerEvents_OnEnterRunModeEventHandler(_debugger_events_OnEnterRunMode);
_debugger_events.OnEnterDesignMode += new _dispDebuggerEvents_OnEnterDesignModeEventHandler(_debugger_events_OnEnterDesignMode);

For now ignore all but OnEnterBreakMode, this will be called whenever we enter any type of breakmode (stepping, breakpoints, exceptions etc). At this point we can do something quite nifty... any expression you can type into a watch window, can be evaluated here for example

Debugger debugger = _application_object.Debugger;
Expression exp = debugger.GetExpression("&g_my_global_foo,d");

if (exp.IsValidValue)
{
    Int64 value = Convert.ToInt64(exp.Value.ToString(), 10);
    System.Diagnostics.Debug.WriteLine(exp.Value.ToString());

    // do other fancy things here
}

At this point its really up to you to choose how you communicate this information to any external process. Personally my first simple solution was old school environment variable. This works perfectly well for a limited project, simply push your symbol address to the environment variable and let any other process (EEAddin for instance) read that environment variable and use it. Should you choose to publish the root address of your global systems for instance one might conceivably achieve full global memory addressing on your target.
The OnEnterRunMode hook should be used to "create" any resources you need on events "Attach" and "Launch"
The OnEnterDesignMode hook should be used to "destroy" those resources for events "Detach, EndProgram & Stop Debugging"

Debugging the debugger addin isn't difficult but is not obvious. Given the setup (Client Project) using (Debugger Addin) which launches Watch Window DLLS (EEAddin) the debugging method i used is:

Open the Debugger Addin project and build/run. This should open up an instance of VS2010 with your Addin loaded (check under Tools->Addin Manager
Within this second instance of VS2010 open up your Client Project, Build and run.
At this point breakpoints within the Debugger Addin are possible and you should be able to trap the incoming events.

Notes:

using environment variables will work for very simple projects however for production code i would recommend using process specific memory mapped files or another less hacky method
VS2008 can be supported however this has to be done as a C++/ATL project and requires a LOT of boilerplate code. Jason Weiler @ Airtight discusses this here and played a large part in enlightening me to this process.
As previous with EEAddin, this framework is not very forgiving and will require you to write defensively in all aspects. C# does handle a lot of this for you however the C++/ATL version will not. If you are writing an addin framework for a large group of programmers i would suggest significant boilerplate on all code.
the DTE framework is VERY powerful, anyone using it should read through the docs thoroughly to appreciate everything it can do and share what you do if you can :D

Becoming a Console Programmer : Extending The Watch Window (EEAddin)

2011-05-21T18:41:00.001-05:00

So we established that its possible to re-format what a Watch Window shows you, make things look better and therefore easier to consume. What we cannot do with that method tho is any real processing... but we can with a little more work.

consider the example

// blogtest.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"
#include <stdio.h>
#include <intrin.h>

enum e_union_test_case
{
    k_unset,
    k_type_string,
    k_type_int,
    k_type_float,
};

struct u_test_level1
{
    int m_type;
    union u_data
    {
        const char *m_string;
        int m_count;
        float m_value;
    }m_data;

    void set(const char *string)
    {
        m_type= k_type_string;
        m_data.m_string= string;
    }

    void set(int int_value)
    {
        m_type= k_type_int;
        m_data.m_count= int_value;
    }

    void set(float float_value)
    {
        m_type= k_type_float;
        m_data.m_value= float_value;
    }

    void print(void)
    {
        if (m_type == k_unset)
        {
            printf("unset\n");
        }

        if (m_type == k_type_string)
        {
            printf("%s \n",m_data.m_string);
        }
        if (m_type == k_type_int)
        {
            printf("%d \n",m_data.m_count);
        }
        if (m_type == k_type_float)
        {
            printf("%f \n",m_data.m_value);
        }
    }
};

struct s_test_level2
{
    static const int k_num_elements= 32;

    u_test_level1 m_elements[k_num_elements];
};


int _tmain(int argc, _TCHAR* argv[])
{
    s_test_level2 test;

    memset(&test,0,sizeof(test));

    test.m_elements[0].set("my foo");
    test.m_elements[1].set(7);
    test.m_elements[2].set(3.14159265358f);    // everyone loves Pi

    return 0;
}

similar to before setup a basic console project and drop in this code. Compile and run to the breakpoint and drop "test" into your watch window opening up the m_elements member... you should see something like.

not very readable at all... the print function shows what we "might" do to display these elements by using "m_type" to change the formatting options...

EEAddin is an option here, (Expression Evaluation Addin). This allows the autoexp.dat [AutoExpand] section to call into a dll for the preview display string. The dll can output any string it desires into the provided char buffer (obeying the limits of course).

you can find an example EEAddin within the VS2010 install, assuming you have the samples extracted (most installs will have a zip file) you will find the solution at

C:\Program Files (x86)\Microsoft Visual Studio 10.0\Samples\1033\VC2010Samples\C++\Debugging\EEaddin

NOTE: the example does not work out of the box, you must edit "ADDIN_API" and make it

#define ADDIN_API __declspec(dllexport) __stdcall

then move the HRESULT return to the opposite side of the ADDIN_API on use. Optionally you could achieve the same results by altering the project compiler options.

now paste the code

enum e_union_test_case
{
    k_unset,
    k_type_string,
    k_type_int,
    k_type_float,
};

struct u_test_level1
{
    int m_type;
    union u_data
    {
        const char *m_string;
        int m_count;
        float m_value;
    }m_data;

    void print(char *result_buffer, int result_size)
    {
        if (m_type == k_unset)
        {
            sprintf_s(result_buffer, result_size,"unset\n");
        }
        if (m_type == k_type_string)
        {
            sprintf_s(result_buffer, result_size,"str:\n");
        }
        if (m_type == k_type_int)
        {
            sprintf_s(result_buffer, result_size,"int:%d \n",m_data.m_count);
        }
        if (m_type == k_type_float)
        {
            sprintf_s(result_buffer, result_size,"flt:%f \n",m_data.m_value);
        }

    }
};

HRESULT ADDIN_API AddIn_blogtest( DWORD dwAddress, DEBUGHELPER *pHelper, int nBase, BOOL bUniStrings, char *pResult, size_t max, DWORD reserved )
{
    DWORD nGot;

    u_test_level1 example;

    // read file time from debuggee memory space
    if (pHelper->ReadDebuggeeMemoryEx(pHelper, pHelper->GetRealAddress(pHelper), sizeof(u_test_level1 ), &example, &nGot) != S_OK)
        return E_FAIL;
    if (nGot != sizeof(u_test_level1))
        return E_FAIL;

    

    example.print(pResult, max);

    return S_OK;
}

Into the existing "timeaddin.cpp" file and add the export to both the header (timeaddin.h) and the def file (eeaddin.def).

Compile this and it will generate EEaddin.dll, copy this dll to your VS2010 IDE folder (C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE).

Add the line

u_test_level1=$ADDIN(eeaddin.dll,AddIn_blogtest)

to your autoexp.dat right below the previous example, below the [AutoExpand] tag.

debug your application... your new display should be

each element now displays using the m_type correctly. We have an issue however, the string is not immediately available. This is due to the EEAddin working in a different memory space than the executable it is accessing. You'll note that the EEaddin DLL reads memory from the application manually using the supplied address AND by dint of the autoexp.dat line we know the size of the type we're representing. We must therefore pull any pointed at string data over manually in order to display it.

the new code is

enum e_union_test_case
{
    k_unset,
    k_type_string,
    k_type_int,
    k_type_float,
};

struct u_test_level1
{
    int m_type;
    union u_data
    {
        const char *m_string;
        int m_count;
        float m_value;
    }m_data;

    void print(char *result_buffer, int result_size)
    {
        if (m_type == k_unset)
        {
            sprintf_s(result_buffer, result_size,"unset\n");
        }
        if (m_type == k_type_string)
        {
            sprintf_s(result_buffer, result_size,"str:%s\n",m_data.m_string);
        }
        if (m_type == k_type_int)
        {
            sprintf_s(result_buffer, result_size,"int:%d \n",m_data.m_count);
        }
        if (m_type == k_type_float)
        {
            sprintf_s(result_buffer, result_size,"flt:%f \n",m_data.m_value);
        }

    }
};

HRESULT ADDIN_API AddIn_blogtest( DWORD dwAddress, DEBUGHELPER *pHelper, int nBase, BOOL bUniStrings, char *pResult, size_t max, DWORD reserved )
{
    DWORD nGot;

    u_test_level1 example;

    // read file time from debuggee memory space
    if (pHelper->ReadDebuggeeMemoryEx(pHelper, pHelper->GetRealAddress(pHelper), sizeof(u_test_level1 ), &example, &nGot) != S_OK)
        return E_FAIL;
    if (nGot != sizeof(u_test_level1))
        return E_FAIL;

    static const int local_buffer_size= 1024;
    char local_buffer[local_buffer_size];

    if (example.m_type == k_type_string && example.m_data.m_string != 0)
    {
        // pull over the string to our local buffer
        pHelper->ReadDebuggeeMemoryEx(pHelper, (DWORDLONG)example.m_data.m_string, local_buffer_size, local_buffer, &nGot);

        example.m_data.m_string= local_buffer;
    }

    example.print(pResult, max);

    return S_OK;
}

this code uses the pointer within the original "example" and pulls over an arbitrary length data stream (1kb) relying upon the string being null terminated. It then points our local copy of example.m_data.m_string to that local buffer before calling our new print(...) function which will now output the string as before. Our new display is:

which is exactly what we need.

Notes:

EEAddin is not forgiving

if you do something heinous within an EEAddin entrypoint it WILL take out the devenv that loaded it. Within Addins you MUST program defensively at all points, assume you will be sent bad data, assume your strings will not be null terminated, assume that the pointers you need to follow are bad and take you into no mans land. Verify everything. You'll note that i did not do this in my example but that is merely for the purposes of keeping the code samples short and sweet.

Debugging the EEAddin

load up the EEAddin project, Attach to the instance of devenv (debug it) that your Main project runs in, put breakpoints into your Addin entrypoint then hit debug on your application. Each time your Watch Window updates your EEAddin dll is loaded/called/unloaded so expect a lot of calls in the array case.

Global Symbols are not accessible

Accessing client memory requires the physical address of said client memory, if this address isn't accessible from the element you're debugging then you have no way to get to that address. An example of this limitation would be a handle into a global manager. The client code can call the manager to get the object, the debugger can only display the value of the handle, EEAddin can only access the internals OF the handle. It does not have knowledge of the manager and thus it cannot access the object itself.

Not all Endianess was made equal

if your target is not the same endianess as PC (little) such as xbox 360 then you must switch the endianess of data before reading it.

Alignment & Pointer size

you will have to manually handle both alignment differences (target <=> client) and pointer size differences. In my support thus far i have chosen 2 pathways. Some structures i have made cross platform entirely by supplying a construct that is 64bit and aware of its pointer size. For other types where this isn't possible i have used an element of the next topic to disclose to the DLL the size of our pointer.

Becoming a Console Programmer : Extending The Watch Window (autoexp.dat)

2011-05-21T16:17:00.001-05:00

Watch windows are important, they show us our data in various forms and generally enable us to debug effectively, sometimes however they need help.

Setup a simple windows console project and use the following code:

struct s_test_level1
{
    int m_count;
    char* m_string;
    float m_value;

    void print(void)
    {
        if (m_string)
        {
            printf("%s %d %f\n",m_string, m_count, m_value);
        }
    }
};

static const int k_num_elements= 32;
struct s_test_level2
{
    s_test_level1 m_elements[k_num_elements];
};

int _tmain(int argc, _TCHAR* argv[])
{
    s_test_level2 test;

    memset(&amp;test,0,sizeof(test));

    test.m_elements[0].m_string= "my foo";
    test.m_elements[0].m_count= strlen(test.m_elements[0].m_string);
    test.m_elements[0].m_value=3.14159265358f; // everyone loves Pi

    test.m_elements[0].print();

    return 0; // &lt;&lt;= breakpoint here
}

now run the program putting a breakpoint on indicated line. Open up a watch window and drop test into it opening up the m_elements array, you should see something like

This is a really simple but even now we see some details we don't always need... lets reformat it.

open up the file

VS2010 PC: C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Packages\Debugger\autoexp.dat
VS2008 PC: C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\Packages\Debugger\autoexp.dat
Xbox360: C:\Program Files (x86)\Microsoft Xbox 360 SDK\bin\win32\autoexp.dat for 360

search for "[AutoExpand]" and add the line

s_test_level1=<m_count,d>, <m_string,s>, <m_value,f>f

save that file and re run the test program.... you should now see

a much easier to read version of the same data.

All the original information is still available should you need it but this new format is much easier to quickly consume.

This form of watch window help is somewhat limited however, you can only interpret existing data, remove superfluous detail, do some VERY rudimentary math and generally make things cleaner and easier to consume; see the next post for MORE.

All of the help required to use autoexp.dat is within the file itself. Note that the "[Visualizer]" section is currently not available for xbox 360 targets but is VERY powerful for other targets.

enjoy and feel free to ask questions.

Incoming Topics (what should I blog about next?)

2011-05-20T09:11:00.006-05:00

So i've not had much time recently for various reasons and i apologize to those who were following avidly. I should however have some time very soon and wanted to give people the change to decide which topic i talk about... so go for it... comment away (these topics are for x86, x64 and xbox360 tho i cannot directly discuss xbox360 almost all code applies to all 3 targets)

Extending "Watch Window" Vizualization
Visual Studio debugger plugins
How to get YOUR object to construct first
Creating a "Premain" function that runs before global constructors
Suggestions?

Static assert with a "real" message.

2010-08-07T22:09:00.004-05:00

static asserts are important for many reasons and are heavily used in most large codebases esp when said codebase has many target environments. They inform the programmer when an assumption fails at compile time allowing for the runtime code to avoid expensive verification.

#define COMPILE_ASSERT(x) extern int __dummy[(int)x]

Which is typically used as

COMPILE_ASSERT(sizeof(int) == sizeof(unsigned));
COMPILE_ASSERT(sizeof(int) == sizeof(char));

the second example if obviously wrong and provides the output.

error C2466: cannot allocate an array of constant size 0

Which is not very useful other than as a signal that something bad occurred.

Now consider the form

#pragma warning (disable: 4483)

typedef char __identifier("compile assert:");

#define COMPILE_STRING_ASSERT(condition, name) \
typedef __identifier("compile assert:") __identifier(name)[1]; \
typedef __identifier("compile assert:") __identifier(name)[(condition) ? 1 : 0];

define COMPILE_ASSERT(condition) COMPILE_STRING_ASSERT(condition, #condition)

The above uses the identifier keyword which is technically an error however the pragma warning disables the error (yes it doesn't make sense). This keyword allows the user to define ANY identifier, valid C++ or not.

COMPILE_ASSERT(sizeof(int) == sizeof(char));

Now provides the output

'compile assert: sizeof(int) == sizeof(char)[1]'

which is a much more palatable message esp when considering automatic build systems.

Note2

__identifier is specific to Visual Studio and may not be supported by other compilers.
VC++ 2010 introduces static_assert for all targets, tho it is less powerful than using __identifier.
SNC may support both, i will try to confirm.

Becoming a console programmer : Math Libraries Part 2

2010-08-07T12:52:00.008-05:00

Implementation

Generally speaking implementation of a math library is not determined by the programmer DOING the implementation but by the clients of the system and the platforms that require support.

Given the myriad targets currently available AND the similarly complex client base i always recommend a relatively complex abstraction built on simple rules:

separate the client facing api into semantic types
define those types using platform specific typedefs or defines
implement using a cross platform api with platform specific implementations.

for example we might want the client to use matrix, point, vector and proxy type for float constants.

the semantic types define their functionality in terms of the low level api which in turn defines a set of common operations that all the semantic types can use at will. This means that to provide support for a new platform for ALL semantic types (which there are usually far more than 4) the programmer simply has to satisfy the lowest level api.

That low level api is typically relatively small providing wrappers for some of the basic operations such as add, mul, madd, permute, shuffle/vsel etc whilst also implementing some of the more complex but common operations such as normalize, permute_operation etc.

To flesh things out for the client side the programmer should define operators and functions that provide mathematically sensible and desirable operations such as

vector = point - point,
point = point + vector * scale.

the proxy type mentioned above would be a simd representation of a single float allowing the math itself to remain in registers.

The key to maintaining long mathematically complex code in registers is in ensuring the semantic types are well defined within themselves and as a whole.

there is a lot more to be said on the subject of math libraries however without infringing on company policies both new and old i cannot expand further. The above should however provide a good start and from there a good programmer will run with it and discover for themselves the rest.

enjoy.

Becoming a console programmer : Math Libraries Part 1

2010-07-05T09:28:00.007-05:00

Writing a Math Library

Over the years i've had to write, re-write and optimize math libraries far too often. Sometimes 2-3 times within 1 project. This might sound strange given that time is money however, counter intuitively, that fact is the driving force behind so many re-writes.

I believe that its possible to write an "ok" math library that will work for "all" use cases but nigh on almost impossible to write an "optimal" math library to suite all cases. Large and still growing programming teams, changing platforms and multiple simultaneous platforms and differing requirements across target executables compound the issue to the point where its simply too complex.

Gustavo Oliveira discusses some of the considerations here which can be roughly summed up as

use intrinsics
return by value
declare data purely
provide operators and functions
inline everything
replicate results in SIMD

I don't agree with all of the above as hard rules but prefer to judge based on each specific platform and target build.

1. Use Intrinsics

Intrinsics are functions whose implementation is handled entirely by the compiler. See Wiki for more. Most compilers provide the ability to write inline assembly routines and this can be very useful for pieces of code that require significant hand optimization however the for short functions such as those seen in vector math libraries, the bulk of the programming work is in the handling of the calling convention and how the return values are provided and interpreted. Due to the myriad entrypoints to support for math libraries it quickly gets out of control and unmanageable.

Intrinsics hide these issues, the compiler has intimate knowledge of the intrinsic function, how it should be called, how it returns, how it can be combined with other instructions and every possible combination you might otherwise have to handle with inline assembly. It does this optimally every time (IME).

Use them.

2. "XXX by Value"

Return by value, on this one i both agree and disagree. It isn't hard and fast in all cases, especially so in cases where a function has more than 1 resultant value. Generally speaking however you should, where possible, write code that returns register sized values as "return by value". Under optimization the compiler will attempt to keep the value in a register and not commit to memory thus making the code faster. Certain platforms are however limited to a small set of registers for SIMD math and thus storage back to memory is more prevalent. I recommend investigating this for your platform(s) of choice.

The XNAMath library handles this by specifying function parameters that can be altered per platform. For example

XMFINLINE XMVECTOR XMVector2IntersectLine
(
    FXMVECTOR Line1Point1,
    FXMVECTOR Line1Point2,
    FXMVECTOR Line2Point1,
    CXMVECTOR Line2Point2                                  <== different type
)

FXMVector is defined to be a constant, pass by reference type, for platforms that don't support passing in registers, under x86 we are limited to passing the 1st to 3rd parameters as registers so only the first three params to this function use this type. The typedef is thus defined as

typedef const XMVECTOR FXMVECTOR;

for x86 / xbox 360.
On xbox 360 the 4th parameter is defined as

typedef const XMVECTOR CXMVECTOR;

whereas on x86 its defined as

typedef const XMVECTOR&  CXMVECTOR;

This allows the actual code to be completely independent of the underlying hardware limitations. Under x64 the PC has more registers to use thus the x86 limited types can be defined to pass by value without changing ought but the typedef.

This platform agnostic setup becomes very important as more and more platforms need support.

Note: On some platforms its possible to define this as a directive to the compiler. On xbox-360 for instance the compiler provides a pragma (passinreg) applying to a single float, double or __vector4. This informs the compiler that it should "prefer" using registers over temporary storage and helps avoid LHS (Load Hits Store).

3. Declare data purely

This is a good general rule however brings with it issues that are hard to avoid. The theory is that the compiler can better optimize a native type than a wrapper type, which is generally the case apart from types supported by extended instructions such as VMX. On 360 the passinreg pragma does not require a pure type to be used provided the data is the only member of the class being created. This allows you to define your type as

__declspec(passinreg)
class my_foo_vector
{
private:
    __vector4 m_vector_data;

public:

    // my foo vector interface
};

and the compiler will treat it as if it were the __vector4 type. This is similarly the case for float/double types and is enormously useful.

What this means is that we can wrap the datatypes and better define their syntax and semantics without negatively impacting the resultant code generation. Typical types defined by a math library might be

point4d - 3d position in space, w = 1
direction4d - 3d direction vector, w = 0
quaternion - simple quaternion container
sphere - 3d position, w= radius
plane - 3d normal, w= distance
euler4d - euler angles, w not used.
float4d - x=y=z=w

each of the types represents a specific behavior and has specific interfaces both local to the type and to allow generation of them. For instance

direction4d generate_direction(point4d_const_in a, point4d_const_in b);
point4d offset_point(point4d_const_in a, direction4d dir, float4d length);

Once a high level math library is defined in terms of behavior rather than type it is much more usable by general programmers who then don't need to worry about the implementation itself, only that the behavior they require is well defined by the type they choose.

So, i would recommend that you investigate your platform and make the call, sometimes platforms don't provide any type of "passinreg" functionality and then it becomes a judgement call. Personally i prefer a wrapped type than a pure type simply because it allows me to limit the functionality i provide to the clients of my system.

4. Operators vs Functions
we can all see that the code

position = position_a + direction_b * scalar_c;

is easy to read and its precedence well defined. Just like general math terms, the more complex things get the more difficult it is to understand and thus we expand things out to more meta terms (temporaries). The compiler will do the same when presented with a long sequence of operators.

IME however... it doesn't help to re-factor that long sequence of operators into a long(er?) sequence of function calls. I am willing to believe there are circumstances where this occurs however the general case that i examine when writing code doesn't show it.

For now i recommend writing all sensible operators, of course those defined as "sensible" are often up for discussion. I tend to do +-*/ and all variants leaving it at that. I highly recommend AGAINST ever overloading operators to supply cross product / dot product.

5. Inline everything.

Another general rule that i agree with (but for those pesky debug builds). Personally i choose to force the inlining of all base level math functions.

Most development occurs in a debug form, we've written the code and it didn't work first time, or we've come back to fix an edge case 6months after writing the code. For most game logic code the math library will be a significant support system and in many cases it is >50% OF said code.

In a release build it is highly likely that most of our math functions will boil down to a small set of instructions, in many cases a single instruction acting on registers alone. This is perfect for our needs in a release environment. Due to the nature of math code we tend to use a lot of functions for very little actual work per function. In a full debug build the compiler will turn off all optimisations, this includes the general use of pass-by-register and return by register, it will also NOT inline functions defined as "inline" while (most of the time) maintaining the inlining of code defined as "forceinline". The net result is we get a LOT more actual code for a client function using the math library. The code is horribly inefficient, dumping to memory after every operation which results in significant LHS issues.

Vectorized inline reliant math in debug is slow, so don't inline in debug.

What i choose to do is define the math library as inline for all builds other than debug, in debug we instance the functions into a release optimized lib that the client executable links against. The math code itself isn't inlined so it doesn't bloat AND the compiler will honor the calling conventions of the code. It won't be anywhere near as fast as the full release inline versions but you can expect ~40% of the speed of said build completed to ~5-10% for full debug with inlining.

6. Replicate

The dot product mathematically provides a scalar value but in code this value most often used AS an operand to a vector operation. Consider reflection

i  - [in] A floating-point reflection vector.
n - [in] A floating-point ray direction vector.
v = i - 2 * dot(i, n) * n

a naive implementation might do

// Result = Incident - (2 * dot(Incident, Normal)) * Normal
float f_dot = dot_product(i , n);    // vector -> memory
vector4 result= i - (2.0f   * f_dot * n);

this will roughly boil down to

// simd register -> memory
float f_dot = dot_product(i , n);   
// LHS (~25cycles)
// memory -> float register + mul -> memory
float tmp= 2.0f * f_dot
// LHS (~25cycles)
// memory -> simd register + replicate
vector4 promote= replicate(tmp);   
// simd mul, simd subtract
vector4 result= i - (promote * n);

for which we get 2 LHS hits. (360 PPC chip experience, not sure on PC side)
Now if we consider the instructions above its not a huge jump to see how to remove the LHS problems.

// simd register return from dot_product
float4 f_dot4 = dot_product(i , n);   
float4 f_dot4_2 = f_dot4 + f_dot4;
// simd mul, simd subtract
vector4 result= i - (f_dot4_2 * n);

this is a relatively simple change that removes some very expensive operations & delays... it can be taken further with the use of vnmsubfp which will perform result= - ((b * c) - a).

the general principle is that vector data should remain registers/vector storage as long as possible.

Part 2 will discuss implementation details.

Becoming a console programmer : Structure Alignment

2010-06-17T20:14:00.014-05:00

This is an area that all programmers should be familiar with however i've met and worked with a fair few who never had to care... until they had to care.
consider:

struct foo1
{
    char m_char;
};

struct foo2
{
    int  m_int;
    char m_char;
};
struct foo3
{
    long long m_longlong;
    int  m_int;
    char m_char1;
    char m_char2;
};

using default settings under x86 these three structs are 1, 8 and 16 bytes respectively. What is actually allocated is


struct foo2
{
    int  m_int;            // offset 0 : size 4 bytes
    char m_char;           // offset 4 : size 1 byte
    char m_pad[3];         //                             3 bytes padding
                          // 8 bytes
};
struct foo3
{
    long long m_longlong;  // offset 0  : size 8 bytes
    int  m_int;            // offset 8  : size 4 bytes
    char m_char1;          // offset 12 : size 1 byte
    char m_char2;          // offset 13 : size 1 byte
    char m_pad[2];         //                            2 bytes padding
                          // 16 bytes total
};

now lets take foo3 and jumble those members a little.

struct foo4
{
    char m_char1;
    long long m_longlong;
    char m_char2;
    int  m_int;
};

One might expect foo4 to be the same size as foo3 however it is not, it comes out to 24 bytes, the padding is as follows.

struct foo4
{
    char m_char1;           // offset 0  : size 1 byte
    char m_pad7[7];         //                          7 bytes padding
    long long m_longlong;   // offset 8  : 8 bytes
    char m_char2;           // offset 16 : 1 byte
    char m_pad[3];          //                          3 bytes padding
    int  m_int;             // offset 20 : 4 bytes
};                          // 24 bytes total

each type contained within a struct must begin on a natively aligned boundary, so the 8 byte "long long" must start on an 8 byte offset within the struct but be after m_char1 so the compiler inserts 7 bytes of "padding" that the user cannot directly access to ensure this.

to inspect this for yourself use the code


printf("\n(%d,",offsetof(foo4, m_char1));
printf("%d,",offsetof(foo4, m_longlong));
printf("%d,",offsetof(foo4, m_char2));
printf("%d)",offsetof(foo4, m_int));
printf(": %d",sizeof(foo4));

output : "(0, 8, 16, 20): 24"

Manual Alignment
It is often beneficial to align the base of a structure beyond its native requirements. I won't go into the reasons for this here (thats a later subject) but it does change the internals of a struct when you do this.
Consider:

struct _CRT_ALIGN(16) foo_aligned4
{
    char m_char1;
    long long m_longlong;
    char m_char2;
    int  m_int;
};

the

    _CRT_ALIGN(16)

informs the compiler that this structure should be 16 byte aligned (and therefore a multiple of 16 bytes in size), this is actually a wrapper around a declspec (see below for further reading)

In the case of foo4 this doesn't change the offsets of any of the internals but it does change the size of the struct

struct _CRT_ALIGN(16) foo_aligned4
{
    char m_char1;           // offset 0  : size 1 byte
                            //                          7 bytes padding
    long long m_longlong;   // offset 8  : 8 bytes
    char m_char2;           // offset 16 : 1 byte
                            //                          3 bytes padding
    int  m_int;             // offset 20 : 4 bytes
                                                      8 bytes adding
};                          // 32 bytes total

if we were to then include an instance of foo_aligned4 inside another struct, said struct will inherit the same aligment. In general a structure will always be aligned to the largest alignment of its components.

struct foo_composite
{
    char m_char1;                // offset  0: size 1 byte
                                //                             15 bytes padding
    foo_aligned4 m_foo_aligned4; // offset  16: size 32 byte
    long long m_longlong;        // offset  48: size 8 byte
    char m_char2;                // offset  56: size 1 byte
                                //                             3 bytes padding
    int  m_int;                  // offset  60: size 4 byte
                                // total 64 bytes
};

Now given our composite structure of 64 bytes lets see how small we can make it without altering the client facing elements.

struct _CRT_ALIGN(16) foo_aligned4_opt
{
    long long m_longlong;            // offset  0: size 8 bytes
    int  m_int;                      // offset  8: size 4 bytes
    char m_char1;                    // offset  12: size 1 byte
    char m_char2;                    // offset  13: size 1 byte
    char m_pad[2];                   // offset  14: size 2 bytes
};                                   // 16 bytes total


struct foo_composite_opt
{
    foo_aligned4_opt m_foo_aligned4;// offset  0: size 16 bytes
    long long m_longlong;           // offset  16: size 8 byte
    int  m_int;                     // offset  24: size 4 byte
    char m_char1;                   // offset  28: size 1 byte
    char m_char2;                   // offset  29: size 1 byte
    char m_pad[2];                  // offset  30: size 2 bytes
                                   // 32 bytes total
};

Our new composite struct is 32 bytes rather than 64, every member still exists but in a different configuration.
You may notice that I manually inserted the padding into the struct, this isn't required but personally i prefer it in cases where i require specific alignment or optimal structure size.

So key points are

structures inherit the largest alignment of their elements
elements will always begin at an offset that is a multiple of their alignment
the compiler will insert padding into your structure to achieve alignment requirements
structure size will always be a multiple of its alignment
to reduce alignment padding, arrange structures largest to smallest in alignement requirements.

Note that normal allocators, such as malloc, C++ operator new, and the Win32 allocators return memory that will most likely not be sufficiently aligned for __declspec(align(#)) structures or arrays of structures. I recommend writing your own allocator either as a wrapper around these or as a region allocator to achieve alignment above default (4 or 8 bytes usually).

further reading -_CRL_ALIGN, __alignof,

Becoming a console programmer : LHS

2010-06-17T08:36:00.004-05:00

Zak Whaley pointed me at this article on "Some Assembly required" discussing float-> int conversion and what occurs when its done following onto the LHS article within the same site which describes relatively well what "Load Hits Store" (LHS) is and how to avoid it using the restrict keyword.

Later when i discuss math libraries i will come back to this with respect to vector <=> float interchange and how to effectively remove it from transient use.

Becoming a console programmer

2010-06-16T21:39:00.004-05:00

Recently on a games industry forum, a friend who teaches over in the Netherlands (not Denmark :p) was asking us (programmers) what we would ensure a PC programmer moving to Console Development should know, before considering themselves a "console programmer"

None of these particularly apply to console programming however i've found over the years that programmers who come form a pure PC background often are lacking in this area so i personally consider them important.

Programmers new to console should strive to achieve:

an awareness of memory consumption - PC programmers tend to be somewhat liberal with memory allocation and stack usage. On Consoles we're often faced with very limited memory resources that shrink with time (due to the game getting larger) rather than the opposite on PC where virtual memory allows incredibly large memory footprints.
knowledge of alignment - internal to structures and how it pertains to their size AND how not paying attention can massively increase cache misses
appreciation of the cost of type conversions both in terms of float <=> integer, float <=> simd and the assumed simpler integer to integer conversions thar can cost significantly more than is assumed.
full understanding of what a Load Hits Store (LHS) is, how it occurs, how to avoid it and how to spot it in everyday code.
full understanding of the implications of a cache miss, what it is, how to avoid them and how to fill the time around the unavoidable ones.
full understanding of what a branch is, how it affects the cpu, how to avoid using them and what patterns should be avoided to minimize them in general purpose code
understand what select functions are (fsel, vsel family) and how/when to use them.

SIMD is important however there are MANY companies that write a math lib once and never look back, i've been guilty of this myself however now i know i was wrong back then and no doubt in many ways what i've written most recently is still wrong.

writing it once implies that when you wrote the lib you knew everything there was to know then and nothing changed in the years afterwards.... yes many games will be fine with an "ok" math lib however a "good" math lib can change the performance of a game and an entire team.

re-write it as often as you can afford learning from the mistakes each time... so much is based upon it that this is one of the few systems that i'd recommend this behavior on.

so a final point

learn simd math in all its forms, learn the extra instructions that most don't use and what they provide, learn how to use #.INF to your advantage and most of all use instructions such as vsel / shuffle/ permute to their fullest advantage.

I will be posting expansions on the above points over the next few weeks depending on demand.

Becoming a programmer - thinking algorithmically

2009-11-19T09:58:00.002-06:00

[duped from a post i made on www.thechaosengine.com an industry only forum]

last time I taught someone how to think algorithmically I asked them... what is a seemingly simply task ... that you do ALL the time.

they chose - make a cup of tea.

I asked them to write down, in absolutely the order they would do it, everything they need in order to do it, everything they do to go from no cup of tea...to a fresh cup in hand.

each time they finished I asked them to follow their own instructions using only the inputs they specified, only the instructions they had written down....

took about 20 iterations to get it right but each time they got better.

I then asked them to repeat that process on their own with other simple tasks....

I did this before introducing any language or limits... simple english....

2 weeks later I sat in on them going through the same process for "checking & replying to email"... 100% correct.

after that... developing small programs in C became a syntactic problem only... write down the english as comments... flesh it out with the C.

The demise of the "Low Level" programmer.

2008-12-14T18:56:00.008-06:00

[updated 8/9/2011]

When I started programming many of the elements we take for granted now, did not exist. There was no DirectX and not many compatible libs were available for the free compilers of the day. So I had to write my own code for most basic programs, keyboard handlers, mouse handlers, video memory accessors, rasterizers, texture mappers, blitters… the programs I wrote then were 100% my own code and I had to be able to handle anything and everything.
Personally I’ve always been interested in what was going on under the hood so this suited me just fine. I always dug into the details and I almost always end up programming as close to the bone ON the hardware (or OS) as I possibly can both to eek out as much performance as possible AND to satisfy my own hunger for knowledge.
Combine the two and what you get today is someone who enjoys spending 5 days making that single function 20x faster, who revels in reducing the memory footprint of the primary data structure by 1 byte per element across the entire program whilst simultaneously writing a pre-caching system to avoid the special case issues…. who… well you get the picture… I’m an OCD level sport optimizing geek. To such a degree that I keep a notepad by my bed to take notes when I wake up from a “bug fix” dream or a “eureka optimization” doze…yes i’m that sick.
Over the last decade I’ve been involved in the hiring process at many studios and in more recent years I’ve noticed a pattern. Knowledge of what is generally considered “low-level” programming is waning. Many programmers know enough to get through a C# or C++ test, but don’t understand something as basic (and important) as the behavior of memory or god forbid a cache. They don’t seem to grasp that one must understand the native environment you’re working in before going ahead and writing a program to run within it. The intricacies of floating point vs fixed point math are completely lost on them as the term “fixed point” brings about a blank stare; floating point numbers are best right?. I once mentioned bit shifting to an experienced engineer of 10 years and was devastated by the complete lack of basic understanding.
It depresses me that so much of what I consider to be essential is simply not being taught anymore. I’m not talking about assembly language per se; even those of us who used to spend hours writing assembly now more often opt to use intrinsics built into compilers to avoid the stress and complication. What I’m talking about is simply the understanding of WHAT is happening when someone does i++ and not ++i, why one might opt to stripe a memory copy/set in certain circumstances.
So here goes… a list of things I believe all console programmers (and recommend to all programmers as good reading) should fully understand with links to educate where possible. (feel free to suggest more/better links)

Floating Point Numbers

They are very useful but often used in situations where they simply don’t suit the solution the programmer is attempting to write. The following links should provide some background and info on where they are not so useful, what the pitfalls are and sometimes even how to avoid them.
http://www.cprogramming.com/tutorial.html#fptutorial
http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/
http://www.codeproject.com/KB/recipes/float_point.aspx
http://drdobbs.com/184402741?pgno=4
http://users.tkk.fi/jhi/infnan.html

Fixed Point Numbers

Fixed point math is mildly old school but it is VERY useful both to understand its makeup and to use. Sadly because it is considered old school many of the online sources are out of date.
http://x86asm.net/articles/fixed-point-arithmetic-and-tricks/
http://gameprogrammer.com/4-fixed.html

Processor Cache Behavior / Memory

Bit Shifting

Branch Prediction

This may be lower level that people think they need to go… but they’d be wrong. Understanding how the hardware you’re programming for treats branches can affect performance to a HUGE degree… far more than most programmers may appreciate re: death by a thousand cuts
http://cellperformance.beyond3d.com/articles/2006/04/background-on-branching.html
http://igoro.com/archive/fast-and-slow-if-statements-branch-prediction-in-modern-processors/
http://www.k8gu.com/ece.umn.edu/documents/classes/ece362-branch-prediction.pdf
http://www.cs.ucr.edu/~gupta/teaching/203A-09/My6.pdf

Sorting

This isn’t really low-level but something i consider “basic” and its an area where many programmers are simply lacking in understanding. Do yourself a favor and play around with this link, read the links it sends you to for each algorithm and try to grasp when each might be used, the properties as described and the next time you need to sort something… consult it.
http://www.sorting-algorithms.com/
another good link with sub links: http://corte.si//posts/code/visualisingsorting/index.html
a funny one (but still strangely useful), the bubble sort dance: http://www.youtube.com/watch?v=lyZQPjUT5B4

So there you go, I’ve likely missed some aspects that should make the list but if you can grasp the above then you’re more likely to get noticed at interviews and you’ll certainly be a better programmer for it.

Edit 8/8/2011 – It seems I didn’t do a very good job at explaining the core audience for the article, I apologize for that. The programmers I would like to see learning and absorbing these “low level” details are those who would see themselves working in console games. The HW is fixed for long periods of the time and resources become scarce usually within 1 game cycle; understanding how to better utilize resources becomes a key part of the job.
While i believe that most programmers would benefit from understanding more about the medium they work in it is certainly not required
Hopefully you enjoyed the article in the spirit it was intended.
Andy Firth

Working with Instant Messenger.

2008-10-01T13:26:00.007-05:00

Personally i could not work without instant messenger (IM) in all of its guises. I use trillian on my main dev machine to talk to friends, colleagues and peers around the world. I do it without thinking and i do it without negatively affecting my work; it is a tool of life and work. I use it to discuss tech within the game/engine, to setup meetings, to discuss scheduling issues, to setup tests... essentially anything work related (and some not work related things).

Recently i did a poll on a well populated games dev only forum. I started the poll based entirely on a single conversation with a friend at R* San Diego who said that R* do not allow IM usage. After a few hours it became apparent that around 10% of developers do not allow IM usage in any form, the other 90% largely given open access.

So why is it some companies simply don't allow it?

It seems that IM usage is a worry to the powers that be who think maybe details that should not be discussed outside of work might be discussed over IM which could indeed be considered a security leak if the receipient is not employed by said company, on the other hand what about internal only IM. I would ask, how likely is it that someone would spend the time (and money) to infiltrate another persons messenger account in order to log the conversations that would require that person to be logged on in the first place. I don't know how easy/difficult this is (and maybe my opinion would change if i knew) but it seems very unlikely that this is happening and thus i question the decision to remove such a useful tool from developers.

I also heard some rumblings that certain companies believe IM is a distraction and they need to remove any and all of those lest the peons get too engrossed in something other than work. Personally i think that people who use IM as a distraction at the expense of their work will likely find another distraction if / when IM is removed.

In the past i've worked for Activision and EA, two of the largest publishers, both of whome encourage IM usage. EA has an SIP setup internal to each studio that provides access to ALL. ATVI don't actively help people setup IM but they do not discourage it either, many of the studio level heads use it. It is a useful element of everyday developer life.

So, my opinions are known... what are yours?

First!!

2008-09-30T20:59:00.000-05:00

So I've been reading blogs for a while now of various people and today a friend of mine asked if i had one... upon hearing i didn't he went ahead (with my permission) and published on his blog some of my thoughts and helpful hints regarding Vista64... I had always thought blogs took a long time however his post took about 30 seconds longer than it took for me to tell him my info...

so i figured its high time i made one myself... hopefully i post something useful here soon.