Doom3world

Doom 3 MetaMod

I'm writing a metamod for the Doom3 engine.

A brief description of DMM:


The system uses SourceHook. If you've ever written a plugin for MetaMod:Source for the Half-Life2 engine, you are already familiar with SourceHook. This allows you to hook any virtual function you desire, multiple times if necessary. Since the Doom interface is primarly virtual functions, this works perfect. You can declare a "hook" of say some function Printf() in idGame, and when that function is called anywhere your function is called. Your function can choose to just observe this event, modify it, or completely override the original Printf() call (ie, you can stop the real function from being called if you want).

The website for DMM is: http://doommod.sf.net/
DMM is open source, and released under the GPL, as are plugins.
Currently I have not released any code, but it does work.

Screenshots are available here: http://doommod.sf.net/ss/
There are only two at the moment, shows very basic DMM interface (1 plugin is loaded called dmm_stub).

A more detailed description is available at: http://doommod.sourceforge.net/index.php?nav=whatisit
You'll have to forgive me if anything there is inaccurate relating to the engine setup (please correct me if I'm wrong), I'm fairly new to the Doom engine, but I'm a veteran to C/C++.

In any event, currently I'm trying to rebuild the SDK. As many of you know, the code is quite heavily coupled. Therefore if you wanted your plugin to say, deal with some physics thing, you may have to include the headers for the physics code and possibly compile and link the core physics implementation.

Breaking up the SDK is a primary concern for me at this point in development; it is key to get an SDK where the headers can be easily used by plugins, otherwise the benefit of plugins greatly decreases. As of now (about 2 days work with a friend of mine, CyberMind, who created a similar project Quake3 MultiMod for quake3 http://q3mm.org/ ) I have 74 SDK files split off and operating properly.

I do need help though. As you guys are coders for the Doom engine, what parts of the SDK do you feel are critical to break off and fix? I have idLib almost done, as well as things like idGame and the like.

I'm also open to suggestions and criticisms, this is my first time attempting to implement any type of "metamod" system (or any game for that matter).

Note: I'm hoping creating DMM and getting it out to you guys will really help in the development of new mods (well plugins would be the word for it with DMM). I've noticed it's very difficult at this time to create a small mod, such as the flashlight one, and use other smaller mods alongside it, such as the infinite run or something similar. I beleive this system would bring modularity to the mod community, and help produce quality works that are easy to distribute and use.

I'm not familiar with sourcehook, but will it be possible to abstract this so that it can run on any OS? It would be a pity if it was windows-only...

Actually I'm running Gentoo Linux. I have not yet ported DMM to Windows (there should only be a few small changes), SourceHook however is fully compatible with both Windows and Linux.

SourceHook is easy to use. Here's a code segment from my dmm_stub plugin which intercepts any calls to idCommon->Printf().



The first line:

Creates a compile-time function which will handle how to deal with Printf() calls.

The line:

Is where my plugin tells SourceHook that I want to intercept idCommon->Printf() calls. My custom function I want to be called before the real Printf() is test().

The line:

Tells SourceHook that I do not want to stop the real Printf() from being called. There are a few different return macros which allow you to do things like stop the real Printf() from being called, or to override what the real Printf() return should be (obviously there is no return for a void function, but it's useful for functions that return information you want to handle yourself).

It's not too difficult after you step in and hook two or three functions, you get the hang of it.

Their official documentation is at: http://www.sourcemm.net/?go=docs&doc=coding
Most of it is relevant to DMM, except for a few of the "Other Macros" and "Global Variables" which are specific to MetaMod:Source.

It's a very powerful system, especially for Doom where virtual functions are all over the place

this sounds awesome. sure we will have to make most of the methods on the SDK classes virtual but they really should be anyway. good work.

Can you tell us how this exactly works? I don't want to look to the code right now, just to see how it works. One point I'm really interested in is, wether it is possible to also hook into functions that are inaccessible because we don't have the sourcecode for it. From your description here in the postings it is not clear to me what exactly is possible and what not.

It still sounds pretty cool to me.

I didn't write SourceHook (the part of the code that's responsible for actually seeking out and modifying the vtable pointers), but from what the author tells me you can hook any function where you have:

1. a virtual function you want to hook
2. the function prototype (in the SDK headers)
3. a pointer to an instance of the object you want to hook

Most of the stuff from the engine API are in the form of virtual functions, as is the game API.

If you don't have the SDK handy, here's a real quick breakdown of those.

The game API:


Unfortunately game_local (the actual game class) inherits from this class and therefore the newly added functions are not virtual and unhookable. However, any engine to game calls will only use the above functions, as the engine doesn't know about how game_local is constructed. Therefore you should still have complete access to everything game-wise.

The engine code is much more in depth, but here's a quick rundown of that section. The following structure is given to a game from the engine when it is loaded:


Most (if not all, I don't recall off the top of my head) are virtual interfaces. A quick example:


In my previous post I used an example where I was hooking idCommon::Printf() that was supplied by the engine from the gameImport_t structure

Hopefully this answers your question. It's pretty nice just about everything in terms of engine<->game interaction is provided in a virtual context, so DMM plugins should have the ability to do just about anything you would want to do with them.

The execution of DMM is relatively straight forward. Normally...

1. the engine finds and loads the game, passing to it the engine API
2. the game does all its startup stuff
3. the game returns its API to the engine

With DMM in the mix, the procedure is modified to be:

1. the engine finds and loads DMM, passing to it the engine API
2. DMM finds and loads the game, passing to it the engine API
3. the game does all its startup stuff
4. the game returns its API to DMM
5. DMM finds all plugins
6. DMM loads a plugin, passing to it the engine API and game API
7. go back to the previous step until all plugins are loaded
8. DMM returns the game API to the engine

After loading, besides unloading on shutdown, DMM's job is to manage how plugins should behave (ie, paused, unpaused, etc) and provide a console interface for user control. Other smaller abilities may be added if required, but DMM is more-or-less idle after loading, the plugins are where all the "work" is done.

edit

You don't need the source code of a function to hook it, just an instance to the object and the prototype. For example, no source to the idCommon::Printf() function is available, but the prototype and instance is so it is hookable via SourceHook.

If you're interested in how SourceHook works, I have a very basic understanding of it. You can email the author for a detailed explaination, but this might help also...
To hook a function you must declare with a macro (see my previous post) what the function you are hooking is called, and the parameters and return types it takes. This macro creates a compile-time function which is used to determine how to proceed with handling the hook you make to it later.
When you actually declare a hook with SourceHook, SourceHook will go to the address in memory of the instance of the object you specify, and attempt to locate its vtable. Within the vtable it will look for the function you asked it to, and modify the pointer to the function normally called to point to SourceHook. When that function is called, since the vtable pointer is now directed at SourceHook, SourceHook will go through a list it has stored internally of what functions should be called at this point and in what order. Typically, there are pre-functions, the real function, then post-functions, called in that order. When you hook a function you can specify if you want it to be pre or post executed. Unhooking a function will instruct SourceHook to relocate the vtable entry it found previously, and modify the pointer to the hooked function to be the original pointer. This in effect causes no additional overhead of invoking a hooked function (except the little that is produced by SourceHook's internal list, and that of the actual functions that have to be invoked).

Actually I'm mostly interested in modifying the behaviour of renderSystem->CaptureRenderToFile, which is virtual, so that SourceHook should work. But I don't know if it will help me in this case, because I need to modify it's behaviour. As I understand this, I can only hook into a function to get a kind of trigger, when I want to do something extra, but it could still be helpfull to me.



We will see. What I need is to redirect the rendering, which is done to a file, into some memry block instead. Don't know if this would be possible.




I will definitley ask him, though I have some ideas how this may work. I once wrote such a function myself, but it was quite different. For my code I wrote a disassmbler, that disassmbled the first few instructions and move them to some other location. Then I overwrote the original code with a jump pointing to my own function and after I was finished with my own code, it called the original code which I saved before. But I doubt that SourceHook uses this approach, because it only works with virtual functions.

Thanks for the info!

In worst case you could just move entire functions from the SDK to your plugin, and modify them as needed, then tell tell SourceHook not to call the originial function, thereby replacing the SDK's version with your custom one. A benefit of this is that your compiled plugin would be much smaller than the SDK, and easily distrubutable if you wanted to hand it out.



I'm relatively new to this area, seems very interesting to me. You may be talking about Detours, which is similar to SourceHook but can hook nonvirtual functions be overwriting a few opcodes in the function to make a jump to your custom one

SourceHook only overwrites the function pointer in the vtable. It sets the vtable entry to its own function (generated by the SH_DECL_HOOK* macros), which iterates through the pre hooks and invokes them, then invokes the original function (if required), then iterates through the post hooks and invokes them.

Hooking non-virtual functions is a much bigger issue (as you said); you actually have to overwrite the function code to make it work properly. This can be difficult with weird functions (such as functions which start with a relative jmp). BAILOPAN has posted some posts about hooking non-virtual functions on http://www.sourcemod.net/devlog

I don't like overwriting code because it's very platform specific and it doesn't work with things like PaX in the kernel.

Yes. That's why I wrote the dissasembler, but you are right about relative jumps. I didn't consider them at that time. It's not a big problem though, because nowadays most programs are compiled, and they have a standard header how they start in 99% of the time. Still to make it bulletproof it would need some polishing.



Problem is, with my mod I get a performance hit of about 25 FPS on average, because I can't make a rendersnapshot into memory. If I could avoid this, I would have no second thoughts using such a a method. On Linux it's easier, because I could tell the users to create a small RAM disk, and then link the outputfile to it, so the overhead would be much less, but this doesn't work on Windows, because Doom 3 translates all pathnames and I can't change that, and there are no links like on UNIX to make that trick.


WOW! When I just wrote the above I had an idea, how I can use this SourceHook to actually achieve what I need! If the pathtranslation algorithm is also a virtual function, then I'm set. I can use a RAM disk, intercept the path translation, and redirect it to the RAM disk, without D3 getting aware of it. I will definitely try this, and see wether the performance improves or not. It might well be that the drivercode may cause most performanceoverhead, but I doubt it. I guess this is really the physical disc access, so I need to avoid it at all costs.

I guess that para- just saved my day.

I just wanted to start evaluating this sourcehook stuff, so I downloaded the sourcode and moved the relevant files over to my project. But I get a lot of errors. I guess this is something simple like a define, a compilerswitch or some buildoption. Does anybody know how to resolve these? It also would be interesting to get a fully working copy of your current source if you don't mind. This could also help me determine how to include this code in my mod.



There are a lot more, but I guess they will go away if I know how to remove the above errors.

Hmm. Hmm hmm.

1) It looks like you're not using the newest sourcehook version (which doesn't use STL due to libstdc++ version problems on linux).

2) What compiler are you using? I'd guess it's a MSVC 6 but I might be wrong. Anyway, I've never tested it with MSVC 6 and I'm almost sure it shouldn't work. gcc/mingw or MSVC >= 7 is ok.

3) It looks like idlib #define vsnprintf on linux; Just add an #ifndef vsnprintf .. #endif block around #define #vsnprintf that somewhere in sourcehook.h

I guess the newest version is on CVS? I downloaded the sourcetar from the homepage which was version 1.02 or something, I think.



Visual Studio NET 2003, but later I will use it on Linux as well. We will port the code once we have a stable version.



Don't know if this is the problem, but I will take a look at this as well.

Thanks.

Well, you can get the newest SH files from http://www.tcwonline.org/cgi-bin/viewcv ... ourcehook/ . Note that the interface has changed so it won't work if you compiled the manager with the old version and the plugin with the new version.

Anyway, MSVC 2003 is 7.1 so that should be ok. Interesting, I don't have the doom3 sdk myself so I'm afraid I can't do much; do id use the STL? My best bet would be something in idlib making STL not compile or you messed something up =P

sparhawk, take a look at src/idlib/Str.h.



In either case, I hope to be making an initial release of DMM soon. I'm aiming for a release candidate before or during this weekend, which will include DMM, a full SDK header tree, precompiled SDK libraries, and an example plugin which hooks a function with SH.

I hope you keep us update here, because I think this is a very interesting project and could help a lot of coders.

Now I found out what the problem was. I put it in the game DLL. After putting it in idLib and doing some minor changes, it compiles now.

On to the next step.

In the above posting with the sample code, where did you get that PLUGIN_INIT() from? I couldn't find it anywhere.

Edit: I really would appreciate it if you could post a fully working sample, because it's rather frustrating to work with these bits and pieces. It seems that you already did a lot of backgroundwork which is not aparent from your posting, when I try to match it up with SourceHook. And I didn't really find a good documentation on how exactly SourceHook is supposed to be used.

There really isn't any formal documentation for SourceHook (yet). PM originally wrote it specifically(?) for a similar MetaMod device for the Source Half-Life2 engine. This is apparent when you try to use SourceHook, you find it's implemented in such a way that you can assoicate various hooks with various "plugins" (quoted because the act of using modules isn't necessary, they simply denote modular behavior).

Anyhow, because of this background, documentation specific to SourceHook wasn't really needed. The author of SourceMM (the other metamod engine for Half-Life) worked closely with PM to integrate the features.

So, at this point the best way to figure out how to use it is either to look at the SourceMM code, my code (which I will be releasing within a few days, or at least putting in the pubicly viewable CVS), or the SourceHook code itself.

I can say this though, you'll need to declare a global pointer of type SourceHook::ISourceHook called g_SHPtr. This should point to an instance of SourceHook::CSourceHookImpl.



This macro is specific to DMM. I use it to help keep writing plugins simple, as plugin authors will not need to modify the code has.



The two g_* variables are specific to SourceHook, the rest is for DMM. I would put my code into the CVS so you can read through it (it's pretty clean and easy to skim through IMO), but I'm not quite ready to plublicly release anything yet. Once I tie up a few loose ends I'll make an initial commit and if you'd like, I'll paste the link here for you to check out.

Basically to make SourceHook work, you need:
1) #include "sourcehook.h"

2) Declare a global SourceHook::ISourceHook pointer called g_SHPtr (you can use a different name, but then you have to #define SH_GLOB_SHPTR [your global variable name] before including sourcehook.h)

3) Declare a global variable of type SourceHook::Plugin which should be called g_PLID (again, you can customize the name by #defining SH_GLOB_PLUGPTR before including sourcehook.h; also note that _PLUGPTR is somewhat misleading as it doesn't matter what it is - the only thing that matters is that it's different for each plugin). If your system is not plugin-based, just set this to 0. I might add a #define which disables plugin-functionality later.

4) Somewhere, you'll need to make an instance of the SourceHook::CSourceHookImpl and assign a pointer to it to the g_SHPtr variable (preferably BEFORE doing any SH_ADD_HOOK operation or similar; otherwise it will crash). I usually use a global but you can allocate it on the heap if you want.

5) Then "declare" the hooks outside of any functions (you can declare them in namespaces though) using the SH_DECL_HOOK* macros

6) Then you can add / remove hooks using the SH_ADD_HOOK* / SH_REMOVE_HOOK* macros.

If you're writing for SourceMM, it does step 1, 2, 3 and 4 for you (you have to use some macro though). I think Doom3 Metamod will provide similar functionality.

I hope I didn't forget anything; feel free to ask any questions.