Message Boards

Topic: Major engine restructuring.  (Read 14613 times)

Offline Zorro

  • Posts: 65
  • Professional n00b
    • View Profile
Major engine restructuring.
« on: May 13, 2007, 21:41:12 »
Today and yesterday I have gone deeper into the core of Odamex then I have ever gone before.  I began hoping that I would be able to better understand what makes Odamex tick, but after tinkering with it for a few hours, I had realized that warm fuzzy feeling had been missing for a while.

A few of the community members have asked why Odamex development has been coming along so slow; The answer is simple: the internals of Odamex are a mess.  Enough of a mess to scare away (possibly many) potential developers.  The current state of the doom engine can only be described as an unstable jenga tower.  As more work is done, it seems the tower is made more unstable and blocks are continually placed on top.

We are currently at the point where the tower is about to topple, The brave ones have stopped placing bricks on top.  The average coders have stopped long ago.  I think the only solution is to let it topple over and take all the bricks, one by one, and put it back together from the ground up.

The doom engine we have now is the result of 13 years worth of mistakes and oversights.  It does not need patchwork, it needs a reworking.  It needs to be torn apart and redesigned as a client-server engine which lends itself to code reusability and sanity.

Instead of running away, I would like to lend a hand in this.  The only way for odamex to succeed is getting this done as soon as possible.
^^ this post has wasted part of your life ^_^

Offline NovaFlame

  • Posts: 16
    • View Profile
Re: Major engine restructuring.
« Reply #1 on: May 13, 2007, 23:36:21 »
Nice way of putting it.

Even though I am still a beginning to learn the workings of Doom (Odamex has taught me some stuff while going through the source), I would like to try and help as much as I can.

This is starting to sound real interesting!
« Last Edit: May 13, 2007, 23:38:01 by NovaFlame »

Offline deathz0r

  • Administrator
  • Posts: 61
  • kick it oldskewl
    • View Profile
    • deathz0r's Place of DOOM
Re: Major engine restructuring.
« Reply #2 on: May 14, 2007, 07:15:47 »
While I'd love to help participate in this, I would like to point out that we do aim to stick close to doom2.exe gameplay physics as closely as possible. We don't want to remove any game-affecting bugs like wallrunning, silent BFG and pseudo-random seeding. We put in effort to reinstate these bugs, and I'm sure we'd hate to see them removed.

I should also mention that there's roughly 100,000 lines of code. This will not be an easy task and is definitely a long-term project, and will require a separate branch.

One more thing, the first task for this will be to remove any reference to ReplaceString. denis mentioned ages ago that ReplaceString can cause serious memleaks. There's only 1-3 references left, but they are incredibly hard to remove.
« Last Edit: May 14, 2007, 07:20:26 by deathz0r »

Offline Zorro

  • Posts: 65
  • Professional n00b
    • View Profile
Re: Major engine restructuring.
« Reply #3 on: May 14, 2007, 11:04:09 »
Quote
While I'd love to help participate in this, I would like to point out that we do aim to stick close to doom2.exe gameplay physics as closely as possible. We don't want to remove any game-affecting bugs like wallrunning, silent BFG and pseudo-random seeding. We put in effort to reinstate these bugs, and I'm sure we'd hate to see them removed.

I understand this completely.  What defines doom, or any game for that matter, is not the content, but the mechanics.  If we were to do things like cut out things like wallrunning and silent-bfg, we might aswell be playing counterstrike on D5M1.

Quote
I should also mention that there's roughly 100,000 lines of code. This will not be an easy task and is definitely a long-term project, and will require a separate branch.

It will be difficult, but it is either that, or work with the existing codebase, which is also a daunting task.  Alot of the written code is completely reusable, we just need to logically analyze and actively restructure things.

Actually, the idea of a separate branch may not be a bad idea at all.  A task this large would need revision control, and at the same time, we would not want to break odamex in the main repo, to avoid completely blocking development.
^^ this post has wasted part of your life ^_^

Offline Russell

  • Developer
  • Posts: 70
    • View Profile
Re: Major engine restructuring.
« Reply #4 on: May 16, 2007, 17:18:56 »
A job like this will be a substantial amount of work, there are factors like (deathz0r already said some)
1) Gameplay mechanics
2) Crossplatform issues
3) Time (how long it will take)
4) Dehacked, boom features which rely on the current infrastructure of the doom engine to work properly.
5) The new design (also taking into account the above points)

I'm sure there are more, a rework like this requires a serious amount of thought as you probably know, to break it down, the following systems (and subsystems) need to be reimplemented.

Systems
1) Wad loading
2) Renderer
3) Automap
4) Menu system
5) Tick system
6) AI, Monsters
7) God knows what else.

Subsystems:
1) Video
2) Sound
3) Input
4) Network

Now you have the stuff that almost makes doom, but if something like this is going to be attempted, it will need to be extendable, because there is no point in doing something like this if its going to be just as limited.
Here are a few ideas I had:

1) Multithreading for all systems and subsystems, computers with SMP/Multiple CPU cores can take advantage of it
2) Module/plugin system, similiar to doomsday, without modules, it'd just be straight odamex with boom and whatnot, but then it could be used to add additional functionality like gameplay modes, bots, ingame launchers etc.
3) Whatever else is here

And jeez, if I could think of anything else, I would :P

Offline Zorro

  • Posts: 65
  • Professional n00b
    • View Profile
Re: Major engine restructuring.
« Reply #5 on: May 16, 2007, 18:57:42 »
You post a lot of important points.  After a little chat on IRC, it would make more sense in trying to achieve our goal to start from scratch, although it will take longer.  We need to keep in mind that a decent portion of the code in doom is perfectly fine.  Code I would put in this category is mostly implementation code (as opposed to interface code).  for example, the audio interface works fine, but may provide a horrible interface (I'm not saying this is true, it is an example).  We can definitely borrow the audio interface's implementation and provide a better replacment interface to the already existing code.

I guess the biggest thing we can do to make this project possible (I was tempted to use the word "easier," but it is my guess that this will be far from easy) would be to keep everything slotted, and avoid sideways dependancies.  This way, for example, we can use the existing renderer with no problems, and when cardboard comes out, make the switch without raising any conflicts with other existing code.  These are the types of improvements we will be trying to make if this gets off the ground at all.

We should finally keep in mind, as Russell pointed out, that though we are trying to provide the classic "look and feel," we do not want the engine to be limited to that.  Things such as game modes and rules, the console, access to the display, should all be coded in a way where extension would be easy and non-destructive.

One thing I think needs to be addressed is the way client/server interaction is implemented.  Most engines work one of two ways.  Either the server controlls the game and the clients display it, or the clients control the game and the server relays it.  In odamex, both implementations are present, and one depends on the other in arbitrary points.  We should either go one way or the other, but the functionality should not *ever* require multiple implementations as it does currently. (perhaps this is a side effect of the ongoing client/server->common separation operation?  Either way, the client/server and common code should have a clear and thick line of distinction, and this will be very hard to achieve if lines and functions are randomly moved one at a time.)
« Last Edit: May 16, 2007, 19:02:43 by Zorro »
^^ this post has wasted part of your life ^_^

Offline Zorro

  • Posts: 65
  • Professional n00b
    • View Profile
Re: Major engine restructuring.
« Reply #6 on: May 18, 2007, 19:04:36 »
Ok, I have come up with a pleasant way of making a thick server design work elegantly while maintaining the clients ability to run independent of any server.  In doom, everything is driven by the ticker procedures which are invoked with each loop.  This is a bad design as it lends itself to separation of subsystems, but if used properly, it could redefine how the engine functions.  Read this chat log (between me and a friend who is developing an engine, about the design I've been thinking of using)

Quote
<Zorro> basically you have the self-sufficient gamestate (not self-modifying), then a dynamic and modify-able list of 'tickers' which either modify it or use it to preform tasks such as rendering
<Zorro> the difference between a client and server can be determined by the list of tickers running
<Zorro> I'm not sure if that's been done or not, but it seems like a nice idea
<Zorro> in theory, you could add a ticker which would implement additioinal gameplay features (such as CTF or whatever) through dynamic linkage, or put the game logic ticker in it's own library

The gamestate would define where instances of what object can be found, the state of the map, and any other variables that contain information that other tickers will need (for example, screen resolution, which would be set by either the console or the menu tickers, but used by the renderer.

Tickers truly would act as modifyers applied to the state.  For example, in a local game, first would come the input ticker which would post events to a buffer, then the game logic ticker, which would use that information, then the sound ticker, then the render ticker.

Tickers should depend only on the gamestate, never on each other (in the finished product).  By being independent, problems are isolated and modifications are no longer engine hacks.

As I said, by using different tickers, you can get different behavior.  This will make the code we write higly valuable and reusable.

Local Client:
*Input
*game logic
*sound
*renderer

Remote Client:
*Input
*game logic
*client network tic
*sound
*renderer

Host Client:
*Input
*game logic
*server network tic
*sound
*renderer

Dedicated Server:
*game logic
*server network tic
^^ this post has wasted part of your life ^_^

Offline Voxel

  • Developer
  • Posts: 22
    • View Profile
    • Voxelsoft
Re: Major engine restructuring.
« Reply #7 on: May 24, 2007, 09:00:22 »
Hi Zorro, you have put forward some interesting ideas in this thread. I'd like to contribute some of my opinions..

THE CODE
---------

As others have pointed out, remaining compatible (we have precision DOOM demo playback compatibility, for example) would be almost impossible without paying homage to the quirks of the original code. This means any roots-up reworking is going to be difficult. Yes, we can build a better game engine easily, but it won't be DOOM. For example, merely changing the order of tickers would break demo playback; taking the engine apart and putting it together "brick by brick" will do so to an unmanageable extent.

Given these restraints, I disagree with you when you say that the engine is a "mess" or a "jenga tower". For a start, Odamex code has amazing stability and portability compared to other ports. The code maintains all its traditional characteristics while accommodating newer techniques where appropriate (e.g. STL for string handling). The improvements have been happening over a long time (we have gone from 150,000 lines to 95,000 lines while improving features). Have a play with the old csDoom versions to understand why Odamex has moved miles ahead.

That's not to say that there aren't problems. Close module coupling and cohesion make modification more difficult than it should be. There is a lot of work to be done, adding proper spectator support and demo playback are two of the big modification challenges.

DEVELOPMENT
-------------

The major reason for slow development is a lack of developer input. That includes me, as I happen to be travelling the country in search of a job while working on a 100 page thesis. Other members of the development team also have important real-world business to take care of. Hopefully we will all have more time over the summer holidays.

We often have people proposing various changes and restructurings. There are plenty of people pulling us in different directions. Plenty of people starting their own ports (and rarely, if ever, finishing them). If you think you can significantly improve the code by rewriting a large part of it, I am quite happy for you to branch the project and look forward to see what you come up with! We'll even host the branch on our SVN.

However, what we really need is dedicated coder time. Not someone to pull in yet another tactical direction, but along with our common strategic goals. A developer that trusts our management and strengthens our team. Someone that actually does the work. Someone who can dig their teeth into a specific tough problem and not let go until it is solved (which may take months of hard work). That's how things get done, how to get credit. If anyone wants such a challenge, just ask :)

We don't need developers who can talk the talk. We don't need developers who can walk the walk. We need developers who walk the walk.

EXTENSIBILITY
------------

I have been thinking about the 'extensibility' problem and how it can be dealt with. The sad truth is that everyone wants some specific mod/tweak and they are not inherently compatible. If we follow all the suggestions, the engine will sprawl with millions of special CVARS and console commands.

The best answer I can come up with is a move towards pure scripting. A translation of the core game logic into Python/Tcl/Lua/<insert language>. This will offload development to whoever wants the change, rather than the Odamex devs. We can then have a classic engine, a CTF engine, and etc. Load them up like a lump and run them. Someone wants flag dropping, silent bfg and no strafe running? Just modify the engine and load the lump onto your server. Client will download the lump and run it. Odamex becomes the technical component that does graphics, sound, engine interpretation, and network connectivity while the pure game code is moved to the WADs. Truly a new standard.

Another benefit of this is, potentially, cheat prevention. A scripted engine can be quickly and subtly modified to test whether it is being circumvented. There are associated security problems of letting downloaded code run, but hey.. your browser runs javascript already. Lots of other questions.. What language should be used? Where do we draw the line between core and script?

Big project to do, but I think it is the best way forward.

NETCODE
--------

A scripted approach will also reduce the need for anti-cheat network code, which many people have been complaining about.

The client and server both use a working engine. This allows for both SP and MP. The client has a copy of the engine for prediction capability. To start with, we had two separate copies of the engine, one in client and one in the server. This has now been moved into a single 'common' engine. That took serious effort. Some lines have been moving around as a result. It may still look messy, but it is an improvement.

The chief reason for a 'blurry' line between client and server is perceived performance. In reality, neither client nor the server replies that the client gets are accurate, in terms of timing. The prior is ping/2 ahead, the latter is ping/2 behind. The player's weapon sound is played at the prior, the bullet hit is actually shown by the latter. Theoretically, both should happen at the same time, the client should delay the sound and do the bullet hit before a server confirmation (which is checked after it arrives). Changes in this area are tricky as there is no common structure defined to deal with the events.

The major hurdle in netcode is verifiability. We need tools to visualise network data streams and how they influence game state. We need a netwide demo recording and playback tool that also records the network data, so that we can play back an entire network game (prom the point of view of multiple clients at the same time), and see where the game state is desynced. Otherwise it is like testing/fixing an electronic circuit without an oscilloscope/voltmeter. Only way to really show how ghost monsters and other problems originate.

There's also an annoying issue in current anti-wallhack code, primarily because the BSP sight checking code is slow and erroneous (p_sight.cpp). There is a much faster way to do it, but the task is tricky and we don't have a developer to take care of it (see DEVELOPMENT above).

MISC
----

Many problems exist outside the main engine domain (e.g. osx midi crashes due to buggy SDL_mixer, tricky wxWidgets versioning), so work should be done to prop up our dependencies in that case. There are some issues that just need people to fix them, especially in audio.

It does not need patchwork, it needs a reworking.  It needs to be torn apart and redesigned as a client-server engine which lends itself to code reusability and sanity.

The only mistake is hotheaded development. If redesign was the answer, one of the other numerous ports would have already succeeded in our place. We should stop concentrating on development and start concentrating on testing. Regression testing and test driven development are the only way to stop making mistakes. We need testing tools that will, for any line change, tell us whether something (a demo, a visual, an audio, a protocol) breaks as a result.

Cheers,
Denis
« Last Edit: May 24, 2007, 09:57:10 by Voxel »

Offline Zorro

  • Posts: 65
  • Professional n00b
    • View Profile
Re: Major engine restructuring.
« Reply #8 on: May 27, 2007, 12:18:06 »
(my opinions have changed a bit in the past week or two)

Yeah, I guess a "rewrite" would be unnecessary.  What we certainly need to do though is examine code for unneeded logic, and more importantly, logic on the client and the server.

As you said, a lot of things (such as demo playback) work perfectly with the engine just as it is.  We should fix up the hole in the tire, not replace the wheel
^^ this post has wasted part of your life ^_^