NeL Manual

Forward

About The Author

Matt Raykowski has been involved in the NeL community since early 2002 and has a broad background of experiencing in many roles including network engineering, system administration, technical support, application development and management. Matt is currently the technical supervisor of corporate application development for Children's Hospitals and Clinics of Minnesota. In 2001 Matt and his intrepid group who later took on the name Ironic Entertainment began designing a game called Werewolf and began searching for a suitable library or set of libraries to meet their needs - their ultimate choice was NeL. Matt is still the lead programmer for the Werewolf project as well as a primary contact for the NeL community. 

Acknowledgments

Introduction

So you picked this book up. Maybe you're wondering what NeL is? Or maybe you're a long time user that just needs a good reference. Either way this book will meet your needs. We'll start out with some discussion about what NeL is, what kind of features it provides, where it came from and why you should use it. Following that you will see a per-module breakdown of functionality, how to use it and how it inter-operates with the rest of the system.

Who This Book Is For

This book is for experienced programmers who are interested in leverage one or more of the modules that NeL provides. Your experience in NeL may vary from none to experienced user - this book will provide useful knowledge to anyone using NeL. 

How This Book Is Structured

The book will begin with a quick chapter on getting started with NeL which will provide a quick start and a brief taste in coding with the NeL framework. Following that the book will be broken into sections dedicated to each of the modules and tools. 

Prerequisites

You should be familiar with ISO Standard C++ and the C++ Standard Template Library (STL.) You should possess a copy of and be familiar with a common compiler for one of the platforms supported by NeL.

• On Windows: Microsoft Visual Studio .NET 2003, 2005 or 2005 Express.
• On Linux (2.6.x, any distribution): GCC 4.0.2 or above. KDevelop is helpful.
• On Mac OS X (Tiger or Panther): a configured and working Xcode environment.
• On All Platforms: CMake 2.4.6 or newer.

Downloading The Code

You can find the code used in this book on the NeL website: dev.ryzom.com For more information on NeL, tips and tricks, or simply to converse with other NeL users please refer to that site for resources such as forums and IRC. 

Contacting The Author

The author may be contacted through his email address matt.raykowski@gmail.com, you may also try contacting him on IRC (irc.freenode.net) in the #nel channel or on the NeL forums as "sfb."

Getting Started

What Is NeL

NeL Features

Origins Of NeL

Installing NeL

Installing NeL On Windows

Presently there is no installer for Windows. 

Installing NeL On Linux

On Debian and Debian-derived distributions run the following command:

sudo apt-get install libnel0 libnel-dev

The libnel-dev package is only required if you will be developing using NeL. If you only need the NeL runtime you will only need to install the libnel0 package. Also there is libnel-dbg which contains the debugging symbols, however it is very large. 

Building NeL From Source

Building NeL On Windows

Checking Out the Source

When working with NeL on Windows you will want to check it out first since most of your required packages will be stored within the NeL source tree, a process that will be explained in the next section. An in depth explanation of source control management is outside of this scope but a brief explanation is required. Our source is divided primarily into three sections: trunk, branches, and tags. Trunk is our "working" tree of code. It is typically the most recent code and is what most developers using NeL (but not necessarily developing NeL) will use. Branches contain variations of trunk from some point in time. NeL framework developers use branches to work on large features that may impact the stability or the ability to compile code over a prolonged period of time. Tags are used to mark points in time of importance, typically we use tags to mark the source code used for a given feature release, e.g. tags/NEL_0_6_0 would be the 0.6.0 version release.  To check out the source code you will need some Windows-based Subversion client such as TortoiseSVN, which is the preferred client of the Windows using NeL community. Once you have this tool installed you will use it to check out NeL.

A common practice is to create a folder called "Projects" that you will use to hold your project source code. You can check out variations of the NeL source and your own project's source to this directory. While in this directory right click in the folder view and choose SVN Checkout...

 
Once this window opens you will fill out the URL of Repository field and the directory to which you would like NeL checked out.

• URL of Repository:  https://nel.svn.sourceforge.net/svnroot/nel/trunk
• Checkout Directory: C:\Projects\nel-trunk

 
Once you accept these settings it will begin the checkout process. It may warn you that the directory does not exist - this is fine, just click Yes to create the new directory.

The structure of our Subversion repository as of writing this is that all "components" of the NeL framework are immediately under trunk, then each component is broken out within it's respective portion. The components currently provided are nel the actual libraries, nelns the networking system, snowballs2 the technology demo and media which contains the original media (texture maps and native modeller files, e.g. .max files) used for NeL samples and demos.

Installing Required Packages

NeL requires a variety of packages to build and to run. For ease the NeL community maintains a large archive of the majority of required libraries. NeL requires the following libraries: freetype, jpeg, libXML, zlib, OpenGL, DirectX, OpenAL, FMOD, EAX, and potentially more libraries depending on the components you choose to compile. An indepth list of dependencies is provided on the NeL website. The website also provides a link to an archive of dependencies. Download the file external8_date.rar from the NeL SourceForge download site under External Dependencies. Extract this to C:\Projects\nel-trunk\3rdParty and the CMake build system will automatically include these paths in your build. This will meet most of your needs.
 
Additionally you will need to get the latest DirectX 9 SDK from the Microsoft website if you want to build the Direct3D driver (the default behavior.)

Building and Compiling NeL

Before you can begin compiling NeL you will first have to use CMake to generate the build environment. NeL forces developers to build out of source to prevent compilation and build environment changes from cluttering up the source tree. This makes it easier to manage changes and source commits, among a variety of benefits. So you will want to navigate to C:\Projects\nel-trunk\nel folder and create a new folder called cmake. Now start the CMake tool and choose the source directory and binary directory:

• Source Directory: C:\Projects\nel-trunk\nel
• Binary Directory: C:\Projects\nel-trunk\nel\cmake 

Once you have set these you will first click Configure and choose the appropriate generator. In this case we will be building for Microsoft Visual Studio 8 2005. It may prompt you to create the build directories, this is normal.

dd 

Building NeL On Linux

To build NeL on Linux you will first need to install some required and optional packages. Since this book cannot cover all the various Linux distributions and package management systems it will refer to the distribution most commonly used by the author: Ubuntu.

Installing Required Packages

You will need to install a series of packages to begin compiling NeL.

sudo apt-get install g++  subversion cmake libxml2 libxml2-dev libfreetype6 libfreetype6-dev libjpeg-dev xorg-dev x11proto-xf86vidmode-dev

In many cases you will already have these libraries and tools installed. A quick summary of what we installed is the compiler, a client to get the source, the tool to generate the build files, an XML parser and a variety of libraries used by the 3D system. Theoretically the 3D-related stuff is optional as you do not have to build every module but there will rarely be an instance where you will want only one module built.

Installing Optional Packages

There are a few packages that are considered optional but are suggested to be installed anyway.

sudo apt-get install libopenal0a libopenal-dev libalut-dev libgtk2.0-dev libglade2-dev libglade2-0

What we are installing is the packages necessary for the OpenAL sound system and a completely optional GTK library so that they NeLNS daemons and other services will run in a GUI rather than as a crude console application.

Checking Out the Source

The Subversion tool we installed earlier is what we will use to get the source code. An in depth explanation of source control management is outside of this scope but a brief explanation is required. Our source is divided primarily into three sections: trunk, branches, and tags. Trunk is our "working" tree of code. It is typically the most recent code and is what most developers using NeL (but not necessarily developing NeL) will use. Branches contain variations of trunk from some point in time. NeL framework developers use branches to work on large features that may impact the stability or the ability to compile code over a prolonged period of time. Tags are used to mark points in time of importance, typically we use tags to mark the source code used for a given feature release, e.g. tags/NEL_0_6_0 would be the 0.6.0 version release. Here is how you check out the code:

$ pwd
/home/me/sandbox
$ svn co https://nel.svn.sourceforge.net/svnroot/nel/trunk nel-trunk
[omitting svn output]
$ ls
nel-trunk

The structure of our Subversion repository as of writing this is that all "components" of the NeL framework are immediately under trunk, then each component is broken out within it's respective portion. The components currently provided are nel the actual libraries, nelns the networking system, snowballs2 the technology demo and media which contains the original media (texture maps and native modeller files, e.g. .max files) used for NeL samples and demos.

Building and Compiling NeL

For the purpose of this section we will focus on the NeL library. The steps performed here can more or less be performed verbatim on the other modules. In order to compile NeL we will first need to generate the build environment. The NeL projects prefers that you use the CMake environment instead of the autotools and MSVC environment files delivered. Reliance on these files is not suggested as they will be deprecated and finally deleted soon. Our CMake configurations requires that you build out of source so that your build files do not litter the source tree. This makes development and management of source changes in NeL much easier and more convenient. So assuming the path used earlier we will need to move to the NeL source directory, create a directory to build in which we will call cmake for simplicity and finally we will build.

$ cd /home/me/sandbox/nel-trunk/nel
$ mkdir cmake
$ cd cmake
$ cmake -G "Unix Makefiles" ..
-- Looking for XOpenDisplay in /usr/lib/libX11.so;/usr/lib/libXext.so
-- Looking for XOpenDisplay in /usr/lib/libX11.so;/usr/lib/libXext.so - found
-- Looking for gethostbyname
-- Looking for gethostbyname - found
-- Looking for connect
-- Looking for connect - found
-- Looking for remove
-- Looking for remove - found
-- Looking for shmat
-- Looking for shmat - found
-- Looking for IceConnectionNumber in ICE
-- Looking for IceConnectionNumber in ICE - found
-- Configuring done
-- Generating done
-- Build files have been written to: /home/me/sandbox/nel-trunk/nel/cmake

NeL also has a number of configurable options you can toggle. For example if you are building a server and have no need for the 3D libraries you could simply disable compilation of them. NeL also provides some optional support which is not by default compiled such as GTK displayers for non-Windows environments. A common scenario is to change the installation prefix, which defaults to /usr/local - you may want to install it to your home directory such as /home/me/nel - here's an example of toggling some optional support and changing the installation prefix:

$ cmake -G "Unix Makefiles" -DCMAKE_INSTALL_PREFIX=/home/mattr/nel -DWITH_GTK=ON ..

Finally you will need to build NeL, which is a simple make command:

$ make
Scanning dependencies of target nelmisc
[  0%] Building CXX object src/misc/CMakeFiles/nelmisc.dir/event_emitter.o
[  0%] Building CXX object src/misc/CMakeFiles/nelmisc.dir/eval_num_expr.o
[  0%] Building CXX object src/misc/CMakeFiles/nelmisc.dir/window_displayer.o

Quick Start Introduction

Modules: NLMISC

Basics

Handling Command Line Options

(TBD, code needs to be finished)

Using Logging

The debugging system allows to log information at various levels. To get started you will want to include the header file "nel/misc/debug.h". This header contains all of the macros necessary to begin logging. The logging facilities provide five different logging levels: Error, Warning, Info, Debug and Assert. Each logging level can be accessed using it's associated logging macro. Each of these will take an expression (denoted by exp in the table below) and this expression is typically in the format that is passed to function printf.

As you can see above these methods allow you to display strings variable arguments such as if you were using the printf function. In the default behavior, the text passed to these macros is displayed into STDOUT except in RELEASE mode. The text logged is also written into a file called "log.log" in the working directory. These macros print the strings with an information header.

Example :

Because NeL allows you to create multi-threaded programs these macros use mutual exclusions (mutex) to ensure no data is corrupted and the displayed text not interlaced. You will become very familiar with using the nlinfo macro as it is the most commonly used macro.

Using Config Files

Config files in NeL provide you with a powerful way to set up your programs at runtime. There are two parts to working with config files: the config file format and the CConfigFile class. There is a fantastic sample for this, so feel free to look in the samples directory for more hands on information. NeL makes working with config files a breeze so we'll cover the code portion first.

The Config File Code

Retrieving Config File Values

Most of the work done on a config file will be done through the CConfigFile class.

Retrieving variables from the config file is almost that simple. The config file system allows you to convert config file values to a variety of types:

Working With Config File Arrays

This covers the essentials of working with config files but there's a whole lot more to config files. You can work with arrays in config files as well. When a parameter in the config file is declared as an array you can access it using the size method and then the asInt(int) method. You're not locked in to asInt, though, you can use all of the type conversion methods - the int parameter is what is important. Here is an example of accessing an array from the config file:

You can see above that I was able to get the "myArray" variable and convert a specific member of it to the string type.

Something that is important to note: if you try to get a variable that doesn't exist or convert it to a type that NeL cannot convert it to (such as trying to convert a string to an integer) the config file system will throw the exception EConfigFile. If you are loading config files that may be changed or accessing variables that may not be in the config file it is highly recommended that you place you getVar calls within a try/catch block to compensate for potential config file errors.

Setting Config File Variables

The config file system allows you to update existing config files. The benefit to this is that you can update or fix config files from within your code automatically. There is only one downside to writing a config file from code: any formatting that you have in the config file and any comments you have are completely lost. With that said, here is a quick example of how you can set a config file variable and save it:

Config File Callbacks

One last thing that config files allow you to do is monitor whether a file or variable has changed. This is useful if you want to drive runtime activity from the file or automatically adopt changes from the config file. The config file system provides two ways of monitoring changes to the config file system - by config file or by config file variable. To implement these you will need to create a function that can be called by the config file system in the event of a change.

By Config File

By Config File Variable

Once you have defined the methods you are interested in you simply need to register the callback with the config file system:

Miscellaneous Config File Options

The config file class also has a few options to help you out. One of the handiest ones is the UnknownVariables public member. This vector of strings lists all of the variables that were requested but not found in the config file. You can use this method during troubleshooting to see if you have a misconfigured config file or to see what variables your code is calling that are not found. This can be extremely handy for finding typos within your code.

Also handy for troubleshooting is the display() public method. Call this method to make the config file system print out all of the variables that it loaded to to the InfoLog.

The Config File Format

The config file format uses C++ style comments:

Really the format for a config file is pretty straight forward. Here is a good example:

var1 = 123;			// var1 type:int,    value:123
var2 = "456.25";		// var2 type:string, value:"456.25"
var3 = 123.123;			// var3 type:real,   value:123.123

// the resulting type is type of the first left value
var4 = 123.123 + 2;		// var4 type:real,   value:125.123
var5 = 123 + 2.1;		// var5 type:int,    value:125

var6 = (-112+1) * 3 - 14;	// var6 type:int, value:-347

var7 = var1 + 1;		// var7 type:int, value:124

var8 = var2 + 10;		// var8 type:string, value:456.2510 (convert 10 into a string and concat it)
var9 = 10.15 + var2;		// var9 type:real, value:466.40 (convert var2 into a real and add it)

var10 = { 10.0, 51.1 };		// var10 type:realarray, values:{10.0,51.1}
var11 = { "str1", "str2", "str3" }; // var11 type:stringarray, values:{"str1", "str2", "str3"}

var12 = { 133, -1 };	// var12 type:intarray, values:{133,-1}

var13 = { 1, 0.5, -1 };

Basic Debugging

In addition to the logging facilities NeL provides some basic debugging tools and the basis of some exception handling. There are two main debugging macros and one exception that are useful in every day development: nlstop, nlerror() and the EFatalError exception. You can also find an informative sample called debug in the NeL source tree that covers this subject as well as some basic logging.

The nlstop macro is the most basic of the two. It requires no parameters and simply stops the code at that point. The nlstop macro perform no login in non-Debug build modes but it is important to be familiar with it and use it when building in Debug modes such as Debug or ReleaseDebug. In the Microsoft IDEs the macro will opt to set a breakpoint in the code and exit you if you are running within the IDE. This is a great way to programmatically set breakpoints for failure code rather than a simple error. The nlstop macro does throw the EFatalException exception, but unlike nlerror() (which will be covered next) it does not allow you to provide any contextual information.

The nlerror() macro is a useful variation in that you can use it immediately halt execution of the program but still provide useful, context sensitive logging. Because nlerror() calls abort() it should be used sparingly and only in situations where continued execution of code could cause problems, predictable crashes and other things of that nature. An important feature of nlerror() is that it emits an EFatalException exception that you can catch. This is tremendously useful for any cleanup that may be necessary upon a fatal error. Here's an example:

String Utilities

The NeL framework provides a wealth of string utilities for string manipulation and conversion. Four of the most important features are the toString and fromString methods as well as the CSString and ucstring classes. There are also many useful utility functions that we will also cover.

String Conversion

The string conversion utility really comes down to two components in NeL: a series of overloaded functions called toString() and fromString(). The names of these functions is pretty self-explanatory - the toString() functions convert a parameter to a string value and the fromString() functions convert a string to their native value. Listing all of the available overloads would take too much space but here is a quick demonstration using the nlinfo() logging facility.

You can see how simple it was to convert a primitive into a string for logging or debugging. But you could also use these facilities in other practical applications - especially the fromString() functions. You could use this to parse user input into native primitive members or variables for use in game logic, for example.

The CSString Class

The CSString class is a very useful string utility class. It provides a wealth of utility functions too numerous to name all here and explain each of there uses. It would be better to quickly peruse the API documentation to see all of the methods available. A quick summary however is appropriate. You can create a new version using several versions of the constructor which access C-style char strings, C++ strings and a couple variations of formatted strings.

Once you have a CSString object created you can use it as you would any C++ style string however with the added benefit of several utility methods that allow you to split the string several ways, crop the string, manipulate case and several ways to do case-insensitive searches and compares.

The ucstring Class

The ucstring class is a variation of the standard C++ string class developed within NeL to support unicode strings. For this reason you will not find the ucstring class in any of the NeL namespaces. Not much explanation it necessary for this since if you know how to use std::string you will know how to use ucstring. There are three important additional methods provided on this class which merit explanation: toString(), fromUtf8() and toUtf8().

There are two variations of the toString() method: one which returns a std::string value and one which takes a std::string argument. The former you would use to convert a ucstring into a std::string for portions of your code which are not unicode-aware. The latter you would use if you need to convert a standard string into a unicode string for portions of code which are strictly unicode.

The fromUtf8() and toUtf8() are used for conversion between UTF8 and UCS2 strings.

Utility Functions and Classes

For people not using the CSString class there are also a number of functions in the NLMISC namespace for string manipulation. Below is a table of interesting functions for std::string and ucstring users.

•  bool testWildCard (const char *strIn, const char *wildCard)
  • Checks strIn using wildCard - a wild-card string. Case-sensitive.
• void splitString(const std::string &str, const std::string &separator, std::vector< std::string> &retList)
  • Splits str into a vector (retList) of strings using a separator. Also splitUCString() accepts ucstring instead of std::string. Case-sensitive.
• bool strFindReplace(T &str, const T &strFind, const U &strReplace)
  • For both std::string and ucstring find a sub-string strFind in str and replace instances of it with strReplace. Case-sensitive.
• std::string toUpper (const std::string &str)
  • Also toLower(), same syntax. Converts the entire case of str.
• std::string stringFromVector (const std::vector< uint8 > &v, bool limited=true)
  • Converts a vector of bytes into a readable string. Unprintable characters are replaced with '?'.
• std::string bytesToHumanReadable (const std::string &bytes)
  • Converts a string representing bytes into a human readable string.
• std::string bytesToHumanReadable (uint32 bytes)
  • Converts an integer into a human readable string representing bytes.
• uint32 humanReadableToBytes (const std::string &str)
  • Converts a string with a human readable byte representation into the byte amount.
• std::string secondsToHumanReadable (uint32 time)
  • Converts a number of seconds into a human readable string.
• uint32 fromHumanReadable (const std::string &str)
  • Converts a human readable time or bytes into a number of seconds.
• void explode (const std::string &src, const std::string &sep, std::vector< std::string > &res, bool skipEmpty=false)
  • Splits a string src by a separator sep into a vector res. The skipEmpty parameter informs explode to not replace the vector if there are no matches.

Internationalization

There are three tools you can use in implementing internationalization in NeL-based applications. There's the CI18N class, the STRING_MANAGER and finally unicode string class ucstring which was covered in an earlier section. The ucstring class is useful for providing unicode strings but in and of itself performs no internationalization.

The CI18N Class

The CI18N class is probably the most useful internationalization tool provided by NeL. A quick introduction to this in the form of code will illustrate how easy it is to use this tool:

Internationalization Sample

fr.uxt

// This is the name of the language.
LanguageName    [Français]
Hi                              [Bonjour]
PresentI18N             [%s est fière de vous présenter le système d'internationalisation de NeL]
ExitStr                 [Appuyez sur <entrée> pour quitter]

en.uxt

// This is the name of this language.
LanguageName    [English]
Hi                              [Hello]
PresentI18N             [%s is proud to present you NeL Internationalisation system]
ExitStr                 [Press <return> to exit]

sample.cpp

As you can see above you only need to create a key-value pair for a string ID (in the example this is referenced in the get() method) and the internationalized string.

Internationalization File Format

The file format for the NeL internationalization system is pretty simple but is very powerful. It allows you to comment your files as well as providing you a powerful preprocessing system. These files should always be named languagecode.uxt where valid language codes are flexible but good examples are en, fr, and de as seen in the sample above.

Labels and and label texts are the core component and most important part of the file format. As soon in the above example and following examples a label is the first set of alphanumeric characters on the line. The label text is indicated by enclosing it following label with the [ and ] characters. Label text can have newlines in them, allowing the response to be several lines long. You can also indicate the use of things or explain the purpose behind the file using comments. Comments come in the form of simple C-style comments.

In addition to a useful comment system the file format also has a system of preprocessor definitions. These allow you some limited programmatic flexibility (through defines and if-statements) as well as tremendously useful include system. It is important to note that the if-statements adhere to scope and therefore can be nested. Listed below are the available preprocessor definitions. The preprocessor if-statements have increased importance when using the #optional statement as well as the load proxy (discussed in the section about the CI18N class) as these inject some dynamics into the loading of your internationalization strings.

Using the CI18N Class

The CI18N class is a relatively simple string manipulation class. In the example provided above we saw how simple it was to load a flat file, specify the current language code, and then begin getting internationalized strings. In a typical application you will first load a translation file using the load(languageCode) method and then use the  get(labelName) method to retrieve translated strings. The get() method returns a ucstring and can contain printf style tokens (e.g. %s.) In the above example you saw how we used printf tokens to send the translated string and arguments to a variable argument method such as printf or nlinfo.

There are a couple handy helper methods that you may find yourself using. The most commonly used is the hasTranslation(labelName) method. This is a quick way of checking to see if a label is available before you attempt to use it, especially if the label's availability is governed by and #optional include or one of the various if-statement preprocessor definitions. One thing that is important to note if you are a long-time NeL user is that the getLanguageNames() method is no longer in the API. NeL lifted the artificial limitation of language codes.

One of the last large features of this system is the "load proxy" interface. The internationalization system allows you to proxy string loading into a custom class that implements the CI18N::ILoadProxy interface. This is used to bypass the default call to the readTextFile() method. Using an implementation of this interface you can call the readTextFile() method how you see fit, process the file text before you load it, or load in custom text using the readTextBuffer() method. There are a few common reasons for doing this such as loading the file and then merging "newer" data from a server or overriding the parameters to readTextFile() so that it performs preprocessing.

The STRING_MANAGER Utility - TODO

ddd 

File and Path Management

The NeL framework provides a wealth of tools for both local and virtual file system manipulation and creation. In this section we'll cover the utility singletons CPath and CFile, reading and writing files on the local filesystem and the how to create and access the two virtual filesystems: BNP (Big Files) and XML Pack.

Using Path and File Utilities

There are two core tools for managing paths and files: CPath and CFile. The CPath singleton is a designed to abstract file-system operations and searches on a high level, including logical virtual filesystems. The CFile singleton is more low-level and designed for direct, local operating system access.

The CFile Utility

The CFile utility is tremendously useful when you need lower level direct access to the local file systems. It is, for the most part, an informational class providing various methods to retrieve a filename (separated from the path,) a path (separated from the filename,) file size and permissions. But it also includes some tremendously useful methods for file system operations such as copyFile() and method for file comparison and so on. Realistically checking the CFile API will provide more information than can be provided here but we will cover in a code example a couple simple scenarios.

The above example really covers the vast majority of uses of this utility. There are a couple advanced items that are important to note: temporary file creation and file callbacks.

The CFile utility provides a method called getTemporaryOutputFilename(src,dest). You provide the method with a source file name and it will update the argument dest with a unique filename: e.g. testfile.txt becomes testfile.txt.tmp. Subsequently calls to the method will then begin to produce testfile.txt.tmp.1 and testfile.txt.tmp.2 and so on.

The file callbacks are a useful way to monitor whether a file has changed. To harness this you need to create a function that accepts a string argument and then use the addFileChangeCallback method. Here is a quick example.

The CPath Utility

This utility is the most commonly used utility in the NeL file toolbox. It creates a logical layer above the physical file system and any virtual file systems such as Big Files and XML Packs. The goal is to provide developers with as simple an interface as possible and still maintain a high level of transparency and usability. In the course of typical usage you will first add search paths, possibly add some file or file extension remappings, perform lookups and finally you may have a need to reset the whole CPath object.

Adding Search Paths

There are several methods at your disposal for adding search paths. You'll typically add a search path so that the system can find common datafiles, configuration files and so on. So it is important to add search paths early in the program execution. An important concept for search paths is the alternative directories options. These are directories that are added to the search order but pre-cached. This means that they will only be searched during lookups once all primary search paths have been exhausted.

File Name and Extension Remapping

dd 

Performing Lookups

dd 

Resetting the Object

dd

Input and Output

ddd

Creating and Using Big Files

The Big File or BNP system is a logical virtual filesystem that allows packagers to place several files into one physical file. Access the files packaged within this .bnp file is completely transparent to developers of NeL-based applications as long as the bnp file is in one of the search paths. The CPath system completely abstracts finding and accessing these files from the developer. Before you can access BNP packages you will first need to know how to create them. Next you'll need to know some slightly advanced usage in accessing the BNP packages.

Creating BNP Files

In order to create .bnp files you will first need a copy of the bnp_make executable. If you do not have this you can compile it as part of the optional NeL tools. This tool is very simple has has three main actions: pack (/p), unpack (/u) and list (/l). It also accepts two optional parameters -if and -ifnot that take a wildcard string and filter the files that will be copied into the BNP file. Take the following directory for example:

$ pwd
/home/me
$ ls -l bnptest/
total 100
-rw-r--r-- 1 me me 61745 2008-05-19 12:47 gnu.shape
-rw-r--r-- 1 me me 28998 2008-05-19 12:47 sky.shape

Here we have a directory with two shape files in it that we want to package into a single BNP file. Here's the simple command to accomplish this.

$ bnp_make /p bnptest
Treating directory: /home/me/bnptest
adding /home/me/bnptest/gnu.shape
adding /home/me/bnptest/sky.shape

Loading and Access BNP Files

Whenever you add a search path to CPath it will automatically discover any bnpfiles in that search path. Alternatively if you have specific files that are not in your search path you can manually add them using the addSearchBigFilemethod. You will rarely need to manually add these to your CPath search paths, but the method is provided for flexibility, in the instance that a file is downloaded by a patcher after execution, for example.

Accessing BNP files is no different than using CPath to find any normal file. Rarely will you use CPath::lookup to find a BNP file, however you will use it to find any file contained within a BNP file. If you used the .bnp that we created in the previous section then loading a file from a BNP file is as simple as the following example:

You can see how simple it was to find a file and load it into a CIFile. While you will rarely load a shape using CIFile and shape loading will be covered in a later section this does show how easy it is to lookup a file from within a bnp and load it. There is the possibility that you have two files with the same name, although this should be avoided at all costs. If you need to troubleshoot one quick way to verify which file you have received is to check the path of the file returned by the CPath::lookup call:

Creating and Using XML Packs

The XML Pack mechanism is useful for packaging several different XML files in a single packed XML file. They function within NeL similar to Big Files and are abstracted by the CPath system. In this section we'll cover how to create XML packs, how to load them and how to access and use them.

Creating xml_pack Files

In order to create xml_pack files you will first need a copy of the xml_packer executable. If you do not have this you can compile it as part of the optional NeL tools. This tool is very simple and only has three (3) command line options: -p (pack), -r (recurse) and -u (unpack). Assume you have the following contents in your current directory:

$ pwd
/home/me/xmltest
$ ls -l
total 8
-rw-r--r-- 1 me me 25 2008-05-15 15:26 bar.xml
-rw-r--r-- 1 me me 26 2008-05-15 15:26 foo.xml

To convert these two files into a single XML Pack you would simply run this from the above directory:

$ xml_pack -p

This creates a new xml_pack file called xmltest.xml_pack in your current directory using every file in the directory. The xml_packer tool will add every file in your directory except other .xml_pack files. If you have multiple sub-directories you can also have the xml_packer tool add the files in those directories as well by adding in the -r command line option. Finally you can extract any xml_pack files you have back into their directories using the -u or unpack option.

Loading and Access XML Packs

Whenever you add a search path to CPath it will automatically discover any xml_pack files in that search path. Alternatively if you have specific files that are not in your search path you can manually add them using the addSearchXmlpackFile method. Useful scenarios of this are if you are transmitting XML Packs of containing dynamic or updated data you may want or need to use the method to one-off add the file.

Accessing XML Packs is no different than using CPath to find any normal file. Rarely will you use CPath::lookup to find an XML Pack, however you will use it to find an XML file contained within an XML Pack. If you used the XML Pack that we created in the previous section then loading an XML file from an XML Pack is as simple as the following example:

You can see how simple it was to find a file and load it into a CIFile. There is the possibility that you have two files with the same name, although this should be avoided at all costs. If you need to troubleshoot one quick way to verify which file you have received is to check the path of the file returned by the CPath::lookup call:

Time Tools

The NeL framework provides a number of tools for timing and profiling NeL-based applications and games. The most commonly used tool is the hierarchical timer which allows you to see overall execution of code as well as atomic breakdowns of scopes or sections of code within the hierarchy. There are many ways to use this and customize it to your specific needs. NeL also provides a much simpler tool called a stop watch, which meters the execution time between two points in code execution. Both of these options will be covered in this section.

Hierarchical Timer

NeL provides a handful of very useful timer tools for troubleshooting and profiling the performance of NeL-based applications. The most commonly used tool is the hierarchical timer. The hierarchical timer is comprised largely if the CHTimer class and a handful of utility macros: H_AUTO, H_AUTO_USE, H_AUTO_DECL, H_BEFORE and H_AFTER. The hierarchical timer gives you the ability to gather time statistics for an entire function or a block of code and provides some methods for managing the data the timer collects as well as reporting this data someplace useful (typically the log.)

In most scenarios you will use H_AUTO, H_BEFORE and H_AFTER so we'll describe how to use these macros first. H_AUTO is the most often used macro in this toolkit. One thing you'll also notice in the code is that the macros have a block of text. This block of text identifies the "scope" that we're benchmarking. This will be explained in more depth later on. Here is an example of these three macros in use:

So you can see that in the method 'usingHAuto' we simply used the H_AUTO macro. This creates an instance of the timer object within the scope of the method and when the method exits and the object is destroyed it will report back to the CHTimer class. This over simplifies the process but is essential enough to describe how to use the methods for timing and benchmarking. In the method 'usingHBeforeAfter' you can see that rather than relying on scope to determine the timing interval we specify the block of code we want to benchmark. This is handy if you have a specific block of code that is plaguing you and you don't want the other activities in that method or tree of methods to contaminate your timing benchmarks.

H_BEFORE/AFTER obviously have an application that H_AUTO cannot provide - benchmarking a specific block of code regardless of scope. But there's one large benefit that H_AUTO has over its before/after counterpart - you never have to remember to provide the H_AFTER macro. Take this scenario for example:

You can see in the above example that you need to make sure to call H_AFTER before every single exit point in your method or else you could (and will) end up with a huge mess in your benchmarking. 

Finally there's the H_AUTO_DECL and H_AUTO_USE macros. These very important macros are used when you want to lump multiple blocks of calls together without having to rely on the strategic placement of H_AUTO calls whenever these blocks of code are called. Here's a quick example of how these blocks of code can be lumped together:

These macros alone will not produce the benchmarking and timing results you require. Ultimately you will need to start using the CHTimer class. This powerful class provides you a variety of useful methods to manage this process. First you will need to know how to start, stop, reset and display the benchmarking and timing statistics. Use the following code as an example:

That is the essentials of using the hierarchical timer classes. Please refer to the API documentation for some more information on the CHTimer class. There are several methods on that class to display the benchmarking and timing data in meaningful ways. It is also important to note that these macros perform no logic when NeL is compiled in RELEASE mode.

Stop Watch

The stop watch class is used for performance measurement and statistics. It allows you to measure the time elapsed with controls like start, stop, pause, and resume. It gives you more flexibility to measure specific metrics at the loss of the automation and reporting abilities of the hierarchal timer. It's very handy for troubleshooting where the hierarchal timer is useful for profiling and overall performance monitoring. The stop watch controls are fairly basic, see this example for a common scenario:

The stop watch system also gives you a handful of other methods to control how you accumulate time. Above we saw the use of start and stop and how to use the getDuration method to display the amount of time that elapsed between the start and stop call. Here we'll see how you can pause execution to skip a bit of code and resume accumulating time later. Also we'll see how you can artificially add time to the stop watch using the addTime method which can be useful if you're skipping a block of logic and want to assume that it took a predetermined amount of time. Finally you can start and stop a stop watch several times and output the average time that elapsed over each of the calls.

In the above examples we saw two ways to display the stopwatch timing information: getDuration and getAverageDuration. The method getDuration will only provide you with time information in milliseconds for the period after the last call to stop. That means if you call start and stop several times - you will only see the last, most recent call to stop. The other handy display method, getAverageDuration, display the average elapsed time for all calls to start and stop.

Advanced

Events and Managing Input

NeL provides a sophisticated yet easy to use event system to provide your application with a variety of events. The event system is comprised of four primary components:

• Events
• Event Listeners
• Event Emitters
• The Event Server

Events

Events are all based upon the CEvent class and have a unique ID using the CClassId interface. The header nel/misc/events.h has a full listing of available events and the event hierarchy and their corresponding unique IDs.

Event Listeners

Event listeners are classes that implement the IEventListener class. The interface defines the basic callback mechanism necessary for the event server to distribute events appropriately. Their implementation is fairly straight-forward and dependent upon your needs. Here's a brief example:

You can see that the event listener interface requires the operator overload for callbacks. In this example we have assumed that the event is a CEventChar event.

In addition to the simple interface the NeL framework also provides an asynchronus event listener class called CEventAsyncListener. This handy utility class allows you to receive events through it and then make checks within your code on-demand. Rather than writing an event listener based on the interface that gets called at the emitting of an event you can check with the asynchronus listener at various points in your code.

This listener implementation only has five methods of interest to users since it focuses on keyboard input:

• addToServer - this method takes an event server as an argument and adds itself to this server.
• removeFromServer - this method takes an event server as an argument and removes itself from this server. 
• isKeyDown - this method checks to see if the specified key (see the TKey enum) is currently down.
• isKeyPushed - this method checks to see if the specified key has been pushed, the default (which can be overriden) is to only check for keys that have been pushed and released.
• reset - this method resets the "down" states stored within this listener.

Event Emitters

Event emitters are the systems that perform the low level work of getting input and information from the OS or hardware and then providing it to the event server. Event emitters are based upon the IEventEmitter interface. Typically developers using NeL will not need to author event emitters as the framework already has implementations of the most commonly used input methods.

The Event Server

The Event Server is at the heart of the event system within NeL.

TODO

Working With Events

ddd

Keyboard Input

todo

Mouse Input

todo

Game Device Input

todo

Customizing the Event System

ddd

Streams

todo

Serialization

NeL has a complex serialization system which was inspired by the serialization system provided in Java. At the root of this system is the concept that serializable classes implement a common method that can be relied upon to import and export data from a variety of sources and ultimately marshall a new object. The serialization system is much more than converting an object to a flat-file and a flat-file back into an object. The serialization system actually harnesses the stream system so that it serializes an object into a binary data buffer in memory so that the serialized object can be handled in whichever way you as a developer deems appropriate. You can write your own stream implementation or use any of the stock ones, such as writing to a flat-file, but you can also harness the abilities of NLNET to send the object across the wire from client to server and vice versa.

The Serial Method

The first step to enabling a class to be serializable is to ensure that it implements the serial method. Take the following example:

This may seem too simple to be true - but this is essentially what is necessary to enable a class to be serializable. Note that the object we serialized into the stream was a string. With the NeL systme you can only default-serialize primitives such as strings, integers, and so on. Any complex types you must write serial methods for. Assume that the above example contained an instance of the class CBar and you wanted that to be serialized. You would need to ensure that CBar also had a serial method and that it was listed in the serial method of the CFoo class.

There are some exceptions to the primitives rule and the main, and most important, exception is that of containers. The stream interface provides methods to serialize containers and specialty containers. You can serialize most any STL containers using the streams interface, see this example:

That summarizes what is needed to work with the vast majority of serialization situations. Beyond this you will need to understand how to serialize pointers, polymorphic objects and hierarchical objects.

Serializing Pointers

When you encounter a pointer in your code that you would like to serialize whether it is a primitive or a non-primitive class or structure you will need to approach it slightly differently. NeL needs to be called with the serialPtr and serialContPtr methods so that it knows not to serialize the value of the object being passed (which may, in most cases, be the address of the object) but the actual object. NeL will output a value corresponding with a pointer and the data and in the event that the pointer is NULL NeL will output a zero (0). See this quick example:

NeL is intelligent in that while serializing data it will keep track of references to objects, preventing circular references as well as preventing duplicate work in the event that two objects reference the same object. Each time a pointer is de-referenced for writing NeL checks a table of the previous pointers processed and bypasses writing the data. At read-time the data structures are faithfully reconstructed.

Polymorphic Objects

There are going to be many instances in which you are working with an interface that a concrete object implements with no knowledge of what that concrete implementation is. NeL provides a system that will be discussed in detail later in this handbook for working generating classes. To understand the polymorphic capabilities of NeL serialization a very brief primer is required in NeL's classable system.

NeL has an interface called IClassable and two macros: NLMISC_DECLARE_CLASS and NLMISC_REGISTER_CLASS. For serializable objects you will actually implement IStreamable which extends IClassable. By implementing IStreamable and declaring/registering your class you can now generate your concrete object with a simple factory call.

The example above may seem convoluded at first - especially if you are unfamiliar with the classable system. We first created an interface that extends the IStreamable class which allows the serialization and stream system to manage this object as well as making the object capable of being generated by the classable class factory, this allows NeL to instantiate a new object of the concrete type by name, regardless of the interface that is being used. Next we create a concrete implementation of IBaseClass and a simple example class that serializes our new class. You'll see that we also used a new serialization method: serialPolyPtr. This method informs NeL that the object being serialized is polymorphic and that it should reference the classable system to identify the actual concrete type before serializing the object into a stream.

Handling File Format Evolution

Todo

NeL File Headers

Todo

Threading

The NLMISC module of NeL has a mixture of tools for programmers to make their applications threaded and to ensure thread-safety. This ranges from interfaces and classes to enable simple thread execution and monitor to controls to ensure thread-safety.

IRunnable Objects

The interface IRunnable is a basic interface that serves as the basis for most threaded logic. If you have any logic that you want executed you will create a sub-class of IRunnable. Here is the IRunnable interface:

You can see that the interface is fairly simple. All you're required to do is implement the run method with your logic. You don't have to implement the getName method but it is highly advisable that you do for troubleshooting. You will soon have logs filled with a thread name of "NoName" and a thread ID that outside of a debugger makes no sense. Below is an example of some logic created within IRunnable that can be threaded.

Thread Management

Simply having an instance of an IRunnable object does you no good - you must have some way to create a thread and manage that thread. This is where the IThread class comes into play. The IThread class provides an interface to store an instance of a thread that you will manage but also provides a couple useful static methods. We'll start with the static methods: create and getCurrentThread. Both of these static methods return an instance of an IThread object. The method getCurrentThread will provide you with the instance of the thread your code is currently executing in but the most important method is the static create method.

The create method is the method you use to convert IRunnable objects into threads. It takes an IRunnable object and a the stack size. If the stack size you provide is 0 - then it will use the default stack size as determined by the OS. Only provide a stack size if you have a specific reason for doing so - in nearly all scenarios providing only the runnable instance is sufficient.

Once you have created a thread object it isn't automatically running - you will have to manipulate it as you see fit which includes starting it. In addition to a method to start it the IThread class provides you with a few more essential methods for controlling the thread: start, terminate, wait and isRunning. The start method starts the thread, terminate will kill the thread before it reaches it's natural exit point, wait will force your current logic to block (wait) for the thread to complete and finally isRunning provides a bool return value to tell you if the object is currently running. Assuming that you have created the runnable from the previous example, here is an example of how you would manipulate the HelloLoopThread:

Task Manager

Not all instances of IRunnable must be convverted into thread objects. Another alternative is the task manager system. You can see an example of this at work in the CAsyncFileManager class. This class is a perfect example of implementing performance critical components of a game in a thread-safe way.

Mutexes

When you create a mutex you will typically use the generic CMutex which is actually a CFairMutex. You do have a variety of mutex types available but in almost all instances you should be fine using the CMutex class. There are a couple important things to note about the use of any of the NeL mutex implementations. You must never attempt to share a mutex across processes. You also must not call enter in succession - each enter must be followed by a leave before calling enter again.and always enter and leave in the same order.

• CFairMutex (or CMutex) - this is a fair mutex.
• CUnfairMutex - this is a classic mutex and can be assumed to not be fair.
• CFastMutex - this is an unfair mutex with low-level optimizations for speed.
• CSharedMutex - this is a method of sharing a mutex between processes without explicitly sharing the handle. This is very advanced.

Using a mutex with NeL is very simple, to illustrate here's a brief example:

In almost all scenarios you will use CMutex but in some special, specific cases you may use CFastMutex where speed is of the essence and you are not worried about thread-lock.

Synchronized Objects

So you can see above how you quickly used the low level mutex classes to make a block of code thread-safe. But there's a much easier and more reliable way of making a specific variable thread-safe. The synchronized objects templates are the most common way of implementing thread safety in NeL. By wrapping your variable in the provided templates you effectively force any access to the variable to be thread-safe. There are two main synchronized object templates:

• CFairSynchronized (CSynchronized) - this is uses a fair mutex to ensure thread-safety.
• CUnfairSynchronized - this uses an unfair mutex to ensure thread-safety.

Of the two templates the most frequently used one is CSynchronized. CUnfairSynchronized is only used in cases where performance is of a concern and thread-lock is not considered to be an issue. The following example shows how to use the CSynchronized template to protect a member:

Library Loading

NeL provides a set of classes and macros to ease the loading and initialization of library loading. This is a system that NeL refers to as "Pure NeL Libraries." The essence of this system is the CLibrary class and the INelLibrary interface. As a client developer (using a pure NeL library) you will use the CLibrary class extensively. As a plugin/module developer you will use the INelLibrary interface. Before you can use a plugin or module you must create it.

Pure NeL Library Creation

Your first step in creating a pure NeL library will be to create the library class that will be loaded. This class inherits from the interface INelLibrary and provides a handful of methods to the library loading mechanism. There are two important methods to be implemented on this class: onLibraryLoaded and onLibraryUnloaded. They both accept a bool for a parameter. This parameter (called firstTime and lastTime, respectively) specifies the order in which this class is loaded and unloaded. Finally there's an important macro that you will need to use called NLMISC_DECL_PURE_LIB - this does the hard work of exporting your library in a manner that the CLibrary class can access it. Here's an example of a pure NeL library:

The method onLibraryLoaded is called after each loading of the library. The pure NeL library system keeps track of how many places have requested this class to be loaded so that you can perform critical initial loading logic as well as any logic you may want to execute every time something requests this library. It is totally up to you as a plugin/library implementor. A common use of this is demonstrated in the sample above - the loading of any assets or allocation of any new objects necessary. There doesn't need to be any logic in this method, you could implement it as:

The method onLibraryUnloaded is called after each unloading/release of the library. The parameter "lastTime" will be true if this call is the last remaining reference to the library. You would commonly do things such as save out data, delete allocated objects and other cleanup things. Think of it as a library-level destructor. As with the onLibraryLoaded the system doesn't require you to put any logic in this method, simply implement it as shown above - with no logic.

Finally there's the NLMISC_DECL_PURE_LIB macro. You don't really need to know much about this other than that it must be issued in your implementation so that the CLibrary system can access the symbol to load your newly created INelLibrary implementation. 

Pure NeL Library Loading

While you may never create a plugin/library using the INelLibrary interface explained above hopefully you will find need to load a plugin using this system. Loading a plugin using the CLibrary system is fairly easy. The system does provide you with a variety of useful utility methods as well. Before we get to the meat of loading a library we should talk about a couple of the utilities that are provided through the static public members of the CLibrary class

The two most important methods provided are makeLibName and addLibPath. The makeLibName method takes a library name (e.g. mymodule) and outputs a new string with a decorated NeL library name. This means that it will convert a library name into a NeL-compliant library filename. It decorates the name in accordance with the platform and compilation mode. On Windows in ReleaseDebug mode mymodule becomes "mymodule_rd.dll" and on Linux it becomes "libmymodule_rd.so." Because it is possible that you will load a library before you have had a chance to populate CPath with search paths the CLibrary method provides you with the addPath and addPaths methods so that you can provide it with information on how to find the dynamic library to load.

Now that we understand some of the basic utility functions we will create a new CLibrary handle and use the loadLibrary method to actually load a library. Before explaining how all of this works together an example might be helpful:

The loadLibrary method takes 4 parameters: the name of the library, whether to add NeL decoration, whether to use the library paths and finally whether this library handle "owns" the library being loaded.

If you used makeLibName yourself then you would set the 2nd parameter to "false" so that it doesn't attempt to re-decorate your library name. These first two parameters are pretty self explanatory. The 3rd parameter tells the library handle whether to use the paths that you added using the addPath or addPaths methods. If this is true it will go through each path in the order added to the library handle. The final parameter, ownership, should always be true unless you are doing something very advanced. This specifies whether or not this handle owns the symbols that are loaded by this. You could potentially use this parameter to some advanced uses but it disables your ability to unload the library with this handle. This last point is very important - if you load a library specifying an ownership of false and then attempt to free the library your code will assert. Finally it will return true if it successfully loaded this library.

Internally the CLibrary system tracks whether or not the library that was loaded was a pure NeL library or not. While at first blush you may have assumed that this system was written specifically for pure NeL libraries it has the ability to dynamically load and unload non-NeL libraries but you lose a fair amount of functionality. It is still suggested to load any dynamic libraries within NeL applications with this system for consistency and to avoid openly calling LoadLibrary or dlopen directly.

When you have loaded a pure NeL library you can retrieve the library author's INelLibrary object. This is important if you know specifically what the library implementation is. Some library authors may expose a handful of public methods on the library that are useful to its implementors. See the following example:

Exposing Symbols

Having taken the time to load the library does you little good if you have no method of exposing functions from the DLL. In the following example we'll illustrate how you can create an interface to represent your library API, implement the logic, and then expose the ability to create these classes from outside code after loading the library. A couple key concepts to keep in mind is that this technique is based upon your creation function definition existing in your source (client) code and the implementation of this creation function existing within your DLL.

At first this seems dauntingly complicated but in reality it's very simple. It's based on three premises: you have a pure virtual interface providing the API required for end-users of your library, that you provide a function to create concrete instances that implement this API and that you expose this function's name and function pointer typedef to users implementing your API. It's easiest to assume that you'll understand interface-based design and instead focus on the mechanics of exposing this library to users.

The first step is to define the function pointer such that it has the correct return value and arguments to create your new object. This typedef must exist in your header that your users will be including. If you included tool.h instead of itool.h you as a user would have a number of linking errors due to missing the sybmols for the Tool class. Function pointer typedefs are pretty simple, but important here. We provided a typedef that returned a pointer to a ITool object and required no arguments.

The next step is to name this function - you need to provide a string name, preferably in the form of a const char in your public header which is itool.h in this instance and finally you will to implement a function with this name using the interface defined by the function pointer - a function returning a pointer to an ITool object with no arguments named createIToolInstance. It is also interesting to note that we used the NL_LIB_EXPORT macro. In Unix compilers you do not need to specify anything to tell the compiler to expose the symbols in the shared library, however in Windows you will need to use __declspec(dllexport) and this is logic that NeL performs for you through this macro.

Finally when you're ready to use your new library you will use your CLibrary handle to retrieve the creation function symbol through it's string name and then call it to invoke it's logic, which in this case is to return a new Tool object. The call to getSymbolAddress in the example above and the cast of the resulting object demonstrates this functionality.

Library Versioning

If this is a library where changes to the API could occur and be shipped with code not recompiled against the import library you will more than likely want to implement library versioning. This is a technique in which you specify a version number in your concrete implementation and only increment it when you believe that functionality changes or API changes will break code using the former library version. This uses the exact same technique as the creation function example above with a small twist.

This uses, as stated above, the same technique for exposing symbols as the creation system but we have a small wrinkle. The version is defined in itool.h - this is important to note because both the software implementing the library and the library itself are using this version. However if you changed the API, incremented the version, and then ran the new library with the old software (meaning, you changed the API but did not recompile the software using the API) it has the ability to compare the version the library is using versus the API version it's build to expect. So you could do a comparison when you load the library:

Application Contexts

Having discussed the pure NeL library system in the section above it is now important to discuss the application context system and safe singletons. Applicaiton contexts are a way for objects to traverse address spaces via relocation. Most Unix systems such as Linux use Position Indenendent Code (PIC) as a method to ensure that an application and all of its shared libraries are operating in the same address space. In overly simple terms when Linux loads a shared library it automatically relocates the symbols address spaces.

Windows systems do not implement PIC and so each shared library (DLL) operates in an independent address space. To illustrate this imagine you have a singleton called CFoo and it has two methods, one called addFoo and the other called printFoos. This obscenely simple singleton will allow you add strings to a list and then print them off. The problem with Windows (and other platforms that do not support PIC) comes up when the application and a library or two libraries use the same singleton - the end up with different images of world based upon their address and execution space. Below is an example of the kind of behavior you would exhibit on a Windows-based system:

Notice that the CBar library outputs something differently than the application itself? To overcome this the NeL developers created the NeL Context system and the concept of safe singletons.

Application and Library Contexts

Applications created using the NeL network services macros automatically have an application context but all other applications may need to explicitly create one. All you need to do is create a new instance. You can also check using the INelContext interface if you want to avoid creating a context unless necessary. Pure NeL libraries based on the INelLibrary interface will also receive a copy of the loading application's context.

Safe Singletons

Safe singletons are singletons that are aware of the NeL context and can be used reliably in multiple libraries as well as the application. The NeL framework provides a couple useful macros to ease the creation of safe singletons: NLMISC_SAFE_SINGLETON_DECL, NLMISC_SAFE_SINGLETON_DECL_PTR and NLMISC_SAFE_SINGLETON_IMPL. The difference between NLMISC_SAFE_SINGLETON_DECL and NLMISC_SAFE_SINGLETON_DECL_PTR is that the former provides a reference to the singleton object and the latter provides a pointer to the singleton object. References are preferred as they tend to be a safer option to the pointer variation.

Using these macros is pretty simple. When your your class definition, simply declare the DECL-variant you wish to implement in the private scope of your class definition. Then in the source where you define your concrete implementations of methods of your class use the NLMISC_SAFE_SINGLETON_IMPL macro.

Class Management

NeL provides a robust class factory system that it uses in a various places. The most notable place that it uses it is in the stream system (described in another section.) If you want classes that can be created by the stream system via poly pointers or if you want to add new shape/model types to the 3D system you will want to know how to use the class management facilities of NeL. This class factory system is actually very simple and involves two macros, an interface and a simple class factory: NLMISC_REGISTER_CLASS, NLMISC_DECLARE_CLASS and IClassable. Here's a simple sample to demonstrate:

In the above example we inherited IClassable on the class that we want the class registry to create and used the NLMISC_DECLARE_CLASS macro to automatically create required methods for the class registry system. Then in main we used the NLMISC_REGISTER_CLASS macro which actually registers this class with the registry for future creation requests. If you fail to use this macro and attempt to call createClass method the class registry will throw the EUnregisteredClass exception. If you accidentally call this macro twice in a runtime the class registry will throw the ERegisteredClass exception - which simply means this class is already registered. It is important to catch these exceptions so that you can be proactive in finding errors in your code. This is the essence of the class registry system. Do not be confused by the CClassId class - this class will be explained in later sections.

Many plugin developers will put the NLMISC_REGISTER_CLASS macro calls in their implementation of INelLibrary's onLibraryLoad where firstTime is true - this means that as soon as the plugin is loaded by the client developer all of the classes within that plugin are registered with the class registry. 

Custom Logging

You may want to customize the logging system directly for your own needs - include "nel/misc/log.h".

Initialization

If your program is not a "service" in the NeL terms (i.e. if it is not built on NLNET::IService),
first call createDebug() to create the basic global loggers ErrorLog, WarningLog, InfoLog, DebugLog and AssertLog (the ones that are used by the nlerror, nlwarning, nlinfo, nldebug and nlassert macros).

Then, you can attach some displayers to the global loggers (NLMISC::CLog objects).

• NLMISC::CStdDisplayer is for stdout (the console, the VC++ output window...). It is attached by default to all of the five logger objects mentionned above.
• NLMISC::CFileDisplayer is for a file.
• NLMISC::CMsgBoxDisplayer is for a message box.
• NLNET::CNetDisplayer is for a logging server (see CLogService in the nelns documentation).
• You can can create your own displayer, in order to print the logged text onto a new target (for example, see the class CChatDisplayer in Snowballs 0.2) or to customize the filter on the header.

Example (we assume CNetDisplayer allows to log via the network):

Logging Information

How do you log a string without repeating the header? You can use the methods on NLMISC::CLog.

Example :

Commands

Basic Command Creation

Commands are a powerful real-time way of handling user command input. Commands are put to heavy use in the NLNET service code as you'll read later but can be used in many places. There are two main macros that govern the creation of commands in NeL: NLMISC_COMMAND and NLMISC_CATEGORISED_COMMAND. NLMISC_COMMAND calls NLMISC_CATEGORISED_COMMAND with a default category of "commands" so we'll discuss NLMISC_CATEGORISED_COMMAND indepth. Below is the definition of the macro.

The NLMISC_CATEGORISED_COMMAND macro takes four parameters. The category parameter simply specifies the category of the command within the help system. This is particularly useful if you have a program or service with a large number of commands. The name parameter specifies the name of the command that the user will call to execute the command - this will make more sense later when we discuss using the NLMISC::ICommand::execute static method. The last two arguments, help and args are also for the help system. The help argument is a string that you provide to advise the user on what the command is and how to use it. The args argument is a string that you provide to explain the type and number of arguments to provide to the command system. Here is an example of a fully functioning command:

The block of logic defined above is actually your specialization of the abstract method NLMISC::ICommand::execute. When the system calls your command it will pass this block of logic the following arguments for you to work with:

• rawCommandString - A std::string of the raw command string (as the user inputted it.)
• args - A vector of std::string objects representing the arguments the user provided.
• log - A reference to the CLog the system would prefer you to use.
• quiet - A bool value representing whether or not the system would prefer that you not output logging
• human - A bool value representing whether the input originated from a human user.

Executing Commands

Once you have created commands if your application is a NLNET service you may begin using them immediate from the service console. However if you're programming a new console application or even a game you can capture user input and pass it directly the command system. This is a quick and easy way of creating a command console for clients. See the following code for an example of executing commands outside of the service infrastructure:

You see here how we quickly developed an application in which a user can execute commands that you have created with a small amount of code.

All commands should following the naming convention of NeL functions, for example: updateSomething and displayStuff.

Advanced Command Techniques

The macros, interfaces and examples above will cover the vast majority of developer needs. But you will find circumstances where your command needs some special attention. There are two more advanced topics to cover with NeL commands: command friendship and object command handlers.

Command Friendship

At the core of the command system are the macros mentioned above which dynamically generate a new class for your command. If this command needed access to some private members of an object that, for example, displays the current thresholds you would need your command to be a friend of this other class. NeL provides two handy macros to accomplish this and all you need to do is use them in your class definition.

Custom Command Handlers 

The custom commands handler is a much more extensive subject.

TODO FINISH TALKING ABOUT THIS

AND THIS 

Variables

NeL provides a variable management layer. This variable management is an extension of the command system explained earlier and comes in two flavors: basic and dynamic. All variables have a naming convention to match the NeL naming convention for public members such as: MyVariable, SomeValue or UsefulSetting.

Basic Variables

A basic NeL variable is a way to expose a concrete primitive to the command system for real-time interaction with the user. The following snippet of code allows the user to modify a variable using the command system as explained in the previous section:

This simple tidbit of code would allow a user to change the value of MyVariable by simply calling the ICommand execution method with "MyVariable 3" or to retrieve the current value by simply calling it with "MyVariable." In essence any variable you create with this macro becomes a simple command that can be called to set or display the value. You can see here that the basic variable macro takes three parameters: variable type, variable name (which must match a concrete variable name as in the example,) and a brief description of the variable which will be accessible under the variables category of the command help system.

Dynamic Variables

Dynamic variables provide you a little more flexibility in the process of getting and setting the value of the variable. One of the core goals of this is to provide you access to a variable which you do not have direct access to (such as a private member of a class.) With a dynamic variable you do not need a variable or class dedicated to the macro/command - you could use some other object within your simulation. If you use an existing class you must implement get and set methods which match the type that you will define in the NLMISC_DYNVARIABLE macro. One of the great advantages to this over the basic variable is that you as an implementor are allowed to define the getter and setter methods which allows you to perform some logic. You may want your setter method to check the limits of the value being passed or to clamp it to a reasonable value. Here's an example of a dynamic variable at work:

In the example above we created a class and then created an instance of it in the global scope. This could easily have been a class with static members or a singleton, we didn't have to create an instance at the global scope - it just has to be accessible in some way from the dynamic variable class/logic.

Built-In Variables

The NeL framework provides a variety of existing built-in variables for use. Below is a comprehensive list of the variables provided by the framework. In a later section specifically about the NeLNS system there will be even more built-in variables available for use that are specific to the various provided services.

 When developing your game or application you should keep a table of available variables similar to this for a quick reference.

Modules: NL3D

Introduction

The NeL 3D library includes 3D audio and video rendering modules.

The objective of the 3D library is to provide an architecture framework for housing the code modules required for representing a virtual universe, and its contents, in 3D.

Nevrax is developing the code modules required to represent the universe that their product is based around. This universe includes animated characters, objects and special effects in a variety of environments including undulating terrain, towns and the insides of buildings.

Portability

NeL 3D library is structured in layers built on top of a common driver layer. The Nevrax team developed the OpenGL and DirectX implementation of the driver layer. Of course, only the OpenGL implementation works on GNU/Linux.

NeL 3D is currently tested on Microsoft Windows platforms but compiled on GNU/Linux

Target Platforms

There are background loading and dynamic level of detail systems that apply to all components of the 3D library, allowing the software to be adapted to run efficiently on different specifications of hardware.

The target test machine specification is as follows:

• Intel Pentium 3 1GHz
• Graphics card comprising Nvidia NV20 chipset
• 128MBytes RAM

Nevrax expect their product to run at 30fps at 1024x768 resolution on a machine of this specification.

Features

Undulating Landscapes

• Bezier patch landscape modeling gives NeL enormously more flexibility than traditional height fields. NeL supports vertical faces, overhangs, and so on.
• The ROAM algorithm is used to provide adaptive subdivision of the landscape based on distance from the camera, steepness, etc.
• Geomorphing is used between LODs (levels of detail) to eliminate popping.
• Landscape coloring effects include vertex coloring and application of pre-calculated shadow maps. Pre-calculated shadows can have soft edges while real-time shadows obviously can not. Shadows can be colored and faded at run time
• Lightmaps are used for landscape lighting to avoid lighting changing with LOD transitions
• Landscape texture mapping at a constant 1.5cm/ texel, with the possibility of a second complete additive texture layer. Texture continuity breaks due to bilinear filtering have been eliminated.
• Landscape is dynamically streamed into memory with background loading
• 'Snap To Ground' functionality allows objects that are on the ground to remain attached to the surface as it geomorphs
• Displacement maps (also known as geometric noise) - these are taken into account during lightmap generation which means that they can still be 'seen' at low levels of detail
• Dynamic lighting

Portal-based Streets and Interiors

• Static light maps: up to 3 layers of light map simultaneously - each layer can be enabled, disabled, faded or colored in real time
• Tools: light map generation: max plugin: support for different lamp types, ray tracing for shadows, projected images (like a slide projector), support for colored translucent objects influencing light color (eg colored glass), support for saturation (with lighting at > 100%), mixing of vertex colors into light map calculation
• Use of portal algorithm for view determination

Characters and objects

• Component based object construction, which allows the assembly of multiple "parts" of objects into a single mesh around a single skeleton
• Multi-Resolution Meshes, to provide a smooth reduction in polygon count as objects retreat into the distance. Full implementation of the technique presented by Hope, in a recent paper, with geomorphing. This implementation provides superior results for low detail LODs and smooth transitions between LODs.
• Optimized management of extremely low detail 'billboard' versions of objects
• The LOD system is capable of working with a subset of the LODs. This means that the high detail LODs don't have to be loaded if there is insufficient memory or if the client machine is not sufficiently powerful to make use of them.

Animation

• Skinning: support for skinning with up to 4 matrices per vertex

• Smooth transitions between animations
• Controllable blending of multiple animations (eg 70% walk + 30% limp)
• Application of different animations to different parts of skeleton (eg run + punch)
• Animation types (so far): bones, material & light parameters
• Animation export plugins for 3DStudio Max: feature more or less complete set of 3DStudio animation controllers (bezier, TCB, linear, etc) and support for biped (Character Studio) skeleton
• Skinning export plugins for 3DStudio Max: support both physique (character studio) & com_skin2 skinning data
• Blended Shape animation, to provide morphing and lip-sync style animations

Materials

• Support for environment mapping, multi-texturing, texture-coloration.
• Tools: more or less complete support for 3ds max materials
• Vertex and pixel shader implementations for GeForce3 and above

Particle System

• Support for various particles representations, including dot, line, triangles fans, bitmaps, shockwave, ripples, meshes.
• A variety of emitters are available : mesh, cone, plane, points and sphere emitters.
• Forces can be applied to particles
• Support for simple collision shapes for particles
• Support for LOD with distance
• Tools: custom WYSIWYG editor in the ObjectViewer
• Particle systems can cast light into the scene and be lit by scene lights
• Integration of tools with Max and with animation exporters

Other Special Effects

• Screen distortion effects
• Real time shadow casting
• Water rendering effects

Technical Features

• Exploitation of 3D graphic accelerator cards' TNL, pixel shader and vertex shader capabilities.
• Adaptive memory management with background hard disk data streaming.
• Adaptive texture and polygon detail to manage CPU load, GPU load and video memory constraints.

Architecture

The rendering process is composed of several traversals. Each traversal acts on their models using virtual methods. The rendering process is done following this pipeline:

• Transform animation traversal
• Transform traversal
• View determination and culling traversal
• Detail animation traversal
• Load balancing traversal
• Lighting traversal
• Render traversal

Transform Animation Traversal

This traversal must evaluate the transforms animated value to get the final parent relative world matrix of each model. We must do this pass to know where each model is.

Transform Traversal

This traversal composes each transforms matrix with the parent one to get the final world matrix. This pass is used to perform hierarchical animation of models.

View Determination and Culling Traversal

This traversal clips models that are not visible. It uses clusters/portals for interiors and a frustrum clipping for exteriors.

Load Balancing Traversal

For each model traversed this traversal evaluates what rendering resources the model requires. It counts the number of primitives that it needs to have rendered and amount of vram used by their textures.

Detail Animation Traversal

This traversal computes the animation for all values not animated in the first pass (transform animation). Animation is done in 2 passes because the first must be done to get models' world bounding box used in view determination. The second pass is done only on visible objects.

Lighting Traversal

This traversal computes dynamic lighting for each visible model with non-clipped lights.

Render Traversal

Render is the most complex traversal.

• If the model uses skinning, it will compute bones world matrices. Matrices have been set by second animation pass but they need to be combined with their parents.
• If the model has an inverse kinematics solver attached, it is evaluated next.
• If the model supports resource load balancing, the final count of rendering primitives is adjusted in function of the distance from the camera.
• The primitives to send to the driver are built.
• The primitives are rendered by the driver.

Basics

The Driver

Working with Fonts

Managing the Scene

Manipulating the Camera

Working with Shapes

Working with Skeletons

Managing and Playing Animations

Advanced

Working with Lights

Managing Landscape

Managing Landscape Loading

Managing and Manipulating Vegetation

Managing The Environment

Creating and Manipulating the Sun

Creating and Manipulating Fog

Creating and Manipulating the Sky

Shape Banks

Manual Primitive Drawing

CEGUI NeLRenderer

Working With Particles and Particle Systems 

Modules: NLNET

Summary

The NeL networking module is designed as a six layer architecture with each layer starting at zero building upon the previous layer and providing the developer with increasing functionality. We'll cover all six layers through the course of this section, including the two deprecated layers, layer two and layer four. The goal of the NeL networking module is to provide a simple elegant interface to facilitate the three primary forms of network communication: client to server, server to client and server to server. While most developers using NeL will only use layers three through six the NLNET module provides developers with direct access to all layers.

0 - Data Transfer Layer

This is the bottom layer of the NeL networking module and is commonly referred to as Network Layer 0 in the NeL 5 layer architecture. It is responsible for abstracting the local API (POSIX, WinSock, etc) from the rest of the networking system. As developer using NeL you will not be expected to work with classes in this layer, such as CSock, CTcpSock and CUdpSock. Only NeL developers seeking to add functionality, such as IPv6 or some other protocol, would work at this layer.

1 - Data Block Management

This is commonly referred to as Network Layer 1. This layer is responsible for buffering data for presentation to the lower level transports. It interacts with the generic NeL serialization and streaming system to send and receive data to the higher layers. It also contains the multi-threaded listening system for services. This layer should not typically be used directly by developers using the NeL framework.

Server Structure

The server, for example CBufServer, provides a single receive queue (see "Receive FIFO Buffer" on the object diagram below), and one send queue per connection ("Send FIFO Buffer"). Internally each connection is associated with a receive buffer to handle non-blocking receiving of incomplete data blocks (in CServerBufSock). The actual receives and sends are done by in layer 0 by a socket, such as CTcpSock.

Every connection is managed by a receive thread such as CServerReceiveTask. Instead of having one thread per connection, which would slow the system down and limit the maximum number of connections, there is a pool of threads that handle several connections. For example 30 threads handling 30 connections allow 900 simultaneous connections. If only one thread was in charge of all the connections the select() operation would take too much overhead for a large number of sockets.

Launching The Server

The server CBufServer starts a listening socket CListenSock which is handled by a particular thread CListenTask. It can then accept incoming connections.

Accepting Connections

When a connection is accepted the server advertises the connection by pushing a connection event into the receive queue. Then it dispatches the associated socket to a receive thread from the pool (or a new one).

Reading Data

The user of this layer, which is usually a higher NeL Net layer, makes a callCBufServer::dataAvailable() to check for incoming data. If a connection or disconnection event is found at the top of the receive queue the associated system callback is called. Then the socket is logically connected (CBufSock::connectedState() is true) in case of a connection event.

If dataAvailable() returned true, the user callsCBufServer::receive(data,&sockid). The second argument tells which connection the data is coming from. It can reply to it directly by calling CBufServer::send(replydata,sockid).

The user must call CBufServer::update() for the system to work properly.

Sending Data

Sending data is buffered as well. The moment when the data is actually sent depends on the triggers specified. By default the time trigger is enabled with a value of 20ms. It means that the data will actually be sent after 20 milliseconds, providing that the update() method is called evenly. The user can also specify a size trigger. The size trigger is  when the size in the send buffer exceeds the specified size the data is sent. Eventually the user can force sending by calling flush().

How the Non-Blocking Receiving Works

Each receiving thread, CServerReceiveTask, performs a select() on its sockets. When incoming data is reported the methodCServerBufSock::receivePart() tries to read a block, which is made up of a length prefix and a payload buffer. If the actual data received is smaller than expected it is retained in the receive buffer for later completion. When it is complete it is pushed into the main receive queue.

How the Non-Blocking Sending Works

All data from a send queue is copied into a buffer and then sent. If the sending was not entirely done or it would block part of the buffer is kept for later sending.

Handling a Disconnection

When a receive thread detects that a socket is disconnected it pushes a a disconnection event into the receive queue by the nextCBufServer::update(). When the disconnection event is processed, at the top of the receive queue, the socket is added to the synchronized set of connections to remove of the thread. It will be effectively removed before its next select.

Wake-up Pipes

A Unix pipe is added in every select set. This is so that the select can be stopped when there is a new connection to add to its set or when the server is required to exit. Similarly, the listen thread performs a select on the listen socket and the wake-up pipe.

Under Windows, the wake-up mechanism is not implemented. Instead, the select timeouts are shorter.

Thread Synchronization

Because of the sharing of data among different threads some mutexes are used to synchronize data access/modification on items:

• The receive queue
• The thread pool
• The connections
• The set of connections to remove
• The connected property of sockets

To ensure a fair access to the variables the GNU/Linux implementation uses a semaphore instead of a pthread_mutex.

Gathering Statistics

Several methods are provided in CBufServer to know how many bytes have been read and written.

Client Structure

The client provides one receive queue and one send queue. The receiving is done is a separate thread, CClientReceiveTask, but the actual sending, flush(), is done in CBufClient::update(). The socket is in blocking mode.

Unlike in the server there is no connection advertisement. A disconnection event is pushed into the receive queue when the receive thread or the sending detects the disconnection. It is also important to note that the client does not remove the socket after disconnecting. It removes it at destruction time so the user can reuse the same CBufClient object by calling again connect() after having disconnected.

2 - Serialized Data Management

This layer is commonly referred to as Network Layer 2. This layer, for all intents and purposes, has been deprecated from active use. It exists for legacy code and should not be used directly for development. The purpose of this layer was to facilitate serialization and streaming of data between higher level code and the low level network transports. This is almost entirely facilitated through the NeL serialization and streaming functionality in the NLMISC module.

3 - Message Management

ddd

4 - Inter-Service Message Management

This layer was the original implementation of inter-service message management. It provided a simple API, CNetManager, that allowed services to send and receive messages. This original implementation didn't scale and meet all of the requirements of the original design and was eventually superceded by the unified network system, described below. The supporting classes still exist within NeL but are deprecated and it is suggested that they are no longer used.

5 - Unified Network Services

This is the top layer of the NeL networking module and is commonly referred to as Network Layer 5.

6 - Network Module

ddd

Common Functionality

ddd

Sending Emails

One of the common utilities that NeL provides is a very simple email (SMTP) interface for sending information emails. This could be useful for situations like crash reports. This system has two functions involved in its use:

The setDefaultEmailParams sets default values for the SMTP server to use, the From email address and the To email address. You can call this method to set those for all subsequent calls to sendEmail - this allows you to send empty strings to this function thus using the default.

Do not use the setDefaultEmailParams function from within a NLNET service since the service already accomodates this through config file variables. If you wish to use sendEmail from within a service you need to change the following config file variables in the service config file:

• DefaultEmailSMTP
• DefaultEmailFrom
• DefaultEmailTo

Actually sending emails through this function is very easy. Note that if you configured the defaults using a config file (for services) or through a call to setDefaultEmailParams"" you can use empty strings (e.g. "") for the first three parameters: server, from and to. The next two arguments, subject and body, are required and should reflect the subject and body of the email you would like to send. The final two arguments, attachedFile and onlyCheck, are optional. The onlyCheck flag is false by default. It tells the email functionality to mock sending the email but not actual complete the transmission and is useful for testing but not likely to be used in production code.

The attachedFile argument is also option by should contain the path and filename of the file you want to send. The email functionality will look up the file, determine the mime-type based on a limited hard coded list and then uuencode it so that it appears as a standard attachment in standards-complant mail clients.

Message Management

The message management layer and it's founding class such as CCallbackClient and CCallbackServer will be used extensively by the higher layers, chiefly layer 5 which handles and manages the NeL network services. However you may use these classes to facilitate TCP connectivity between your services and a client or tool. In many of the available examples the "player client" uses a CCallbackClient to connect to the frontend service's CCallbackServer, which then receives and filters player messages and submits them to the rest of the game services as appropriate.

In addition to this use of the message management functionality it is, as said earlier, the basis of functionality for the higher layers so it is very important to understand how to create and receive messages and receive them through callbacks and callback arrays.

Using Messages

The work done to facilitate serialization and streaming in the NLMISC module has not gone to waste, it is also hard at work here in the NLNET module and is the core concept behind the CMessage class. In its simplest form the CMessage class has a name and is a stream of variables loaded in a specific order. A quick example in code may clarify:

You can see it is as simple as the serialization system is explained in earlier chapters: you call the object's serial method with an argument of the variable you would like to serialize. Since this class is based on the CMemStream class from NLMISC you may use any of the documented serialization methods, including those to serialize containers. Just like in ordinary serialization, order is critically important. When reading a message you must read the variables in the order that they were added:

Messages also have a number of other useful methods for figuring out more information about them. For example depending on upon whether a message is being sent or received reflects what the serial methods do: if a message is being sent the variables are added to the stream and if the message is being received the data is read from the message stream and assigned in to the variable. You can verify this through the isReading() method of the message.

Messages allow you to change their stream formats - the way in which the data is encoded when the message is streamed to the destination. The default option is UseDefault, but you can specifically choose Binary or String. The default string mode is configurable through CMessage::setDefaultStringMode(bool). The default upon compilation is false - meaning that messages will prefer to use Binary mode unless specifically asked to use a different format using the setType() method. It is important to note that when CMessage is building its header it will change the mode to Binary and then change the mode back to the stream format you have chosen - meaning that the header should always be formatted in Binary.

If you set the type (message type, not stream format) via setType and the message is in reading mode it will only change the message name and stream format flags but will not rebuild the header. When building a new message a setType call is implicit. Typically the default stream format is sufficient and the streaming system will elect to change stream formats dynamically as befits the data streaming.

NeL also has four message types: OneWay, Request, Response and Except. The default message type is OneWay. The concept of mesasge types is specific to layer 6 of the NeL network architecture.

Understanding Callbacks

Callbacks live at the core of the message management layer. Callbacks are comprised of three parts: the callback function, the callback array and the server or client utilizing the callbacks.

Callback Functions

Callback functions are the easiest concept to explain.  There are two function templates used for callbacks: unified network callbacks and unified message callbacks. Unified network callbacks are callbacks used to receive notice about service up/down events. Unified message callbacks are what you would call when a certain message type (signified by message name, not literal message type) are received. These functions receive the message that was sent, read it in and produce some sort of logic.

There are a number of callback types available in NeL's networking framework:

Below are some of the function template definitions:

A message callback is the function called whenever a message name is mapped to the function through a callback array. A service callback is called whenever the function is provided to the unified network system as a method to call when services by a specific short name come up and down. This will be explained in more detail in the section about NeL services and the unified network system. Below is a brief example of the two callback templates in implementation:

Callback Arrays

The callback arrays are simply a way of mapping a message name to a callback function. There are two primary callback arrays used when writing NeL network clients or services: TUnifiedCallbackItem and TCallbackItem. The latter is used with CCallbackServer and CCallbackClient and the former is used by the unified network system.

Callback arrays are only used for unified message function mapping and never for unified network functions - the latter are always assigned directly to a short name through the unified network system. Here is an example callback array for the above examples:

You can see we took the message name we expected to receive from the other service and mapped it to our function that implemented the function specification for unified message callback functions. Both callback array types use this same message-to-function mapping.

Callback Clients and Servers

To bring this all together and demonstrate the concepts before we move on to the layer 5 unified network services we'll demonstrate a simple client/server using the callback client and server.

NeL Network Services

The heart and power of the NeL network module is layer five, Unified Networking. This is the delivered framework providing functionality for server-to-server communication as well as the necessary functionality to participate in a NeL Network Services (or NeLNS) shard. The NeLNS services will be discussed in detail in future chapters but the one service that will is critically important to this chapter is the Naming Service, or NS. The Naming Service provides a universal lookup system for the unified network classes, it allows you as a service developer to programmatically find other services within your shard by name.

Creating Services

The first step in working with NeLNS is understanding that NeLNS provides the essential framework for a server but provides no game functionality. For example the Snowballs technology demo requires three services in addition to NeLNS: a frontend service (FES), a position service (POS) and a chat service (CHAT). The FES is the most common service - every shard will need an implementation of at least one FES, but the other two services implement game logic.

To create a service you will first need to define your service class. This class must inherit the NLNET::IService class - this is the basis for all NLNET services. Following the class definition and implementation of methods you will need to, in your source file, declare the class as a service using the NLNET_SERVICE_MAIN. This macro has seven (7) arguments:

1. Service Class Name: e.g. CTestService
2. Short Service Name: e.g. "TS"
3. Long Service Name: e.g. "test_service"
4. Service Running TCP Port: e.g. 37000
5. Service Callback Array: e.g. EmptyCallbackArray
6. Service Default Config Directory: e.g. "/opt/nel/etc"
7. Service Default Log Directory: e.g. "/opt/nel/var/log"

The default config directory and default log directory can be an empty string ("") in which case the service will use the current working directory, or in otherwords it'll look for its configuration and write its logs to the directory that the service was run from. In the example above we used "EmptyCallbackArray" for the service callback array. This is a delivered callback variable that has no callbacks listed for the service. Callbacks will discussed in a later section. The long service name is important because the service will search the default configuration directory for this name plus .cfg for its configuration. The short service name is how you will refer to all running instances of this service. Finally the service class name is the class you defined earlier. The only method that requires implementation is the init() method, however it need not require any logic. Here is a brief example in code:

There are three methods that you may implement that the service system will call and what are some common uses:

This is enough to get your service running but isn't getting your service to do critically important tasks like receive messages from other services. To accomplish this we will need to learn about callback and setting up callback arrays.

The notion of callbacks starts in an earlier layer of the NeL network module: layer 3, the message management layer. This layer, which will be discussed

Working With Unified Networking

ddd

Commands and Variables in Services

ddd

Creating Network Modules

ddd

Modules: NLPACS

ddd

Basic Concepts

ddd

Move Containers

ddd

Move Primitives

ddd

Move Primitive Properties

ddd

Move Primitive Types

Move primitives have two essential types: 2DOrientedCylinder and 2DOrientedBox. All collision primitives in PACS are two dimensional and extrude infinitely on the Z axis, therefore height does not matter.

For o

Move Primitive Reaction Types

ddd

Move Primitive Trigger Types

ddd

Move Primitive User Data

ddd

Creating and Loading Move Primitives

As illustrated when discussing move containers we can manually create a move primitive and add it to the move container. This is, however, incovenient for static data. Landscape for example will have hundreds if not thousands of instance groups with collision data. During the process of creating content the content developers and artists will also create primitive objects within their editor (e.g. 3DStudio MAX.) The exporter has the ability to save PACS primitives out to a file

TODO finish talking about this, maybe screen shots?

World Images

ddd

Advanced Concepts

ddd

Retrievers

Retriever Bank

A retriever bank is typically loaded from a file with the .rbank extension and represents a collection of Local Retrievers.

Global Retriever

ddd

Local Retriever

A local retriever represents

Local Retriever Types

Landscape

ddd

Interior

ddd

Anatomy of a Local Retriever

Chains

ddd

Surfaces

ddd

Tips

ddd

Topologies

ddd

Face Selection Grid

ddd

Global Retriever

ddd

Collision Descriptions and Triggers

ddd

Global and Local Position

ddd

Primitive Blocks

ddd

Modules: NLGEORGES

Summary

Georges is a module of NeL that is used heavily in the loading of data into the game system. It's based on two core concepts: Forms and Sheets. A form is an XML representation of the data which requires a loader to translate into in-game data. A sheet is a method of packing the forms and referencing them by a unique ID. This is useful when communicating sheet information between the client and the server.

Georges Forms

Georges is a versatile and flexible form loader built upon XML. A basic Georges form is comprised of three basic elements:

• The Form itself.
• The Definition (DFN) files.
• And Type (TYP) definitions.

To better understand how the pieces of a form come together it is easiest to learn the files in reverse order. Forms rely on definitions which rely on types. A type is the simplist piece of configuration in a Georges form.

Types

Types are the smallest block of information in creating a Georges form. Before doing anything else you will need a set of basic types and an understanding of creating custom ones for your data types. First we'll show you two samples. These are types that can be used in a real application but are simple enough to see the basics of a type and give some examples elements to describe.

• int.typ - A basic integer.

boolean.typ - A basic boolean.

All type files should contain only one TYPE element but can contain several DEFINITION elements under TYPE. You can place other elements inside of a TYPE tag. For example some of the NeL tools will add a COMMENT and a LOG entry into the block to track changes on the type. Most of the properties of a type are self explanatory, but the DEFINITION elements need some extra commentary. DEFINITION's have a Label and a Value. The Label is the information used in form comparison and validation. The Value is what is returned when an atom is requested via the API for an atom matching that label. This is a way to create a list of valid values that don't adhere to minimum and maximum, for example the string-based boolean type above.

All types can contain the following properties:

• Type - the type of data this will return. All Types must have this property.
• UI - the type of UI widget an editor would use to edit this. All Types must have this property.
• Default - the default value of elements/atoms using this data type.
• Min - For numeric types the minimum valid value.
• Max - For numeric types the maximum valid value.
• Increment - For numeric values the increments the UI should edit in.

There are a number of valid data types a type can provide, they are:

• UnsignedInt
• SignedInt
• Double
• String
• Color

The following are valid UI types that Georges will parse and provide to any tools using it:

• Edit
• EditSpin
• NonEditableCombo
• FileBrowser
• BigEdit
• ColorEdit

Definitions

Definitions define the types of arrays, structures and atoms available in the form and the type of information they provide. All forms require one definition file at the minimum, the first definition file is loaded based upon the form's file extension. For example if your form file is named sample.config_data Georges will load the config_data.dfn definition file (and any definitions or types it utilizes) before it processes the form data. Definition files have two main elements:

• ELEMENT - this describes the data contained in an array, struct, or atom.
• PARENT - this forces a definition to load other definitions as dependencies. This is handy of you have some often-used atoms.

Lets see an example definition file:
sample_config.dfn

Elements are the most complicated piece of a definition as they have a number of variations depending on the type of data their atoms, etc. would contain. Elements with a type of "Type" will represent atoms that contain values. Elements require a Name, Type, and Filename property at the minimum. Elements represent three different types of data types: structs, arrays, and atoms. If you look ahead to forms, these different types of data may make more sense. In the XML above we see five examples of atom types, two arrays, and a structure. You can see the full set of these files in the sample in Subversion, it may help make some of this make more sense.

Elements For Atoms

When setting up an element for an atom you need to name it (this is the name will be the one that the atom will use in the form,) you need to define it's type as "Type," and you need to define a filename. The filename defines the type of data (see types, above) any atoms using this element will use. Elements for atoms are the most simple types of element definitions. Note that you can also set the default value for an atom here, overriding the default value defined by the type.

Elements For Structs

In our example we also define a named struct. In the form when you do <STRUCT Name="PositionData"> you need to create an element to describe this struct. Here is the example from above:

The name of the element must match the name of the struct in the form. You will set your type to Dfn and define the definition file's filename in the Filename property. This tells Georges this is a struct and how to determine what atoms are members of it. Finally you need to flag Array="false" so that Georges knows it's just one struct, not a named array of them.

Elements for Virtual Structs

Georges elements also support a method of defining virtual structures. In this instance your definition allows the form designer to choose the definition for the specific entry. The element type for this is DfnPointer and is used with the VSTRUCT form item. A virtual struct can, like a normal struct, be declared as an array as well. Here's an example:

You may have noticed that we skipped specifying the filename for the definition - this is because we will specify the definition file name in the form itself! This allows us to have entries in which the data is less loosely bound but requires more effort on the part of the programmer writing the form loader.

Elements For Arrays

There are two types of arrays to consider when creating an element entry for an array.

Arrays of Atoms

Arrays of atoms are the easiest types of array to define. Take the following snippet from the definition example above:

In essence what we've done was created an element definition for an atom and used the Array="true" property to tell Georges to expect a list of strings. You'll see an example of defining this array in the section on creating and editing forms.

Arrays of Structs

An array of structs is slightly more complicated, but still fairly simple in syntax. Instead of defining the type as "Type" you would define it as "Dfn." This tells Georges that it should expect a list of structures containing the elements listed in the Filename. Look at the following snippet from the XML above:

We've informed Georges here that an array named "CoolFilesInfo" is going to contain a list of structs with the info contained in the definition file "coolfilesinfo.dfn." The sample for Georges in Subversion contains that file and some code on accessing it through the Georges API.

Element Information

Elements contain the following properties.

• Name - the name of the element as it will be used in the form.
• Filename - the filename of the file to be loaded to define this element's type.
• FilenameExt - this is a file mask that is used most commonly when the UI type of the specified type is "FileBrowser." When the tool opens a file browser it should use this file mask for selection.
• Type - specifies the type of dependency this element has for its type.
• Default - specifies the default value for this element.
• Array - this is a true/false string. This says that the form is going to specify multiple items of this element type in an array.

Valid Element Type values are:

• Type - if this value is specified the definition loads the TYPE specified in the Filename property of the Element.
• Dfn - for structs and arrays, defining the atoms they will contain.
• DfnPointer - the DfnPointer element type is used in relationship to VSTRUCT entries.

Forms

Using the information from above about types and definitions you can finally get down to working on creating your form. Below is a trimmed down version of the form used in the Georges sample from Subversion. Notice that the file extension of the form below is the filename of the extension list above in the definitions section.

default.sample_config

Looking at the example form above you should start to see how forms reference elements in definitions. Some important things to note about forms. The first element must always be either a STRUCT or a PARENT and parents should come before STRUCTs. You should never put any other element type directly under a FORM. In the code example you will see that you can access the ATOMs in the first (default) struct by referring to them in dot notation such as ''".TestVal1"'' - the dot denotes they're a member of the root structure.

Atoms

ATOMs represent individual data types, they define values you will use in your application. Atoms all have a Name property and a Value property. A name for an atom may be left out only if it is a member of a named array. Look at the ''TestArray'' example - the atoms contained in that array inherit their name (and thus element info) from the array.

Arrays

Arrays in a form are simple. All they require is a name. This name allows Georges to find the element type from the definition and thus atom or definition information required to describe the items in the array. In the example above I provided two types of arrays. An array of structures and an array of atoms. Building these in the form is as straight forward as the example. Building the definitions is explained above in the definitions section. Arrays can be accessed programmatically by their name as well using bracket and bracket/dot notation. In the case of ''TestArray'' you can access the data using ''TestArray'' and in the case of ''CoolFilesInfo'' you could access it's member data as ''CoolFilesInfo.Ranking'' - examples of processes like this are given in the code sample.

Structs

Structs are used in a number of ways already described in detail. But they can also be used in a standalone type to be accessed in dot notation as described in the arrays section. You can see this in the ''PositionData'' example in the form. Members of this structure can programmatically be accessed via the root node using the name ''PositionData.x'' or any of the other members.

Virtual Structs

Virtual structs are essentially the same as structs except that instead of their structure being defined by the overall hierarchy of form definitions you may define their structure using the DfnName property within the form itself. This gives you a certain amount of flexibility in allowing form designers to enter abstract information. In the example above I provided a hypothetical property element in which the property could be say a string or maybe it's represented by a whole different data structure and you want to allow for the different data atoms to be defined.

The flip side to this flexibility is that it now requires the form loader to check the type of the structure by definition file name and then generate the appropriate class or load data in the appropriate order.

Form Information

Forms contain the following elements:

• ATOM
• STRUCT
• VSTRUCT
• ARRAY
• And PARENT

Georges Loading

dd

Sheets

dd

Modules: NLLIGO

NeL Network Services (NeLNS)

summary of nelns

Overview

dd

Login Service (LS)

dd

Welcome Service (WS)

dd

Naming Service (NS)

dd

Frontend Service (FS)

dd 

Admin Services

Admin Service (AS)

Admin Executor Service (AES)

Admin Web Page

Tools

3DSMAX Exporter

Summary

dd

Getting Started

dd

Exporting Information

ddd

Exporting Mesh

ddd

Exporting Skeletons

ddd

Exporting Skeleton Weights

ddd

Exporting Instance Groups

ddd

Exporting Model Animations

ddd

Exporting Scene Animations

ddd

Exporting Collision Surfaces

ddd

Creating Accelerators

Summary

ddd

Creating A Cluster

ddd

Creating A Portal

ddd

Creating Landscape

Summary

ddd

Create Landscape

ddd

Paint Landscape

ddd

Creating Vegetation

ddd

Creating Water Surfaces

The NeL 3D framework has a concept of water pools and wave management. The basis of this is creating a water surface in 3DSMAX and exporting it appropriately. Water display in NeL works best on G eForce 3 and newer video cards. On older cards it will default to a more crude rendering of water. Creating water is accomplished through three primary steps within the 3DSMAX plugin:

1. Create a convex polygon to represent the water.
2. Apply a NeL Material to the shape.
3. Configure the Water Material.

Create The Water Surface

The first step is to create the water surface. In 3DSMAX you will need to create a new convex polygon for each body of water you will be presenting. This object must have a z-value of 0.

TODO what type of surfaces (in MAX) can be used? quad patch?

Applying the Water Material

Next you will have to all a NeL Material to the water shape. Once this is applied navigate to the NeL Texture section. The texture slots are allocated in this way for NeL water:

• Texture 1 - An environment map that is reflected by the water. This is obtained by rendering the top hemisphere of the skydome.
• Texture 2 - Used to set an alternate texture for the environment map.
• Texture 3 - An environment map that is reflected by the water when viewed from underwater. This is essentially the same as Texture 1.
• Texture 4 - Used to set an alternate texture for the environment map when viewed from underwater. This is essentially the same as Texture 2.
• Texture 5 - A heightmap used to deform the environment map of the water. The bumpmap is build from this heightmap.
• Texture 6 - A heightmap used to deform the bumpmap generated by Texture 5 for water animation.
• Texture 7 - The alph and colors used to modulate the final color and alpha of the water.

It is possible then to blend between the two sets of maps for night and day transitiion. You can use the Object Viewer to view the shape using the night/day dialog box.

Configuring the Water Material

Under the NeL Material section there is a sub-section for Water. Here is where you configure various information about the water material.

TODO describe this better.

Tile Editor

Object Viewer

Media Pipeline