O3DE In Windows Subsystem for Linux

  • Thu Jan 27 2022 05:14:37 GMT+0000 (Coordinated Universal Time)
  • Lua Pong

Playing in a new engine

O3DE is the new (ly rebranded) game engine released into open source by Amazon. You can read more about it here These articles will try not to cover the exact material already available in the documentation, but hopefully will supplement them!

As a relatively new engine, the documentation is a little sparse and so it's been a fun exercise to explore and play with this engine with minimal guidance. There are a few resources that are available, but it's still very much a work in progress.

Working the past few days in O3DE has reminded me a lot of the first few weeks at a new company, where you are tasked with getting up to speed on a large piece of software with a lot of internal history but not a lot of commercial grade documentation available.

First Step

Getting started on any large piece of software like this can be daunting, but generally speaking your best first step is to take advantage of whatever onboarding documentation you are provided. In this case we have two official resources:

The official documentation provides 2 methods of installation for 2 different platforms. You can install either from source by cloning the repository, or you can download and official stable release. Right now they primarily support Windows and Linux although Mac support seems just around the corner.

Interestingly for me, as of the time of this writing, there is only one official release (21.11 or 2111.1 depending on where it's written) which you can get here. Unfortunately, there is a defect in this build that prevents the editor from launching to me. The defect was subsequently fixed here and merged into master but a new release has not been cut - which means...

Building from Source

Surprisingly the documentation for this was fairly straight forward, however, I did run into a little bit of confusion because I code almost exclusively for windows in WSL, ie: my command line (bash) and editor (vim), and tools (grep, etc) are all done in the linux subsystem. Fortunately since WSL is so cool I can easily call out to a standard cmd.exe when I need to do window-y specific things (like trigger builds from MSVC or launch a gui application that I don't want to run like poop in my x11 emulator).

Of course, this does mean that I can't just copy and paste the instructions from the website and I need to have a very small amount of additional information to understand which pieces need to be run in windows and which I can run from WSL directly.

Building for Windows (WSL) Instructions

Here are the official steps offered for setting up O3DE from source (or github).

For the most part, I follow the instructions for build for linux but there are some changes that I needed to make.

Step 0 - Installing Dependencies / Prerequisites

Make sure you follow the steps described in BOTH the linux and windows sections.

This means having CMake installed for both systems

  • on linux via apt and

  • on windows either through an installer or (my perferred method) through something like chocolatey.

(The error appeared to be a problem where some portion of the toolchain was looking specifically for 2019 even when my CMake was configured to generate for 2022. My thought at the time was something was accidentally hardcoded but since I didn't investigate at that moment, I am not sure.)

You will also need Microsoft Visual Studio installed (Community is fine) but as of writing Visual Studio 2022 did not work for me (although I need to try again after resolving some other issues). I did install 2019 and use that instead (as we'll see when we generate projects) and this did work for me.

You can also install the linux toolchain (if you don't already), it won't get in the way of anything.

Step 1 - Create a package directory

Create a package directory in a writeable location. The following examples use the directory $HOME/o3de-packages.

I had some strange behavior when I put my packages directory into the WSL filesystem. Basically, what I think is happening is that the way the O3DE gems system works is it pulls packages down and as part of the configuration step prebuilds these dependencies outside of your project, then by seting cmake library / include path magic, points each target at the prebuild modules living in the packages.

Unfortunately, although WSL filesystem lives within windows the default location uses a network drive \\wsl$\Ubuntu-20.04 which confuses / causes problems with various aspects of the tooling.

So for this step, make sure you put your packages directory in the main windows filesystem ie: /mnt/c/o3de/packages will put this on the equivalent C:\o3de\packages folder. Also note: WSL is case sensitive, windows is not.

Step 2 - Get the python runtime

Here I ran the instructions as described with no issue.

python/get_python.sh

Step 3 - Use CMake to Generate and Build

Rather than the cmake command they use, we want to trigger cmake in windows to generate windows projects. Instead, I use this command:

cmd.exe /c cmake -S . -G "Visual Studio 17" -B$BUILD_DIR -DLY_3RDPARTY_PATH=${PATH_TO_PACKES_DIR}|| exit cmd.exe /c cmake --build $BUILD_DIR --target Editor --config ${CONFIG} -- -m || exit

Where you need to set:

PATH_TO_PACKAGES_DIR=C:\O3DE\o3de-packages BUILD_DIR=build/windows CONFIG=profile

To the appropriate values for your build. You can parametarize the config value (if you put this in a shell script like I did) and take notice that since cmake will be running in windows that we put the windows style path as the path to the packages. Additionally note that we need to escape the window style path delimeter \\.

The first line generates a Visual Studio project build. You can try Visual Studio 18 as your generator, its possible that the defect I encountered was due to other issues that have been resolved. I'll attempt this later.

The second line builds the Editor project, which we can then access in:

${REPO_DIR}/build/windows/bin/profile/Editor.exe

If called from WSL it will automatically forward the request to windows and properly open the window.

Step 4 - Register the Engine

Similar to cmake this command needs to be run on windows. You have to be careful here and understand how O3DE coordinates multiple installations on the same machine. All packages and engine references are coordinated through a manifest file held in:

  • Linux: ${HOME}/.o3de/o3de_manifest.json
  • Windows: C:\Users\{Your User Name}\.o3de\o3de_manifest.json

It sounds obvious, but things like registering engines, gems, and projects needs to be called with the appropriate o3de.py script (the windows version for windows, and the linux version for linux) otherwise you'll be registering things for the other platforms. I made the mistake of trying to get both linux and windows working at the same time, and getting both of these manfiest files thoroughly mixed up before realizing my error.

To run the registration command in linux (wsl) use the provided command from the repo directory:

scripts/o3de.sh register --this-engine

However to run this to update the windows manifest make sure you call out to cmd.exe so the python script properly picks up the correct platform. I have an alias to a script that does this:

# in a file called wo3de.sh, placed in my path
CMD=wslpath -w ~/development/o3de/engine/scripts/o3de.bat
cmd.exe /c $CMD "$@"

Notice how i use wslpath -w to convert my wslpath into the appropriate windows path. Then I tell cmd.exe to execute the o3de.bat batch file which just calls into the python script for us forwarding all the arguments.

So, when placed in my path, I can call:

wo3de.sh register --this-engine

Now I can use scripts/o3de.sh to call WSL commands, and wo3de.sh to call into windows.

In the next article we will look at Creating a Start Menu