Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
September 1, 2014
arrowPress Releases
September 1, 2014
PR Newswire
View All
View All     Submit Event





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
Building for Linux, the smart way
by Leszek Godlewski on 04/03/14 01:19:00 pm   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

 

The Linux land has a reputation, especially among developers used to Windows, of being – let's say – somewhat savage, uncivilized. We've all heard the ghost stories: things being downright broken, lack of documentation and general despair; people coming, exclaiming: "what the fuck?!" and going right back. 

Well, I've been exposed to the Unix philosophy, coding practices and general way of things for just under 10 years. This experience makes me comfortable with the platform, but I've still had a lot to learn in the past year in making the transition from Linux development as a hobby to making a living out of it. I'd like to share some of the less obvious tricks I've put up my sleeve – this time regarding the build system.

Unless mentioned otherwise, I will assume a 64-bit Debian system, "testing" branch (this is what SteamOS is based off of).

Multilib

This section may not apply to you if you don't meet these two conditions:

  • you still need to provide 32-bit binaries,
  • you are wary of the build environment provided by Steam Runtime (because e.g. you like alternating between gcc and Clang, or want to use gold for linking – more on that later).

If you're fine with 64-bit only or the SR toolchain, you may skip all the way to Clang.

If you, however, try building a 32-bit program on a 64-bit system (or the other way around) with the stock toolchain, you will most likely fail to build it, despite setting the correct architecture on it with a -m32, -m64 or a -march switch. Apparently, you are somehow missing the right libc and libgcc packages, even though they may seem to be installed.

The point of failure may be different, if any dependencies apart from the standard library are used in the program. That is usually a trivial issue, though, fixed by enabling multiarch and getting the :i386 package (for Debian, more on that here).

I used to maintain a schroot (kind of like a VM, but native – the kernel and some parts of the file system are shared with the host) with a 32-bit variant of the system just for the sake of building 32-bit binaries; however, there is a better way.

Enter gcc-multilibInstall this package and suddenly the 32/64-bit cross-compiles start working as expected. It's magic!

Clang

Okay, this one has been repeated ad nauseam recently, but it really makes a world of difference; also in terms of build times. A full rebuild of my current project takes:

  • with gcc: 3m47s
  • with Clang: 3m05s

You may say ~40 seconds is not a big deal; it is to me.

Not mentioning the not-so-obvious benefit of having another point of reference. And, given the fact that Clang aspires for full command line compatibility with gcc and usually builds everything that gcc can (it even defines the __GNU__ macro alongside __clang__, and supports the GNU extensions to the language), it really is a drop-in replacement.

It brings a lot of good and can be quickly switched back for gcc. What's not to like?

Linking

This build step is a major source of grief for anyone who's ever worked with a AAA codebase, as linking several hundred of object files and libraries can take ages. The Linux situation is made even worse by its aging linker, the GNU linker a.k.a. ld, which is single-threaded, requires objects and libraries to be specified in the order of dependency etc.

gold

The good news is that we are not doomed to use GNU ld. A few years back a small team at Google recognized the problem and decided to fix it, resulting in the development of GNU gold – a brand new linker built from scratch, whose most prominent feature is that it's multi-threaded.

Switching from ld to gold in most cases is a question of a command line switch for GCC/Clang, or at most making a symlink. Here's the command I use to link my current project's binary on a Debian system, stripped of all irrelevant options:

clang++ -B/usr/lib/gold-ld

For my current project, switching the linker from GNU ld to gold took the link time down from 18 to 5 seconds on my Xeon E3-1240 v3.

Library groups

Another thing the GNU ld is infamous for is the seemingly unreasonable requirement of specifying libraries in the order of reverse dependence (i.e. an object's or library's dependencies follow the object or library; there is a great explanation of that on Stack Overflow, if you're interested). This can get very annoying once you have to deal with circular dependencies, and while there is a way to do this "properly" – specify the library again – I'd much rather not have to think about it.

Turns out there are command-line switches that do just that (see the ld manual) – we can define a group of archives:

--start-group archives --end-group
 The archives should be a list of archive files. They may be either explicit file names, or -l options.

Or the short-hand version:

-( archives -)

Just dump your libs within the brackets and you're set.

There is a caveat, though: this causes the linker to perform an exhaustive search through all the libs in the group until all symbols are resolved. The manual warns of this having a possible performance impact. I haven't noticed any in my project, but just keep that in mind.

Debugigng symbols

Bruce Dawson has given an excellent talk about Linux debugging at Steam Developer Days 2014, so I'm not going to repeat here what he said. Instead I'll just reiterate the two main points about symbols:

  • use objcopy --only-keep-debug and --add-gnu-debuglink to split the debug symbols from the binary,
  • use the build IDs (GNU ld option --build-id, see manual) and a network storage to set up a symbol server.

There is one thing that Bruce has not mentioned in the talk, though, and which is – in my opinion – crucial to daily work. Update: He did suggest it to me in a tweet post-factum, though, so kudos to him anyway!

Caching the gdb-index

When working with a large codebase, such as a triple-A-grade game engine, you may notice that the debugging symbols may be... well, somewhat heavy:

$ ls -sh ADVGame-Linux32-Debug.dbg
244M ADVGame-Linux32-Debug.dbg
$ ls -sh SuperSecretProject.dbg
254M
SuperSecretProject.dbg

Ouch.

What's even worse is that those ~250 megs need to be loaded by the debugger and processed to build an internal data structure for quick symbol lookup; with the files above, this can take 10 to 30 seconds on my machine, and it happens every single time you load the binary in GDB.

There goes fast iteration time.

We can, however, cache this data structure offline instead and fold it into the build process! I think it's a fair tradeoff between build time and runtime, considering that the latter almost always implies the index generation cost anyway. The GDB manual describes the method well and concisely; here's what I'm using for my project (assuming SuperSecretProject is the game binary):

mkdir -p $(OUTPUT_PATH)/gdb-index
gdb -batch -ex "save gdb-index $(OUTPUT_PATH)/gdb-index" SuperSecretProject
objcopy --add-section .gdb_index=$(OUTPUT_PATH)/gdb-index/SuperSecretProject.gdb-index --set-section-flags .gdb_index=readonly SuperSecretProject SuperSecretProject

Some trivial variable substitution makes this look less like a magic incantation. I promise!

As David Konerding has pointed out in the comments, newer releases of the gold linker can build the gdb-index for you at link time, which should be faster and is obviously easier (no GDB batch mode commands hodgepodge!). It's now a question of appending --gdb-index to your linker command line, or -Wl,--gdb-index, if you're using the gcc or clang compiler driver to link.

Thanks, David!

Conclusions

Hopefully, at least some of these tricks have been new and useful to you. If that's the case - glad I could help! If not – why not share your tricks in the comments?

Also, I know what some developers are thinking: we shouldn't need to be doing all this, we shouldn't need to do so much research and documentation crunching just to do such basic things. And you're probably right. I'd like to explore this non-orthogonality between the Unix philosophy and the game developer mindset in a future blog post.


Related Jobs

Playtika Santa Monica
Playtika Santa Monica — Santa Monica, California, United States
[09.01.14]

Sr. BI Developer
Wargaming.net
Wargaming.net — Hunt Valley, Maryland, United States
[09.01.14]

Engineering Manager
Wargaming.net
Wargaming.net — Chicago, Illinois, United States
[09.01.14]

Engineering Manager
Wargaming.net
Wargaming.net — Hunt Valley, Maryland, United States
[09.01.14]

Graphics Software Engineer






Comments


D Scott Nettleton
profile image
I've been compiling on Linux for years, and I still found this helpful. Thanks for sharing!

Leszek Godlewski
profile image
Cool, glad you found it useful!

Matías Goldberg
profile image
Yeah, this is useful! Thanks!

Travis Jones
profile image
Neat post. The debugging symbol magic is certainly new to me. Looking forward to further posts on this topic in the future.

Leszek Godlewski
profile image
Then I highly recommend Bruce Dawson's talk about this (linked in the text). He's covered it really in-depth, just left out this tiny bit about gdb-index.

Funnily enough, he suggested me to try gdb-index on Twitter in the first place, so I shoud probably credit him for that, too. :)

Paco Barter
profile image
Very useful post. Thanks for sharing!

David Konerding
profile image
Leseck,

THanks for the recent post on porting games to linux. I wanted to mention that the Gold linker already has support for writing the GDB index directly so you don't need to invoke GDB during the build process.

http://gcc.gnu.org/wiki/DebugGNUIndexSection

Leszek Godlewski
profile image
You're right, I did not know that! Thanks! I will update the relevant part of the post. :)

Bram Stolk
profile image
Working around the circular dependencies with library groups is, in my opinion, not addressing the problem itself.
It pays to eliminate all circular dependencies.
Doing so will also clean up the architecture, and include trees, causing much faster builds.

One way to avoid a lower level module depending on a high level module is by the use of callbacks.


none
 
Comment: