Clarification of built-in GLX extension

[paulmelis]

I'm trying to understand the GLX extension. In the 2.2 beta1 release notes the following is stated:

The TurboVNC Server now includes a GLX extension that uses the swrast DRI driver installed on the system to perform unaccelerated OpenGL rendering. [...] Note that your system must have Mesa 8.x or later installed in order for the swrast DRI driver to be available. On systems that do not have vendor-specific GPU drivers installed, or on systems that provide a libglvnd-enabled build of Mesa, the TurboVNC Server's GLX extension will use direct rendering, which improves performance and compatibility.

The text I highlighted doesn't make sense to me. If you do NOT have vendor-specific GPU drivers installed (e.g. NVIDIA's binary blob) then the TurboVNC GLX will use direct rendering? How does the presence of e.g. NVIDIA drivers influence the software-based rendering GLX extension in TurboVNC?

[dcommander]

To explain that requires a brief explanation of the architecture of GLX. The GLX extension to X11 originally provided a way for applications to send OpenGL commands over the wire to be rendered on a remote X server, so there is both a client and a server component to GLX. The server component is part of the X server, and the client component is part of libGL, which the 3D application calls. The server and client components can potentially have completely different implementations of OpenGL. If the X server is on a different machine than the application, then "indirect rendering" must be used-- sending the OpenGL commands through the X11 wire protocol to the remote X server, and the pitfalls of that approach were the original reason why VirtualGL was created. If the X server is on the same machine as the application, then it is possible to use direct rendering-- basically bypassing the X server and sending the OpenGL commands directly to the driver, which renders (or instructs a GPU to render) the commands to the appropriate place on the screen. However, that requires that both the client and server OpenGL implementations be the same. When you're using the TurboVNC Server or a similar X proxy, the server OpenGL implementation is Mesa. Thus, you can only use direct rendering if one of the following applies:

1. The client OpenGL implementation is also Mesa. This will not be the case if you install the nVidia or AMD proprietary drivers (unless the system is GLVND-capable-- see below.)
2. You are using VirtualGL, which interposes the GLX API and thus redirects GLX and OpenGL commands to the 3D X server, which presumably has a GPU-accelerated server OpenGL implementation that matches the client OpenGL implementation.
3. Your system and drivers support GLVND, which allows multiple client OpenGL implementations on the same system and which allows the client OpenGL implementation to be chosen based on the server OpenGL implementation. On a GLVND-capable system with, say, the nVidia drivers, the nVidia client OpenGL implementation will be used when rendering to the 3D X server, but the Mesa client OpenGL implementation will be used when rendering to an X proxy (i.e. when running OpenGL applications in the X proxy without VirtualGL.)

On older systems, such as RHEL 6, that are not GLVND-capable, installing a proprietary OpenGL stack makes that proprietary stack the only available client OpenGL implementation. Thus, you can't use direct rendering with the TurboVNC Server's software OpenGL implementation in that case (although, if you're installing a proprietary OpenGL stack, that probably means you have a GPU, so you should really be using VirtualGL.) The workaround is to install the Mesa libGL implementation in a non-default directory and point LD_LIBRARY_PATH to that directory when using the TurboVNC Server's software OpenGL implementation. RHEL 7+ and other modern Linux distributions support GLVND, so this is a non-issue on such distributions.

[dcommander]

It also bears mentioning that I resisted adding a built-in/server OpenGL implementation to TurboVNC for many years, since TurboVNC was primarily used along with VirtualGL. I ultimately relented because of compositing window managers, such as GNOME 3, that require OpenGL. Even on systems that are configured to use VirtualGL, the TurboVNC Server's built-in OpenGL implementation may still be used for the window manager, in which case you could run into the aforementioned indirect rendering problem if the system isn't GLVND-capable. However, since very few supported Linux distributions remain that aren't GLVND-capable (public updates for RHEL 6 end later this year), the server OpenGL implementation in TurboVNC 3.0 will no longer support indirect OpenGL rendering by default.