Demystifying vertex colors in 3ds Max

Written by Jesper Tingvall, Product Expert, Simplygon

Disclaimer: The code in this post is written using version 10.2.10100.0 of Simplygon and 3ds Max 2021. If you encounter this post at a later stage, some of the API calls might have changed. However, the core concepts should still remain valid.

Introduction

In this blog post we are going to optimize a game character in 3ds Max. We are going to use two sets of vertex colors; one which the game utilized and one which control the reduction.

Male human game character.

Prerequisites

This example will use the Simplygon integration in 3ds Max, but the same concepts can be applied to all other integrations of the Simplygon API. However you will probably not encounter the same 3ds Max specific problems.

Problem to solve

We have a game character which we want to optimize. In our game we use vertex colors to differentiate clothes from the character's skin. The optimized mesh must keep these colors.

Male human game character where skin is black and colors are white.

If we do a 25% reduction via the user interface in 3ds Max we get this LOD. It is quite a heavy reduction so not intended to be viewed up this close. However, just to be safe we want to allocate some more vertices to the face.

Original mesh and bad looking LOD.

Solution

The solution is to add a secondary vertex color set to control the reduction.

Vertex paint face

Humans pay close attention to faces, so we want to keep more vertices in that area. To achieve this we are going to use map channel three to weight paint the face. First every vertex is colored grey, then we paint a white T-shape on the face. This is the area where humans are looking most at.

Grey game character with a white T painted on his face.

After painting these are all map channels in our mesh. Zero is first vertex color set which we use to differentiate between skin and clothes. Three is vertex color set we will use to control reduction.

Map channel info dialog in 3ds Max.

Preserve more then one vertex color set in 3ds Max

If we process the character with a standard 25% reduction pipeline in UI after painting we get the following output. Something is definitely wrong.

Yellow game character.

What has happened is that in 3ds Max every channel >0 is counted as an UV channel. Our color has thus lost its blue value since only first 2 values are kept; U and V coordinates, corresponding to red and green values.

We need to use sgsdk_SetIsVertexColorChannel to indicate that map channel 3 should be used as vertex color channel and not another UV channel. To use this function we need to perform the optimization via scripting. sgsdk_SetIsVertexColorChannel should be called before exporting from 3ds Max into Simplygon.

def main():
    # Initialize Simplygon
    sg = simplygon_loader.init_simplygon()

    # Set map channel 3 to vertex color type
    rt.sgsdk_SetIsVertexColorChannel(3, True)
    
    # Export scene to file
    bResult = rt.sgsdk_ExportToFile(export_path, False)

    # Reduce exported file
    run_pipeline(sg)
    
    # Importing
    bResult = rt.sgsdk_ImportFromFile(processed_path, True, True, True)
    print('Import result: ', bResult)
    
    # De-initialize Simplygon
    sg = None
    gc.collect()
    print('Finished')

With sgsdk_SetIsVertexColorChannel in place we can see that the third vertex color set is preserved.

LOD of game character with preserved grey and white vertex colors.

Use vertex color to control reduction

To tell Simplygon to use our newly preserved vertex color set to guide optimization we need to access the reduction pipeline's VertexWeightSettings. First we are going to enable using vertex colors to guide optimization by setting SetUserVertexWeightsInReducer to True. We also need to specify what vertex color set we are going to use. In 3ds Max we do not have names for color sets like in Maya. Instead we refer to them via their mapping channel ID. In our case we use map channel three as our vertex color set, thus we set SetWeightsFromColorName to 3.

def run_pipeline(sg):
    pipeline = sg.CreateReductionPipeline()
    
    # Set reduction ratio
    reduction_settings = pipeline.GetReductionSettings()
    reduction_settings.SetReductionTargetTriangleRatio(0.25)
    
    # Use vertex color 3 to control optimization
    weight_settings = pipeline.GetVertexWeightSettings()
    weight_settings.SetUseVertexWeightsInReducer(True)
    weight_settings.SetWeightsFromColorName("3")
    
    pipeline.RunSceneFromFile(export_path, processed_path, Simplygon.EPipelineRunMode_RunInThisProcess) 

After using vertex color set three as weights to prioritize the face we get a much more nice looking LOD.

Male human game character with OK looking LOD.

The observant documentation reader might notice that WeightsFromColorComponent's default value is red, and think that we should be able to use vertex color set 3 to guide optimization without specifying it is a color channel. This is however not the case as in Simplygon UVs and vertex color sets are stored separately. We can not use an UV set to control reduction.

Result

After weight painting the face, marking that map channel as a vertex color channel and using it to control reduction we get a much more nice looking LOD for 25% reduction.

Male human game character LOD.

If we look at our original vertex color set we can see that it is kept as well.

Male human game character LOD with black skin and white clothes.

Complete script

# Copyright (c) Microsoft Corporation. 
# Licensed under the MIT license. 

from pymxs import runtime as rt
from simplygon10 import simplygon_loader
from simplygon10 import Simplygon
import gc

export_path = 'C:/Temp/ExportedScene.sb'
processed_path = 'C:/Temp/ProcessedScene.sb'

def run_pipeline(sg):
    pipeline = sg.CreateReductionPipeline()
    
    # Set reduction ratio
    reduction_settings = pipeline.GetReductionSettings()
    reduction_settings.SetReductionTargetTriangleRatio(0.25)
    
    # Use vertex color 3 to control optimization
    weight_settings = pipeline.GetVertexWeightSettings()
    weight_settings.SetUseVertexWeightsInReducer(True)
    weight_settings.SetWeightsFromColorName("3")
    
    pipeline.RunSceneFromFile(export_path, processed_path, Simplygon.EPipelineRunMode_RunInThisProcess) 

def main():
    # Initialize Simplygon
    sg = simplygon_loader.init_simplygon()

    # Set map channel 3 to vertex color type
    rt.sgsdk_SetIsVertexColorChannel(3, True)
    
    # Export scene to file
    bResult = rt.sgsdk_ExportToFile(export_path, False)

    # Reduce exported file
    run_pipeline(sg)
    
    # Importing
    bResult = rt.sgsdk_ImportFromFile(processed_path, True, True, True)
    print('Import result: ', bResult)
    
    # De-initialize Simplygon
    sg = None
    gc.collect()
    print('Finished')


if __name__== "__main__":
    main()
⇐ Back to all posts

Request 30-days free evaluation license

*
*
*
*
Industry
*

Request 30-days free evaluation license

*
*
*
*
Industry
*