Written by Jesper Tingvall, Product Expert, Simplygon

Disclaimer: The code in this post is written using version 10.2.10100.0 of Simplygon and Blender 3.6. If you encounter this post at a later stage, some of the API calls might have changed. However, the core concepts should still remain valid.

Introduction

In this blog we'll showcase how to use a mapping image with multiple output materials. This enables you to perform material merging on an asset with both opaque and transparent materials.

Prerequisites

This example will use the Simplygon integration in Blender, but the same concepts can be applied to all other integrations of the Simplygon API. If you run it in other integrations you need to change the material casting settings as described in this blog.

Problem to solve

We have an asset and want to reduce draw calls by merging materials. The asset contains both transparent and opaque materials. Since it costs more to render with a transparent material than an opaque one we want multiple output materials; one for all opaque materials and one for all transparent materials. Other assets could have other reasons for multiple output materials, like certain parts of the asset rendered using specific shaders.

The assets we are going to use in this blog is Coffee Cart 01 and Alarm Clock 01. The assets have been modified to suit the topic of the blog better. We have 6 different materials, 2 transparent and 4 opaque. Transparent parts are on the coffe jugs and alarm clock's glass cover. The materials have 4 channels: diffuse, normals, roughness and metalness.

Solution

The solution is to use an aggregator processor with material merging using material casters. What we are going to do different from ordinary aggregation is that our mapping image will contain multiple output materials.

Aggregation with multiple output materials

The first step is to define how we want to split up the materials, define which materials that should be merged together. First we define an Enum containing the different material types we want to aggregate. If we wanted to have more output materials we would expand this class.

# The different kind of material channels we are going to bake our asset into.
class MaterialTypes(Enum):
    OPAQUE = 0
    TRANSPARENT = 1

After that we add a function that can tell which MaterialTypes our input material belongs to. If we wanted to expand to more output materials we would add more logic to this function. Right now it uses IsTransparent to deduce material type. IsTransparent will be True for transparent or cutout materials.

def get_material_type(scene: Simplygon.spScene, material_id: int) -> MaterialTypes:
    """Return which material type the material is."""
    material = scene.GetMaterialTable().GetMaterial(material_id)
    if material.IsTransparent():
        return MaterialTypes.TRANSPARENT
    else:
        return MaterialTypes.OPAQUE

We then introduce a function which returns a list of all MaterialTypes present in our input scene. We will use this to decide which materials to have in our output scene. The reason why we introduce a little bit of extra logic here and not just hard code (for example output material 0 is all opaque materials and 1 are all transparent materials) is that some assets does not have transparent materials, or perhaps no opaque materials. This way we avoid adding extra logic for handling those kinds of assets.

def get_material_types(scene : Simplygon.spScene) -> list[MaterialTypes]:
    """Returns a list of all material types in scene."""
    material_types = []
    for j in range(0, scene.GetMaterialTable().GetMaterialsCount()):
        material_type = get_material_type(scene, j)
        if not material_type in material_types:
            material_types.append(material_type)
    return material_types

Lastly this function lists finds which index a MaterialType will have in the output scene. Since we refer to materials by index this is needed.

def get_material_type_index(kind : MaterialTypes, material_kinds : list[MaterialTypes]) -> int:
    """Returs the index in material table of a specific material kind. Used to map MaterialTypes -> integers."""
    return material_kinds.index(kind)

With our helper functions in place we can create our AggregatorProcessor which will handle merging the geometries together and aggregating the UV coordinates. We specify that it should MergeGeometries. With that flag set all geometries are merged into one, enabling us to save draw calls.

def create_aggregator_processor(sg: Simplygon.ISimplygon, scene: Simplygon.spScene, material_types : list[MaterialTypes]):
    """Create aggregation processor with mapping image set to split materials into different material types."""

    # Create the aggregation processor. 
    sgAggregationProcessor = sg.CreateAggregationProcessor()
    sgAggregationProcessor.SetScene( scene )
    sgAggregationSettings = sgAggregationProcessor.GetAggregationSettings()
    mapping_image_settings = sgAggregationProcessor.GetMappingImageSettings()
    
    # Merge all geometries into a single geometry. 
    sgAggregationSettings.SetMergeGeometries( True )

Since we want to merge the materials we need to generate mapping images. A mapping image is used to transfer data from the original geometry to optimized geometry. It also handles generating the new set of UV coordinates. To generate the new set of UV coordinates we use chart aggregator which will use the original UV from each model in the asset and layout them on a new UV chart.

    # Generates a mapping image which is used after the aggregation to cast new materials to the new 
    # aggregated object. 
    mapping_image_settings.SetGenerateMappingImage( True )
    mapping_image_settings.SetApplyNewMaterialIds( True )
    mapping_image_settings.SetGenerateTangents( True )
    mapping_image_settings.SetUseFullRetexturing( True )

    mapping_image_settings.SetTexCoordGeneratorType( Simplygon.ETexcoordGeneratorType_ChartAggregator )
    sgChartAggregatorSettings = mapping_image_settings.GetChartAggregatorSettings()
    sgChartAggregatorSettings.SetChartAggregatorMode( Simplygon.EChartAggregatorMode_SurfaceArea )
    sgChartAggregatorSettings.SetSeparateOverlappingCharts( False )

    # Set input material count to number of materials in input scene.
    material_count = scene.GetMaterialTable().GetMaterialsCount()
    mapping_image_settings.SetInputMaterialCount( material_count )

In most cases we only have one mapping image but by setting SetOutputMaterialCount we can create several mapping images, one for each output material. Which output material an input material should belong to can be specified with SetMaterialMapping. We set this for every material present in the input scene. To figure out what output material ID it should have we use the index of that material type in our material_types list. This is done with get_material_type_index that we introduced above.

    # Set output material count to each material kind present in the input scene.
    mapping_image_settings.SetOutputMaterialCount( len(material_types))
 
    # Set material mapping where all materials of a certain kind maps to the same output material.
    for j in range(0, material_count):
        kind = get_material_type(scene,j )
        new_id = get_material_type_index(kind, material_types)
        mapping_image_settings.GetInputMaterialSettings(j).SetMaterialMapping(new_id)

If we run the aggregation processor with the specified mapping image setting we get a scene that has two output materials. Opaque output material is colored in gray and transparent material is colored in blue.

Material casting

Now it is time to transfer textures to our new materials. We start by defining a table of all material channels we want to bake as well as what color space they use. We'll do this as a list of tuples where first part is channel name and second part is if the texture is in sRGB color space. The name of material channels differ from integration to integration so if you are porting this code you need to change them.

# Name of all material channels we want to bake
MATERIAL_CHANNELS = [("Diffuse", Simplygon.EImageColorSpace_sRGB), ("Roughness", Simplygon.EImageColorSpace_Linear), ("Metalness", Simplygon.EImageColorSpace_Linear), ("Normals", Simplygon.EImageColorSpace_Linear)]

# Blender specific name for Normal channel.
NORMAL_CHANNEL = "Normals"

We start by creating some helper functions. Most of the material channels are transfered using color casters. Some of the channels use sRGB color space and some don't. If you notices that your colors are a bit off after optimization then incorrect sRGB flag is probably the reason. For transfering the normal channel we use a normal caster. We set the normal caster settings to correspont with how Blender uses normal maps. We specify what mapping image to use as well as input textures and materials. The function lastly returns file path to the newly baked texture.

def cast_channel(sg : Simplygon.ISimplygon, scene : Simplygon.spScene, mapping_image : Simplygon.spMappingImage, channel_name : str, sRGB : bool, output_file_name : str) -> str:
    """Cast material channel to texture file. Returns file name."""
    caster = None

    # Special case if channel is a normal channel. Then we use a normal caster.
    if channel_name == NORMAL_CHANNEL:
        caster = sg.CreateNormalCaster()
        caster_settings = caster.GetNormalCasterSettings()
        caster_settings.SetMaterialChannel(channel_name)
        caster_settings.SetOutputImageFileFormat( Simplygon.EImageOutputFormat.png)
        
        # Blender specific normal casting settings.
        caster_settings.SetGenerateTangentSpaceNormals(True)
        caster_settings.SetFlipGreen(False)
        caster_settings.SetCalculateBitangentPerFragment(True)
        caster_settings.SetNormalizeInterpolatedTangentSpace(False)
        # Normal maps are per default non-srgb.

    else:
        caster = sg.CreateColorCaster()
        caster_settings = caster.GetColorCasterSettings()
        caster_settings.SetMaterialChannel(channel_name)
        caster_settings.SetOutputImageFileFormat( Simplygon.EImageOutputFormat.png)
        caster_settings.SetOutputSRGB(sRGB)

    caster.SetMappingImage( mapping_image)
    caster.SetSourceMaterials( scene.GetMaterialTable() )
    caster.SetSourceTextures( scene.GetTextureTable() )
    caster.SetOutputFilePath(output_file_name)

    caster.RunProcessing()
    return caster.GetOutputFilePath()

Create output material

Now it is time to setup the output materials. As always we'll start by creating some helper functions. First we'll create a function which imports an image file and creates a texture we can add to our scene's texture table.

We specify the FilePath and ColorSpace for the texture. Name is also specified and is used by the material to refer to this texture. Thus it is important that it is unique. We use the file path as texture name.

def create_texture(sg : Simplygon.ISimplygon, file_path : str, color_space : int) -> Simplygon.spTexture:
    """Create a texture from file_path."""
    new_texture = sg.CreateTexture()
    new_texture.SetFilePath( file_path )
    new_texture.SetName(file_path)
    new_texture.SetColorSpace(color_space)
    return new_texture

To use the texture in a material we need to create a shading network which specifies how the material should be rendered. Our shading network consist of a single ShadingTextureNode. We specify which texture to sample with SetTextureName and which UV coordinate to use with SetTexCoordLevel.

def create_shading_network(sg : Simplygon.ISimplygon, texture_name : str) -> Simplygon.spShadingNode:
    """Create a simple shading network which displays a texture of name texture_name."""
    texture_node = sg.CreateShadingTextureNode()
    texture_node.SetTexCoordLevel( 0 )
    texture_node.SetTextureName( texture_name )
    return texture_node

We have two output materials, opaque and transparent. We create a helper function which setup material specific settings depending on what MaterialTypes it is for. The only setting we need to use is this example is setting the BlendMode.

def setup_output_material(material : Simplygon.spMaterial, material_type : MaterialTypes):
    """Set correct settings for output material depending on material type."""
    if material_type == MaterialTypes.TRANSPARENT:
        material.SetBlendMode(Simplygon.EMaterialBlendMode_Blend)

Once we have all helper functions we can do material casting for every MaterialType in our input scene. We start by create a material and adding it to the material table. Then we iterate through all MATERIAL_CHANNELS and cast it to a texture file. To get the mapping image corresponding to the given output material we use GetMappingImageForImageIndex with output material index. The difference between that one and the more common GetMappingImage is that GetMappingImage always returns the mapping image with index 0, while we can specify what index to fetch with GetMappingImageForImageIndex.

Once the texture is cast we create a texture from it, use that as input to out shading network and lastly setup if it should be opaque or transparent.

def cast_materials(sg : Simplygon.ISimplygon, scene : Simplygon.spScene, processor : Simplygon.spAggregationProcessor, material_types : list[MaterialTypes], material_table : Simplygon.spMaterialTable, texture_table : Simplygon.spTextureTable):
    """Cast all materials in scene to new texture and material tables."""

    for j in range(0, len(material_types)):
        print(f"Creating output material for {material_types[j]}...")

        # Add new material for each kind
        new_material = sg.CreateMaterial()
        new_material.SetName(f"{material_types[j]}")
        material_table.AddMaterial( new_material )

        for channel in MATERIAL_CHANNELS:
            # Cast texture to file for specific mapping image
            print(f"Casting {channel[0]}...")
            sRGB = channel[1] == Simplygon.EImageColorSpace_sRGB
            casted_texture_file = cast_channel(sg, scene, processor.GetMappingImageForImageIndex(j), channel[0], sRGB, f"{TMP_DIR}{channel[0]}_{j}" )
            print(f"Casted {casted_texture_file}")

            # Create a texture from newly casted texture file.
            texture_table.AddTexture(create_texture(sg, casted_texture_file, channel[1]) )
    
            # Create material from texture
            new_material.AddMaterialChannel( channel[0] )
            new_material.SetShadingNetwork( channel[0], create_shading_network(sg, casted_texture_file) )

            setup_output_material(new_material, material_types[j])

After casting materials we get diffuse, normal, roughness and metalness textures. Currently we bake both material types to 4k textures, which for the transparent material is a bit of a waste. An improvement would be to bake transparent textures to a more appropriate texture resolution.

Putting it all together

Now it is time to glue it all together to a function which optimizes our scene. First we analyze the material kinds in the input scene. These different material kinds are used in the aggregation processor to create two mapping images, one for opaque materials and one for transparent materials. After the aggregation has been done we cast all materials and create output materials. Lastly we assign the new textures and materials to our scene.

def optimize_scene(sg: Simplygon.ISimplygon, scene : Simplygon.spScene):
    """Optimize scene with aggregator."""

    material_types = get_material_types(scene)
    sgAggregationProcessor = create_aggregator_processor(sg, scene, material_types)
 
    # Start the aggregation process.     
    print("Running aggregator...")
    sgAggregationProcessor.RunProcessing()

    # Create temporary material and texture tables the output scene will use.
    sgMaterialTable = sg.CreateMaterialTable()
    sgTextureTable = sg.CreateTextureTable()

    # Cast all materials
    cast_materials(sg, scene, sgAggregationProcessor, material_types, sgMaterialTable, sgTextureTable)

    # We can not clear texture table above since we still are using it for material casting. Now when all materials are casted we can assign new material table and texture table to scene.
    scene.GetTextureTable().Clear()
    scene.GetMaterialTable().Clear()
    scene.GetTextureTable().Copy(sgTextureTable)
    scene.GetMaterialTable().Copy(sgMaterialTable)

Result

The result after optimization is an asset that has two draw calls; one for all opaque parts and one for all transparent parts. Since we have kept almost the original texture resolution we can not see any difference between the original asset and optimized asset. This makes our new tool suitable for optimizing even LOD0 assets. A good use case for it is optimizing static scenes where we have draw call issues, but texture memory to spare.

Lastly we can compare the optimized asset to the input scene in numbers. No matter how many input materials or textures we can merge them into only two draw calls.

Knowing how to split output material after aggregation is a useful thing to have in your toolbox as there always are certain assets where some materials need to have a custom shader.

Complete script

# Copyright (c) Microsoft Corporation. 
# Licensed under the MIT license. 

from enum import Enum
import bpy
import gc

from simplygon10 import simplygon_loader
from simplygon10 import Simplygon

# Temporary file names.
TMP_DIR = "C:/Tmp/"
IN_FILE = "input.glb"
OUTPUT_FILE = "output.glb"

# Texture size. Change parameters for quality
TEXTURE_SIZE = 4096

# Name of all material channels we want to bake
MATERIAL_CHANNELS = [("Diffuse", Simplygon.EImageColorSpace_sRGB), ("Roughness", Simplygon.EImageColorSpace_Linear), ("Metalness", Simplygon.EImageColorSpace_Linear), ("Normals", Simplygon.EImageColorSpace_Linear)]

# Blender specific name for Normal channel.
NORMAL_CHANNEL = "Normals"


# The different kind of material channels we are going to bake our asset into.
class MaterialTypes(Enum):
    OPAQUE = 0
    TRANSPARENT = 1


def export_selection(sg: Simplygon.ISimplygon, file_path: str) -> Simplygon.spScene:
    """Export the current selected objects into Simplygon."""
    bpy.ops.export_scene.gltf(filepath = file_path, use_selection=True)
    scene_importer = sg.CreateSceneImporter()
    scene_importer.SetImportFilePath(file_path)
    scene_importer.Run()
    scene = scene_importer.GetScene()
    return scene


def import_results(sg: Simplygon.ISimplygon, scene, file_path : str):
    """Import the Simplygon scene into Blender."""
    scene_exporter = sg.CreateSceneExporter()
    scene_exporter.SetExportFilePath(file_path)
    scene_exporter.SetScene(scene)
    scene_exporter.Run()
    bpy.ops.import_scene.gltf(filepath=file_path)
    

def setup_output_material(material : Simplygon.spMaterial, material_type : MaterialTypes):
    """Set correct settings for output material depending on material type."""
    if material_type == MaterialTypes.TRANSPARENT:
        material.SetBlendMode(Simplygon.EMaterialBlendMode_Blend)


def get_material_type(scene: Simplygon.spScene, material_id: int) -> MaterialTypes:
    """Return which material type the material is."""
    material = scene.GetMaterialTable().GetMaterial(material_id)
    if material.IsTransparent():
        return MaterialTypes.TRANSPARENT
    else:
        return MaterialTypes.OPAQUE
    

def get_material_types(scene : Simplygon.spScene) -> list[MaterialTypes]:
    """Returns a list of all material types in scene."""
    material_types = []
    for j in range(0, scene.GetMaterialTable().GetMaterialsCount()):
        material_type = get_material_type(scene, j)
        if not material_type in material_types:
            material_types.append(material_type)
    return material_types


def get_material_type_index(kind : MaterialTypes, material_kinds : list[MaterialTypes]) -> int:
    """Returs the index in material table of a specific material kind. Used to map MaterialTypes -> integers."""
    return material_kinds.index(kind)


def setup_output_material_settings(output_material : Simplygon.spMappingImageOutputMaterialSettings):
    """Set settings for output material"""
    output_material.SetTextureWidth( TEXTURE_SIZE )
    output_material.SetTextureHeight( TEXTURE_SIZE )


def cast_channel(sg : Simplygon.ISimplygon, scene : Simplygon.spScene, mapping_image : Simplygon.spMappingImage, channel_name : str, sRGB : bool, output_file_name : str) -> str:
    """Cast material channel to texture file. Returns file name."""
    caster = None

    # Special case if channel is a normal channel. Then we use a normal caster.
    if channel_name == NORMAL_CHANNEL:
        caster = sg.CreateNormalCaster()
        caster_settings = caster.GetNormalCasterSettings()
        caster_settings.SetMaterialChannel(channel_name)
        caster_settings.SetOutputImageFileFormat( Simplygon.EImageOutputFormat.png)
        
        # Blender specific normal casting settings.
        caster_settings.SetGenerateTangentSpaceNormals(True)
        caster_settings.SetFlipGreen(False)
        caster_settings.SetCalculateBitangentPerFragment(True)
        caster_settings.SetNormalizeInterpolatedTangentSpace(False)
        # Normal maps are per default non-srgb.

    else:
        caster = sg.CreateColorCaster()
        caster_settings = caster.GetColorCasterSettings()
        caster_settings.SetMaterialChannel(channel_name)
        caster_settings.SetOutputImageFileFormat( Simplygon.EImageOutputFormat.png)
        caster_settings.SetOutputSRGB(sRGB)

    caster.SetMappingImage( mapping_image)
    caster.SetSourceMaterials( scene.GetMaterialTable() )
    caster.SetSourceTextures( scene.GetTextureTable() )
    caster.SetOutputFilePath(output_file_name)

    caster.RunProcessing()
    return caster.GetOutputFilePath()


def create_texture(sg : Simplygon.ISimplygon, file_path : str, color_space : int) -> Simplygon.spTexture:
    """Create a texture from file_path."""
    new_texture = sg.CreateTexture()
    new_texture.SetFilePath( file_path )
    new_texture.SetName(file_path)
    new_texture.SetColorSpace(color_space)
    return new_texture


def create_shading_network(sg : Simplygon.ISimplygon, texture_name : str) -> Simplygon.spShadingNode:
    """Create a simple shading network which displays a texture of name texture_name."""
    texture_node = sg.CreateShadingTextureNode()
    texture_node.SetTexCoordLevel( 0 )
    texture_node.SetTextureName( texture_name )
    return texture_node


def create_aggregator_processor(sg: Simplygon.ISimplygon, scene: Simplygon.spScene, material_types : list[MaterialTypes]):
    """Create aggregation processor with mapping image set to split materials into different material types."""

    # Create the aggregation processor. 
    aggregation_processor = sg.CreateAggregationProcessor()
    aggregation_processor.SetScene( scene )
    aggregation_settings = aggregation_processor.GetAggregationSettings()
    mapping_image_settings = aggregation_processor.GetMappingImageSettings()
    
    # Merge all geometries into a single geometry. 
    aggregation_settings.SetMergeGeometries( True )
    
    # Generates a mapping image which is used after the aggregation to cast new materials to the new 
    # aggregated object. 
    mapping_image_settings.SetGenerateMappingImage( True )
    mapping_image_settings.SetApplyNewMaterialIds( True )
    mapping_image_settings.SetGenerateTangents( True )
    mapping_image_settings.SetUseFullRetexturing( True )

    mapping_image_settings.SetTexCoordGeneratorType( Simplygon.ETexcoordGeneratorType_ChartAggregator )
    chart_aggregator_settings = mapping_image_settings.GetChartAggregatorSettings()
    chart_aggregator_settings.SetChartAggregatorMode( Simplygon.EChartAggregatorMode_SurfaceArea )
    chart_aggregator_settings.SetSeparateOverlappingCharts( False )

    # Set input material count to number of materials in input scene.
    material_count = scene.GetMaterialTable().GetMaterialsCount()
    mapping_image_settings.SetInputMaterialCount( material_count )

    # Set output material count to each material kind present in the input scene.
    mapping_image_settings.SetOutputMaterialCount( len(material_types))
 
    # Set material mapping where all materials of a certain kind maps to the same output material.
    for j in range(0, material_count):
        kind = get_material_type(scene,j )
        new_id = get_material_type_index(kind, material_types)
        mapping_image_settings.GetInputMaterialSettings(j).SetMaterialMapping(new_id)
    
    # Set output material settings.
    for j in range(0, len(material_types)):
        setup_output_material_settings(mapping_image_settings.GetOutputMaterialSettings(j))

    return aggregation_processor


def cast_materials(sg : Simplygon.ISimplygon, scene : Simplygon.spScene, processor : Simplygon.spAggregationProcessor, material_types : list[MaterialTypes], material_table : Simplygon.spMaterialTable, texture_table : Simplygon.spTextureTable):
    """Cast all materials in scene to new texture and material tables."""

    for j in range(0, len(material_types)):
        print(f"Creating output material for {material_types[j]}...")

        # Add new material for each kind
        new_material = sg.CreateMaterial()
        new_material.SetName(f"{material_types[j]}")
        material_table.AddMaterial( new_material )

        for channel in MATERIAL_CHANNELS:
            # Cast texture to file for specific mapping image
            print(f"Casting {channel[0]}...")
            sRGB = channel[1] == Simplygon.EImageColorSpace_sRGB
            casted_texture_file = cast_channel(sg, scene, processor.GetMappingImageForImageIndex(j), channel[0], sRGB, f"{TMP_DIR}{channel[0]}_{j}" )
            print(f"Casted {casted_texture_file}")

            # Create a texture from newly casted texture file.
            texture_table.AddTexture(create_texture(sg, casted_texture_file, channel[1]) )
    
            # Create material from texture
            new_material.AddMaterialChannel( channel[0] )
            new_material.SetShadingNetwork( channel[0], create_shading_network(sg, casted_texture_file) )

            setup_output_material(new_material, material_types[j])


def optimize_scene(sg: Simplygon.ISimplygon, scene : Simplygon.spScene):
    """Optimize scene with aggregator."""

    material_types = get_material_types(scene)
    aggregation_processor = create_aggregator_processor(sg, scene, material_types)
 
    # Start the aggregation process.     
    print("Running aggregator...")
    aggregation_processor.RunProcessing()

    # Create temporary material and texture tables the output scene will use.
    new_material_table = sg.CreateMaterialTable()
    new_texture_table = sg.CreateTextureTable()

    # Cast all materials
    cast_materials(sg, scene, aggregation_processor, material_types, new_material_table, new_texture_table)

    # We can not clear texture table above since we still are using it for material casting. Now when all materials are casted we can assign new material table and texture table to scene.
    scene.GetTextureTable().Clear()
    scene.GetMaterialTable().Clear()
    scene.GetTextureTable().Copy(new_texture_table)
    scene.GetMaterialTable().Copy(new_material_table)


def process_selection(sg : Simplygon.ISimplygon):
    """Remove and bake decals on selected meshes."""

    # Export scene from Blender and import it 
    scene = export_selection(sg, TMP_DIR+IN_FILE)
    
    # Aggregate optimized scene
    optimize_scene(sg, scene)
    
    # Import result into Blender
    import_results(sg, scene, TMP_DIR+OUTPUT_FILE)


def main():
    sg = simplygon_loader.init_simplygon()

    process_selection(sg)
    sg = None
    gc.collect()


if __name__== "__main__":
    main()