Written by Jesper Tingvall, Product Expert, Simplygon

Disclaimer: This post is written using version 10.0.4600.0 of Simplygon and Blender 2.93. If you encounter this post at a later stage, some of the API calls might have changed. However, the core concepts should still remain valid.

Introduction

In this spooky blog post we will optimize away spider web decals from a scene. While it might seem unfitting given the release date of this post it is a well-known fact that most of the spider webs being displayed are not real and just for decoration.

Prerequisites

This example will use the Python Simplygon API in Blender, but the same concepts can be applied to all other integrations of the Simplygon API.

Problem to solve

We want to bake transparent decals into surface below. This benefit of this is to save draw calls and not use transparent shaders. We do not want to change the geometry so using remeshing is not an option.

Solution

To bake decals into underlying geometry we will do as follow. First we will remove decal geometry by traversing our scene and removing geometry nodes. Once that is done we will generate unique UVs coordinates in our scene using an aggregation processor. After that we will generate a mapping image from original scene via surface mapper. Lastly we will cast the color channels from the original scene to our optimized one.

Instead of using pipelines which does lots of things behind the scenes we are going to use processors and handle the material casting by ourselves.

Export scene from Blender

First we export the selected parts as a gltf file from Blender using bpy.ops.export_scene.gltf. We can then use a Scene Importer to import it as a Scene.

def export_selection(sg, file_path):
    """Export the current selected objects into Simplygon."""
    bpy.ops.export_scene.gltf(filepath = file_path, use_selection=True)
    sceneImporter = sg.CreateSceneImporter()
    sceneImporter.SetImportFilePath(file_path)
    sceneImporter.Run()
    scene = sceneImporter.GetScene()
    return scene

Detect and remove decals

We will regard a mesh to be a decal if every material it contains is transparent. First we'll introduce a helper function which checks if a material is transparent via IsTransparent.

def is_material_transparent(scene, material_id):
    """Returns if material is transparent."""
    material = scene.GetMaterialTable().GetMaterial(material_id)
    return material.IsTransparent()

Using that function we can determine if a mesh is a decal using criteria we specified above; every material is transparent.

def is_mesh_decal(scene, scene_mesh):
    """If all materials on our mesh is transparent we consider mesh to be a decal."""
    materials =  scene_mesh.GetGeometry().GetMaterialIds()
    for j in range(0, materials.GetItemCount()):
        if not is_material_transparent(scene, materials.GetItem(j)):
            return False
    return True

We then go through the scene recursively and adds any mesh considered being a decal to a specified decal selection set.

def select_decals(scene, scene_node, decal_set):
    """Add all child SceneMesh nodes considered to be decals to selection set."""
    for i in range(0, scene_node.GetChildCount()):
        child = scene_node.GetChild(i)
        if child.IsA("ISceneMesh"):
            scene_mesh = Simplygon.spSceneMesh.SafeCast(child)
            if is_mesh_decal(scene, scene_mesh):
                decal_set.AddItem(child.GetNodeGUID())    
                
        select_decals(scene, child,  decal_set)

When helper functions are created we can remove all decals from the scene. First we create a selection set and fill it with all decals in the scene. We can then remove them via RemoveSceneNodesInSelectionSet.

    # Create a selection set containing all decals
    decal_set = sg.CreateSelectionSet()
    decal_set_id = optimized_scene.GetSelectionSetTable().AddSelectionSet(decal_set)
    select_decals(optimized_scene, optimized_scene.GetRootNode(), decal_set)

    # Remove decals from optimized scene
    optimized_scene.RemoveSceneNodesInSelectionSet(decal_set_id)

Here is our scene with decals removed.

Generate unique UVs

For being able to bake decals into geometry below our entire scene needs unique UV coordinates. To generate this we will use an aggregation processor. First we set SetMergeGeometries to False which will keep our scene as individual meshes.

To generate unique UV coordinates we ask the processor to generate a mapping image with SetTexCoordGeneratorType set to Simplygon.ETexcoordGeneratorType_Parameterizer. Another way of doing this with ETexcoordGeneratorType_ChartAggregator would be to set SeparateOverlappingCharts set to True.

def aggregate_scene(sg, scene, texture_size):
    """Aggregate all meshes in scene to create unique UVs."""
    # Create the aggregation processor. 
    aggregation_processor = sg.CreateAggregationProcessor()
    aggregation_processor.SetScene( scene )
    aggregation_settings = aggregation_processor.GetAggregationSettings()
    
    # Do not merge into one mesh
    aggregation_settings.SetMergeGeometries( False )

    # Use mapping image to generate unique UVs.
    mapping_image_settings = aggregation_processor.GetMappingImageSettings()
    mapping_image_settings.SetGenerateMappingImage( True )
    mapping_image_settings.SetApplyNewMaterialIds( True )
    mapping_image_settings.SetGenerateTangents( True )
    mapping_image_settings.SetUseFullRetexturing( True )
    mapping_image_settings.SetTexCoordGeneratorType( Simplygon.ETexcoordGeneratorType_Parameterizer )
    
    # Set output texture size
    output_material_settings = mapping_image_settings.GetOutputMaterialSettings(0)
    output_material_settings.SetTextureWidth( texture_size )
    output_material_settings.SetTextureHeight( texture_size )
    
    aggregation_processor.RunProcessing()

Mapping image from original scene to optimized scene

To transfer the decals from our original scene to our aggregated one we are going to use a Surface Mapper. A Surface Mapper creates a mapping image from one scene (or geometry) to another. A mapping image describes how to project surface from one model to another one. It contains multiple layers, so not just the first hit is included. That means transparent and thin objects that is close to surfaces.

Depending on how far away the decals are we might need to use SetSearchOffset and SetSearchDistance to include them in the mapping image for underlying geometry.

def create_mapping_image_between_scenes(sg, original_scene, optimized_scene, texture_size):
    """Use surface mapper to generate mapping image from original_scene to optimized_scene."""

    surface_mapper = sg.CreateSurfaceMapper()
    surface_mapper.SetSourceScene(original_scene)
    surface_mapper.SetDestinationScene(optimized_scene)

    surface_mapper.SetSearchOffset(1)
    surface_mapper.SetSearchDistance(2)

    mapping_image_settings = surface_mapper.GetMappingImageSettings()
    output_material_settings = mapping_image_settings.GetOutputMaterialSettings(0)
    output_material_settings.SetTextureWidth( texture_size )
    output_material_settings.SetTextureHeight( texture_size )
    
    surface_mapper.RunSurfaceMapping()
    return surface_mapper.GetMappingImage()

Casting materials

We are now going to do material casting manually using the mapping image we created above.

First we create a Color Caster and set SetOutputFilePath. We also specify what channel it should cast via SetMaterialChannel. This name is different depending on what integration you are using, so in case it is not Blender it has another name.

Lastly we ask the color channel to run casting by calling RunProcessing.

def perform_color_casting(sg, scene, mapping_image, material_channel, temporary_image_path ):
    """Setup and run a color caster material casting."""
    caster = sg.CreateColorCaster()
    caster.SetMappingImage( mapping_image )
    caster.SetSourceMaterials( scene.GetMaterialTable() )
    caster.SetSourceTextures( scene.GetTextureTable() )
    caster.SetOutputFilePath( temp_path + material_channel + ".png")

    caster_settings = caster.GetColorCasterSettings()
    caster_settings.SetMaterialChannel( material_channel )
    caster_settings.SetOutputImageFileFormat( Simplygon.EImageOutputFormat_PNG )

    caster.RunProcessing()
    return caster.GetOutputFilePath()

The output is a texture containing all surfaces in the scene, as well as any decals close to them.

We will now assign the texture to our scene. We'll add a function which adds material to scene.

First step is to create a texture using CreateTexture and add it to the texture table, as well as using the texture we casted above.

Once we have the texture in our texture table we can refer to it in our materials. First we create a shading network containing a simple Shading Texture Node referring to our newly created texture. Then we add a material using that shading node as it's shading network for a specified material channel. If we would have more material channels, like normal maps, we would need to add them here as well.

def add_material_to_scene(sg, scene, material_channel, texture_path):
    """Add a new material with specified texture to scene"""

    # Create texture from texture_path and add it to scene
    texture = sg.CreateTexture()
    texture.SetName( material_channel )
    texture.SetFilePath( texture_path)
    scene.GetTextureTable().AddTexture( texture )

    # Create shading node using texture we created above
    shading_node = sg.CreateShadingTextureNode()
    shading_node.SetTexCoordLevel( 0 )
    shading_node.SetTextureName( material_channel )
    
    # Create material using shading node we created above and add it to scene
    material = sg.CreateMaterial()
    material.AddMaterialChannel( material_channel )
    material.SetShadingNetwork( material_channel, shading_node )
    scene.GetMaterialTable().AddMaterial( material )

Import optimized scene to Blender

After processing we want send the result back into Blender. We will do this with a SceneExporter and re-use the same gltf file we using during exporting.

def import_results(sg, scene, file_path):
    """Import the Simplygon scene into Blender."""

    scene_exporter = sg.CreateSceneExporter()
    scene_exporter.SetExportFilePath(file_path)
    scene_exporter.SetScene(scene)
    scene_exporter.Run()
    bpy.ops.import_scene.gltf(filepath=file_path)

Putting it all together

Once all our helper functions are created we can put them together into a function which takes a scene, detects and remove decals, then bakes them into the underlying geometry.

def process_selection(sg):
    """Remove and bake decals on selected meshes."""

    # Export scene from Blender and import it 
    file_path = temp_path + file
    original_scene = export_selection(sg, file_path)

    # Create copy of original scene we will work with
    optimized_scene = original_scene.NewCopy()
    
    # Create a selection set containing all decals
    decal_set = sg.CreateSelectionSet()
    decal_set_id = optimized_scene.GetSelectionSetTable().AddSelectionSet(decal_set)
    select_decals(optimized_scene, optimized_scene.GetRootNode(), decal_set)

    # Remove decals from optimized scene
    optimized_scene.RemoveSceneNodesInSelectionSet(decal_set_id)
    
    # Aggregate optimized scene
    aggregate_scene(sg, optimized_scene, texture_size)

    # Create a mapping image from original_scene to optimized_scene
    mapping_image = create_mapping_image_between_scenes(sg, original_scene, optimized_scene, texture_size)

    # Bake color texture
    color_texture = perform_color_casting(sg, optimized_scene, mapping_image, color_channel, temp_path)
    
    # Clear textures and material from optimized_scene
    clear_scene_material_and_texture_tables(optimized_scene)
    
    # Add color material to scene
    add_material_to_scene(sg, optimized_scene, color_channel, color_texture)
    
    # Import result into Blender
    import_results(sg, optimized_scene, file_path)

Result

This is the result after processing our scene. All spider web decals has been removed and baked into the environment. As all object share material and texture in scene it is very cheap to render via a simple opaque shader.

Complete script

# Copyright (c) Microsoft Corporation. 
# Licensed under the MIT license. 

import os
import bpy
import gc

from simplygon10 import simplygon_loader
from simplygon10 import Simplygon


file = "scene.glb"
temp_path = "c:/tmp/"
texture_size = 4096
color_channel = "Basecolor"


def select_decals(scene, scene_node, decal_set):
    """Add all child SceneMesh nodes considered to be decals to selection set."""
    for i in range(0, scene_node.GetChildCount()):
        child = scene_node.GetChild(i)
        if child.IsA("ISceneMesh"):
            scene_mesh = Simplygon.spSceneMesh.SafeCast(child)
            if is_mesh_decal(scene, scene_mesh):
                decal_set.AddItem(child.GetNodeGUID())    
                
        select_decals(scene, child,  decal_set)


def is_mesh_decal(scene, scene_mesh):
    """If all materials on our mesh is transparent we consider mesh to be a decal."""
    materials =  scene_mesh.GetGeometry().GetMaterialIds()
    for j in range(0, materials.GetItemCount()):
        if not is_material_transparent(scene, materials.GetItem(j)):
            return False
    return True


def is_material_transparent(scene, material_id):
    """Returns if material is transparent."""
    material = scene.GetMaterialTable().GetMaterial(material_id)
    return material.IsTransparent()


def export_selection(sg, file_path):
    """Export the current selected objects into Simplygon."""
    bpy.ops.export_scene.gltf(filepath = file_path, use_selection=True)
    sceneImporter = sg.CreateSceneImporter()
    sceneImporter.SetImportFilePath(file_path)
    sceneImporter.Run()
    scene = sceneImporter.GetScene()
    return scene


def import_results(sg, scene, file_path):
    """Import the Simplygon scene into Blender."""
    scene_exporter = sg.CreateSceneExporter()
    scene_exporter.SetExportFilePath(file_path)
    scene_exporter.SetScene(scene)
    scene_exporter.Run()
    bpy.ops.import_scene.gltf(filepath=file_path)
    

def clear_scene_material_and_texture_tables(scene):
    """Clean scene's texture and material table."""
    scene.GetTextureTable().Clear()
    scene.GetMaterialTable().Clear()


def perform_color_casting(sg, scene, mapping_image, material_channel, temporary_image_path ):
    """Setup and run a color caster material casting."""
    caster = sg.CreateColorCaster()
    caster.SetMappingImage( mapping_image )
    caster.SetSourceMaterials( scene.GetMaterialTable() )
    caster.SetSourceTextures( scene.GetTextureTable() )
    caster.SetOutputFilePath( temp_path + material_channel + ".png")

    caster_settings = caster.GetColorCasterSettings()
    caster_settings.SetMaterialChannel( material_channel )
    caster_settings.SetOutputImageFileFormat( Simplygon.EImageOutputFormat_PNG )

    caster.RunProcessing()
    return caster.GetOutputFilePath()


def add_material_to_scene(sg, scene, material_channel, texture_path):
    """Add a new material with specified texture to scene"""

    # Create texture from texture_path and add it to scene
    texture = sg.CreateTexture()
    texture.SetName( material_channel )
    texture.SetFilePath( texture_path)
    scene.GetTextureTable().AddTexture( texture )

    # Create shading node using texture we created above
    shading_node = sg.CreateShadingTextureNode()
    shading_node.SetTexCoordLevel( 0 )
    shading_node.SetTextureName( material_channel )
    
    # Create material using shading node we created above and add it to scene
    material = sg.CreateMaterial()
    material.AddMaterialChannel( material_channel )
    material.SetShadingNetwork( material_channel, shading_node )
    scene.GetMaterialTable().AddMaterial( material )
    
    
def aggregate_scene(sg, scene, texture_size):
    """Aggregate all meshes in scene to create unique UVs."""
    # Create the aggregation processor. 
    aggregation_processor = sg.CreateAggregationProcessor()
    aggregation_processor.SetScene( scene )
    aggregation_settings = aggregation_processor.GetAggregationSettings()
    
    # Do not merge into one mesh
    aggregation_settings.SetMergeGeometries( False )

    # Use mapping image to generate unique UVs.
    mapping_image_settings = aggregation_processor.GetMappingImageSettings()
    mapping_image_settings.SetGenerateMappingImage( True )
    mapping_image_settings.SetApplyNewMaterialIds( True )
    mapping_image_settings.SetGenerateTangents( True )
    mapping_image_settings.SetUseFullRetexturing( True )
    mapping_image_settings.SetTexCoordGeneratorType( Simplygon.ETexcoordGeneratorType_Parameterizer )
    
    # Set output texture size
    output_material_settings = mapping_image_settings.GetOutputMaterialSettings(0)
    output_material_settings.SetTextureWidth( texture_size )
    output_material_settings.SetTextureHeight( texture_size )
    
    aggregation_processor.RunProcessing()

    
def create_mapping_image_between_scenes(sg, original_scene, optimized_scene, texture_size):
    """Use surface mapper to generate mapping image from original_scene to optimized_scene."""

    surface_mapper = sg.CreateSurfaceMapper()
    surface_mapper.SetSourceScene(original_scene)
    surface_mapper.SetDestinationScene(optimized_scene)
    
    
    surface_mapper.SetSearchOffset(1)
    surface_mapper.SetSearchDistance(2)
    
    mapping_image_settings = surface_mapper.GetMappingImageSettings()
    output_material_settings = mapping_image_settings.GetOutputMaterialSettings(0)
    output_material_settings.SetTextureWidth( texture_size )
    output_material_settings.SetTextureHeight( texture_size )
    
    surface_mapper.RunSurfaceMapping()
    return surface_mapper.GetMappingImage()


def process_selection(sg):
    """Remove and bake decals on selected meshes."""

    # Export scene from Blender and import it 
    file_path = temp_path + file
    original_scene = export_selection(sg, file_path)

    # Create copy of original scene we will work with
    optimized_scene = original_scene.NewCopy()
    
    # Create a selection set containing all decals
    decal_set = sg.CreateSelectionSet()
    decal_set_id = optimized_scene.GetSelectionSetTable().AddSelectionSet(decal_set)
    select_decals(optimized_scene, optimized_scene.GetRootNode(), decal_set)

    # Remove decals from optimized scene
    optimized_scene.RemoveSceneNodesInSelectionSet(decal_set_id)
    
    # Aggregate optimized scene
    aggregate_scene(sg, optimized_scene, texture_size)

    # Create a mapping image from original_scene to optimized_scene
    mapping_image = create_mapping_image_between_scenes(sg, original_scene, optimized_scene, texture_size)

    # Bake color texture
    color_texture = perform_color_casting(sg, optimized_scene, mapping_image, color_channel, temp_path)
    
    # Clear textures and material from optimized_scene
    clear_scene_material_and_texture_tables(optimized_scene)
    
    # Add color material to scene
    add_material_to_scene(sg, optimized_scene, color_channel, color_texture)
    
    # Import result into Blender
    import_results(sg, optimized_scene, file_path)


def main():
    sg = simplygon_loader.init_simplygon()

    process_selection(sg)
    sg = None
    gc.collect()


if __name__== "__main__":
    main()