Written by Jesper Tingvall, Product Expert, Simplygon

Disclaimer: The code in this post is written using version 10.0.1400.0 of Simplygon and Python 3.9.7. If you encounter this post at a later stage, some of the API calls might have changed. However, the core concepts should still remain valid.

Introduction

With Simplygon 10.0 we introduced tessellated attributes. In this post we'll showcase how to use them to speed up high resolution remeshings.

Prerequisites

This example will use the Simplygon API in Python, but the same concepts can be applied to all other integrations of the Simplygon API.

Problem to solve

We want to remesh an asset at a very high resolution. One reason could be that our primary use case is not to reduce the polygon count. Instead we want to clean up topology and remove interior data, generating a watertight mesh with a single unique UV map. However, remeshing at high resolutions require lots of memory and can take very long time. We would like to speed up this process.

This is the asset we will process. It is a kit bashed collection of rocks. After processing, all internal geometry will be removed, but we want to keep all the surface geometric detail. As showcased from wireframe it is very high poly, 6 998 112 triangles.

Solution

By using Simplygon's tessellated attributes, we can instead remesh at a lower resolution. This will create a topologically closed mesh with all interiors removed. After remeshing, we can then transfer the surface geometry details onto a the remeshed mesh to get a high polygon remesh with all geometric detail restored.

Import and export

First we are going to introduce helper import and export functions. One change from Simplygon 9 to Simplygon 10 is that RunImport and RunExport are renamed to Run and returns an EErrorCode. With that error code we can more easy detect what is wrong if it fails.

def import_scene(sg, asset_file):
    scene_importer = sg.CreateSceneImporter()
    scene_importer.SetImportFilePath(asset_file)

    import_result = scene_importer.Run()
    if Simplygon.Failed(import_result):
        raise Exception("Import failed")

    return scene_importer.GetScene()

We also add an export scene function with error handling.

def export_scene(sg, scene, export_path):
    scene_exporter = sg.CreateSceneExporter()
    scene_exporter.SetExportFilePath(export_path)
    scene_exporter.SetScene(scene)

    export_result = scene_exporter.Run()
    if Simplygon.Failed(export_result):
        raise Exception("Exporting to " + export_path + " failed")

Remeshing pipeline

The low polygon mesh is created using a remeshing pipeline. We can use SetHole Filling to clean up any small holes that might be in the original mesh.

def create_remesh_pipeline(sg, screen_size):
    remeshing_pipeline = sg.CreateRemeshingPipeline()
    remeshing_settings = remeshing_pipeline.GetRemeshingSettings()

    remeshing_settings.SetOnScreenSize(screen_size)
    remeshing_settings.SetHoleFilling(Simplygon.EHoleFilling_Medium)
    remeshing_settings.SetHardEdgeAngle(90.0)
    remeshing_settings.SetForceSoftEdgesWithinTextureCharts(True)
    return remeshing_pipeline

To be able to transfer textures as well as displacement data for tessellated attributes we set up a mapping image.

To generate a mapping from old mesh to new we need to set SetGenerateMappingImage and SetGenerateTexCoords to True. With SetTextureWidth and SetTextureHeight we can set the texture size for our generated textures.

Our asset is opaque and do not have any thin geometry which we want to merge into underlying geometry. To speed up execution we can set SetMaximumLayers to 1. This will cause the mapping image to only care about first hit when it maps from high poly to low poly mesh.

def setup_mapping_image(remeshing_pipeline, texture_size):
    # Setup generation of a mapping image. This will be used to cast texture
    # data to a new image, as well as the displacement data to the tessellated attribute
    mapping_image = remeshing_pipeline.GetMappingImageSettings()
    mapping_image.SetGenerateMappingImage(True)
    mapping_image.SetGenerateTexCoords(True)
    mapping_image.SetTexCoordLevel(0)
    mapping_image.GetOutputMaterialSettings(0).SetTextureWidth(texture_size)
    mapping_image.GetOutputMaterialSettings(0).SetTextureHeight(texture_size)
    mapping_image.SetMaximumLayers(1)

Color caster

To transfer the color we add a color caster. For our asset we do not need any alpha channel in our output texture so we set SetOutputPixelFormat to EPixelFormat_R8G8B8 (RGB with 8 bit depth per color channel).

def add_color_caster(sg, pipeline, channel_name):
    #/ Set up to cast the Diffuse color channel (just like a regular remesh)
    color_caster = sg.CreateColorCaster()
    color_caster_settings = color_caster.GetColorCasterSettings()

    color_caster_settings.SetDilation(10)
    color_caster_settings.SetMaterialChannel(channel_name)
    color_caster_settings.SetOutputPixelFormat(Simplygon.EPixelFormat_R8G8B8)

    pipeline.AddMaterialCaster(color_caster, 0)

If we process our asset with the remesh pipeline and color caster from above using screen size 150 and texture resolution 4096 we get following result at 730 triangles. As expected the resulting mesh is lacking details, but those will be added in next step.

Attribute tessellation

To add detailed geometry to our low poly mesh we are going to use attribute tessellation. First step is to enable attribute tessellation for our pipeline by setting SetEnableAttributeTesselation to True.

To decide when a triangle should be tessellated can can set different SetAttributeTessellationDensityMode. The one we will use is EAttributeTessellationDensityMode_RelativeArea. In that mode we can control when a triangle should be tessellated by SetMaxAreaOfTessellatedValue. In that mode the entire area of our geometry is normalized to 1, and we can specify how large each tessellated triangle should be in relation to that.

We are inverting that number to instead use how many triangles should be used per geometry in total. However it will also obey the additional constraints we put on it below.

To controls how much a triangle can be tessellated we use following two functions. We set MinTesselationLevel to 0, allowing triangles to remain not tessellated. Each tessellation value divides up one triangle into 4. So with SetMaxTesselationLevel set to 10 one triangle can be tessellated up to 4 ^ 10 = 1 048 576

With SetOnlyAllowOneLevelOfDifference set to True the level of tessellation will only differ by one between neighbouring triangles. So if a triangle is tessellated to level 2; its neighbours will only be tessellated to either 1, 2 or 3. This settings makes borders between triangles look more nice.

With SetMaxTotalValuesCount we can set the absolute max tessellated triangles a geometry in our scene is allowed to have. We set it to 4 000 000.

def setup_attribute_tesselation(remeshing_pipeline, min_subsamples):
    attribute_tesselation_settings = remeshing_pipeline.GetAttributeTessellationSettings()
    attribute_tesselation_settings.SetEnableAttributeTessellation(True)

    attribute_tesselation_settings.SetAttributeTessellationDensityMode(
        Simplygon.EAttributeTessellationDensityMode_RelativeArea)

    # In DensityMode RelativeArea the total area of our geometry is 1. 
    # Thus we can calculate target area of our tesselated triangles from how many tesselated triangles samples we want using formula.
    max_area_of_tesselated_value = 1.0 / min_subsamples
    attribute_tesselation_settings.SetMaxAreaOfTessellatedValue(
        max_area_of_tesselated_value)

    # Force only one level of difference in tesselation level between subdivided triangles
    attribute_tesselation_settings.SetOnlyAllowOneLevelOfDifference(True)

    # Set minimum tessellation level to 0, (which means untesselated).
    attribute_tesselation_settings.SetMinTessellationLevel(0)

    # Set maximum tessellation level to 10, (which allows one triangle to be tesselated up to 4 ^ 10 = 1 048 576 subvalues)
    attribute_tesselation_settings.SetMaxTessellationLevel(10)

    # Safe guard max allowed tesselated triangles per geometry. 
    attribute_tesselation_settings.SetMaxTotalValuesCount(4000000) 

Displacement caster

We use a Displacement caster to bake a displacement map which we save into our tessellated attributes. This is similar to how one would generate a displacement map, but in our case we set SetOutputToTessellatedAttributes to True which saves it to attributes instead of a texture.

Along with that we also need to set attribute tesselation sampling settings. Via SetAttributeFormat we can specify how the displacement data should be stored. With it set to EAttributeFormat_F32vec3 we get full 3d displacement. We can also set it to EAttributeFormat_U16 which would displace it along the normal. To have a more smooth and better looking result, at cost of performance, we set SetSupersamplingCount to 16.

Lastly we set SetSourceMaterialId to 0. This allows our caster to read displacement data from our first material.

def add_displacement_caster(sg, pipeline, scene, displacement_channel_name):
    displacement_caster = sg.CreateDisplacementCaster()
    displacement_caster_settings = displacement_caster.GetDisplacementCasterSettings()

    displacement_caster.SetScene(scene)
    displacement_caster_settings.SetMaterialChannel(displacement_channel_name)

    displacement_caster_settings.SetOutputToTessellatedAttributes(True)
    attribute_tesselation_sampling_settings = displacement_caster_settings.GetAttributeTessellationSamplingSettings()
    attribute_tesselation_sampling_settings.SetSourceMaterialId(0)
    attribute_tesselation_sampling_settings.SetAttributeFormat(
        Simplygon.EAttributeFormat_F32vec3)  # F32vec3 or U16 are allowed
    attribute_tesselation_sampling_settings.SetSupersamplingCount(16)

    pipeline.AddMaterialCaster(displacement_caster, 0)

Tessellate scene

If we would export the scene after adding tessellated attributes and displacement caster we would not notice any difference between that one and the low poly remeshing. That is because our exporter does not export any tessellated attributes. To get them out of Simplygon we would either need to write our own exporter, or tessellate the scene prior to export. We can do this via NewTesselatedScene which takes a scene with tessellated attributes and create a new tessellated scene from it.

def create_tesselated_scene(sg, scene):
    tesselation = sg.CreateAttributeTessellation()
    return tesselation.NewTessellatedScene(scene)

Putting it all together

Now we have all the functions and it is time to put them together. First we do a low-poly remeshing with displacement data saved in tessellated attributes. Then we tessellate it and save it.

def process_file(sg, asset_file, output_file):
    scene = import_scene(sg, asset_file)

    # Remeshing for target screen size 150 pixels.
    remesh_pipeline = create_remesh_pipeline(sg, 150)

    # Use texture size of 4096x4096
    setup_mapping_image(remesh_pipeline, 4096)

    # Cast new texture from Diffuse color channel.
    add_color_caster(sg, remesh_pipeline, "Diffuse")

    # Setup tesselated attributes for scene. Use at least 100 000 tesselated triangles.
    setup_attribute_tesselation(remesh_pipeline, 100000)

    # Add displacement caster casting into tesselated attributes
    add_displacement_caster(sg, remesh_pipeline, scene, "Displacement")

    # Run remeshing and both casters
    remesh_pipeline.RunScene(
        scene, Simplygon.EPipelineRunMode_RunInThisProcess)
    
    # Uncomment this line to export low poly mesh
    # export_scene(sg, scene, "StonePile_lowpoly.obj")

    # Create and export tesselated scene
    tesselated_scene = create_tesselated_scene(sg, scene)
    export_scene(sg, tesselated_scene, output_file)

Result

After tessellating and exporting our model looks like this and contains 214 216 triangles. We have an even tessellation of our model which contains small surface details. This can best be seen at the edges if our mesh.

The execution time is compared to a high resolution remeshing very quick.

Remeshing at 150 pixels with tessellated attributes

To get a mesh of equal quality with just remeshing we need with our asset go up to around 3080 pixels resulting in 186 572 triangles.

Remeshing at 3080 pixels

As we can see it is significantly faster to use tessellated attributes. It also needs less memory during processing.

One additional benefit with this approach is that we can use the low poly mesh as a LOD, as it shares textures with the tessellated high poly mesh.

Complete script

# Copyright (c) Microsoft Corporation.
# Licensed under the MIT license.

from simplygon10 import simplygon_loader
from simplygon10 import Simplygon


def process_file(sg, asset_file, output_file):
    scene = import_scene(sg, asset_file)

    # Remeshing for target screen size 150 pixels.
    remesh_pipeline = create_remesh_pipeline(sg, 150)

    # Use texture size of 4096x4096
    setup_mapping_image(remesh_pipeline, 4096)

    # Cast new texture from Diffuse color channel.
    add_color_caster(sg, remesh_pipeline, "Diffuse")

    # Setup tesselated attributes for scene. Use around 100 000 tesselated triangles.
    setup_attribute_tesselation(remesh_pipeline, 100000)

    # Add displacement caster casting into tesselated attributes
    add_displacement_caster(sg, remesh_pipeline, scene, "Displacement")

    # Run remeshing and both casters
    remesh_pipeline.RunScene(
        scene, Simplygon.EPipelineRunMode_RunInThisProcess)
    
    # Uncomment this line to export low poly mesh
    # export_scene(sg, scene, "StonePile_lowpoly.obj")

    # Create and export tesselated scene
    tesselated_scene = create_tesselated_scene(sg, scene)
    export_scene(sg, tesselated_scene, output_file)


def import_scene(sg, asset_file):
    scene_importer = sg.CreateSceneImporter()
    scene_importer.SetImportFilePath(asset_file)

    import_result = scene_importer.Run()
    if Simplygon.Failed(import_result):
        raise Exception("Import failed of " +
                        asset_file + ": " + str(import_result))

    return scene_importer.GetScene()


def create_remesh_pipeline(sg, screen_size):
    remeshing_pipeline = sg.CreateRemeshingPipeline()
    remeshing_settings = remeshing_pipeline.GetRemeshingSettings()

    remeshing_settings.SetOnScreenSize(screen_size)
    remeshing_settings.SetHoleFilling(Simplygon.EHoleFilling_Medium)
    return remeshing_pipeline


def setup_mapping_image(remeshing_pipeline, texture_size):
    mapping_image = remeshing_pipeline.GetMappingImageSettings()
    output_material_settings = mapping_image.GetOutputMaterialSettings(0)
    mapping_image.SetGenerateMappingImage(True)
    mapping_image.SetGenerateTexCoords(True)
    mapping_image.SetTexCoordLevel(0)
    mapping_image.SetMaximumLayers(1)
    output_material_settings.SetTextureWidth(texture_size)
    output_material_settings.SetTextureHeight(texture_size)


def setup_attribute_tesselation(remeshing_pipeline, max_subsamples):
    attribute_tesselation_settings = remeshing_pipeline.GetAttributeTessellationSettings()
    attribute_tesselation_settings.SetEnableAttributeTessellation(True)

    attribute_tesselation_settings.SetAttributeTessellationDensityMode(
        Simplygon.EAttributeTessellationDensityMode_RelativeArea)

    # In DensityMode RelativeArea the total area of our geometry is 1. 
    # Thus we can calculate target area of our tesselated triangles from how many tesselated triangles samples we want using formula.
    max_area_of_tesselated_value = 1.0 / max_subsamples
    attribute_tesselation_settings.SetMaxAreaOfTessellatedValue(
        max_area_of_tesselated_value)

    # To make triangles match in seams, force only one level of difference in tesselation amount.
    attribute_tesselation_settings.SetOnlyAllowOneLevelOfDifference(True)

    # Allow triangles to remain untesselated.
    attribute_tesselation_settings.SetMinTessellationLevel(0)

    # Allow one triangle to be tesselated up to 4 ^ 10 = 1 048 576 triangles.
    attribute_tesselation_settings.SetMaxTessellationLevel(10)

    # Safe guard max allowed tesselated triangles per geometry. 
    attribute_tesselation_settings.SetMaxTotalValuesCount(4000000) 


def add_color_caster(sg, pipeline, channel_name):
    color_caster = sg.CreateColorCaster()
    color_caster_settings = color_caster.GetColorCasterSettings()

    color_caster_settings.SetMaterialChannel(channel_name)
    color_caster_settings.SetOutputPixelFormat(Simplygon.EPixelFormat_R8G8B8)

    pipeline.AddMaterialCaster(color_caster, 0)


def add_displacement_caster(sg, pipeline, scene, displacement_channel_name):
    displacement_caster = sg.CreateDisplacementCaster()
    displacement_caster_settings = displacement_caster.GetDisplacementCasterSettings()

    displacement_caster.SetScene(scene)
    displacement_caster_settings.SetMaterialChannel(displacement_channel_name)

    displacement_caster_settings.SetOutputToTessellatedAttributes(True)
    attribute_tesselation_sampling_settings = displacement_caster_settings.GetAttributeTessellationSamplingSettings()
    attribute_tesselation_sampling_settings.SetSourceMaterialId(0)
    attribute_tesselation_sampling_settings.SetAttributeFormat(
        Simplygon.EAttributeFormat_F32vec3)  # F32vec3 or U16 are allowed
    attribute_tesselation_sampling_settings.SetSupersamplingCount(16)

    pipeline.AddMaterialCaster(displacement_caster, 0)


def create_tesselated_scene(sg, scene):
    tesselation = sg.CreateAttributeTessellation()
    return tesselation.NewTessellatedScene(scene)


def export_scene(sg, scene, export_path):
    scene_exporter = sg.CreateSceneExporter()
    scene_exporter.SetExportFilePath(export_path)
    scene_exporter.SetScene(scene)

    export_result = scene_exporter.Run()
    if Simplygon.Failed(export_result):
        raise Exception("Exporting to " + export_path +
                        " failed: " + str(export_result))


def main():
    sg = simplygon_loader.init_simplygon()
    sg.SetGlobalEnableLogSetting(True)

    process_file(sg, "stone/StonePile2.obj", "StonePile_out.obj")
    del sg


if __name__ == "__main__":
    main()