Comparing material casting and aggregation
Disclaimer: The code in this post is written on version 9.1.225 of Simplygon. Should you encounter this post running a different version, some of the API calls might differ. However, the core concepts should still remain valid.
Introduction
In Simplygon, you have two different options when it comes to transferring materials from source to optimized geometry. You can either cast the materials pixel for pixel using the paramterizer, or use the chart aggregator to use the existing UVs when merging the materials. This makes for some interesting choices that we will explore in this post.
Material casting
When casting materials using the parameterizer, a new set of UVs will be generated for the asset. When it comes to remeshings, parametrized material casting is the only option. The remesher generates a completely new mesh and cannot use the UVs from the original.
Since parametrization will generate unique texture coordinates for the whole geometry (no overlaps) you might need large texture resolutions to preserve visual fidelity. Hence, it is best used on objects meant for distant view, or when there aren't much overlaps in the original, for example scanned content.
Material aggregation
Material casting using the chart aggregator, uses the UV shells from the original asset and merges them into one atlas. Among others, you have the option to seperate overlapping charts and redistribute the texture density using the chart aggregator mode.
Chart aggregation would be the preferred option when you are creating object representations that will be used close to the camera. It typically leads to higher texture quality than parametrization, which means that you get away with less texture space when using this method. The option is available to you when you are reducing or aggregating assets.
Above left is an example of an aggregated material, to the right cast materials.
Creating new materials in script
Let us create a simple script where we try both the parametrizer and the chart aggregator on a couple of assets. We will also be using an automatic approach to adding material casters. This approach might not work for all cases, but it's a good starting point for you to start exploring the different options at hand.
1. Create the template script
We'll create a Python file called matrial_casting_example.py. As always in our posts we use this template to start our script:
from simplygon import simplygon_loader
from simplygon import Simplygon
def process_file(sg, asset_file):
def main():
sg = simplygon_loader.init_simplygon()
print(sg.GetVersion())
process_file(sg, <input assset>)
del sg
if __name__== "__main__":
main()
For this example we will be defining four functions. These are:
- add_default_casters - responsible for adding the casters that's needed to transfer the materials in the scene we're processing.
- setup_mapping_image - sets up the settings on the mapping image to use the transfer method we're looking for.
- process_pipeline - sets everything up on a pipeline, runs the processing and exports the results.
- process_file - starts the processing.
2. Automatic material casters
We need to define a function that automatically assigns material casters to the pipelines, derived from incoming asset. It's quite likely that you will have to modify this function to fit your needs. In our case we used an FBX file as input, meaning that the function is fitted for that file format.
The function GetMaterialChannelsWithTextureInputs is very useful to achieve this. It will return all material channels that have texture inputs. With this little snippet of code we can loop over all those channels. If your materials are using color as inputs in some channels, they will not be included in this list and you need to make sure that those are also included.
materials = scene.GetMaterialTable()
texture_channels = sg.CreateStringArray()
materials.GetMaterialChannelsWithTextureInputs(texture_channels)
for i in range(0, texture_channels.GetItemCount()):
channel = texture_channels.GetItem(i)
There are a range of different material casters in the Simplygon SDK, and it might be an idea to explore them to better understand what tools you have at your disposal. For this example, we will only be using:
If you want to add a default color caster to a pipeline, this is the code you need.
caster = sg.CreateColorCaster()
caster_settings = caster.GetColorCasterSettings()
caster_settings.SetMaterialChannel(channel)
pipeline.AddMaterialCaster( caster, 0 )
Now we have enough to define our add_default_color_casters function. Here it is:
def add_default_casters(sg, pipeline, scene):
"""
Finds all the material channels with textures connected to them and creates the appropriate caster for the channel.
"""
materials = scene.GetMaterialTable()
texture_channels = sg.CreateStringArray()
# Get a list of all channels with texture inputs
materials.GetMaterialChannelsWithTextureInputs(texture_channels)
normal_map_added = False
for i in range(0, texture_channels.GetItemCount()):
channel = texture_channels.GetItem(i)
if channel == Simplygon.SG_MATERIAL_CHANNEL_NORMALS:
caster = sg.CreateNormalCaster()
caster_settings = caster.GetNormalCasterSettings()
caster_settings.SetGenerateTangentSpaceNormals(True)
normal_map_added = True
elif channel == Simplygon.SG_MATERIAL_CHANNEL_OPACITY:
caster = sg.CreateOpacityCaster()
caster_settings = caster.GetOpacityCasterSettings()
elif channel == Simplygon.SG_MATERIAL_CHANNEL_BASECOLOR or channel == Simplygon.SG_MATERIAL_CHANNEL_DIFFUSE :
caster = sg.CreateColorCaster()
caster_settings = caster.GetColorCasterSettings()
# GLB files stores opacity in the alpha channel of the color map.
caster_settings.SetOpacityChannelComponent(Simplygon.EColorComponent_Alpha)
else:
# For all other channels we'll just add a straight forward
# pixel to pixel color caster.
caster = sg.CreateColorCaster()
caster_settings = caster.GetColorCasterSettings()
print("Adding a material caster for channel: "+channel)
caster_settings.SetMaterialChannel(channel)
pipeline.AddMaterialCaster( caster, 0 )
# Finally, if there wasn't any normal map we should add one if it's a remeshing.
if pipeline.IsA("IRemeshingPipeline") and not normal_map_added:
caster = sg.CreateNormalCaster()
caster_settings = caster.GetNormalCasterSettings()
caster_settings.SetGenerateTangentSpaceNormals(True)
caster_settings.SetMaterialChannel(Simplygon.SG_MATERIAL_CHANNEL_NORMALS)
pipeline.AddMaterialCaster( caster, 0 )
You can see that we have a few different channel names as options, it all depends on what the incoming file format uses as channel naming. If you are creating the Simplygon scene programmatically, you will most likely need to adjust the names to match your channels.
A final note on this function, is the IsA function. All Simplygon objects can be asked whether they are of a certain class. In this context we use that to ensure that a normal map is added to remeshing pipelines.
3. Setup the mapping image
The pipeline contains an object called mapping image, which dictates how the material casting is processed. We need to set that up for our example as well. Let's define a function to get that done.
def setup_mapping_image(sg, pipeline, scene, textureSize):
mapping_image_settings = pipeline.GetMappingImageSettings()
mapping_image_settings.SetGenerateMappingImage(True)
mapping_image_settings.SetTexCoordName("MaterialLOD")
material_settings = mapping_image_settings.GetOutputMaterialSettings(0)
material_settings.SetTextureWidth(textureSize)
material_settings.SetTextureHeight(textureSize)
if pipeline.IsA("IAggregationPipeline"):
# In aggregation processes we want to keep the orginal UVs and
# merge all textures onto one atlas. This saves UV space if there's
# a lot over overlapping UV charts.
mapping_image_settings.SetTexCoordGeneratorType(Simplygon.ETexcoordGeneratorType_ChartAggregator)
mapping_image_settings.SetApplyNewMaterialIds(True)
mapping_image_settings.SetGenerateTangents(True)
mapping_image_settings.SetUseFullRetexturing(True)
chart_agg_settings = mapping_image_settings.GetChartAggregatorSettings()
# We can play around with this parameter if we want to ditribute the UV space in a different
# way compared to the original.
chart_agg_settings.SetChartAggregatorMode(Simplygon.EChartAggregatorMode_TextureSizeProportions)
# Enable the chart aggregator and reuse UV space.
chart_agg_settings.SetSeparateOverlappingCharts(False)
For a remeshing pipeline we will be using the default settings apart from texture size. When it comes to aggregation, we'll flip from the default parameterizer to the chart aggregator. As mentioned earlier in this post, SetChartAggregatorMode gives you the option to redistribute the texture density to fit your needs.
4. The rest
Finally we just want to tie it up toghether with two orchestrating functions. They are responsible for loading, remeshing and aggregating the asset and finally export the results. Here they are:
def process_pipeline(sg, pipeline, scene, output_file):
print("Setting up material casters...")
setup_mapping_image(sg, pipeline, scene, 1024)
add_default_casters(sg, pipeline, scene)
pipeline.RunScene(scene, Simplygon.EPipelineRunMode_RunInThisProcess)
#Export the hollow shell into the output file.
scene_exporter = sg.CreateSceneExporter()
scene_exporter.SetExportFilePath(output_file)
print("Writing output to "+output_file)
scene_exporter.SetScene(scene)
scene_exporter.RunExport()
def process_file(sg, asset_file):
# Load the asset we want to process
scene_importer = sg.CreateSceneImporter()
scene_importer.SetImportFilePath(asset_file)
if scene_importer.RunImport():
# Process the asset
remesh_pipeline = sg.CreateRemeshingPipeline()
remesh_scene = scene_importer.GetScene()
agg_pipeline = sg.CreateAggregationPipeline()
aggregation_scene = scene_importer.GetScene().NewCopy()
print("Running remeshing...")
process_pipeline(sg, remesh_pipeline, remesh_scene, asset_file.split('.')[0]+'_remesh.fbx')
print("Running aggregation...")
process_pipeline(sg, agg_pipeline, aggregation_scene, asset_file.split('.')[0]+'_aggregated.fbx')
else:
print("Failed to load "+asset_file)
Running the script
Let's run the script on asset with a lot of overlapping UVs. The gold mine pictured below has those traits.
Below there's a look at the texture and a close up of the asset after using the chart aggregator when transferring the materials.
As you can see the details have been preserved quite well, and we now have everything in one atlas allowing us to draw the asset in one draw call. The geometry hasn't been modified at all.
The remeshed version of the gold mine looks a lot coarser when zooming in on it.
On the other hand we now have a watertight mesh that's only 4000 polygons (original is 135,000), which is really suitable for a far away representation.
We could also apply reduction to the results from the aggregation (and create a hollow shell), however it will not be able to get as low as the remesher.
Conclusions
When choosing between the chart aggregator and parametrizer, the default choice should be the former. If you're looking for to create unique UVs for the resulting objects (for example, casting ambient occlusion) both options works, but you must set SeparateOverlappingCharts to True for the aggregator in those cases.
When you are looking to create a distant proxy, and polygon reduction is your primary motive, remeshing would be normally be the best approach. In that case, the parametrizer would be your only option. However, the level of detail on the textures shouldn't be much of a problem as the asset will be far so away from the camera.
The script
Here is the complete script.
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT license.
from simplygon import simplygon_loader
from simplygon import Simplygon
def add_default_casters(sg, pipeline, scene):
"""
Finds all the material channels with textures connected to them and creates the appropriate caster for the channel.
"""
materials = scene.GetMaterialTable()
texture_channels = sg.CreateStringArray()
# Get a list of all channels with texture inputs
materials.GetMaterialChannelsWithTextureInputs(texture_channels)
normal_map_added = False
for i in range(0, texture_channels.GetItemCount()):
channel = texture_channels.GetItem(i)
if channel == Simplygon.SG_MATERIAL_CHANNEL_NORMALS:
caster = sg.CreateNormalCaster()
caster_settings = caster.GetNormalCasterSettings()
caster_settings.SetGenerateTangentSpaceNormals(True)
normal_map_added = True
elif channel == Simplygon.SG_MATERIAL_CHANNEL_OPACITY:
caster = sg.CreateOpacityCaster()
caster_settings = caster.GetOpacityCasterSettings()
elif channel == Simplygon.SG_MATERIAL_CHANNEL_BASECOLOR or channel == Simplygon.SG_MATERIAL_CHANNEL_DIFFUSE :
caster = sg.CreateColorCaster()
caster_settings = caster.GetColorCasterSettings()
# GLB files stores opacity in the alpha channel of the color map.
caster_settings.SetOpacityChannelComponent(Simplygon.EColorComponent_Alpha)
else:
# For all other channels we'll just add a straight forward
# pixel to pixel color caster.
caster = sg.CreateColorCaster()
caster_settings = caster.GetColorCasterSettings()
print("Adding a material caster for channel: "+channel)
caster_settings.SetMaterialChannel(channel)
pipeline.AddMaterialCaster( caster, 0 )
# Finally, if there wasn't any normal map we should add one if it's a remeshing.
if pipeline.IsA("IRemeshingPipeline") and not normal_map_added:
caster = sg.CreateNormalCaster()
caster_settings = caster.GetNormalCasterSettings()
caster_settings.SetGenerateTangentSpaceNormals(True)
caster_settings.SetMaterialChannel(Simplygon.SG_MATERIAL_CHANNEL_NORMALS)
pipeline.AddMaterialCaster( caster, 0 )
def setup_mapping_image(sg, pipeline, scene, textureSize):
mapping_image_settings = pipeline.GetMappingImageSettings()
mapping_image_settings.SetGenerateMappingImage(True)
mapping_image_settings.SetTexCoordName("MaterialLOD")
material_settings = mapping_image_settings.GetOutputMaterialSettings(0)
material_settings.SetTextureWidth(textureSize)
material_settings.SetTextureHeight(textureSize)
if pipeline.IsA("IAggregationPipeline"):
# In aggregation processes we want to keep the orginal UVs and
# merge all textures onto one atlas. This saves UV space if there's
# a lot over overlapping UV charts.
mapping_image_settings.SetTexCoordGeneratorType(Simplygon.ETexcoordGeneratorType_ChartAggregator)
mapping_image_settings.SetApplyNewMaterialIds(True)
mapping_image_settings.SetGenerateTangents(True)
mapping_image_settings.SetUseFullRetexturing(True)
chart_agg_settings = mapping_image_settings.GetChartAggregatorSettings()
# We can play around with this parameter if we want to ditribute the UV space in a different
# way compared to the original.
chart_agg_settings.SetChartAggregatorMode(Simplygon.EChartAggregatorMode_TextureSizeProportions)
# Enable the chart aggregator and reuse UV space.
chart_agg_settings.SetSeparateOverlappingCharts(False)
def process_pipeline(sg, pipeline, scene, output_file):
print("Setting up material casters...")
setup_mapping_image(sg, pipeline, scene, 1024)
add_default_casters(sg, pipeline, scene)
pipeline.RunScene(scene, Simplygon.EPipelineRunMode_RunInThisProcess)
#Export the hollow shell into the output file.
scene_exporter = sg.CreateSceneExporter()
scene_exporter.SetExportFilePath(output_file)
print("Writing output to "+output_file)
scene_exporter.SetScene(scene)
scene_exporter.RunExport()
def process_file(sg, asset_file):
# Load the asset we want to process
scene_importer = sg.CreateSceneImporter()
scene_importer.SetImportFilePath(asset_file)
if scene_importer.RunImport():
# Process the asset
remesh_pipeline = sg.CreateRemeshingPipeline()
remesh_scene = scene_importer.GetScene()
agg_pipeline = sg.CreateAggregationPipeline()
aggregation_scene = scene_importer.GetScene().NewCopy()
print("Running remeshing...")
process_pipeline(sg, remesh_pipeline, remesh_scene, asset_file.split('.')[0]+'_remesh.fbx')
print("Running aggregation...")
process_pipeline(sg, agg_pipeline, aggregation_scene, asset_file.split('.')[0]+'_aggregated.fbx')
else:
print("Failed to load "+asset_file)
def main():
sg = simplygon_loader.init_simplygon()
print(sg.GetVersion())
process_file(sg, "goldmine.fbx")
del sg
if __name__== "__main__":
main()