Multimodal visualization

1. Built-in visualizations#

SmooSense infers the best way of displaying visual contents based on the column name, type and sample values.

1.1 URL#

A column will be inferred as URL if it is string type and has nonnull sample values starting with http://, https://, s3:// or ./

1.2 ImageUrl, VideoUrl, PdfUrl, AudioUrl#

URL columns have specific rendering based on their file name extension.

1.3 Bbox#

To visualize bounding box:

• Save the bounding box as list of float or integers.
• Make sure the column name contains bbox
• Make sure there is another column named image_url as the background image.

2. Plug in your own visualization#

If the built-in visualization is not enough, you can serve your own visualizer as an API, and integrate into SmooSense using iframe. Strings starting with iframe+https:// or iframe+http:// will be recognized as iframe url.

Here we show several examples:

2.1 Bounding box#

https://cdn.smoosense.ai/viz-bbox.html?
image=https://cdn.smoosense.ai/000000130579.jpg
&bboxes='[{"bbox":[176,187,64,57],"label":"GT"},{"bbox":[162,183,107,68],"label":"pred"}]'
&name=baseball glove
&autorange=true

2.2 Robot motion using SmooViz#

We will be open-sourcing a library for lightweight visualization of robot motion. Here is an example:

https://viz-robot-cdn.smoosense.ai/single.html?
modelName=unitree_g1
&dataUrl=https://viz-robot-cdn.smoosense.ai/unitree_g1/example.json