IEEE Consumer Electronics Magazine - March 2018 - 89

with the geometry. The point of view is that of the user (what
is projected to the HMD).
At the beginning of the procedure, as seen in Figure 4(a), the
user grabs the stylus of the haptic device and freely moves the
virtual replica in the augmented space. The replica is rendered as
a virtual object, and its position/attitude is updated according to
the haptic device input in each frame. The only haptic feedback
in this phase is the force due to the contact between the stylus tip
and the geometry. The force, which is computed according to the
algorithm in the previous section, prevents the interpenetration
between the stylus and the geometry. Once in contact, if the user
moves the tip along the surface, he or she can feel the shape and
its geometrical features with a high level of detail and realism.
On the other hand, the user can also decide to detach from the
surface, and, in this case, the tactile feedback ends.
By pressing a button on the haptic stylus, detailed in Figure
4(b), the user can sketch geometrical points (morphing control
points) on the surface. In this phase, to ensure a correct and
precise snapping between the points and the surface, the haptic
feedback renders a bidirectional force that constrains the stylus
tip on the surface. While the standard unilateral contact interaction produces a reaction forces only for preventing penetration, the snapping forces also prevents detaching. From a
mathematical point of view, this two-side interaction is
achieved imposing (2) without the condition d c $ n c 1 0.
In the third phase, shown in Figure 4(c), the user can
select each sketched point and can displace each along the
relevant direction. The user moves the stylus, which is
attracted by the point using a fictitious force according to (5)
and (6). After pressing the button to confirm the selection,
virtual rulers that guide the displacement of the selected point
along a specific direction are added to the augmented scene.
Virtual ticks also help in the definition of the correct amplitude of the displacement. Both rulers and ticks give a force
feedback, so the user has a tactile perception of their presence. The points can also be moved in a group by defining a
master node, which drives the displacement (with its own
ruler) and a set of slave nodes that follow it.
After the displacement of the points, all of the points of
the initial geometry are morphed by using the entire set of
sketched points as control points, shown in Figure 4(d); this
operation is achieved by updating the coefficient of (9) and
then processing each point to be moved. It is possible to perform synchronous morphing (the RBF morph algorithm is
real-time solved each video) or asynchronous morphing (the
RBF morph algorithm is solved only after the displacement
of all of the control points).

After a morphing act, the user can
review and compare the modified
geometry with respect to the initial
act and set up another morphing
procedure.
After a morphing act, the user can review and compare the
modified geometry with respect to the initial act and set up
another morphing procedure [Figure 4(e)]. The obtained configuration can then be transferred to the CAE setup for multiphysics analysis. The time of CAE evaluation depends on the
computation resources (and high performance computing
power, if any).
To quantitatively evaluate the performance of the methodology, three different mesh sizes of the same geometry have been
processed. The results are summarized in Table 1. The bottleneck of the methodology is the force-feedback processing, which
includes the computational-demanding contact detections between
the various mesh elements and edges. On the other hand, the
morphing algorithm based on the RBF is efficient, and the 1/60-s
frame rate is sufficient to complete both the building of the interpolation function and the update of the mesh. The AR processing is a nearly constant computational burden and is not affected
by the complexity of the mesh.

CONCLUSION
A methodology for performing interactive sculpting of geometrical surfaces has been presented. The methodology
combines the use of AR, haptic rendering, and an RBF morphing algorithm. The proposed application demonstrates
that the integration among different devices is successful
and the computational effort is compatible with a standard
workstation. The use of a morphing strategy based on the
RBF demonstrates that the update of the geometry can be
achieved in real time and does not influence the computational performance. The addition of tactile feedback has tremendously increased the level of realism. A group of ten
different subjects tested the methodology to obtain extended
feedback about performance and usability. None of the subjects possessed past experiences in AR or sculpting, and all
of them properly used the methodology after a few minutes
of training. All participants said that the tactile feedback is
realistic and that the haptic device combined with the

Table 1. A performance comparison among different mesh sizes.
Mesh Size (Number of Nodes)

AR Processing

Force-Feedback Processing

RBF Morphing

500

Real time (< 1/60-s)

5,000

Real time (< 1/60-s)

15,000

Real time (< 1/60-s)

Some lags (~2-s)

Real time (< 1/60-s)

march 2018

IEEE Consumer Electronics Magazine

Table of Contents for the Digital Edition of IEEE Consumer Electronics Magazine - March 2018