Human pointing motion during interaction with an autonomous blimp

ABSTRACT We investigate the interaction between a human and a miniature autonomous blimp using a wand as pointing device. The wand movement generated by the human is followed by the blimp

through a tracking controller. The Vector Integration to Endpoint (VITE) model, previously applied to human–computer interface (HCI), has been applied to model the human generated wand

movement when interacting with the blimp. We show that the closed-loop human–blimp dynamics are exponentially stable. Similar to HCI using computer mouse, overshoot motion of the blimp has

been observed. The VITE model can be viewed as a special reset controller used by the human to generate wand movements that effectively reduce the overshoot of blimp motion. Moreover, we

have observed undershoot motion of the blimp due to its inertia. The asymptotic stability of the human–blimp dynamics is beneficial towards tolerating the undershoot motion of the blimp.

SIMILAR CONTENT BEING VIEWED BY OTHERS EVALUATION OF THE INTEL REALSENSE T265 FOR TRACKING NATURAL HUMAN HEAD MOTION Article Open access 14 June 2021 A WEARABLE LIGHT-TOUCH CONTACT DEVICE

FOR HUMAN BALANCE SUPPORT Article Open access 01 April 2021 NONCONTACT HUMAN-MACHINE INTERACTION BASED ON HAND-RESPONSIVE INFRARED STRUCTURAL COLOR Article Open access 18 March 2022

INTRODUCTION Human pointing motion is one of the major means of indicating intentions. Human pointing devices, such as the computer mouse, are ubiquitous in modern computers. Pointing motion

has been leveraged in human–robot interaction (HRI) as a natural and effective way to communicate with robots1,2,3,4,5,6. Unmanned aerial vehicles (UAVs) are gaining popularity for indoor

surveillance, delivery, and warehouse monitoring, where they are required to operate in close proximity to human. In this context, human needs to interact with UAVs in effective ways, and

pointing motion should be considered as a preferred method of interaction. Some recent studies have reported promising results on the interaction between human and quadrotors and

drones7,8,9,10 leveraging pointing motions11,12. However, these interactions are constrained by safety concerns for the human operator, and there is a lack of mathematical modeling for the

pointing device movements generated by the human during the interaction. The Georgia Tech Miniature Autonomous Blimp (GT-MAB) is a lighter-than-air UAV developed for indoor applications in

human-occupied environments. It consists of a saucer-shaped envelope filled with Helium, and a gondola attached to the envelope. The envelope makes GT-MAB naturally cushioned, not posing any

safety threat to human. Furthermore, GT-MAB keeps itself aloft without the need for consistent propulsion, due to the buoyancy of the envelope. Therefore, its endurance can be several

magnitudes longer than that of a heavier-than-air UAV13. Hence GT-MAB is well-suited for carrying out HRI experiments, which often requires the UAV to operate in close proximity to human,

and also desires sustained airborne presence to perform repetitive missions14,15,16,17,18. This paper investigates the interactions between a human and the GT-MAB in close proximity, taking

advantage of the safety and extended flight time. The human uses a marked wand as the pointing device, which is traced by a localization system. A feedback controller on the GT-MAB achieves

tracking of the human pointing motion. Other than our conference paper19, we have not found similar investigations of human interaction with robotic blimps using a pointing device in the

literature. Through experimental data collected, we proceed to analyze the wand motion generated by human to control blimp movements. Previous works11,12 on human–UAV interaction have not

employed mathematical models for human pointing motion. We apply the VITE model, previously applied to model human motion during interaction with a computer mouse20,21,22 to model the wand

movements. The experimental data verifies that the VITE model is able to capture the major features in human generated wand movement during the interaction with GT-MAB. On the other hand,

our study has shown difference between the motion of the GT-MAB and a computer mouse. Due to inertia, the GT-MAB can not react instantaneously when human changes the direction of wand

movement. This has caused the initial response of the blimp movement being away from the target position, which demonstrates a perceivable undershoot in blimp motion. Through theoretical

analysis, we can show that the closed-loop dynamics of human–blimp interaction is exponentially stable. The stability indicates that the undershoot blimp motion can be tolerated by the

human. Furthermore, the GT-MAB tends to bypass the target position more easily than a mouse cursor displayed on a monitor. We discover that the VITE model may serve as a reset controller

used by the human to reduce the overshoot of the blimp motion. This work is a significant extension to our conference paper19, where we show preliminary results on modeling human pointing

motion with the VITE model, and provide stability analysis. This work presents validation of modeling the pointing motion using the VITE model with experiment data collected by more human

subjects. Further, we establish a connection between the reset control method and the VITE model, and discuss the benefits and limitations of the VITE model as a feedback controller with

resetting operator, comparing with a linear feedback control law. These discoveries have not been reported in our conference paper19, as well as the literature reviewed. The rest of the

paper is organized as follows. “Background: VITE model” section provides a brief introduction on the background of modeling human pointing behavior. The problem formulation is presented in

“Problem formulation” section. “Stability analysis” section presents stability analysis of the closed-loop system. Experiment setting and experiment results are shown in “Parameter

identification and validation” section, and the discussion on human behavior analysis is presented in “Human behavior analysis” section. “Conclusion” section describes the conclusion of the

paper based on the experiment results. BACKGROUND: VITE MODEL The VITE model is a second order dynamic model for human pointing motion20. Consider a human controls the position of a pointing

device, such as a computer mouse, which displays a visible pointer such as a cursor on a monitor. We assume the human intends to move the pointer to a desired location. Let _y_(_t_)

represent the position of the pointing device under human control. Let _u_(_t_) represent the perceived position of the pointer, \(r_t\) denote the desired position of the pointer, then

\(r_t - u(t)\) represents a difference vector describing the difference between the displayed pointer position and the desired position. The VITE model describes the motion of the pointing

device as follows: $$\begin{aligned} \left\{ \begin{array}{ll} \dot{\eta }(t) = \gamma (-\eta (t) + r_t -u(t))\\ \dot{y}(t) = g[\eta (t)]^{+}_{d} \end{array}\right. , \end{aligned}$$ (1)

where \(\eta (t)\) represents an internal state describing how the human perceives the difference vector, which cumulatively integrates the difference vectors over time with a constant gain

\(\gamma \). The operator \([\cdot ]^{+}_{d}\) is used to switch the pointing motion off when the displayed pointer bypass its target. It is defined by the following equation

$$\begin{aligned}{}[v]^{+}_{d}= \left\{ \begin{array}{ll} v, &{\quad} {\text{if}} \langle v,d \rangle \ge 0 \\ 0, &{\quad} {\text{otherwise}} \end{array}\right. , \end{aligned}$$ (2)

where _d_ defines the direction from the pointer to the target position at initial time _t_ = 0. The parameter _g_ is called the go signal, which is a feedback gain describing how the

internal state \(\eta (t)\) results in the pointer motion _y_(_t_). The signature property of the VITE model is that the displayed pointer tends to move beyond the target, generating an

overshoot. This has been confirmed to agree with human behaviors by experimental work on human pointing motion22,23. The pointing task is modeled as a feedback control that generates the

wand position as the input to the blimp system. The blimp position is the output that will be controlled to a desired position. This is considered as an output regulation problem for

controller design. The VITE model is viewed as a feedback control law that produces the wand position _y_(_t_) based on the difference vector \(r(t)-u(t)\). In the context of human computer

interface using a computer mouse, the human drives the displayed cursor position _u_(_t_) to the target position \(r_t\), by controlling the position of the mouse _y_(_t_). We notice that

the VITE model bears similarity with the reset controller in control theory literature24,25. Comparing to regular feedback controllers, a reset controller typically defines a reset

condition. At the time that the reset condition is met, the control effort is set to zero. The control effort of the reset controller switches back to non-zero values if the reset condition

is not met. Due to the switching condition in (2), the VITE model can be viewed as a variant of the reset controller. However, the human considers the pointing task fulfilled when the reset

condition is satisfied, and the control effort _y_(_t_) stays at zero for all time after the reset condition is met. It is known that a reset controller might reduce overshoot in the step

response of a feedback control system26. However, such effect has not been reported in human robot interaction through pointing motion. PROBLEM FORMULATION We investigate human pointing

motion when interacting with the GT-MAB. as illustrated by Fig. 1. The human operator observes blimp position and then moves a marked wand with its position followed by the GT-MAB. The human

will control the blimp towards a target position that is unknown to the blimp. We perform a series of pointing experiments where data on the wand movements and blimp movements are

collected. To simplify the dynamics model, we introduce the Assumption 1 in19, that the horizontal position of the target that the human specifies is the same as the horizontal position of

the blimp. Due to this Assumption, we use _y_(_t_) to represent the vertical location of the wand, _u_(_t_) to represent the vertical position of the blimp, and _r_(_t_) to represent the

vertical location of the target. We will then attempt to model the human pointing motion using the VITE model. ASSUMPTION 1 We assume that when interacting with the blimp, the wand movements

can be modeled by (1). REMARK 1 It has been shown that human exhibits similar motions when reaching and pointing with their arms, mouse pointers and other devices27. VITE model has been

adopted for modeling human pointing motion in many existing work on HCI, e.g.21. Hence we assume that the human blimp interaction behavior can be described by the VITE model as well.

Experimental justification of this assumption is provided in Section Parameter identification and validation. From our previous work14, the vertical motion of GT-MAB can be described as

$$\begin{aligned} \begin{aligned} m \ddot{u}(t) + F_z(t) = f_z(t), \end{aligned} \end{aligned}$$ (3) where \(m = m_{RB}+m_{Az}\), \(m_{RB}\) and \(m_{Az}\) are the rigid-body mass of the

blimp and the added mass for the vertical motion. The term \(F_z(t) = D \dot{u}(t)\) where _D_ is the aerodynamic drag coefficient for the vertical movement. As justified in our previous

paper28, _D_ can be well approximated as a positive constant for indoor miniature blimps. The term \(f_z(t)\) represents the thrust force in the vertical direction. Feedback controllers are

implemented to control the position and heading of the blimp. For convenience, GT-MAB keeps the same heading angle through the entire experiment, and the reference setpoint for horizontal

movement is set to a fixed position, so that it will remain at the same horizontal position throughout the experiment. Setpoint of the vertical motion is set to the height of the wand, thus

enabling the blimp to track position of the wand. The height controller can be described as \( f_z(t) = k_p e(t), \) where _e_(_t_) = _y_(_t_) − _u_(_t_) represents the difference between

the height of the blimp and the height of the wand. \(k_p> 0\) denotes the feedback gain. The blimp dynamics and the human pointing dynamics are connected to form a closed-loop feedback

system as shown in Fig. 1. The first objective of this paper is formulated as the following problem: PROBLEM 1 Prove that the closed-loop system formed by the human pointing dynamics and the

blimp dynamics is asymptotically stable. Concerns on stability might not be of high priority for human computer interface because mouse movement is agile. For the GT-MAB, however, due to

inertia and air drags, the bimp might not react instantaneously to wand movement. A theoretical justification of the asymptotic stability of the closed-loop system will ensure that the

human–blimp interaction experiments can be carried with meaningful and predictable results. After collecting the wand movement data from the experiments, we will identify the parameters of

the VITE model for each human subject. The parameters of the VITE model include the unknown gains \(g, \gamma \), and the unknown reference target positions \(r_t\). The problem is

formulated as follows: PROBLEM 2 Given a set of wand trajectories, denoted as _y_(_t_) collected from multiple experiments, identify the unknown VITE parameters \(g, \gamma \), and the

unknown reference signal \(r_t\). Solution to this problem will also be used to justify that the VITE model can capture the features of human pointing motion in the context of human–blimp

interaction. We have noticed that the VITE model can be viewed as a reset controller used by the human to reduce the difference between the blimp position and the target position. Given many

possible feedback controllers that are able to achieve the same goal, we are interested to know why the VITE model was adopted by human. Specifically, we formulate the following problem to

compare the VITE model with a linear feedback law: PROBLEM 3 Identify the benefits and limitations of the VITE model as a reset controller in the context of human blimp interaction,

comparing to a linear feedback controller without the reset operator. STABILITY ANALYSIS In this section, we show that the closed-loop dynamics formed by the VITE model and the blimp

dynamics are asymptotically stable. We assume that the target position is located at the origin, and blimp’s initial position is lower than the target position. Detailed description and

justification of this assumption is provided in19. Given the blimp dynamics described in (3), and the human pointing motion dynamics as in (1), we introduce an augmented state \(\mathbf {x}

\in \mathbb {R}^4\), where elements of \(\mathbf {x}\) are defined as \(\begin{bmatrix} x_1&x_2&x_3&x_4 \end{bmatrix}^T \triangleq \begin{bmatrix} u&\eta&\dot{u}&y

\end{bmatrix}^T,\) then the closed-loop system dynamics can be written as $$\begin{aligned} \begin{aligned} \dot{\mathbf {x}}&= \begin{bmatrix} x_3 \\ -\gamma (x_1+x_2) \\ - \frac{D}{m}

x_3 +\frac{k_p}{m} (x_4-x_1)\\ g x_2 ^+\\ \end{bmatrix}, \end{aligned} \end{aligned}$$ (4) where \(x_2^+\) denotes the non-negative portion of \(x_2\), that is, \(x_2^+ = x_2\) when \(x_2

\ge 0\), and \(x_2 = 0\) when \(x_2 < 0\). The closed loop dynamics has the following equilibrium set: \(E = \{ x_3 = 0, \hbox{ and } \, x_1 = x_4 = -x_2, \hbox{ and } \, x_2 \le 0\}.\)

We will now examine the stability of the closed-loop system dynamics in (4) considering two cases, the first case where \(x_2 \ge 0\), and the second case \(x_2 <0\). In both cases, the

system dynamics is linear, while a switch in system dynamics happens when \(x_2\) goes from \(x_2 > 0\) to \(x_2 < 0\). Define a vector \(\mathbf {z}\in \mathbb {R}^4\) as \(\mathbf

{z} = \begin{bmatrix} x_1 - x_4&x_1 + x_2&x_3&x_2^+ \end{bmatrix}^T.\) In the case where \(x_2 \ge 0\), the dynamics for \(\mathbf {z}\) is $$\begin{aligned} \dot{\mathbf {z}} =

\begin{bmatrix} 0 &{} 0 &{} 1 &{} -g \\ 0 &{} -\gamma &{} 1 &{} 0\\ -\frac{k_p}{m} &{} 0 &{} -\frac{D}{m} &{} 0\\ 0 &{} -\gamma &{} 0 &{} 0 \\

\end{bmatrix} \mathbf {z} := \mathbf {A}_1 \mathbf {z}. \end{aligned}$$ In the case where \(x_2 < 0\), \(z_4 = x_2^+ = 0\). Thus, \(\dot{z}_4 = 0\), and the dynamics of \(\mathbf {z}\)

can be described as $$\begin{aligned} \dot{\mathbf {z}} = \begin{bmatrix} 0 &{} 0 &{} 1 &{} 0 \\ 0 &{} -\gamma &{} 1 &{} 0 \\ -\frac{k_p}{m} &{} 0 &{}

-\frac{D}{m} &{} 0\\ 0 &{} 0&{} 0 &{} 0 \\ \end{bmatrix} \mathbf {z} := \mathbf {A}_2 \mathbf {z}.\end{aligned}$$ For the first case, the following Lemma holds. LEMMA 1 The

closed-loop system \(\dot{\mathbf {z}} = \mathbf {A}_1 \mathbf {z}\) is exponentially stable if \( k_p > \frac{g(D + \gamma m)(\gamma ^2 + D) - \gamma D^2 - m\gamma ^2 D}{D}\). PROOF We

will prove the above lemma using Routh’s stability criterion. The characteristic polynomial is $$\begin{aligned} \begin{aligned} \det (\lambda I - \mathbf {A}_1)&= \lambda ^4 +

\left(\frac{D}{m} + \gamma \right)\lambda ^3 + \left(\gamma \frac{D}{m} + \frac{k_p}{m}\right)\lambda ^2 + \frac{\gamma k_p}{m} \lambda + \frac{g k_p \gamma }{m}. \end{aligned}

\end{aligned}$$ (5) The matrix \(\mathbf {A}_1\) is Hurwitz if the first column of the Routh array is positive. Denote the first column of the Routh array as \(a_0, a_1, b_1, c_1, d_1\).

\(a_0, a_1, b_1\) and \(d_1\) are guaranteed to be positive given any choice of \(D, \gamma , g, k_p\). If \(k_p > \frac{g(D + \gamma m)(\gamma ^2 + D) - \gamma ^2 - m\gamma ^2 D}{D}\),

then \(c_1>0\) and the system is stable by the Routh stability criteria. \(\square \) Now consider the second case where \(x_2<0\). In this case, \(z_4\) has already converge to zero,

we will consider the stability of the subsystem \(\dot{\tilde{\mathbf {z}}} = \tilde{\mathbf {A}}_2 \tilde{\mathbf {z}}\), where \(\tilde{\mathbf {z}} = [z_1, z_2, z_3]^T\), \(\tilde{\mathbf

{A}}_2\) is the third order leading principal submatrix of \(\mathbf {A}_2\). LEMMA 2 The subsystem \(\dot{\tilde{\mathbf {z}}} = \tilde{\mathbf {A}}_2 \tilde{\mathbf {z}}\) is

exponentially stable for all \(H, g, \gamma , k_p >0\). PROOF The characteristic polynomial is \( \det (\lambda I - \mathbf {A}_2) = \lambda ^3 + (\frac{D}{m} + \gamma )\lambda ^2 +

\frac{D \gamma + k_p}{m}\lambda +\frac{ k_p \gamma }{m}. \) For all \(D_{\omega z}, g, \gamma , k_p >0\), the first column of the Routh array is positive. Therefore, the closed-loop

system is exponentially stable. \(\square \) The above two Lemmas lead to the following Theorem. Detailed proof can be found in19. THEOREM 1 For the system dynamics (4), with \(k_p >

\frac{g(D + \gamma m)(\gamma ^2 + D) - \gamma D^2 - m\gamma ^2 D}{D}\), if the states start from \(x_2 > 0\), the states will exponentially converge to the equilibrium set _E_. PARAMETER

IDENTIFICATION AND VALIDATION In this section, we describe the experiment setting for human blimp interaction, and process the experimental data to justify Assumption 1 and to identify the

unknown parameters of the VITE model. The stability of the closed-loop system is verified wit the parameters identified from the experimental data. EXPERIMENT SETTING AND DATA PROCESSING

PARTICIPANTS Five unpaid participants (2 female, mean age 24 years old) are recruited for the study, all with normal eyesight, and all practiced to be familiar with GT-MAB dynamics. TASK We

asked the participant to determine two target positions at different heights at their will, and then drive a GT-MAB from its initial position to the first target position by moving the wand.

Once the participant is satisfied with the blimp position, he/she will change the direction of wand movement and drive the blimp towards the second target position. The participant was

asked to keep repeating this task for 60 s. One video of the experiment can be found at https://youtu.be/4JavPaOVKio. Note that in some of the experiments, Assumption 1 in19 is violated. Due

to the wind disturbance, the blimp may have horizontal displacement, even though a horizontal station keep feedback control law is implemented. FACILITIES The experiment took place in a

space that is 8 m long, 7.5 m wide, and 3.5 m high. Flex13 cameras from OptiTrack are installed on the walls, at about 3.5 m height. Reflective markers are attached on the top of GT-MAB

envelope as well as on the tip of the wand, so that the OptiTrack system can detect and record position of the wand and the blimp. The OptiTrack system captures the trajectory of the blimp

and the wand at 10 Hz sampling frequency. Control commands are sent to the GT-MAB via wireless communication enabled by the XBee device. The blimp controller is implemented using MATLAB at

10 Hz, on a Core i-7 2.93 GHz PC with 16GB RAM. The blimp and wand trajectories are logged as they are captured by the OptiTrack system. Considering the limited coverage of OptiTrack

cameras, the human and the blimp will stay inside a 4 m long, 4 m wide, and 2 m high space during the experiment. The starting position of blimp is about 1 m away from the human, and is

about 1.5 m high, while the horizontal target position of the blimp will be the same as the starting position, and the vertical target position will be between 0.5 m and 2 m. Figure 2 (Left)

shows a demonstration of the experiment settings. DATA PREPROCESSING It has been shown that when human performs the pointing motion, the pointer velocity data is noisy29. To address this

issue, we filter the wand velocity using a Savitzky–Golay filter with a 4th order polynomial and a window size of 81 (8 s), and calculate the smoothed wand trajectory by integrating the

filtered velocity. The smoothed wand trajectory, denoted as \(\tilde{y}(t)\), is used as the input data for solving the parameter identification problem. We divide the smoothed wand

trajectory \(\tilde{y}(t)\) into several going-up and going-down segments, according to the movement of the wand. Since the trial is considered to be finished once the human stops moving the

wand, indicating that the human is satisfied with the blimp height, the time interval of the experiment can be divided into \([T_1^-, T_1^+]\), \([T_2^-, T_2^+], \ldots , [T_{N}^-, T_N^+]\)

based on the up/down motion of the wand, where \(T_{n}^-\) and \(T_{n}^+\) denote the starting and ending time of the _n_th trial, \(T_{n-1}^+ = T_{n}^-\). The ending time of each trial is

the timestep when the wand stops moving, \(\dot{\tilde{y}}(t)=0\). In each set of experiment data, we delete the data points that do not belong to a complete trial. Figure 2 (Right) shows

one set of blimp and wand trajectories divided into up/down sections. PARAMETER ESTIMATION Let \(\Omega _{u}=\{n\in \mathbb {Z}|\dot{\tilde{y}}(t) > 0, t\in [T_{n}^-, T_n^+]\}\) represent

the set of all going-up sections. Similarly, let \(\Omega _{d}=\{n\in \mathbb {Z}|\dot{\tilde{y}}(t) < 0, t\in [T_{n}^-, T_n^+]\}\) denote the set of going-down sections. Let \(r_{t,u}\)

and \(r_{t,d}\) denote the target position of all the going-up sections and going-down sections respectively. We represent the set of unknown parameters in the VITE model as \({\Phi } =

[\hat{g}, \hat{\gamma },\hat{r}_{t,u}, \hat{r}_{t, d}]\). Given a set of parameters \(\Phi \) as well as the actual blimp trajectory _u_(_t_) from the experiment, a simulated wand trajectory

\(\hat{y}(t)\) can be derived from the following initial value problem, where the wand trajectory is simulated by the VITE model. Further, at the start and the end of each segment, since

the human stops moving the wand, the human internal state must be zero. $$\begin{aligned} \begin{aligned} \dot{\hat{\eta }}(t)&= \left\{ \begin{array}{c l} \hat{\gamma }(-\hat{\eta }(t)

+ \hat{r}_{t,u} - u(t)),\quad {\text {if}}\, n\in \Omega _{u}\\ \hat{\gamma }(-\hat{\eta }(t) + \hat{r}_{t,d} - u(t)), \quad {\text{if}}\, n\in \Omega _{d}\\ \end{array}\right. ,\\

\dot{\hat{y}}(t)&= \hat{g}[\hat{\eta }(t)]_d^{+}, \\ \hat{y}(0)&= \tilde{y}(0), \\ \hat{\eta }(T_{n}^-)&= 0, n\in \Omega _{\rm u} \cup \Omega _{\rm d}. \end{aligned}

\end{aligned}$$ (6) Hence we formulate the parameter identification problem as a constrained optimization problem. The optimization problem is minimizing the difference between the simulated

and the smoothed wand wand trajectory. To solve the optimization problem, we convert the terminal constraint that the human internal state is zero to a penalty term in the cost function, as

follows: $$\begin{aligned} \begin{aligned}{}&\min _{{\Phi }} \sum _{n\in \Omega _{u}}{\int _{T_{n}^-}^{T_n^+}{(\tilde{y}(t) - \hat{y}(t))^2 d t}} + \sum _{n\in \Omega _{d}}{\int

_{T_{n}^-}^{T_n^+}{(\tilde{y}(t) - \hat{y}(t))^2 d t}} + \beta \sum _{n\in \Omega _{u}\cup \Omega _{d}} \hat{\eta }(T_n^+)^2 \\&{\text {s.t. }} \dot{\hat{\eta }}(t) = \left\{

\begin{array}{ll} \hat{\gamma }(-\hat{\eta }(t) + \hat{r}_{t,u} - u(t)), &\quad {\text {if}}\, n\in \Omega _{u}\\ \hat{\gamma }(-\hat{\eta }(t) + \hat{r}_{t,d} - u(t)) &\quad {\text

{if}}\, n\in \Omega _{d}\\ \end{array}\right. ,\\&\quad \ \dot{\hat{y}}(t) = \hat{g}[\hat{\eta }(t)]_d^{+}, \\&\quad \ \hat{y}(0) = \tilde{y}(0), \\&\quad \ \hat{\eta }(T_{n}^-)

= 0, n\in \Omega _{\rm u} \cup \Omega _{\rm d}, \end{aligned} \end{aligned}$$ (7) where \(\beta \in \mathbb {R}\) is a parameter chosen to balance the penalty term and the modeling error. We

set \(\beta = 50\), and solve the above optimization problem using the interior point method30. Since the cost function is non-convex with respect to the parameters, we apply the global

optimization technique31 to avoid the solution converging to a local minimum. The identified parameters are given in Table 1. We identify the VITE model parameters \(\hat{\gamma }, \hat{g}\)

for both participants. The two participants perform the pointing motion with different model parameters. Further, the identified parameters of each of the participants in different trials

show consistency, which is indicated by the small variance of the identified parameters in different experiments. JUSTIFICATION OF ASSUMPTION 1 To verify whether the VITE model is applicable

to human blimp interaction, we divide the up/down sections into two sets, the training set and the testing set. The training set is the set of complete going-up and going-down segments in

the first 35 s, while the test set contains the rest of the trajectory. We first compare the smoothed wand trajectory in experiment with the wand trajectory simulated by the VITE model with

the identified parameters. Since the VITE model is a second order dynamical system, we also compare the smoothed wand velocity in experiment with the VITE model simulated wand velocity.

JUSTIFICATION USING THE WAND POSITION DATA The Root Mean Square Error (RMSE) between the true and the simulated wand trajectory and its variance is given in Table 1. For the two

participants, the RMSE for the training set is less than 7.2% of the total change-of-height of the blimp throughout the experiment. The RMSE of the test set is larger than the training set,

and is about 11.6% of the total change-of-height of the blimp. The small RMSE and the low variance indicate that the VITE model can accurately describe the human blimp interaction, and it

can correctly predicts the human behavior. Figure 3 (Left) shows one set of comparison between the measured and the reconstructed wand trajectory. It can be seen that the two trajectories

match well in both the training and the test set. This indicates that the VITE model simulated trajectory is able to describe and predict the human pointing behavior in human blimp

interaction. JUSTIFICATION USING THE WAND VELOCITY DATA We also provide justification of the VITE model by computing the RMSE between the true and simulated wand velocity. The RMSE is given

in Table 1. The RMSE for the training set is less than 8.5% of the maximum wand velocity throughout the experiment, while the RMSE is less than 15.2% of the maximum wand velocity in the test

set. Figure 3 (Right) shows one set of the measured, smoothed and simulated wand velocity, where the simulated wand velocity is computed by \(\hat{g}\eta (t)\). It can be seen that the

simulated wand velocity matches with the smoothed wand velocity, which shows that the VITE model can qualitatively reproduce the wand velocity. The VITE model matches with the wand velocity

better at the beginning of each segment, in the acceleration phase. There is relatively larger error when the blimp is about to reach the target position, and the human decelerates the wand.

One possible explanation for the modeling error in the deceleration phase is that in the experiment, human is not given a specific target position. Hence the participant does not pay

attention to the accuracy of the pointing motion, and tends to drive the blimp towards the target with higher speed than the VITE model predicts. From the error analysis, we observe that the

human blimp interaction can be described and predicted by the VITE model. This is an expected outcome since the VITE model describes a general relationship between the distance from the

target to the pointer and the neural commands of muscles contraction for human pointing motion in different interfaces. HUMAN BEHAVIOR ANALYSIS In this section, we describe human and blimp

behaviors during the experiments. We will compare these behaviors with the behaivors observed in human computer interface experiments. In order to describe observations of the experiment, we

use the following visualization techniques in existing literature on HCI: TIME-SERIES PLOTS We plot the wand position and velocity over time, to describe the human behavior in time. PHASE

SPACE PLOTS The phase space trajectory describes the evolution of the system state in the VITE model with respect to the system input, which is the blimp position. WAND AND BLIMP MOVEMENTS

Figure 2 (Right) shows one set of experimental data. We observe that the human moves the wand towards the target with accelerated motion at the beginning. As the blimp goes near the target

position, the human slows down the pointing motion, until he/she is satisfied with the blimp position. After the human stops moving the wand, the blimp slows down, and keeps moving up/down

for a short period of time before its speed reduces to zero. As shown in Fig. 4, comparing to the identified target position with the blimp trajectory, the stopping position of the blimp in

each going up/down segment goes over the identified target position. This overshoot can be explained by the VITE model. Let’s take one going-up segment as an example. Suppose the internal

state has not reached zero, then the human internal state can be described by integrating the system input, $$\begin{aligned} \eta (t) = \int \limits _{0}^t \gamma \exp (-\gamma (t-\tau ))

(r_t - u(\tau ))d\tau . \end{aligned}$$ (8) Suppose at time step \(t'\), \(u(t')\) reaches \(r_t\). This is the timestep that the input to (8) goes to zero, meaning that the human

stops accelerating the wand. However, at this timestep, \(\dot{y}(t') = g [\eta (t')]_d^+>0\), since the human acceleration is positive, \(r_t - u(t)>0, \forall t\in

[0,t')\). Hence the human does not stop the wand movements when the blimp reaches the target position. Instead, when human stops moving the wand, the blimp has already moved over the

target position. This explains the overshoot observed in Fig. 4. Moreover, we observe an “undershoot”in the blimp trajectory. Figure 5 (Left) shows the trajectories of the closed-loop

pointing system. The human internal state is plotted against the blimp position. From the plot, It can be seen that the blimp first move in the opposite direction of the target position,

i.e., in the going-up segments, the blimp first goes down before going up towards the equilibrium set. The system state first moves in the “wrong”initial direction, but then it eventually

reverses course and reaches the desired steady state. As shown in Fig. 5 (Right), since the blimp is a second-order system, direction change will take more time than a computer mouse. Hence,

after the human stops the wand and switches to the next target, the blimp takes additional time to reduce its speed to zero, before changing its direction to go to the next target. At the

initial time of each going-up/down segment, the initial condition of the blimp is not zero. The wrong initial direction of the blimp is the natural response of the blimp driven by the

system’s initial states. Since the closed-loop system is exponentially stable, the blimp catches up with the wand trajectory after a short period of time, as shown in Fig. 5 (Right). The

initial response in the “wrong”direction is an undesirable effect in practice. In human blimp interaction experiments, the user might get confused whether the blimp has finished reaching the

previous target, and may suspect that the blimp is malfunctioning. In such cases, incorporating feedback from the blimp in reaction to the human user may provide better interaction

experience18. BENEFIT AND LIMITATION OF THE VITE MODEL We refer to the VITE model without the reset term as the baseline controller, \( \ddot{y}(t) = g\gamma (-\eta (t) + r_t -u(t)).\) It is

easy to prove that the equilibrium of the closed-loop system formed by the base controller and the blimp is the origin. When the blimp reaches the target position, we have \(u = r_t\). In

this section, we aim to identify the benefits and limitations of the VITE model by comparing it against the baseline controller. Compared with the baseline controller, the major benefit of

the VITE model is that it reduces the overshoot of blimp motion. Figure 6 shows comparison between the tracking error of using the VITE model and the baseline controller to track a fixed

target position. Under the VITE control, the blimp motion settles at the equilibrium much faster than the baseline controller, with significantly less overshoot. It has been theoretically

justified that the reset controller reduces overshoot and settling time of a system under control25. However, the equilibrium of the closed-loop system formed by the VITE model and the blimp

is not the origin. Once the reset condition \(\eta = 0\) is satisfied, the human perceives the task finished, and stops wand movement. Therefore, the VITE model will not drive the tracking

error to zero. On the other hand, the baseline controller will drive the blimp position to the origin, achieving better accuracy than using the VITE model. The use of the reset controller

indicates that when interacting with the blimp, human prefers to fulfill the task in shorter time with less overshoot while sacrificing accuracy. This phenomenon is also seen in pointing

motion across various interfaces, i.e. HCI using a mouse29. The human adopts a two-phase mechanism to fulfill the pointing motion, the surge phase and the corrective phase. The surge phase

denotes the initial movement towards the target. In this phase human points in an accelerated motion. After the pointer is adjacent to the target, human uses a slower corrective motion to

let the tracking error reach zero32. CONCLUSION We investigate interaction between a human user and an autonomous blimp by letting the human control the position of the blimp through wand

movements. We verify that the VITE model can describe human generated wand motion when interacting with the blimp. We show that the closed-loop system describing the human–blimp interaction

is exponentially stable. The exponential stability tolerates the undershoot behavior of the blimp caused by its inertia. Moreover, the study suggests that the VITE model, as a special reset

controller, reduces the overshoot of the blimp motion in human–blimp interaction. ETHICS APPROVAL This study was performed with approval granted by office of research integrity assurance,

Georgia Institute of Technology. All experiments were performed in accordance with relevant guidelines and regulations. CONSENT TO PARTICIPATE Informed consent was obtained from all

individual participants included in the study. CONSENT TO PUBLISH The authors affirm that human research participants provided informed consent for publication of Figs. 1, 2(Left), 3, 5, and

video https://www.youtube.com/watch?v=4JavPaOVKio. DATA AVAILABILITY The datasets generated during and/or analysed during the current study are available in the human–blimp-interaction

repository, https://github.com/mengxueHou/. REFERENCES * Gromov, B., Abbate, G., Gambardella, L. M. & Giusti, A. Proximity human-robot interaction using pointing gestures and a

wrist-mounted IMU. In _2019 International Conference on Robotics and Automation (ICRA)_, 8084–8091 (IEEE, 2019). * Tölgyessy, M. _et al._ Foundations of visual linear human–robot interaction

via pointing gesture navigation. _Int. J. Soc. Robot._ 9, 509–523 (2017). Article  Google Scholar  * Sprute, D., Tönnies, K. & König, M. A study on different user interfaces for

teaching virtual borders to mobile robots. _Int. J. Soc. Robot._ 11, 373–388 (2019). Article  Google Scholar  * Saunderson, S. & Nejat, G. How robots influence humans: A survey of

nonverbal communication in social human–robot interaction. _Int. J. Soc. Robot._ 11, 575–608 (2019). Article  Google Scholar  * Van de Perre, G. _et al._ Studying design aspects for social

robots using a generic gesture method. _Int. J. Soc. Robot._ 11, 651–663 (2019). Article  Google Scholar  * Liu, P., Glas, D. F., Kanda, T., Ishiguro, H. & Hagita, N. A model for

generating socially-appropriate deictic behaviors towards people. _Int. J. Soc. Robot._ 9, 33–49 (2017). Article  Google Scholar  * Pfeil, K., Koh, S. L. & LaViola, J. Exploring 3D

gesture metaphors for interaction with unmanned aerial vehicles. In _Proceedings of International Conference on Intelligent user interfaces_, 257–266 (2013). * Bolin, J. _et al._

Gesture-based control of autonomous UAVs. In _Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems_, 1484–1486 (2017). * Rajappa, S., Bülthoff, H. & Stegagno,

P. Design and implementation of a novel architecture for physical human–UAV interaction. _Int. J. Robot. Res._ 36, 800–819 (2017). Article  Google Scholar  * Ng, W. S. & Sharlin, E.

Collocated interaction with flying robots. In _IEEE International Symposium on Robot and Human Interactive Communication_, 143–149 (2011). * Peshkova, E., Hitz, M. & Kaufmann, B. Natural

interaction techniques for an unmanned aerial vehicle system. _IEEE Pervasive Comput._ 16, 34–42 (2017). Article  Google Scholar  * Gromov, B., Guzzi, J., Gambardella, L. M. & Giusti,

A. Intuitive 3D control of a quadrotor in user proximity with pointing gestures. In _2020 IEEE International Conference on Robotics and Automation (ICRA)_, 5964–5971 (IEEE, 2020). * Sebbane,

Y. B. _Lighter than Air Robots_ (Springer, 2012). Book  Google Scholar  * Tao, Q. _et al._ Swing-reducing flight control system for an underactuated indoor miniature autonomous blimp.

_IEEE/ASME Trans. Mechatron._ 26, 1895–1904 (2021). Article  Google Scholar  * Cho, S. _et al._ Autopilot design for a class of miniature autonomous blimps. In _IEEE Conference on Control

Technology and Applications, Kohala Coast, Hawaii_ (2017). * Tao, Q., Cha, J., Hou, M. & Zhang, F. Parameter identification of blimp dynamics through swinging motion. In _2018 15th

International Conference on Control, Automation, Robotics and Vision_, 1186–1191 (2018). * Yao, N. _et al._ Monocular vision-based human following on miniature robotic blimp. In _Proceedings

of 2017 American Control Conference_, Seattle, WA (2017). * Yao, N. _et al._ Autonomous flying blimp interaction with human in an indoor space. _Front. Inf. Technol. Electron. Eng._ 20,

45–59 (2019). Article  Google Scholar  * Hou, M., Tao, Q., Varnell, P. & Zhang, F. Modeling pointing tasks in human–blimp interactions. In _2019 IEEE Conference on Control Technology and

Applications (CCTA)_, 73–78 (IEEE, 2019). * Bullock, D. & Grossberg, S. Neural dynamics of planned arm movements: Emergent invariants and speed-accuracy properties during trajectory

formation. _Psychol. Rev._ 95, 49 (1988). Article  CAS  Google Scholar  * Varnell, P., Malisoff, M. & Zhang, F. Stability and robustness analysis for human pointing motions with

acceleration under feedback delays. _Int. J. Robust Nonlinear Control_ 27, 703–721 (2016). Article  MathSciNet  Google Scholar  * Martín, J. A. Á., Gollee, H., Müller, J. & Murray-Smith,

R. Intermittent control as a model of mouse movements. _ACM Trans. Comput. Human Interact. (TOCHI)_ 28, 1–46 (2021). Article  Google Scholar  * Müller, J., Oulasvirta, A. &

Murray-Smith, R. Control theoretic models of pointing. _ACM Trans. Comput. Human Interact. (TOCHI)_ 24, 1–36 (2017). Article  Google Scholar  * Horowitz, I. & Rosenbaum, P. Non-linear

design for cost of feedback reduction in systems with large parameter uncertainty. _Int. J. Control_ 21, 977–1001 (1975). Article  Google Scholar  * Beker, O., Hollot, C., Chait, Y. &

Han, H. Fundamental properties of reset control systems. _Automatica_ 40, 905–915 (2004). Article  MathSciNet  Google Scholar  * Beker, O., Hollot, C. V. & Chait, Y. Plant with

integrator: An example of reset control overcoming limitations of linear feedback. _IEEE Trans. Autom. Control_ 46, 1797–1799 (2001). Article  MathSciNet  Google Scholar  * Shadmehr, R.

& Wise, S. P. _The Computational Neurobiology of Reaching and Pointing_ (MIT Press, 2005). Google Scholar  * Tao, Q., Hou, M. & Zhang, F. Modeling and identification of coupled

translational and rotational motion of underactuated indoor miniature autonomous blimps. In _2020 16th International Conference on Control, Automation, Robotics and Vision (ICARCV)_, 339–344

https://doi.org/10.1109/ICARCV50220.2020.9305371 (2020). * Müller, J. Dynamics of pointing with pointer acceleration. In _IFIP Conference on Human–Computer Interaction_, 475–495 (Springer,

2017). * Kim, S.-J., Koh, K., Lustig, M., Boyd, S. & Gorinevsky, D. An interior-point method for large-scale \(l_1\)-regularized least squares. _IEEE J. Sel. Top. Signal Process._ 1,

606–617 (2007). Article  ADS  Google Scholar  * Ugray, Z. _et al._ Scatter search and local NLP solvers: A multistart framework for global optimization. _INFORMS J. Comput._ 19, 328–340

(2007). Article  MathSciNet  Google Scholar  * Aranovskiy, S., Ushirobira, R., Efimov, D. & Casiez, G. A switched dynamic model for pointing tasks with a computer mouse. _Asian J.

Control_ 22, 1387–1400 (2020). Article  MathSciNet  Google Scholar  Download references FUNDING The research work is supported by ONR grants N00014-19-1-2556 and N00014-19-1-2266; NSF grants

OCE-1559475, CNS-1828678, and S&AS-1849228; NRL grants N00173-17-1-G001 and N00173-19-P-1412; and NOAA grant NA16NOS0120028. AUTHOR INFORMATION AUTHORS AND AFFILIATIONS * Georgia

Institute of Technology, School of Electrical and Computer Engineering, Atlanta, GA, 30308, US Mengxue Hou, Qiuyang Tao & Fumin Zhang Authors * Mengxue Hou View author publications You

can also search for this author inPubMed Google Scholar * Qiuyang Tao View author publications You can also search for this author inPubMed Google Scholar * Fumin Zhang View author

publications You can also search for this author inPubMed Google Scholar CONTRIBUTIONS All authors contributed to the study conception and design. M.H., Q.T., and F.Z. all performed data

collection and analysis. The first draft of the manuscript was written by M.H. and all authors commented on previous versions of the manuscript. All authors read and approved the final

manuscript. CORRESPONDING AUTHOR Correspondence to Fumin Zhang. ETHICS DECLARATIONS COMPETING INTERESTS The authors declare no competing interests. ADDITIONAL INFORMATION PUBLISHER'S

NOTE Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. SUPPLEMENTARY INFORMATION SUPPLEMENTARY INFORMATION. RIGHTS AND

PERMISSIONS OPEN ACCESS This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any

medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The

images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is

not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission

directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. Reprints and permissions ABOUT THIS ARTICLE CITE THIS ARTICLE Hou, M.,

Tao, Q. & Zhang, F. Human pointing motion during interaction with an autonomous blimp. _Sci Rep_ 12, 11402 (2022). https://doi.org/10.1038/s41598-022-15016-w Download citation *

Received: 19 February 2022 * Accepted: 16 June 2022 * Published: 06 July 2022 * DOI: https://doi.org/10.1038/s41598-022-15016-w SHARE THIS ARTICLE Anyone you share the following link with

will be able to read this content: Get shareable link Sorry, a shareable link is not currently available for this article. Copy to clipboard Provided by the Springer Nature SharedIt

content-sharing initiative