Internet of Things for beyond-the-laboratory prosthetics research

Research on upper-limb prostheses is typically laboratory-based. Evidence indicates that research has not yet led to prostheses that meet user needs. Inefficient communication loops between users, clinicians and manufacturers limit the amount of quantitative and qualitative data that researchers can use in refining their innovations. This paper offers a first demonstration of an alternative paradigm by which remote, beyond-the-laboratory prosthesis research according to user needs is feasible. Specifically, the proposed Internet of Things setting allows remote data collection, real-time visualization and prosthesis reprogramming through Wi-Fi and a commercial cloud portal. Via a dashboard, the user can adjust the configuration of the device and append contextual information to the prosthetic data. We evaluated this demonstrator in real-time experiments with three able-bodied participants. Results promise the potential of contextual data collection and system update through the internet, which may provide real-life data for algorithm training and reduce the complexity of send-home trials. This article is part of the theme issue ‘Advanced neurotechnologies: translating innovation for health and well-being’.


Introduction
The loss of the upper limb affects three million people all over the world [1,2]. People with upper-limb difference

Methods
The system developed in this study includes the upper limb prosthesis in an IoT architecture, as shown in figure 1. The prosthesis controller and the smart devices (PC and mobile) are with the user to control the upper-limb prosthesis and collect control signals and contextual information. They can work independently without internet connection, but connecting the device to the cloud server allows the device to upload the collected data and receive system updates through Wi-Fi communication. The cloud server is developed on the Thinger.io platform to allow remote data collection, storage and visualization as well as the control and update of devices. Users and clinicians can access to the server anywhere by logging into its web portal, so the experiments are no longer limited in the laboratory because of the hardware constraints. More details about each module in the schematic diagram are introduced in the following sections.

(a) System description (i) Hardware
The prosthesis controller was implemented on an Arduino MKR WIFI 1010 (Arduino LLC, USA) because of its low-cost and accessibility [24]. This system enables two-channel myoelectric recording with a 500 Hz sampling rate, signal pre-processing with bandpass filters and notch filter, abstract decoding for real-time prosthesis control. For this work, we used the smoothed envelop of the EMG signal, which is of the order of a few Hz. As such, a lower sampling rate, violating the Nyquist frequency, does not contaminate the data by aliasing [30]. Besides, a wireless communication module was developed based on the Protoson (PSON) protocol [31], which enabled the bi-directional binary data transmission between the server and the device. Furthermore, a micro-SD shield (Arduino MKR SD Proto Shield) and a real-time clock module (HW-084) were connected to the system for data storage and synchronization. While the prosthesis was used, control data were logged to the micro-SD card with timestamps, so users could save the data onboard if Wi-Fi communication was not available. A prototype of the internet-enabled prosthesis controller is shown in figure 2a.
We three-dimensionally printed a by-pass socket (figure 2b). It included a cuff and a twopart quick wrist disconnect (QWD) housing. The cuff was printed flat and thermoformed to encapsulate the lower forearm using a heat gun. The print orientation of the cuff gave it slight intentional flexibility to allow the brace to fit all participants easily.  added to the top of the cuff. The cuff also features an attachment site for the QWD housing to hold the Robo-limb (Ossur, Iceland). The two housing parts were printed separately and affixed to the cuff with bolts. Batteries and the prosthesis controller were placed behind the QWD housing to reduce the length of the cables. An overview of the hardware is shown in figure 2c.

(ii) Prosthesis control algorithm
We could use any prosthesis control method to demonstrate this IoT-based setting. In this proof of principle study, for the first time, we implemented the abstract control method [32,33] for the control of a prosthesis in a real-time experiment. With the support of a computer-based myoelectric interface, users can learn novel ways to map their muscle activity to commands to actuate a prosthesis. We previously described the interface in [10] and [33]. For completeness, we briefly describe the controller here. Figure 2d shows a circular cursor on a two-dimensional task space. The position of the cursor is determined by the mean-absolute value (MAV) of the myoelectric signals from two muscles. Participants can move the cursor from the lower basket (rest state) to a target and hold the cursor within the target for a fixed dwell period to activate a preset grasp on the prosthesis. We assigned power, tripod/pinch and point grasps, and hand open to targets one to four, respectively. We set the state machine such that the prosthesis can accept a new grasp command only when the hand is in the open state and the cursor is in the rest basket. This way we removed the possibility of inadvertent grasps.

(iii) IoT cloud portal and server
The cloud server was developed using the Software as a Service (SaaS) model [34]. It was deployed on the Thinger.io platform [31] that handled data collection, data visualization and device reconfiguration. All registered stakeholders can access the server through a web browser. No local software installation was required. To satisfy the requirements of different stakeholders, the platform offers the possibility of stakeholder-specific dashboards at various granularity. Specifically, two web-based dashboards, one for a user and the other for an experimenter, were developed using the representational state transfer (REST) application programming interface and were saved as templates. We could include new members in the experiment by simply applying the template to their accounts or by personalizing it according to user preferences. The data were securely stored on an Amazon DynamoDB database and separated into virtual storage spaces called Data Buckets. For privacy and security reasons, data buckets were created and managed for each user individually. When a prosthesis controller was connected to the server, its control data were synchronized to the user's data buckets automatically. Users could log into their own accounts to monitor and reset their prostheses, while the clinician could read, modify and download data from users they were responsible for.

(b) Participants and preparation
Three able-bodied participants took part in this proof-of-principle study. They were free from any known neurological or motor disorders. They all had extensive experience of computer-based abstract myoelectric control. This study was their first experience of wearing a prosthesis by-pass socket and controlling a commercial myoelectric prosthesis. Myoelectric sensors were placed on the skin, targeting the flexor carpi radialis (FCR) and extensor carpi radialis (ECR) muscles. Normalized mean-absolute values of the myoelectric signals were calculated and used as control signals following the method described in [33]. Before the experiment and as a familiarization block, participants wore the socket and the prosthesis and practised with the cursor control on the computer interface. The power supply of the prosthesis was off during this block. This way we ensured that the familiarization block is representative of the main study, in terms of the load on the arm.
The familiarization block comprised two stages. The first stage included a practice run and then 40 trials. First, participants practised the movement of the myoelectric cursor in the abstract decoding task space (figure 2d) for 2 min. After that, in each trial, a target was selected and highlighted on the interface, and participants moved the cursor from the basket to hit the selected target within 1.5 s. Participants were instructed to achieve as many trials as they could. They needed to achieve more than 70% trials for each target in order to clear the stage. Throughout this stage, they could see the cursor on the computer screen. At the end of the stage, participants had a 2 min or longer break to avoid fatigue. They were instructed to place their forearm on their legs or the handle of the experimental chair to rest their muscles. The second stage closely followed the first but with the difference that the cursor was no longer visible. The second stage mimicked the prosthesis control in a practical scenario.

(c) Functional assessment
A real-time control pick and place experiment was carried out to investigate the functionality and robustness of the proposed system. The prosthesis controller was connected to the server and data were recorded. This study comprised 16 blocks, comprising two groups of eight. In each block, participants were required to grasp, lift and relocate a series of objects. They were also required to press the touchpad of the experimental laptop with a point grip to indicate the start and stop of the block. Only in one block, a participant did not click the touchpad properly. This block was removed from the analysis. In the first eight blocks, the objects were a compact disc and a drink can. As such, the grasp order was point, tripod, power and point. The order in which they picked the disc and the can was decided by the participant. In the second eight blocks, the objects were a ball and the same drink can. The grasp order in the second eight block was point, pinch, power and point. Similarly, the order in which they picked the ball and the can was decided by the subjects. Note that the change from tripod to pinch grip on the prosthesis was carried out on the server remotely, during the experiment, demonstrating the flexibility of the proposed system. If participants lifted an object with a grasp that was not associated with the object, they had to submit user feedback to the server using the dashboard to report the grasp they intended to use. To avoid fatigue, due to bearing the weight of the hand, a 1 min break was included between each block, during which participants could put down their arms to rest the muscles. Participants could ask for a longer rest if they still required more rest.

(d) Data analysis and system update based on real-life data
The data buckets were exported to the analysis laptop. For each participant, the myoelectric control signals were synchronized with the motor commands and the user feedback based on the timestamps. Motor commands without unexpected grasp labels were considered as the correct commands. Also, we evaluated the feasibility of updating the control algorithm based on reallife data. The correct motor commands in the pick and place experiment were used as the training data. Their angle v from cursor positions was extracted to train three 2-class Naive Bayes classifiers. The optimized decision boundaries between each two adjacent targets located at the position where the angle v has equal probability density between two targets. The new decision boundaries were verified by the unexpected motor commands.

Results (a) Online dashboards for device control and data visualization
The user interface, user feedback dashboard and clinician interface were developed for this study. Each prosthesis controller has an independent user interface and a user feedback dashboard for data visualization, device control and contextual data reporting.
A user interface (figure 3a) includes a prosthesis monitor and an MCI, which shows the current status of the upper-limb prosthesis and the corresponding control signals in real time. It provides the visual feedback of the muscle contraction when the user is trying to control the prosthesis, so it may be used for in-home myoelectric training and the evaluation of the prosthesis settings, such as the calibration levels. The recalibration switch on the screen allows the user to recalibrate the corresponding device following our standard calibration protocol [10]. Regarding the ethical considerations, such as privacy, the user can stop sharing the data with the server by turning off the Data streaming and MAV streaming switches, which stops the device from recording control data and EMG signals but maintain its functions in prosthesis control.
A user feedback dashboard (figure 3b) allows the user to upload the contextual information when they are using the prosthesis. The user presses the four buttons on the top of the dashboard to report which grip they intend to select through the previous muscle contraction, and our system will label the selected grip to the corresponding MAV data. In this way, the user can provide labels for the practical muscle contractions during daily activities and we are able to collect the information remotely for user behaviour analysis and algorithm training. There is a questionnaire on the dashboard for the user to provide contextual information, such as where they are using the prosthesis. As the pick and place experiment was carried out in the laboratory, we did not require the participants to use this function during the tests.
The clinician interface (figure 3c) can be used to monitor and manage all prosthetic devices. It has time-series charts to show the motor commands, user feedback and corresponding timestamps from the data buckets for each device, so when the prostheses have been used and what activities they have been used for can be monitored remotely. The scale of the time-series charts is adjustable, so it can be used to analyse grasp selection for a specific task or to monitor the time of device usage in a longer term. The control panel next to the time-series charts can change the device settings for the selected participant, so the device reconfiguration can be carried out without physical access to the device.

(b) Contextual data collection and synchronization
Data stored in the data bucket can be both visualized on the dashboards and downloaded for offline analysis in CSV, ARFF or JSON formats. Figure 4 shows an example of a pick and place trial operated by one of the participants. After clicking the keyboard with the point grip, the participant picked up the can and relocated it to the target label using the power grip. When picking up the disc, the prosthetic hand grasped with the point grip, which was not expected by the participant. They therefore submitted feedback to the server with their smartphone in the left hand, and then finished the rest of the trial following the instructions. With the timestamps, MAV signals, motor commands and user feedback can be synchronized, as in figure 4a. The user feedback was automatically labelled to its previous motor command as the grip that the participant intended to make. The corresponding cursor trace and the cursor position where the decision has been made (figure 4b) were saved as a new dataset so that prosthetic control data during practical activities can be labelled and collected remotely. Representatives for successful trials and also trials in which the participants provided feedback can be found in the electronic supplementary material, video.

(c) Offline data analysis and system customization
Based on the experimental data, we trained the system to fit the participants' short-term adaptation to the abstract myoelectric controller. The decision-making points for grasp commands (figure 4c) were decoded to its phase angle in the MCI (figure 4d   Table 2 shows the number of unexpected grip feedback that locate the correct target if customized Bayes decision boundaries are applied.

Concluding remarks
Because of the need for unsupervised real-life data [35,36], the number of beyond-laboratory upper-limb prosthetic studies beyond the laboratory has been gradually increasing over the   [37]. These studies collected questionnaires, wear-time, grip count or video clips to investigate prosthetic interventions, device performances and user perspectives in home environments [13,38,39]. However, to fully understand and improve an advanced control algorithm for daily practice, more detailed control data and contextual information are necessary.
In this paper, we demonstrated a proof-of-principle study of applying IoT for prosthetic research beyond the laboratory. We demonstrated the system by a laboratory-based study, but the data collection, device monitoring and system updates were all carried out through Wi-Fi communication. Therefore, it is feasible to repeat the experiments in community settings without any system modification, as we proposed. The developed system saves the prosthetic control data to an online server and visualizes the data in real time through a web-based dashboard. User feedback forms and control panels on the dashboards allow the collection of contextual information and system reconfiguration over the air. Three able-bodied participants participated in the experiments. The experimenter could monitor the data in real time and reconfigure the device remotely.
The current version of this system saves timestamped myoelectric control signals, grasp commands and user intents, which will be sufficient for understanding how often and how well the controller is used in real-life settings. Future development may include expansion of the storage capacity to enable storing raw myoelectric and other sensor signals such as force and accelerometry. Transmitting raw signals in real time may require larger bandwidth and its necessity remains to be established. The raw myoelectric signals allow the experimenter to playback the muscle activity during at-home use and develop comprehensive datasets for further analysis, e.g. for machine learning-based decoding [40,41]. Meanwhile, transmitting the raw data will increase the resource consumption, in terms of bandwidth and storage space, but a large portion of them may not be used in scientific research due to the lack of contextual information [42]. Moreover, raw data transmission increases the power consumption, and it will significantly reduce the battery life of the device [43]. Considering these factors, further discussion may be required to determine whether it is worthwhile to transmit raw data or to convert myoelectric signals into features before transmission especially for high-density EMG settings [44][45][46]. If raw EMG signals are indispensable, they can be saved on the on-board SD card and uploaded to the server periodically. The proposed hardware is capable of the latter.
We respected data security and user privacy in our system design. Following the recommendations of [47], we adopted the default Thinger.io data security system design. First, each prosthesis controller had its ID and device credentials and needed to authenticate itself to the server before collecting or sending data. Besides, data were transmitted securely using the Transport Layer Security protocol. We further implemented user-level security such that the participants can see only their data and experimenters can only access projects for which they were responsible. In the send-home prosthetics research, in addition to cyber security, one should address users' privacy concerns. As the perception of data sharing over the internet is different between people [48], we only collected the minimum data that were relevant to the study. Participants should have options to stop any kind of data sharing at any time, such as using the data-streaming buttons on the dashboards, so they can choose to share data over a specific period or allow us to for continuous monitoring [49]. The proposed hardware is capable of addressing all of these requirements.
With the support of IoT, the system can collect labelled real-life data during the pick and place experiment. These data include the style of the participant when doing specific muscle activities [50], so learning from them may yield a model that better fits the participant. We used the naive Bayes classifier as an example to demonstrate the optimization of the control model over the air. This is an example of how a simple machine-learning algorithm can promote data-driven clinical decision making, in the field of prosthetics. This approach helps in addressing ethical issues when using artificial intelligence in the healthcare domain [51]. Besides, it may increase the participants' trust in the system, which can further enhance their experience and satisfaction with the intervention [52].
It is readily possible to adapt the proposed IoT system to incorporate alternative upperlimb prostheses and controllers. It is also possible to deploy the server on different platforms. Deploying a server on commercial IoT platforms simplifies the development processes by using the built-in functions for designing servers and the ready to use cloud infrastructure for connecting devices. They are feasible for pilot studies, but not all custom features are available in their libraries. As such, we were not able to introduce new modules to the server. For instance, we could not allow participants to add text comments to the motor commands because of the constraints on the widgets. A different approach from the demonstration is to self-host an opensource solution for data recording and user interfacing, e.g. [53]. The server has to be developed from sketch, but it is more flexible in the short run. However, the self-hosted server is a lot more challenging to scale up for long-term clinical trials.
Future works will focus on adapting the system on sent-home unsupervised studies, where participants with limb difference use our devices in their daily activities. The experimental design and the user interface will be co-created with a broad range of stakeholders so that users' needs and ethical considerations within user home environments can be better taken into account, as recommended by [54]. Besides, the real-time system configuration function will be deployed in the studies for not only the decision boundaries but also the gain and window length for each myoelectric channel. We envisage that this platform can be used to expedite the involvement of the clinicians in the research on prosthetics, especially when it comes to tuning the parameters of a prosthesis controller on the fly and with the involvement of users.
Ethics. Ethical approval was granted by the local ethics committee at The University of Edinburgh (Ref: 89177).
The study was conducted in accordance with the Declaration of Helsinki. The subjects gave their informed consents for inclusion before they participated in the study.
Data accessibility. The data are provided in electronic supplementary material [55]. The data that support the findings of this study are openly available in Edinburgh DataShare at https://doi.org/10.7488/ds/3260.