SONY

menu
Search button in the site

Search

Make it possible to touch the world
at a scale of 1 to 10
The precision bilateral control system

The precision bilateral control system is a remote-controlled robotics technology that enables
detailed operations by scaling 10 times while maintaining the sense of a human hand. One of
application is surgical robots, which are expected to develop minimally invasive surgery that
reduces the burden on the patient’s body.

Researchers
Shunsuke Yajima / Atsushi Miyamoto
Robotics

A future in which humans and
robots live in harmony

The precision bilateral control system is a robotics technology that enables micro manipulation beyond the limit of human dexterity by integrating multiple different technologies such as precision mechanical design, high-accuracy force sensors and precision acceleration control.

Under the theme of “A future in which humans and robots live in harmony,” we have been committed to the research and development of a variety of technologies that form the basis for robots that can control various tasks at will in a space where humans are present. The precision bilateral control system has been developed to reach the leading edge in the field of manipulation involving the sense of force with the expectation that it can assist humans even in medical fields which require particularly high level of operability and safety.

Show Image

Make it possible to touch the
world at a scale of 1 to 10

In general, “bilateral control” refers to a control method that simultaneously handles position synchronization and force action and reaction between two devices. There is already an established common scheme to control “position control” and “force control” by acceleration.

The precision bilateral control has 1 to 10 scaling function for position and force. If the operator moves his hand 10 mm, the tip of the robot moves 1 mm, and if the tip of the robot touches the environment with a force of 1 gram-force, a force of 10 grams-force is transmitted to the operator. With this 10 times scale operation, not only can you do detailed work that is difficult for human hands, but you can also handle fragile objects without damaging them. You should feel as if you are working in a miniature world with your body reduced to a tenth of the original size.

Show Video

Click here for the description of the video content.

This is an introduction video that shows the devices and the features of the precision bilateral control system.
Show Video

Click here for the description of the video content.

This is an introduction video that shows the devices and the features of the precision bilateral control system.

Core technologies used in the
precision bilateral control system

The robotics technology is formed by combining multiple elemental technologies such as mechanics, sensors and controls at a high level. The system we are currently developing is no exception. For a robot to work in collaboration with humans in the same space, technology is needed that allows the robot to behave flexibly to external environment changes by controlling its force. Accurate 3D sensing, transmission and real-time control of the force are crucial.

Mechanical design for
increased operability

We think that providing a wide rotational motion range, which is difficult in conventional surgery, by means of robotics technology can make it easier to perform complex surgical procedures. Generally, by consolidating rotating shafts axis at the tip of the mechanism, a wide rotational motion range can be obtained without making the robot larger in size. However, since our system uses a special tip tool equipped with a force sensor, which will be described later, we needed to protect a force sensor part at the tip. Making effective use of a cable drive that transmits the torque of the motor located at the root to the tip via a cable, we consolidated yawing, pitching, rolling and grasping motions to the tip of the arm without compromising the force sensor performance. This enabled us to deliver a rotational motion range as wide as that of a human wrist while keeping the entire mechanism compact in size.

It is not easy to generate an idea about such a mechanism and incorporate it into the design. In addition, in order to create a mechanism suitable for precision bilateral control that we seek to develop, we need to consider:

  • High backdrivability (for smooth force control)
  • Backlash-less mechanism (for higher positioning accuracy)
  • High mechanical response performance (for enhanced operability for the human operator)

Increasing the mechanical response performance, in particular, requires making the mechanism lighter and more rigid while taking into consideration the motion range and size conditions of the robot. Making the mechanism lighter contradicts making it more rigid. So, a considerable amount of development time and cost is required to find optimal values through repeated prototyping. We created a vibration model expressing even the smallest details of the mechanism, such as the reducer, cable and bearing, with mass-spring-damper elements and developed a unique simulator by combining this model with a control model. The simulator repeats a trial-and-error process at high speed, making it possible to optimize a complex mechanism in a short time.

Cable drive mechanism to provide a wide rotational motion range: By concentrating the rotating shaft on the tip of the mechanism using cable drive, a new mechanism with a wide range of rotation can be realized.

High-sensitivity force sensing
(3D optical measurement of
force at the tool tip)

To feed back the sensation of touching the human body’s soft tissue to an operator requires the means to sense a minute change in force on the order of 1 gram-force that is applied to the tool tip. Our conventional method has been to estimate the force at the tip using a force sensor located at the bottom of the tool unit. This is because the tip tool of a precision work robot is small in diameter and does not have enough space to install a conventional force sensor. With this method, however, the inertial force of the tool unit extending beyond the force sensor is observed as noise, making it impossible to detect a small change in the force at the tip.

To solve this problem, we turned to the FBG sensor (optical strain sensor) having a diffraction grating embedded in part of the optical fiber. Despite its extremely thin fiber shape, this special sensor enables highly sensitive measurement of the strain amount in the sensor part. We thought that turning the tip tool itself into a force sensor by means of this FBG sensor would substantially reduce the dynamic noise that has plagued us in the past. Through repeated trial and error, we have developed our own technologies such as the manufacturing process for precisely bonding an FBG sensor to the tool, and the algorithm for estimating the 3D force applied to the tool tip based on the strain amount of each FBG sensor. These efforts have made it possible to sense the tool tip force with ultra-high sensitivity on the order of 1 gram-force.

Force sensor-equipped tip tool: By mounting an FBG type strain sensor on the tip of the tool, dynamic noise is reduced and 1 gf of tip force sense sensing is achieved.

Precise control of position and
force (accurately achieving
the
target acceleration of precise
motion)

A motion control system based on acceleration control is used to implement bilateral control between two robots. Using a disturbance observer, it controls acceleration, the physical quantity at the lowest level of the dynamical system. This guarantees the robustness of motions, which enables the motions for both position tacking and contact with an external environment simultaneously. Furthermore, working with precision at a scale one-tenth of that of human fingertips requires constantly stable motions and high response performance to transmit a small force accurately.

The conventional algorithm is subject to the impact of a modeling error caused as the robot changes its posture. This results in the degradation of the position tracking performance during the operation and the vibration during the contact with an external environment. We solved these problems by incorporating our technological asset called Generalized Inverse Dynamics (GID) into the algorithm. GID is a model-based control algorithm that calculates the driving torque of a robot to enable a certain motion by performing an optimization calculation while taking constraints into consideration. The drive torque involved in a presumable posture change is calculated using GID as needed and adjusted through feedforward compensation. This improved the response performance of bilateral control, allowing us to solve the problems. By combining the high response performance of GID and the robustness of acceleration control, we developed a control system capable of constantly precise operation.

Outline of the precise bilateral control block: The left figure shows the outline of the bilateral control block using the precise acceleration control. The figure on the right plots the force response (top) and the position response (bottom) for precise bilateral control system, and shows that the law of action and reaction of force and coordination of movement between the two robots can be realized at the same time.

Outline of the precise acceleration control block: The top figure shows an outline of a precise acceleration control block that achieves both high responsiveness and high robustness. The plot at the bottom shows that the control performance in a small region was improved by introducing GID.

FPGA-based ultra-high-speed real-
time signal processing

Field Programmable Gate Array (FPGA) circuits, which are programmable integrated circuits, are used for the electrical processing vital for the precision bilateral control system. This has increased the signal processing speed and eliminated control instability caused by communication delay and noise.

To realize a precision bilateral control system that works with precision at a scale one-tenth of that of human fingertips requires high-speed real-time processing of electrical signals. Conventionally, the processing of these signals is implemented in the general-purpose CPU of the host PC, but such an implementation cannot meet the control system requirements because of communication delay and noise. We therefore decided to leave part of the electrical processing to the FPGA suitable for high-speed parallel processing. Since the FPGA is an LSI that allows its internal logic to be designed freely, a circuit configuration optimal for robot motion control can be implemented.

The interface with hardware such as sensors and motor drivers and the signal processing algorithm were formerly run by software. We implemented these on logic circuitry in the FPGA and developed a unique protocol for communication between the host PC and FPGA, thus making the processing 50 times faster than in conventional systems. Moreover, connecting multiple FPGAs in a cascade by optical fiber enables low-latency signal transmission even in multi-degree-of-freedom systems like the precision bilateral control system that consists of a leader robot and a follower robot. This reduces the impact of communication delay and noise, considerably improving the control performance.

Outline of signal processing and communication system using FPGA: Implementation of hardware interface, signal processing algorithm and communication module on FPGA realizes high-speed signal transmission with low latency.

Total optimization design

In Order to integrate the multiple elemental technologies, we have discussed so far into a system, it is necessary to ensure that its performance is sufficient for precise operation. This performance can be represented by three performance targets: position accuracy of 10 μm, force accuracy of 1 gf and interventional mass of 50 g (an indicator of comfortable operation). However, it was difficult to satisfy all of them at the same time due to their conflicting relationships. To overcome this challenge, we adopted a simulation technique to deal with all of the mechanical, electrical and control technologies and calculated an optimal system by setting unique evaluation indicators. This total optimization design has enabled us to develop a system that meets all the performance targets.

Relationship of design parameters conflicting performance targets: A relational diagram showing that it is difficult to achieve multiple target performances at the same time, for example, when the rigidity of the mechanism is increased to achieve the target position accuracy, the mass increases, making it difficult to achieve the target of the intervening mass.

Practical use of surgical robots
that can transmit
even the sense of force

Surgical robot systems are now in increasing use in surgery, and cases of clinical application are also on the rise.
Highly accurate arm manipulation by these systems enables minimally invasive surgery that reduces the burden on the patient by minimizing incisions. Also, they allow intuitive operation, which helps ease the burden on surgeons as well. On the other hand, the existing surgical robot systems force surgeons to rely on vision when operating the robot, leaving the sense of force unavailable.

If our precision bilateral control system comes into practical use as a surgical robot system, surgeons will be able to work in collaboration with a surgical robot while sharing the motions and sense of force with the robot. It will become possible to safety handle with fragile organs in the body, such as blood vessels and nerves, which are difficult to handle with vision alone.

Surgical procedures involve very complex workflows, and what we are required to do from a technological perspective is to evolve those procedures to safer and less invasive ones. To meet this challenge, we intend to promote the collaboration with surgeons and other healthcare workers as well as the cooperation with internal and external partners with medical skills.

Versatile technology that can also
be applied to many other areas
such as outer space and
disaster recovery

While we assume the medical sector to be the first step toward the social implementation of the bilateral control technology, it can be used in various fields where people work through robots and machines. In the current system, the robot arm makes precise movements on a scale of one-tenth that of a human, but magnifying this scale in reverse produces 10 times greater movements and forces. This technology can be used for humans to get work done remotely with the sense of force under extreme conditions such as in outer space, at a disaster site or on the sea floor, as well as for robots to work in a living space safely for nursing care or other purposes by using force.

Researchers

Shunsuke Yajima

Tokyo Laboratory 24

Sony has a diverse workforce that includes not just robot experts but also engineers who excel in many other areas. I find it very meaningful to create new robots by combining the knowledge of these engineers and contribute to society in healthcare and other fields. We set development targets ourselves and take time to achieve those targets. That’s one of the appealing aspects of Sony.

Atsushi Miyamoto

Tokyo Laboratory 24

What makes the Sony R&D Center special is that it has a flexible collaborative system for R&D, as well as a system whereby we can accomplish the entire process from elemental technology development to social implementation in a coherent way. I hope that young engineers who will be working in R&D in Sony will also aim to conduct R&D that will contribute to society.

to the top