Jump to Header Jump to Main Content Jump to Footer

Assistive technology: the conjunction of man and machine

Home > Blog > Assistive technology: the conjunction of man and machine

Assistive technology: the conjunction of man and machine

Darren Gabbert, left, and Gui DeSouza, an assistant professor of electrical and computer engineering, have partnered to make progress in the area of assistive human technologies.

After graduating from the University of Missouri in 1987 with a degree in computer science, Darren Gabbert was hired as a graduate research assistant by MU’s Campus Computing, now the Division of IT. The work that he and others did to find innovative computer-aided learning solutions for persons with mobility, visual and learning disabilities grew into Mizzou’s Adaptive Computing Technology Center, located in the lower level of the Memorial Union, where Gabbert has worked for more than 20 years.

Like most neuromuscular diseases, Gabbert’s condition is progressive. As a quadriplegic, he has hands-free control of his computer using voice recognition. He controls his power wheelchair, seating actuators and his smartphone via an electromyographic (EMG) sensor attached to a single surviving pectoral muscle. “With that muscle, I twitch and the sensor picks up the signal and operates the chair. My personal case study illustrates that while a disease is diminishing physical abilities at a steady rate, assistive technology solutions have been compensating at an even greater rate,” Gabbert said.

“I was born in 1965 with a neuromuscular disease, which adversely affects development of voluntary muscles. My parents moved from rural northwest Missouri to Sedalia so that my sister, who has the same condition, and I could attend the Children’s Therapy Center for our schooling. Disability segregation was the norm in Missouri during the ’70s”

“Though I never walked, it wasn’t until I was 13 years old that I received my first power wheelchair. This was a taste of freedom for which I’ve never lost my appetite. That crude, belt-driven motorized chair was, at the time, leading-edge technology. Resources for maintenance and repairs were scant. However, I have many memories as a boy of my father driving me from Sedalia to Columbia where there was a special lab that provided wheelchair customization and repairs. Dr. James Rathke, an electrical engineering professor at MU, managed this lab out of the basement of what is now Engineering Building West.

“The students and staff that worked in that lab probably had no idea of the impact they had on my life. A wheelchair breakdown meant a loss of freedom. I always left that dreary basement with my freedom returned. Whether it was a repaired wiring glitch or a driving interface modification, these engineers were my first encounter with the importance and potential of assistive technology.”

— Darren Gabbert BS CS ’87

Nearly three years ago, Gabbert met with Harry Tyrer, professor of electrical and computer engineering about the possibility of collaborating on research in areas of assistive technology where gaps exist for supporting the needs of persons with severe disabilities. Tyrer connected Gabbert with Guilherme DeSouza, an assistant professor in electrical and computer engineering. The pair met and agreed to work together on “smart” wheelchair technologies through DeSouza’s Vision-Guided and Intelligent Robotics Lab (ViGIR).

The group of students working in DeSouza’s robotic assistive technology lab — undergraduate, graduate and seniors working in groups on capstone projects — dubbed themselves the RATs. Various student projects in the 2010 winter semester met with success, and almost without exception, the students reported the research to be meaningful because of its potential to impact people on such a personal level, attitudes most certainly influenced by Gabbert’s direct connection to the lab. Some of the projects are detailed at the end of this story.

Gabbert donated a Permobil wheelchair to the lab, and DeSouza has worked with graduate lab assistants Ruizhi Hong, Daniel Conrad and Randy Melloy to make the vehicle into a “smart” wheelchair — dubbed “Permobot” — with vision-guided intelligence that incorporates path planning and obstacle detection. Hardware and software component development was needed to allow the wheelchair to be controlled by an onboard-embedded device that functions somewhat like a server.

Graduate student Luis Rivera and Professor Gui DeSouza developed an algorithm that can separate signals with specific signatures (patterns). In the case of robotic assistive technology, it will be a specific muscular activity. It can use a single electromyographic device to capture the combined activation of the muscles and yet identify what activity was being performed.

Conrad and Melloy designed and produced — by hand — encoder discs, which were recently re-manufactured by ECE freshman Jaime Germer and sophomore Paul Pancoast using a 3-D printer that creates a three-dimensional object from a computer-assisted design (CAD). Hong developed navigation algorithms and worked with Conrad on planar detection — walls and floor — recorded by cameras on the wheelchair that designate objects found outside the planes as obstacles.

“The robot will sense the environment and view the objects as potential fields, or force fields, where having a high potential means that the object will push the robot away from it,” said Conrad. “The chair can also be operated with the joy stick. But autonomous motion is what makes it more versatile.”

Conrad explained that the Permobot operates with LED photo sensors and the discs Melloy designed interrupt the light. Control depends on the pulses of light measuring the positions of the wheels to tell that the chair is moving.

Earlier this semester Conrad and Pancoast worked to develop an interface that can be incorporated into other programs as additional RAT projects are conceptualized and initiated.

Gabbert said he finds DeSouza’s knowledge, energy and vision to be inspiring. His first real exposure to the students in the ViGIR lab was when he was reviewing their progress on the Permobot. “The atmosphere in the lab denoted focused research, innovation and camaraderie. I immediately recognized in these young engineers the zeal of using technical knowledge and skills to make a difference in someone’s life.”

Luis Rivera, another graduate student on the RAT team, is working on a new technique to analyze and classify or find patterns in signals like hand movements to eventually anticipate wheelchair users’ intentions. “It would work with any kind of signal, including EMG signals generated by muscles,” said Rivera, who is a Fulbright scholar from Guatemala, “The purpose is to classify, for example, hand movements by analyzing the signals emitted by the muscles in the arm.”

“All I’m doing now is opening and closing a switch with my EMG sensor,” said Gabbert, “but Luis’ project would take it a step further and recognize patterns. It provides a way for different signals to do different things. It might result in a way of driving the chair that was more fluid.”

Rivera said that his original research in DeSouza’s lab was to use terahertz signals to identify the signatures, or characterize particular substances, like buried explosives. He and DeSouza eventually discussed using the technique for the EMG signals.

Grad student Daniel Conrad, and Paul Panhorst, a sophomore, are working on an encoder to determine the motion of the wheelchair with respect to the ground. Here they count the pulses with the instruments and compare them with the count made by the wheelchair’s on-board computer.

“We have had very promising results with sound signals, but we have to do a lot of tests using EMG signals,” said Rivera, adding that he is incorporating a box-like device into the research that is used to control Gabbert’s wheelchair. “The EMG signals are filtered and processed with this box.”

Rivera is interested in assistive technology because it’s something useful for people. After he graduates, he plans to return to Guatemala, he said, where research in engineering and technology is very limited. “I’d like to return to the University Del Valle, and start a research program there to help people.”

“You don’t realize how difficult it is for a person like Darren,” said DeSouza. “When he first came to a meeting, he had to have someone come to the elevator to push the button when he was ready to leave. Darren would like us to develop a robotic arm that can be installed on a wheelchair that could press buttons, open doors and pick up cups.”

DeSouza said that this need on Gabbert’s part has resulted in a grant proposal to seek funding to pay for the technical details because research that is already being done in the ViGIR lab can be applied to such a project. He is both proud of and impressed by the work his lab assistants are doing.

His ultimate goal is to offer people with disabilities functional options. He said that when someone in a wheelchair is being pushed, they point, nod and speak to indicate their preferences.

“I can envision that gaze control, facial expressions, speech, hand gestures, etc. could all be used as part of a more friendly interface, one that could continuously recognize a person’s intentions and operate with different levels and forms of activation,” DeSouza said. “We would like to incorporate multiple interfaces and cues to allow them to make choices, multi-modal interfaces with human intention recognition.”

DeSouza and Gabbert have a friendly, jocular relationship. DeSouza said he jokes with Gabbert that if they incorporate gaze control, they will have to make sure that other intention cues are in place in case a beautiful woman crosses Gabbert’s path and his wheelchair starts to follow her.

Gabbert also is excited that the research the RATs are doing to discover ways to take limited movement and expand its capabilities can be applied to many disabilities. “These are things that anticipate people’s intentions,” he said. “The same approach could work, for instance, for people with brain injuries.

“For someone to lose physical ability, but gain functional ability is a phenomenal technological success,” Gabbert said. “That is the type of success that motivates students to get involved in person-centered robotic assistive technology.”

“Darren was a match from heaven for the work we are doing: the insights of a person with a disability who is also an IT professional with 20-plus years of experience in assistive technology,” said DeSouza.

Would it be fair to say that this research alliance has changed the focus of the work being done in DeSouza’s lab?

“Yes. I would say that it has,” he said.

Gabbert’s perspective and skills are a good fit for MU’s Adaptive Computing Technology Center

Darren Gabbert and Carmen Schafer demonstrate available technologies at MU’s Adaptive Computing Technology Center where they work with MU students who have a wide variety of disabilities to give them tools that will help them be successful.

Darren Gabbert, a 1987 Mizzou computer science graduate, thought it was too good to be true when he immediately landed a position with what is now the MU Adaptive Computing Technology (ACT) Center. Born with a progressive neuromuscular disease, Gabbert has been dependent on adaptive technologies nearly as long as he has been alive.

“I was able to use my technical skills to give others that ‘taste of freedom,’ the sweetness of which I knew well,” Gabbert said. “The more people are empowered to do, the more they want to do. I still remember the day that I had finished a training session with a blind student using synthesized text-to-speech computer feedback. Before we met, she didn’t think she could make it work, but [during the session] she turned to me and asked, ‘Can’t you make it go faster?’ It was already speaking faster than I could comprehend, but we pushed the limits further. She reads faster than most sighted individuals.

“Finding innovative solutions to make persons with mobility, visual and learning disabilities more productive through computing has become the mission of our efforts,” he said.

ACT Center services are free for MU students who can benefit from the array of products and services they offer.

“Finding practical and innovative solutions is the purpose of this center,” said Carmen Schafer, a user support analyst-specialist with the center, as she and Gabbert demonstrated some of the products available through the ACT Center.

“Jaws for Windows” is a program for visually impaired users that reads aloud what is on the screen. It allows the user to be independent in using a computer, and have access to information at their fingertips. Websites that are developed in compliance with accessibility standards established by the World Wide Web Consortium (W3C) will be more user friendly to a screen-reader user.

MU Discovery freshman Jaime Germer and sophomore Neil Thawani look at a program they are writing to sample data from an electromyographic (EMG) sensor that once processed, will help to define the patterns of muscle activation. Click the image for more details.

“Dragon NaturallySpeaking” is speech recognition software that converts speech into text. It allows the user to open applications and files, create documents, send email, and navigate the web. Gabbert explained that instead of typing errors, the software sometimes recognizes the words incorrectly.

“Instead of typos, you get wordos,” said Gabbert who uses his computer completely hands-free.

The Pulse Pen is a device that takes notes in a special notebook. Once a user has written notes with the Pulse Pen, they can return the pen to any place on a page and it will repeat the recorded lecture. This “pencast” can be uploaded to the computer as images and shared with other users, and may also be saved as MP3 files that can be accessed with iTunes.

These are just a few of the tools in the center’s arsenal of adaptive technologies.

“Every possibility is explored when we do consultations,” said Schafer. “We consider all available technology to see what is useful for individuals. Some of the students we see once; others return for additional help.”

“Sometimes an individual has struggled and then something really clicks and you know it,” said Gabbert.

“The rewards of person-centered technology solutions never get old. Twenty years later, I still find it challenging, fun, and sometimes baffling — techno-nerds thrive on being baffled,” he said. “The ACT Center staff focuses on matching, implementing and training with existing assistive technologies, but technology is always changing and the demands on technology are ever increasing.

“Gui [DeSouza]’s Robotic Assistive Technology initiatives offer opportunities to reach beyond the current limits and provide new solutions.”

Visit MU’s ACT Center on the Web at actcenter.missouri.edu.

Back to Top

Enter your keyword