2008 NCTI Technology in the Works Award Winner — VISO (Voice Input/Speech Output) Calculator

Math is a core language of STEM education. Students who are blind or have low vision have been at a disadvantage with few tools designed for their needs. With the growing prevalence of math mark-up language, MathML, assistive technology researchers knew that a speech-interactive calculator would fill a need and be able to take advantage of new digital content. This 2008 Technology in the Works research project developed an on-screen calculator that shows real promise as a compensatory tool, but even more importantly, as a learning tool.

Participants:

« Dave Schleppenbach gh, LLC Founder and President » Dr. Emily Bouck Assistant Professor of Special Education Purdue University

An On-Screen Calculator: A Clear Need

With the growing prevalence of math mark-up language, MathML, assistive technology researchers knew that a speech-interactive calculator was needed for persons who are blind or have low vision. Existing math editors and software were simply too visually oriented and complex.  NCTI’s Technology in the Works award was, as Dave Schleppenbach puts it, “Just enough impetus to get us off the [dime] to do what we needed — we’ve been interested in doing this for some time.” An equivalent matching award and cost sharing of a percentage of Dr. Emily Bouck’s salary demonstrated Purdue University’s commitment to the need and further leveraged a complex project involving both seminal development and validating research on the fast-track.

The result of their collaboration is the VISO (Voice Input/Speech Output) Calculator, a completely new approach that builds on gh’s existing, proprietary MathSpeak process, which provides a structured, logical lexicon and syntax for both expressive and receptive mathematics in speech.  VISO is a computer-based system for Windows, and the interface has a visual appearance similar to a traditional scientific calculator with entry buttons for numbers, cosigns, logarithms, roots, and so on.  There is a display window where algorithms and functions are shown as operations are entered.  Importantly, there is a built-in accessibility menu for controlling voices (VISO uses the free voices bundled with Windows); speech output speed; colors of screen, text, and highlighting as text is voiced; and other necessary functions. See the VISO Calculator in action .

Dr. Bouck and her assistants at Purdue were sine qua non in providing information from informal research, focus groups, and interviews in order to inform basic functionality, feature sets, and even color schemes within the system.  A prototype was demonstrated at the 2008 NCTI Technology Innovators Conference, and positive responses and feedback from attendees further pushed development in directions to include other disability groups, especially those with learning disabilities.

Midstream Design Changes — A Familiar Refrain

Technology collaborators frequently report the need to modify initial conceptions, and the VISO collaboration proved no exception.  The original plan for the system was more akin to the math editors the collaborators found lacking, but the team realized an on-computer calculator was a better concept as they began confronting technical challenges in more detail.  Emily describes a certain inevitability of evolution from what can be modeled versus the reality of a tangible product:

I think the calculator had to take a different focus than the grant originally proposed, due to challenges faced by the gh team, including time and finances.  We provided input into the alpha and beta versions
and of course provided feedback based on the research we collected (as the project shifted).

The limits of a relatively small research and development grant imposed concomitant constraints, particularly in the face of the grand scope of the vision.  Dave continues, explaining that combining speech input and output for math interaction posed…

…A question that was a lot larger than we ever imagined.  This surprised us because we are recognized as so-called experts in the area already, and we came into it thinking we kind of know what’s going to happen here…  Just when you think you know that is exactly when you should step back and ask questions.

Years of work on the ‘rendering’ side — converting math to speech so that it could be heard, built confidence, but Dave explains,

What we hadn’t thought about was, ‘Well, what happens when they have to do the reverse — when the person actually has to generate math content.’  That is actually quite profound — but something we’re most excited about as an outcome from the project.  We discovered a whole range of things that were really interesting.

Research and Outcomes

Emily was in charge of the research design, implementation, and analysis.  She quickly sums up the study and the results:

We completed a single subject design with three students who have visual impairments to explore the calculator in comparison to students’ typical means of calculation. The results were that students decreased their time to use the VISO calculator, as compared to their standard calculation method, as well as decreased the number of attempts to use the calculator. Students were generally positive towards the VISO calculator, and were excited about it for future potential.

Dave adds that none of the formal subjects were Braille literate, and thus could not use Braille-based access modalities.

With his background in chemistry, he further notes that educational research isn’t as straightforward as it is in chemistry in which you can look for a color change that definitively shows an experimental effect — nor can you calculate the effect of an educational intervention out to a tiny decimal point for extreme precision.  Still, he emphasizes that results obtained in the study on VISO were so strong that it was actually ‘disruptive’ to the statistical analysis, and adds, “It was quite obvious from qualitative observations that there was a profound difference [having access] versus not having the calculator.”

Dave found other results even more profound.  As suggested above, the first was that there needed to be a provision for training users to expressively utilize the lexicon of the MathSpeak language:

We discovered there was no technique other than spoken math for people to enter in information, and we were surprised by the ease with which people picked up the language (of MathSpeak). In fact, informally, we feel people learn to speak math easier than to listen to it.  Related to cognitive psychology, when people are actually generating the data themselves — creating speech — there’s a different thought process than when they are decoding information.

Dave and the gh team, therefore, felt that anecdotal data showed students not just improving in calculation ability and access, but also in their fundamental understanding of the mathematical concepts as a result of more intensive reciprocal speaking and listening processes compared to traditional approaches.  He indicates that students from India involved in the research at various stages, who reported being  taught mathematics in a much more verbal manner than that experienced by most Western students, found this a natural and obvious conclusion, as well.  He believes this has implications for professional mathematicians as well as all math students who could benefit from applying spoken language processes to math,

This is where I get back to the encoding versus decoding issue.  We are challenging students to encode info and speak it into the computer — we are actually pushing them a little bit more than if they were just passively listening. The hope is that whatever the equivalent for illiteracy in mathematics is, call it innumeracy, can be tackled by the forcing of students to take ownership of the process and be more proactive.  What we’re hoping is through this process maybe we build some pathways, cognitive pathways they normally wouldn’t have built.  And those pathways are probably similar to those used to solve the problem itself.  I think there are real benefits from training people in this technique [and] I’ve actually talked with Dr. Nemeth about this a lot.

Emily sums up with another important outcome, “We have submitted a formal paper from the study to a peer-reviewed journal.”

Collaboration Across Organizational Cultures

Purdue and gh are both located in West Lafayette, IN, which helped to support collaboration, as Emily explains,

We communicated mostly through email and face-to-face meetings.  Actual meetings were used for some major issues, which was helpful.  We have also tried to do additional collaborations beyond the NCTI award, though a main developer I worked with at gh has left the company, leading to some challenges.

She continues, emphasizing oft-reported schisms between academic and business contexts,

Definitely the cultures are different. Business needs to be concerned with making a profit, which is different from the grant model in academia. I think it’s harder for businesses to work with grants, especially small ones that may not offset the cost they incur as part of the project.

And even though Dave speaks with pride about the upcoming journal article, Emily knows, “Of course, the focus of academia is really research and publishing, which may not be a major focus for all businesses.”

Dave strongly echoes the notion of a fundamentally “different [accountability] metric used in this business of Assistive Technology,” and emphasizes the aggressive model gh employs for development:

We are notorious at gh for spending money speculatively on R&D, which is very unusual I think for companies in our field. Most companies in AT probably spend 90% of their money on sales and marketing, we probably spend something approaching 90% on R&D…

Despite the occasional organizational mismatch, both sides understood the needs and foci of the other, and Emily concludes,

I enjoyed the project as a whole and my work with them.  It was definitely an interesting experience to work with collaborators outside of the academic world. gh was a great company to work with and it was a very positive experience for all.

A Little SuiteTalk Emerges from Collaboration

Dave laments some of the hardships and unexpected detours during the collaboration, but emphatically indicates that, “In hindsight it was well worth it.  There were many serendipitous discoveries, and it has spawned a lot of ideas for larger grants and new collaborations. There were a lot of intangibles.”

One major concrete outcome, though, was a fundamental shift in product design and marketing for gh. Originally, VISO had been conceived as a potential add on for the company’s seminal product, ghPlayer, which is now being rebranded as ‘ReadHear.’ However, information about the importance of the calculator and combined speech input and output led to the design of a packaged suite of products that will incorporate this capacity throughout, and include VISO.  The new suite, ‘SuiteTalk,’  will include ReadHear, ClickHear, WriteHear, and TestHear, a full range of associated applications for playing content and e-books, navigating the web, composing text, and assessing understanding with speech facilities.

Offering Software as a Service (SAAS)

The research also revealed some economic and marketing issues:

What we discovered in the modern economy is that affordability is a primary concern. There were two issues that we discovered when trying to commercialize with schools that caught us by surprise.  First, you need a wide variety of price points.  Some schools can afford and need expensive packages — some can’t.  Some schools are very sophisticated and can handle complicated installations, but others need just a turnkey solution — again, given the complexity of math, it’s even more an issue.  You have to have a pretty wide spread.

gh is now also offering web subscriptions for as little as $9.95 per month as well as simple annual options, and the company has offerings that are web-based and operate independently of browsers or resident software.  gh products can make large publisher-produced HTML books accessible with speech, and can operate on a mobile phone.  Dave anticipates a launch of some aspects of the suite this October at the Assistive Technology Industry Association (ATIA) convention in Chicago.  He continues,

It’s safe to say if we had not done this research project, we probably wouldn’t be doing this product launch, either.  The technology we discovered really motivated us to start thinking a lot bigger.

The remainder of the Suite is expected to unfold over the next several months, and future developments may include switch-based access and graphing capabilities to take math support even further.