US 9,811,171 B2
Multimodal text input by a keyboard/camera text input module replacing a conventional keyboard text input module on a mobile device
Cüneyt Göktekin, Potsdam (DE)
Assigned to Nuance Communications, Inc., Burlington, MA (US)
Filed by Nuance Communications, Inc., Burlington, MA (US)
Filed on Mar. 5, 2013, as Appl. No. 13/786,321.
Claims priority of application No. 12158195 (EP), filed on Mar. 6, 2012.
Prior Publication US 2013/0234945 A1, Sep. 12, 2013
This patent is subject to a terminal disclaimer.
Int. Cl. G06F 3/02 (2006.01); G09G 5/00 (2006.01); G06F 3/023 (2006.01); G06F 1/16 (2006.01); G06F 3/00 (2006.01); G06K 9/32 (2006.01); G06F 17/28 (2006.01); G06F 3/0488 (2013.01); G06F 3/01 (2006.01)
CPC G06F 3/0237 (2013.01) [G06F 1/1686 (2013.01); G06F 3/005 (2013.01); G06F 3/013 (2013.01); G06F 3/0227 (2013.01); G06F 3/04883 (2013.01); G06F 3/04886 (2013.01); G06F 17/289 (2013.01); G06K 9/3258 (2013.01); G06F 2203/0381 (2013.01); G06K 2209/01 (2013.01)] 16 Claims
OG exemplary drawing
 
1. A method of multimodal text input in a mobile device, the method comprising:
using an original communication interface between an original keyboard module of the mobile device and a third party application to enable communication between a multimodal input module, that replaces the original keyboard module, and the third party application by:
executing the multimodal input module by:
steadily running the multimodal input module in the background of the mobile device and constantly monitoring in the background of the mobile device to detect when a text input field of the third party application is activated; and
responding to detecting that the text input field of the third party application is activated by:
activating a keyboard mode;
displaying an A-Z-keyboard in a first field of a display for text input;
automatically activating a camera mode when the keyboard mode is activated;
capturing an image of written text having characters different from characters of the A-Z-keyboard, reducing a size of the A-Z-keyboard, displaying the A-Z-keyboard reduced in a reduced first field, and displaying the captured image with the written text in a second field of the display of the mobile device, the reduced first field and the second field together occupying a same field size as the first field;
converting the captured image to character text by optical character recognition (OCR) and displaying the recognized character text on the display; and
outputting a selected part of the recognized character text as the input text to the third party application receiving the input text upon a selection of the part of the recognized character text, wherein the outputting to the third party application from the multimodal input module is via the original communication interface to the third party application as between the original keyboard module and the third party application, and wherein the multimodal input module is configured to enable the respective selection to take place by a single keypress or control command, or by a single gesture.