마우스 커서 아래에 있는 텍스트 가져오기

마우스 커서 아래에 있는 텍스트를 가져오는 기법은 여러 가지 방법으로 가능하다. 가장 많이 알려져 있는 방법은 Windows API를 후킹하여 Invalidate한 영역의 텍스트를 가져오는 방식인데, 이것과 달리 많은 사람들에게 알려져 있지 않은 IAccessibility라는 인터페이스를 사용하여 보다 쉽게 마우스 커서 아래에 있는 텍스트를 가져올 수 있다. 문제는 가져온 텍스트를 정확하게 파싱하는 것이 중요하다.



다음은 The Old New Thing에서 소개한 글이다.

invalid-file

[예제 프로젝트]



----------------------------

How to retrieve text under the cursor (mouse pointer)

Microsoft Active Accessibilty is the technology that exposes information about objects on the screen to accessibility aids such as screen readers. But that doesn't mean that only screen readers can use it.

Here's a program that illustrates the use of Active Accessibility at the most rudimentary level: Reading text. There's much more to Active Accessibility than this. You can navigate the objects on the screen, read various properties, even invoke commands on them, all programmatically.

Start with our scratch program and change these two functions:

BOOL


OnCreate(HWND hwnd, LPCREATESTRUCT lpcs)


{


  SetTimer(hwnd, 1, 1000, RecalcText);


  return TRUE;


}








void


PaintContent(HWND hwnd, PAINTSTRUCT *pps)


{


  if (g_pszText) {


      RECT rc;


      GetClientRect(hwnd, &rc);


      DrawText(pps->hdc, g_pszText, lstrlen(g_pszText),


               &rc, DT_NOPREFIX | DT_WORDBREAK);


  }


}

Of course, the fun part is the function RecalcText, which retrieves the text from beneath the cursor:

#include <oleacc.h>





POINT g_pt;


LPTSTR g_pszText;





void CALLBACK RecalcText(HWND hwnd, UINT, UINT_PTR, DWORD)


{


  POINT pt;


  if (GetCursorPos(&pt) &&


    (pt.x != g_pt.x || pt.y != g_pt.y)) {


    g_pt = pt;


    IAccessible *pacc;


    VARIANT vtChild;


    if (SUCCEEDED(AccessibleObjectFromPoint(pt, &pacc, &vtChild))) {


      BSTR bsName = NULL;


      BSTR bsValue = NULL;


      pacc->get_accName(vtChild, &bsName);


      pacc->get_accValue(vtChild, &bsValue);


      LPTSTR pszResult;


      DWORD_PTR args[2] = { (DWORD_PTR)(bsName ? bsName : L""),


                            (DWORD_PTR)(bsValue ? bsValue : L"") };


      if (FormatMessage(FORMAT_MESSAGE_ALLOCATE_BUFFER |


                        FORMAT_MESSAGE_FROM_STRING |


                        FORMAT_MESSAGE_ARGUMENT_ARRAY,


                        TEXT("Name: %1!ws!\r\n\r\nValue: %2!ws!"),


                        0, 0, (LPTSTR)&pszResult, 0, (va_list*)args)) {


        LocalFree(g_pszText);


        g_pszText = pszResult;


        InvalidateRect(hwnd, NULL, TRUE);


      }





      SysFreeString(bsName);


      SysFreeString(bsValue);


      VariantClear(&vtChild);


      pacc->Release();


    }


  }


}





 

Let's take a look at this function. We start by grabbing the cursor position and seeing if it changed since the last time we checked. If so, then we ask AccessibleObjectFromPoint to identify the object at those coordinates and give us an IAccessible pointer plus a child identifier. These two pieces of information together represent the object under the cursor.

Now it's a simple matter of asking for the name (get_accName) and value (get_accValue) of the object and format it nicely.

Note that we handled the NULL case of the BSTR in accordance with Eric's Complete Guide to BSTR Semantics.

For more information about accessibility, check out Sara Ford's WebLog, in particular the bit titled What is Assistive Technology Compatibility.

-----------------------



이외에도 다음 글을 살펴보면 많은 도움이 될 것이다.

Accessibility is not just for people with disabilities

When the topic of accessibility comes up, most people think it's all about people with disabilities. While it's true that people with disabilities were the original target audience for accessibility, more recent developments have highlighted the value of accessible technologies to everyone.

For one thing, making your program accessible means that test automation can manipulate your program in a uniform manner. They can enumerate the buttons, get text for them, press a button programmatically, and so on. Even if you don't use a standard button control but instead opt to go windowless, as long as you expose your controls via accessibility, test automation can find them.

Also, we saw how programs can use accessibility to retrieve text from the screen. I use this in my English/Chinese dictionary program to "pluck" text off the screen and paste it into the "translate this" box, where I can arrow through the sentence and have the program translate each word or phrase on the fly. (Rats, now you know where I'm going with the program.) And once I started using this feature, I discovered to my dismay how many of the programs that I use on a regular or semi-regular basis simply fail to support this simple task. Microsoft products have a much higher success rate, but even there the support is not 100%.

Finally, check out this screencast showing off Vista's speech recognition system. The "Say what you see" feature which Chris discusses at time code 8:52 needs to get the name of every element on the screen so it can take what you say and look for it on the screen. If your program doesn't expose these names, the "Say what you see" feature won't know what the user needs to say to click on your button, and users will say, "Harumph, why doesn't this program work with voice recognition? All my other programs do."

My secret hope is that "Say what you see" will finally be enough to prod people into taking accessibility seriously. Because it's not just for people with disabilities.