Forms Builder - - Use Cases

Ray Pla . GetReal Software, Inc.

Alternating Pictures in "pseudo-realtime" Using Windows SDK By Ray Pla

Presented at Trenton Computer Festival 4/22/2006

©GetReal Software, Inc, 2006

i Table of Contents

Overview...... 1 Programming Considerations...... 2 Operating System and Multithreading Considerations...... 2 Win32 APIs and GUI issues...... 4 Bit-Mapped Graphic Display Concerns...... 5 Application Design...... 6 General...... 6 Flow of Control...... 8 Appendix A – Recommended Reading...... 9

©GetReal Software, Inc, 2006 i Overview

The purpose of this application is to test for the perception of differences in near-identical pictures. The pictures to be compared are read, two at a time, from a file. A blank screen image, used as a spacer, is also read at program start-up. Each group of two pictures is then displayed in rapid succession, first one, and then the other, with an intervening blank screen between the pictures. This rapid alternating of pictures continues until the viewer hits a key or a one-minute time interval elapses.

Program control is based on Windows multithreading and uses the Win32 API set. The control code runs the GUI that allows the user to start the tests and review the results; this code runs in the standard user thread priority. The display code is separate, and runs once the user starts a test; this code runs in a thread where the priority has been boosted to allow it to run without interruptions from other applications running on the machine. Since the display code still runs in the thread range for applications, it does not interfere with system related functions.

Since the program is concerned with the display of complete graphic images, these are loaded into memory once, so that subsequent access will not cause undue delay. The overall intention, once the pictures are being displayed, is to develop a tight loop where each of the pictures are shown for the desired interval, the next image displayed for the desired interval, and so forth. Graphic techniques recently introduced by Microsoft were used to achieve this effect.

Contact Info - - In case of comments or questions, I can be reached at: [email protected]

©GetReal Software, Inc, 2006 1 Programming Considerations

Once it has been established that the application processing is of some duration (in human terms), it is often useful to separate it from the overall GUI and related control elements. In this case, the program is primarily showing a set of pictures. Since showing the pictures is independent of the other application tasks, such as displaying and recording the results, it makes sense to separate these functions into separate threads. This is often done when the user wants to interact with the program while some other processing occurs.

However, this program has a special consideration: it needs to run a task which effectively stops all other interaction with the system until the user has given a response. This makes it necessary for the system to respond only to this program while it is running. By running the display portion at a higher thread priority, keyboard and mouse input are forced to this program’s control and, except for system functions, all processing is diverted to the picture display.

Another important issue that this program had to address was the rapid visual display of a full image. In order to ensure that the images would be displayed quickly, they had to be loaded into memory before they were displayed. Moreover, the actual display was done under control of a Microsoft API, since direct video control is not feasible for an application of this type. The choice of display mechanism was important since “screen flicker” was not acceptable.

Operating System and Multithreading Considerations

Modern operating systems allow processing to occur at differing levels of control. Even when nothing else is happening, an idle process is running. The process is the basic element used by the operating system to control the programs that it allows to run. However, the way a process is created and controlled can differ greatly between operating systems.

Microsoft decided, in the late 1990s, that Win NT was to be the single base operating system that it would support going forward. It is based on a 32-bit model, but, more importantly, it uses a preemptive scheduling system based on an API set known as the Win32 APIs. Although used in Windows 95 and ME, the Win32 APIs were intended primarily to support Windows NT and its successors, and will be discussed in more detail below.

Win NT and its successors, Win 2000, XP and Server 2003, have an overall architecture that is based on the general concept of a process. Each process begins with one thread and that thread, in turn, can start more threads, as needed. A thread is the basic unit of execution under Windows NT, and it runs as an independent unit. The operating system treats each thread as a separate entity and schedules time for each thread separately. Which threads are run, and the order in which they are run, are under the control of an operating system component known as the task manager. (The Win NT operating system actually implements this using a dispatcher mechanism, which is somewhat more general.)

Each thread is allocated a priority, which can be adjusted (to a degree) by the application; the operating system sees it as a running process. When a process is started, it runs at a specified

©GetReal Software, Inc, 2006 2 priority. Most applications start their processes at a level defined as normal. A process can also be started at high or idle; these states are all in the dynamic mode.

Threads run in one of two modes: real-time and dynamic (or application). Application-related threads run in dynamic mode, with extremely rare exceptions; only highly specialized threads will run using real-time mode, since these can interfere with standard operating system functions. Most threads run at the normal priority level, and (at a given level) are scheduled on effectively a first- come-first-served basis, often referred to as round-robin scheduling. In order to achieve performance results, an application will create a new thread, frequently with a priority other than normal (usually higher).

When a program runs and creates more than one thread, these threads usually want to be able to communicate. The most common scheme used in multithreading applications is the boss/worker model, where the “boss” starts the activities of many “workers” and then waits for them to communicate their status, in some cases periodically querying them. This application does not follow that model. Instead, it operates in a more hands-off mode, where the initial thread will start the display task, wait for it to complete, do some work itself, and then restart the display task as needed.

The basic communication mechanism in all threads involves each task waiting on a communication from the other. A thread will normally do some work and then wait for an “event”; the event acts as a notification to a waiting thread. In the standard boss/worker scenario, all threads are effectively active for the duration of the program. The “boss” will usually control the GUI and monitor the activities of the workers, which will be actively working on their respective tasks (such as building segments of a large file, or monitoring multiple data feeds). In the case of the Picture Display program, both threads control the GUI, but they take turns. There are effectively only two threads used by the Picture Display program, and only one of these threads is active at any time; the thread which is not active is waiting for a communication from the other. (In classic computer science terms, these two threads could be considered co-routines – each routine is dependent on the other.)

It should be noted that a thread waiting for an event will be immediately restarted if there are no other threads with a higher priority waiting for service. Normally, an application will run as the foreground application; this will cause an automatic priority boost and generally will be the highest priority application thread. However, if there are multiple programs running and receiving input each program at a given priority level is given a period of time to run (generally called a time-slice, but referred to by Microsoft as a quantum). These are small increments and are based on the HAL; for Intel processors they are 10 ms. each.

©GetReal Software, Inc, 2006 3 In the case of the Picture Display program, the fact that this is the active program with a thread priority boost to well above normal should cause the display thread to always wait no more than one quantum for an event (a timeout or a user keystroke). An additional complication, however, is the fact that Windows is constantly sending messages, and any outstanding Windows messages have to be handled by the thread or it can go into a limbo state, waiting to process a Windows event.

Win32 APIs and GUI issues

As noted in the discussion of operating systems, above, the Win32 APIs were developed largely in support of Windows NT, and they are effectively a wrapper around the operating system features that they support. Because they are primarily concerned with Windows NT features, it is difficult to discuss the Win32 APIs without going into some detail on how Win NT is structured and the specific features that are available to the programmer.

The NT operating system is based on a key controlling program known as the kernel. This is a process (Ntoskrnl.exe) which is loaded as part of system initialization (the boot process). Another important part of the operating system’s components, the HAL (Hardware Abstraction Layer) code, is also loaded during the boot process. The kernel supports things like file handling, memory management and thread control, and is not directly accessible to standard program calls. The HAL, also, is even more of an internal operating system feature, allowing access to external devices only through Microsoft supported interfaces. The Win32 API set (using libraries like Kernel32.dll, Gdi.dll and others) is the Microsoft supported way to access these features.

There are several standard operating system mechanisms that are used to coordinate threads access: mutexes, critical sections, semaphores and events. The first two, mutexes and critical sections, are quite similar, and are both used to protect simultaneous access to data, although critical sections tend to be faster. Both semaphores and events are commonly used communication primitives, and are used for the basic control within the Picture Display program, although a critical section was also used.

As might be imagined, timing is an important consideration when attempting to display pictures at regular intervals. The basic feature provided by the operating system is the sleep function. A system clock is also available for timing purposes. An alternate system feature provided by Microsoft’s Win32 APIs is the WaitableTimer; this feature allows for periodic delays, but it also assumes that the thread is always in a waiting state and, therefore, wasn’t suitable for this application.

©GetReal Software, Inc, 2006 4 The key calls used to support the needed waits were the system call WaitForSingleObject and the Windows call MsgWaitForMultipleObjects (which currently seems to have little available online documentation). The first call is a standard call when waiting for an event in another thread. The second, however, is needed because of the structure of a Windows program. Windows is constantly sending messages which have to be handled by the application program. At some point in your processing, you have to handle these by sending them back to the Windows message handler. The latter call provides a way to get the messages meant for the program, which is effectively suspended while not delaying the running display portion.

Bit-Mapped Graphic Display Concerns

Images are stored in a number of specialized formats. The GIF format is a widely used and relatively straightforward encoding scheme for graphic images. It does not give photographic image quality, but a more important fact is that for quite a while it was entangled in legal complications which made is difficult to use without payment of royalties. (The format was published by Compuserve in 1987 without knowledge that it was based on a process that had a patent pending. This patent was owned by Sperry, later Unisys, who required royalties be paid for use, making commercial development companies skittish about using this format; indeed, other claims, including one from IBM, are also possible.)

The JPEG format is more complex in its formulation, but more accessible in terms of available code. It can be displayed without significant loss, depending on how it was originally saved. In a book on graphics imaging, “Compressed Image File Formats”, by John Miano, a significant amount of already developed imaging code is provided to support the retrieving of a JPEG formatted image.

It is difficult to discuss the JPEG format without going into significant detail, including the fact that it is possible for the format to be lossy. Very briefly, an image is sampled; the sampling can contain from one to as many as 8 interleaved components. The components are subject to a Discrete Cosine Transform applied to 8x8 pixel data units. The resulting transform is subject to a mathematical removal of cosine coefficients; although designed to reproduce the original closely, this quantization process is the step that can introduce loss. The result is subjected to Huffman encoding and saved as a JPEG file. For further discussion, I suggest that you read Miano’s book, or any other book on graphics which discusses how a JPEG image is created.

©GetReal Software, Inc, 2006 5 Application Design

General

The program will allow any number of picture pairs to be displayed. Since the file read by the program containing the names of the pairs of pictures is created by the tester to define the actual picture pairs, the program aborts if a specified file name cannot be found, or if the number of pictures is not even. Once the test is complete (all picture pairs read), it can be re-run without closing the program. Statistics are kept in a separate file, recording the tester and the time used in each test.

The program flow breaks into three separate elements, where the first element controls the second, which, in turn, controls the third. The first, outermost controlling element is the main routine (which, in this case, is WinMain). The second element is the PerceptionTest routine, which controls the display and recording of individual tests for all separate test needed to complete the perception test. The third element is the PerceptionDisplay routine, which controls the alternating display of pictures. (Each of these elements runs on its own thread.)

©GetReal Software, Inc, 2006 6 The overall flow is shown below:

Base Screen

Perception Test

Perception Display

240 ms. 80 ms. 240 ms. 80 ms. Picture 1 Blank Picture 2 Blank

Control Code

Screen Control

 Initial Setup o Threads created for Perception Test and Perception Display (suspended) o Windows elements created and displayed (title, entry for participant and screen button [‘Go’])  Wait on client name entry o If ‘Go’ is clicked-on, an error message is issued asking for this field to be filled in  Wait on ‘Go’ button to be clicked o When clicked, invokes the PerceptionTest routine (signal the thread)  PerceptionTest runs o Reads both image files into memory o If first time, reads in a blank screen image o Invokes the PerceptionDisplay routine (signal the thread) o Sets a one minute timer o Suspends the PerceptionTest thread  PerceptionDisplay runs o Do forever . Display Picture1 . Delay 240 ms. . If signaled, exit this loop . Display Blank Screen . Delay 80 ms. . If signaled, exit this loop . Display Picture2 . Delay 240 ms. . If signaled, exit this loop o Notify PerceptionTest o Wait on DisplayStart event  In PerceptionTest on timeout or notification from PerceptionDisplay o If timeout, signal PerceptionDisplay and wait for notification o Compute and save results o Notify Main Thread o Wait on TestStart event  On Main Thread notification o Build second screen (same as first, replacing test duration for data entry field) (If last display screens were just read, a ‘Test Complete’ is displayed and the ‘Go’ button becomes a yes/no text box) o If not last display screens, restart at the bullet ‘PerceptionTest runs’, above o If last display screens, reinitialize internal data, re-create initial window display and restart at the bullet ‘Wait on Client name entry’, above

These recommended books are provided as a way to understand the programming concepts used, and also to provide insights into the WinNT operating system. These books relate to the use of Win32 API set directly, rather than the .NET framework and its associated classes, which is Microsoft’s currently recommended programming paradigm. While .NET works well for Web-based applications, and can also be used for client- server and desktop applications, the closer you get to the actual operating system, the more you have to know about how it functions and the limitations it imposes.

When an application such as the Picture Display program is written, it is best to keep the possible complications to a minimum, since the task is difficult enough to begin with. To have taken on the addition burden of .NET would probably have resulted in no program. It is not clear whether this type of program can be run in .NET at this point, but the books given here provide insight into the underpinnings of the operating system environment that Windows runs on. This is useful in .NET, if for no other reason than to know how the system might affect the performance of your (fully managed) application.

Microsoft Windows Internals, Fourth Edition by Mark E. Russinovich, David A. Solomon ISBN: 0735619174, Microsoft Press, 2004

This work provides a clear insight into the functioning of all the major elements of the current Microsoft operating system environments. It is not greatly changed from its predecessor Inside Microsoft Windows 2000, Third Edition.

Programming Applications for Microsoft Windows , Fourth Edition by Jeffrey Richter ISBN: 1572319968, Microsoft Press, 1999

This is a classic work on using the various programmatic features provided through the Win32 API set. This book is out of print and currently selling at exorbitant prices on Amazon. I would recommend the third edition, called Advanced Windows. This book covers most of the same features, and is much more easily (and cheaply) available on Amazon.

Programming Windows, Fifth Edition by Charles Petzold ISBN: 157231995X, Microsoft Press, 1998

This is the core book for programming in Windows (prior to .NET). Petzold is known for his books on programming in the Windows environment. He has also published a book for programming in the .NET framework, Programming Windows with C#. This book, as well as Richter’s Applied .NET Programming, are well worth reading.

.NET Framework Solutions, In Search of the Lost Win32 API by John Paul Mueller ISBN: 078214134X, Sybex, 2002

Although this book focuses largely on .NET capabilities, its early sections concentrate on recovering features of the WIN32 API set that may have been left behind by .NET. There are various references on .NET that will also give information on interoperability. msdn.microsoft.com

While not a book, this website is the best place to get technical information on Microsoft’s entire product line: how to program for it, how they believe it should work and any fixes that are known.