Multimedia Driver Supp ort in the FreeBSD Op erating System
Luigi Rizzo James S Lowe
Universita di Pisa University of Wisconsin Milwaukee
Dip di Ingegneria del l Informazione Col lege of Engineering and Applied Science
l rizzo iet unipi it james cs uwm edu
http www iet unipi it luigi http www uwm edu james
Abstract applications which deliver multimedia streams re
quiring synchronization b etween the application and
the hardware
In this pap er we will discuss the motivation de
Acquisition and rendering of multimedia streams re
sign implementation issues and applications for the
quires sp ecial p eripherals such as audio CODECs
audio and the video acquisition drivers which are
and frame grabb ers as well as adequate supp ort in
available in FreeBSD Both systems have b een
the op erating system In fact CODECs and frame
designed from scratch with sp ecial attention paid
grabb ers are relatively simple devices and in prin
to the e ective transfer and synchronization of data
ciple they could b e easily supp orted in an op erating
between the hardware and adv anced applications
system using the conventional read and write
system calls plus a small number of ioctl calls to
The main fo cus in the design of the audio driver
con gure the sp ecial features of the devices How
was for supp orting multimedia applications which
ever synchronization is a fundamental requirement
often require full duplex op eration and have strict
with many advanced applications so the internal
synchronization requirements This suggested the
structure of the driver and the interface it exp orts
de nition of a new software interface simpler to use
must supp ort these requirements in a exible and ef
than the previously existing OSS Voxware driver
cientway Furthermore video streams are highly
The driver is backward compatible with the OSS
demanding in terms of bandwidth so the conven
API due to the large base of applications that use
tional read write interface might not be the
this API
most e cient or e ectiveway to transfer data
The video acquisition driver fo cuses on providing
In recentyears multimedia supp ort in the FreeBSD
exible access to the acquisition device and the abil
op erating system has received a lot of attention
ity to suit the needs of di erent applications from
namely the addition of supp ort for full duplex au
simple TV viewers to video conferencing programs
dio cards and high p erformance video grabb ers com
The driver provides memory mapp ed access to the
municating through the PCI bus Most of these
capture bu er exible supp ort for synchronization
systems have b een designed from scratch In this
with the application and supp orts the various cap
pro cess we had the signi cantadvantage of already
ture formats and scaling capabilities that are avail
knowing which applications would likely use these
able from the video acquisition hardware
devices and what features were required As a con
sequence we could use this information to provide
e cient supp ort for the applications without miss
ing some imp ortant features and without cluttering
Intro duction
the design with features that are of no practical use
For the video acquisition driver wewere essentially
Multimedia applications are b ecoming more and
free to design our software interface from scratch
to the increasing per more p opular everyday due
For the audio driver the existence of a relatively
formance of workstations and networks The ability
to run highly demanding compression algorithms in
The term CODEC comes from a hybrid of the words
software has promoted the developmentofadvanced COmpressor and DECompressor
large base of applications forced us to implement
compatibility interface We decided to lo ok at
a Record buffer Playback buffer
compatibility issues only after the architecture of
Input Output
er was designed so that these issues did not the driv Mixer Mixer RDY RDY
DMA FREE DMA FREE in uence our design
AUX ADC DAC
pap er is structured as follows In Section
The CD CD
we present the audio driver starting with a brief
description of audio hardware followed by a dis
cussion of the various features implemented bythe
Figure A mo del of a typical full duplex audio device
driver and a brief discussion of our design choices
On the left is the record or input section on the right
We discuss the same topics in Section for video
the playback or output section
acquisition supp ort Finally Section discusses the
implementation status of the drivers presented in
system s mainboard as in the case of Intel based
this pap er and Section contains our conclusions
PC s In some other cases the audio device has
an internal DMA engine which acquires control of
the bus and p erforms the data transfer devices on
a PCI bus generally op erate in this way As an al
Audio
ternative the audio device could havesomeinternal
memory where data is bu ered e g under control
of a sp ecialized pro cessor and the main pro cessor
The structure of a typical audio device is shown in
has only to transfer blo cks of data at given intervals
Figure with two logically indep endent systems
p ossibly using programmed I O
providing supp ort for audio record and playback
In some cases the audio device has an on b oard Digi
In the record section analog sources are routed
tal Signal Pro cessor DSP which can b e used to run
through a mixer to the Analog to Digital Con
data compression algorithms o oading the main
verter ADC whose output is stored into a bu er
pro cessor from this task This is in general a useful
The hardware dictates the native resolution or
b ecause some algorithms e g GSM or MPEG en
bits data formats linear or law channels
co ding deco ding are exp ensive and it is cheap er to
mono stereo and sampling rate The software im
run them on a DSP which has b ecome a commo d
plements the bu er and may additionally implement
ity device b ecause it is extensively used in cellular
some data format conversion b efore passing data to
phones and other digital audio devices
the application program
There are large variations in the capabilities of the
The playback section uses a bu er to hold data that
mixers as well Some simple devices just have one
is sent to the Digital to Analog Converter DAC
input and one output channel directly connected to
The output of the DAC is fed to a mixer which
the ADC and the DAC resp ectively with no vol
combines together other analog sources and sends
ume con trols The ma jority of audio devices for
them to the appropriate outputs Similar to record
PC s have a simple multiplexer with only a master
ing the op erating system software implements the
volume control on the input section and a full fea
bu ering strategy and p ossible format conversions
tured mixer with indep endent volume controls for
to supplement the native features supplied by the
the various channels on the output section Newer
hardware
devices have separate full featured mixers on b oth
the input and output paths containing a large se
The DAC and the ADC are usually part of the
lection of sources and destinations along with the
same physical device referred to as the CODEC
ability of p erforming miscellaneous functions such
The transfer of samples between the CODEC and
as swapping left and rightchannels muting sources
the bu ers supplied by the op erating system can b e
etc
done in several ways Most of the time the audio
device uses a DMA channel which is presentonthe
Except for cases where hardware limitations makethe
system share resources thus preventing full duplex op eration
or o dd mixer characteristics
ally simple b ecause the requested op eration e g
setting a volume or selecting an input source gener
Appl. Appl. Appl.
ally takes place in real time and the only signi cant
veaninterface which can ease the p ort
Server issue is to ha
ware to di erent systems Library ing of soft
Device driver
Hardware (CODEC, DSP, ...)
Audio Access Granularity
Figure Possible structure of applications using the
The natural representation of sound is a stream of
audio device Application access to the driver can be
data Our driver emulates this natural representa
direct via a library or through a server
tion by supplying the application with the appropri
ate to ols to manipulate a circular queue rather than
Audio Device Initialization
requiring the application to collect and manipulate
the data in blo cks
The device driver layer in Figure is resp onsible
The audio driver has two mo des of op eration char
for the transfer of data b etween the applications and
acter and block mo de In character mo de the device
the audio hardware Prior to using the audio device
pro duces a stream of bytes and select returns
for data transfer applications will need to acquire
when one byte can b e transferred Toenter the block
the capabilities of the device supp orted sampling
mode not to b e confused with blocking mode which
rates data formats number of channels full du
is an orthogonal feature the AIOSSIZE ioctl is
plex op eration and any other device sp eci c fea
used to sp ecify the granularity to be used for the
tures and set the desired data formats These
select op eration The latter will return success
op erations are usually done by means of ioctl
fully only when at least a whole blo ck of data can b e
calls whose name and format change from system
transferred AIOSSIZE ioctl call can mo dify the
to system Although some approaches e g those
requested value if it is not an acceptable one and it
where all parameters are read or set at once using
returns the blo cksizeinuse Ablock size of or
a single call app ear to b e more elegant and e cient
will set the device driver into character mo de
than others there are in practice no di erences b e
cause this is generally a one time op eration
We want to point out that the AIOSSIZE function
only sp eci es the behavior of the select call
Another one time op eration for audio devices that
both read and write retain the usual byte
include a programmable DSP mightbe the down
granularity We found this to b e a necessary feature
loading of appropriate rmware to the DSP to p er
b ecause it p ermits applications to control the data
form the required function e g running some com
transfer rate with ne granularity rather than b eing
pression or decompression algorithm Dedicated
forced to use multiples of the blo ck size For robust
software interfaces are often used for this purp ose
ness reasons the user shouldn t make further as
and e ciency is generally not a primary concern
sumptions on the b eha vior of read and write
In particular it should not be assumed that they
Finally appropriate ioctl calls are necessary to
always transfer the requested amount of data or
control the mixers which are present in the signal
that they always work on multiples of the blo ck size
paths This is an area which would really bene
Similarly the user should make no assumptions on
t from some standardization e ort given the sig
the internal op eration of the driver e g that the
ni cant di erences in capabilities of these devices
size of DMA transfers equals the value sp eci ed with
However the control of mixer devices is conceptu
AIOSSIZE
There are signi cant di erences among audio cards As
an example some CODECS can only work in full duplex un
der some constraints others have bugs that are triggered by
Non Blo cking I O for Audio
certain op erating mo des and so on The driver can blo cker
roneous requests but the only way to make go o d use of the
available hardware is to have the driver provide a unique iden
Traditionally device drivers provide supp ort for non
ti er for the actual hardware and applications or libraries
blo cking I O This op erating mo de can b e selected
can map these identi ers to a list of features and adapt to
them by issuing the FIONBIO ioctl In non blo cking
be derived by using a real time clo ck In prac mo de read and write op erations will never blo ck
tice this metho d is imprecise b ecause of deviations at the price of p ossibly returning a short transfer
between the nominal and the actual sample rate count Non blo cking reads are p ossible even in
clo ck drifts or bu er over ows under ows which blo cking mo de byinvoking the FIONREAD ioctl
cause samples to b e missed Furthermore variable rst and then reading no more than the amountof
rate compression schemes or dynamically changing available data
bu er sizes might render the use of timers for de
termining queue o ccupation completely useless There is no standard function which is equiva
lent to FIONREAD for write op erations In our
FIONREAD and AIONWRITE only provide limited infor driver wehave implemented such a function called
mation on the status of bu ers and are used mainly AIONWRITE which returns the free space in the
when the application would liketoavoid blo c playbackbu er A write of this manybytes will not king on
blo ck even if blo cking mo de is selected In our im the device We would also liketohave alternative
plementation b oth FIONREAD and AIONWRITE track mechanisms which either notify a pro cess e g us
the status of a DMA transfer ing a Unix signal orwake it up when a sp eci ed
event o ccurs Figure shows in detail the record
playback bu ers The b oundary between the
read() and
FREE and READY regions whichwe call the cur
CAPTURE READY DMA FREE
rent DMA pointer moves with time Applications
FIONREAD
y be interested when the current DMA p ointer trigger ma
write()
reaches a given p osition relative to either the b egin
PLAYOUT FREEDMA READY ning or the end of the bu er and b e noti ed when
ent o ccurs as well as knowing the exact p o AIONWRITE this ev
trigger
sition of the current DMA p ointer relative to either
end of the bu er
Figure A view of the record and playback bu ers
with the indication of events which trigger the actions
To supp ort these functionalities wehaveintro duced
requested with AIOSYNC The thickvertical bar indicates
a single new ioctl AIOSYNC which takes the
the current p osition of the DMA p ointer Also indicated
sp eci cation of an event the current DMA p ointer
are the e ect of read and write and the values
reaching a given p osition in either bu er and an
returned by FIONREAD and AIONWRITE
action to execute when the event o ccurs The event
can b e sp eci ed relative to either end of the bu er
while the action can b e any of the following in all
Audio Synchronization
cases up on return the current DMA p ointer will
b e rep orted relative to the same bu er end as used
in the request
It is imp ortant for some applications to know the
exact status of the internal bu ers in the device
No op eration This function is blo cking unless
driver both in terms of ready and free space be
the event has already o ccurred and will return
cause some other activities could be synchronized
when the desired event o ccurs This function
with the audio streams As an example a player
is very p owerful and exible it can b e used to
program maywanttoavoid the condition that the
wait for a bu er to drain or ll up to a cer
playback bu er b ecomes empty in the middle of op
tain level or just to rep ort the status of the
eration or a telephone like application might need
transfer duplicating to some extent the infor
to control the amount of data bu ered in order to
mation supplied by FIONREAD and AIONWRITE
limit the end to end latency
although with AIOSYNC we can read the current
size of either the FREE or the READY region
In principle when the data transfer o ccurs at a con
of the bu er
stant nominal rate the amount of data bu ered can
Send a signal This function is non blo cking
This can happ en in blo cking mo de as well
Weshouldhave really used the name FIONWRITE b ecause
It schedules a signal to b e sent to the pro cess
this function is very general and not p eculiar to audio devices
when the bu er reaches a given mark This pro
E g it could b e useful on tty devices on network so ckets
vides an asynchronous noti cation which can
and everywhere we have some amount of bu ering in the
hardware or the kernel b e handled while a pro cess is active or wakes
The last ioctl we use to supp ort synchroniza up a pro cess blo cked on a system call
tion is AIOSTOP The function takes the indication
Wakeup a selecting pro cess This function is
of a channel and immediately susp ends the trans
non blo cking It causes a pro cess blo cked on
fer on that channel ushing the content of the ker
a select calltobewoken up if the desired
nel bu er The return value from the function is
event o ccurs Note that this action is not an
the amountofdataqueued in the bu er when the
exact duplicate of the previous one while a
channel was stopp ed This function allows the ap
signal scheduled with the previous function
plication to susp end a recording when it decides that
can wake up a selecting pro cess there is a p o
no more data is required and directly supp orts the
tential race condition in that the sequence
PAUSE function in audio players It is resp onsibil
ity of the application to reload any data that w as
ioctl fd AIOSYNC
ushed in the play bu er There is no explicit func
ret select
tion to start a transfer after a pause b ecause this
action is implicit when issuing a read write
mightbeinterrupted in the middle and the sig
or select call
nal b e delivered b efore the select call The
problem can b e solved but at the price of some
Function Description
obfuscation in the co de With this function
FIONBIO Selects blo cking or non blo cking
we simply request a select for exceptional
mo de of op eration for the device
conditions on the le descriptor sp eci ed us
FIONREAD Returns the amount of data which
ing the fourth parameter of select towake
can b e read without blo cking
up when the desired event o ccurs This makes
AIONWRITE Returns the amount of data which
us a ected by timing issues b ecause the event
can b e written without blo cking
AIOSSIZE Selects character or blo ck mo de of
is p ossibly logged in the device driver and re
op eration for the device setting the
p orted to the application as so on as select
threshold for select to return
is invoked
AIOGSIZE returns the blo ck size currently in
use
AIOSYNC Schedules the requested action re
AIOSYNC covers all practical needs for synchroniza
turn signal or enable select at the
tion and the cost of implementing the di erentno
o ccurrence of the sp eci ed event
ti cation metho ds is minimal b ecause they share al
Returns the current status of the
most the same co de paths
bu er
AIOSTOP Immediately stops the transfer on
The resolution of the AIOSYNC calls dep ends a lot on
the channel and ushes the bu er
Returns the residual status of the
the features of the underlying hardware On some
bu er
devices the DMA engine can b e reprogrammed on
the y to generate an interrupt exactly when the de
read Returns at most the amount of data
requested Might return a short
sired event o ccurs On other devices this cannot b e
counteven in blo cking mo de Also
done so if the desired event falls within the b ound
starts a paused recording
aries of an already started DMA transfer there is
write Writes at most the amount of data
no alternative but to p erio dically p oll the status of
requested Might return a short
the transfer In this case the resolution whichcan
counteven in blo cking mo de Also
be achieved dep ends on the granularity of the sys
starts a paused playback
tem s timer b ecause the p oll is generally done once
select In character mo de will return when
per timer tick Common values for the timer fre
at least one byte can b e exchanged
quency corresp ond to a granularityof to ms
with the device In blo ck mo de
which are acceptable for the coarse synchronization
will return when at least a full blo ck
of streams
of the size sp eci ed with AIOSSIZE
can be exchanged with the device
Consider that ms corresp onds at the sp eed of sound
Also start a paused recording
to ab out meters which is comparable to the distance b e
tween players in an orchestra the refresh rate of video devices
is in the to ms range so a synchronized video out
Table Functions supp orted by our audio driver for
put with higher resolution would b e useless moreover non
data transfer and synchronization
real time op erating systems would make a higher resolution
useless b ecause of the jitter in scheduling pro cesses
We b elieve that memory mapp ed access to the au For reference Table summarizes all the functions
dio bu er is not imp ortant in a mo dern system related to synchronization and data transfer sup
where the memory bandwidth is orders of magni p orted by our driver There is not a function to set
tude greater than the required data rate Provided the size of the kernel bu er in the device driver we
a suitable synchronization mechanism exists such did not implementitonpurpose becausewedonot
as AIOSYNC the pre initialization of the bu er de b elieve it to b e desirable or useful In fact bu er
scrib ed ab ove can b e easily implemented in the ap ing within the device driver serves to avoid requir
plication using the conventional read write ing applications to communicate with the CODEC
interface also gaining in programming clarity Ad at many small intervals The amount of bu ering
ditionally for sp ecial applications such as audio con required to avoid loss of data dep ends mainly on
ferencing the sp eed and load of the system and is much b et pre lling the bu er can be e ciently
ter decided by the op erating system rather than done in the driver itself as weinfactdo Finally
by the application Also these bu ers are used by separate pro cesses or threads can b e used to gener
the DMA engine and reside in non pageable mem ate background audio in a exible way
ory so it is again up to the OS to decide how many
resources can b e dedicated to this purp ose In many systems access to the audio device is medi
ated through a library whichprovides additional
It follows from the ab ove that applications with sp e functionalities such as mixing multiple streams
cial bu ering requirements can not rely on kernel playing entire les in the background etc This
resources which mightnotbeavailable to the same approach is certainly advisable although a libraries
degree on all systems and will need to implement can only exp ort and simplify the use of functionali
bu ering on their own this also gives them greater ties existing in the device driver
control over bu ers and improves p ortabilityofthe
co de These are the same reasons which suggest Another p opular approach for audio applications
not to include functions to manipulate the content is to mediate access to the audio device through
of the internal bu ers of the device driver a server pro cess similar to the X Windows
server The very nature of audio p oses some limita
tions to this approach Multiplexing audio output is
not as simple as for video where multiple indep en
Related Work
dent windows can be created Thus mechanisms
are required to move the fo cus of the server from
one application to the other either manually or au
There is unfortunately relatively little published
tomatically The second more imp ortant problem
work on audio device drivers Most work on multi
is related to the real time nature of audio medi
media devices fo cuses on video acquisition and ren
ating data transfers through an additional pro cess
dering which has more demanding requirements in
and p ossibly through a communication channel can
terms of pro cessing and data movement overhead
intro duce further unpredictable delays in the com
Most op erating systems implement a primitive in
munication with negative e ects on some applica
terface to the audio hardware giving only access to
tions
the basic features of the CODEC and with little
or no supp ort for synchronization
The mapping of kernel bu er in the pro cess mem
Video acquisition
ory space has gained some p opularity in recent
times on the grounds that this technique can save
some unnecessary copies of data Having the
bu er mapp ed in memory also gives the false sen Video acquisition devices capture data from ana
sation that programs can gain functionality As an log video sources enco ded as PAL NTSC or SE
example the typical use of memory mapp ed bu ers CAM standard color signals The typical structure
in audio conferencing programs is to pre initialize of these devices is describ ed in Section
the playback bu er with signi cant data e g white
noise or silence to minimize the e ect of missing The analog enco ding standards for video signals are
audio packets For games things can b e arranged so based on the sensitivityofthehuman eye with re
that some background music is placed in the bu er sp ect to intensity luminance and color chromi
and played forever without further intervention nance information Human eye receptors nerves
to suit the needs of the applications in both color are called rods and cones The ro ds detect lumi
space representation e g YUV or RGB and data nance or grey scale and are used for p eripheral vi
arrangement e g planer or packed sion The cones detect color details and far away
ob jects There are less cones than ro ds and the
The YUV representation is oriented on the human cones have signi cantly less spatial resolution
p erception of visual information and it is the pre As a consequence analog video standards enco de
ferred format for doing compression or similar pro separately the luminance and chrominance signal
cessing The RGB representation is based on the re and may utilize less bandwidth for chrominance in
pro duction of color information for p eripherals and formation
it is b etter suited for applications likedisplaying on
CRTs The most common formats are RGB and able Luminance ADC Video YUV as describ ed in T
Input
GB stands for Red Green and Blue which are the
Chrominance ADC R
principle signal comp onents of color cameras scan
ners or CRTs
Bus
stands for Cyan Magenta Yellow and Black
Decoder Scaler Formatter CMYK
h color comp onentisremoved from
Interface It describ es whic
white to generate a certain color K is de ned as
the minimum of CMY The relationship of CMY
to RGB us given as
I/O Bus
C R
M G
Figure Typical Video Acquisition Board
Y B
YUV is a color di erence comp onent representation Y
stands for luminance information and is compatible
Video Hardware Description
to a grey or black and white signal U and V are the
color di erence signals U C B Y
B
and V C R Y The direct and
R
Analog PAL NTSC or SECAM standard color sig
inverse matrixes for YUV and RGB conversion
are
nals are deco ded scaled to the desired value sent
through a formatter and written to system memory
Y R
via the I O bus interface as shown in Figure The
U G
deco der contains a luminance pro cessor a chroma
V B
pro cessor and a synchronization and clo ckproces
and
sor The deco der also extracts timing information
from the video signals The output from the de
Y R
co der consists of samples of the luminance Y and
U G
V B
chrominance U V comp onents of the signal to
gether with timing information The scaler accepts
HSI HSV orHSL stands for Hue Saturation and
the YUV data from the deco der interp olates the
Intensity V for Value or L for Luminance The
samples scales the video downward to the desired
Intensity Luminance or Value is equivalent to the
size and lters the data in b oth the horizontal and
Y value of YUV space The Hue and Saturation
vertical domains It is extremely imp ortant that
describ e the chrominance UV plane in p olar co
this function is done by the hardware b ecause this
ordinates by means of a vector its length S
op eration is exp ensive and scaling the image to the
Saturation and its angle H Hue
desired size b eforehand also reduces the amountof
data to b e transferred to memory
Table Color Representations
Data can b e arranged bypacking samples together
Video Color Space Representation
in a pixel by pixel basis packed format or in
a comp onent by comp onent basis planer format
The formatter p erforms digital color space conver
The choice dep ends on the typ e of pro cessing that
sion of the YUV data into several output formats
the application requires Packed formats are pre
ferred when doing color space conversion or render
ing and pro cessing is done on all comp onents of
YUV Y is sampled at every pixel U V are
each pixel Planer formats are generally used for
sampled at every second pixel horizontally on each
line This format may either packed or planer
computations that involve only one of the comp o
The packed pixel format has the following order
nents at a time e g DCT in order to improvethe
U Y V Y U Y V Y The planer
pattern of access to memory
format is a widthheight Y plane followed by
width height U and V planes
Table describ es the color formats supp orted bythe
YUV Y is sampled at every pixel U V are
video acquisition driver in FreeBSD Other formats
sampled at every second pixel horizontally on each
such as YUV YUV and RGB are
line and every second pixel vertically The format
utilized by some video capture hardware but are
is planer with a widthheight Y plane followed by
not supp orted by our driver Many of these formats
a width height U and V planes
and data typ es are describ ed on the FOURCC
YUV Y is sampled at every pixel U V are
home page
sampled at every fourth pixel horizontally and
every fourth pixel vertically The format is
planer with a widthheight Y plane followed by
a width height U and V planes
Frames and Fields
RGB There are two formats for RGB data
The most commonly used format is RGB
Analog video is interlaced and captured as two
Each format represents bits p er pixel
elds Each eld is denoted by either the odd
RGB Each comp onent red green or
or even designation NTSC elds are separated
blue are represented by bits giving di er
by th second and PAL elds are separated by
entlevels of each or p ossible colors the
th of a second To capture a full size frame two
most signi cantbitis The format is packed
elds b oth even and o dd are interlaced and stored
as follows xRRRRRRGG GGGBBBBB
in memory Thus a NTSC input signal generates
RGB The red comp onent is represented
frames second and a PAL input signal generates
by bits the green by and the blue is
frames second
represented by bits The format is packed
as follows RRRRRGGG GGGBBBBB this
The time di erential b etween even and o dd elds in
format is not supp orted by the driver but in
a frame may create some undesirable e ects in the
cluded here for completeness
stored video image These interlaced elds cause
RGB Each comp onent red green or blue are
high frequency comp onents to b e intro duced when
represented by bits giving di erent levels of
compression algorithms are used There are several
each or million colors The format is packed
metho ds of eliminating these undesired e ects one
as follows xxxxxxxx RRRRRRRR GGGGGGGG
simple metho d which is commonly used is to only
BBBBBBBB
capture and enco de the even or odd elds in a
frame
Table Common Digital Video Formats
Applications Interface to the Video
Hardware
Application programs may request one of three
Format Full CIF QCIF
video capture metho ds single frame capture
NTSC x x x
asynchronous continuous capture or synchronous
PAL x x x
pip elined capture which captures into a circular
bu er with low and high water marks Along with
Table Standard Square Pixel Video Sizes
the typ e of capture the application program must
sp ecify the size of the image to capture and what
format see Table to store the captured data Ta ble describ es the standard video sizes
of active frames falls b elowthe low water mark Video Capture Mo des
It is the applications resp onsibility to clear the
active bitmap and decrementthenumber of ac
The FreeBSD video acquisition driver provides three
tive frames count
mo des of capture op eration the conventional Unix
read interface memory mapp ed single capture
Frame size Rows Columns Depth
or asynchronous continuous capture and memory
Number of frames Numb er of frames in the bu er
mapp ed synchronous multi frame ring bu er cap
Low water mark Capture stops when the number
ture
of active frames is than this
number
High water mark Capture starts when number of
Conventional Unix read interface This
active frames is this number
metho d of capture may b e used with little pro
Active bitmap Bitmaskofactive frames kernel
gramming cost and is used for capturing a sin
sets application clears
gle frame of data In this mo de the applica
Number of active Countofactive frames kernel in
tion op ens the device sets the capture mo de
frames crements application decrements
and size and uses the read system call to
Frame
load the data into an application de ned bu er
Frame N
The hardware copies video data into a system
memory lo cation When a full frame is avail
Table Memory Map of Synchronous Capture Mo de
able the system copies this data from system
memory into the user data bu er and returns
control to the application program
The memory mapp ed interface to video data was
motivated by the overhead of copying the video data
Memory mapp ed single capture or asyn
from system space to user space Dep ending on the
chronous continuous capture In this
size and format of the requested data there maybe
mo de the application op ens the device sets
asubstantial savings of b oth CPU time and mem
the capture mo de and size memory maps the
ory bandwidth by avoiding this copy and directly
frame bu er into the user pro cess space and is
accessing the data More detail on this sub ject is
sues either the single capture or the continuous
discussed in the p erformance section
capture ioctl to load the data into the mem
ory mapp ed bu er The hardware copies video
Multiple frame bu ers are useful so that the applica
data into a system memory lo cation In single
tion program can utilize the current frame while the
capture mo de the ioctl call returns once
system is capturing the next frame On slower ma
the data capture is complete In continuous
chines several frames may b e captured while the ap
capture mo de the ioctl call returns imme
plication is pro cessing a previously acquired frame
diately and the memory bu er is up dated con
By utilizing multiple frame bu ers the application
tinuously until the application requests video
can elect to skip intermediate frames and pro cess
capture b e discontinued
the most recently acquired frame while maintaining
synchronization with the video stream
Memory mapp ed synchronous multi frame
ring bu er capture The application op ens
the device sets the geometry memory maps the
Video Synchronization
common control structure and the data de nes
the frame size numb er of frames high and low
water marks and starts synchronous continuous
There are several synchronization to ols that the
capture mo de
video device driver supplies to application pro
Table describ es the memory mo del used by
grams
the video capture driver to synchronize the ker
nel with the application program The kernel
Signals The driver will send a signal to the ap synchronizes by incrementing the countofthe
each time a frame is com plication program number of active frames and marking the frame
plete This is useful for asynchronous noti ca as active in the active bitmap If the number of
tion of frame completion in continuous mo de active frames exceeds the high water mark the
or frame available in synchronous mo de driver ceases to capture data until the number
TV viewer may consume the entire memory band Time stamps The driver will app end a time
width Other applications that require compression stamp to each frame The time stamp is the
and decompression of the video data will takead time the capture of the frame was completed
ditional memory bandwidth and CPU p ower Time is expressed in seconds and microseconds
since midnight hour January This
Because of the bandwidth required for full size full is useful for combining video streams with time
motion video MBytes second limited mem stamp ed audio streams
ory bandwidths and available CPU p ower it is dif
Common Bitmap In synchronous capture mo de
cult to achieve full motion compression of video in
when a frame is captured the frame number is
real time without hardware supp ort
marked as activein a bitmap It is up to the
application to clear the bit This is useful for
determining the most recent frame as well as
howmany frames have b een captured since the
Availability and Implementation
last comparison of the bitmap
Status
Video Performance Issues
The audio driver presented in this pap er is now a
standard comp onentoftheFreeBSD op erating sys
tem and it is in widespread use with most legacy
The video capture driver is able to capture frames
applications already working with it either unmo di
into system memory at full NTSC or PAL sp eeds
ed or with minor mo di cations to the audio mo d
provided the I O bus can sustain the data rates re
ule Because the new driver has better supp ort
quired for full sp eed capture These rates are shown
for full duplex op eration than the previously ex
in Table With the intro duction of the Intel Tri
isting Voxware driver audio conferencing to ols like
ton chip set TX full sp eed full frame video
vat and rat can now b e used on a wide
acquisition was p ossible on the PC
range of hardware Compared to the previously ex
isting driver the new one has a much simpler con
guration b ecause a single device pcm entry
Analog Sp eed Size Rate RGB
in the kernel con guration le brings in supp ort for
Format f s pix f MBytes sec
anumb er of di erent cards The task of identifying
NTSC x MB s
the correct card typ e and doing sp eci c resource
PAL x MB s
con guration is now left to the driver rather than
to the user at system con guration time
Table Data Rates for Video Acquisition
Newer and beta releases of the driver to supp ort
more cards and x bugs are available from the au
One p ossible application for the video acquisition
thor s home page At the time of this writing
device would be to have the acquisition hardware
all of the supp orted cards use the services of the ISA
place the digitized video data into memory and then
DMA controller to supp ort DMA op erations As a
copy this data to the video display device e g aTV
consequence most of the functionalities describ ed in
viewer This application would not have to com
this pap er could b e supp orted by using some simple
press the data but would have to move the data
co de to fetch the transfer status from the ISA DMA
from memory to the frame bu er on the video dis
controller In order to obtain the asynchronous no
play device
ti cations needed to wake up sleeping pro cesses two
approaches have b een followed If the audio device
Memory bandwidth on a common PC can range
supp orts interrupting a DMA op eration on the y
from MB s to MB s More recent PC
then the device is reprogrammed to generate an in
interface chipsets have memory bandwidths on the
terrupt when the desired event o ccurs When this
order of MB s On some machines just
periodic handler is scheduled to is not p ossible a
the simple movement of the video data such as a
pro cess the event within one timer tick from its o c
At the time of writing of this pap er a newly intro duced
currence The overhead for the p erio dic handler is
Intel chip set BX has b een rep orted to have a memory
bandwidth of MB s and CPUs are running at Mhz and display video streams on a single PC without hardware
Even at these rates it will be di cult to enco de deco de assistance
only sp ecify the external interface of the de very small and the resolution is ms with the de
vice driver do not rely or make assumptions fault timer frequency Hz
ab out the internal structure of the driver or of
the hardware Do not exp ort information which The video acquisition driver is part of the standard
could lead to non p ortable co de to b e written distributions of FreeBSD Separate device
entries are required for the two most p opular PCI
keep the numb er of functions small
based frame grabb ers the Matrox meteor which
basedonthe Philips chipsets and the bktr which
avoid duplication of functionalities in the inter
based on the Bro oktree chipsets
face so that there is no doubt on what is the
preferred metho d to achieve a given result
Many applications can utilize the digitized video
provided by the video acquisition driver in FreeBSD
We believe we have achieved the ab ove goals be
The motivation for writing the driver was to work
cause our interfaces are small p owerful and simple
with network streaming video conferencing pro
to use and resulted in a very compact implementa
grams such as nv and vic Both of these
tion
programs use the video acquisition driver in contin
uous capture mo de and incorp orate software com
Hardware compression and decompression solutions
pression algorithms Because of the p erformance of
will have to b e implemented in order to allowreal
PC memory architecture see section and the
time storage and retrieval of video data Cur
amount of data see Table full frame full rate
rently there are to o many compression algorithms
data compression is di cult to achieve on a current
and standards and to o few low cost b oards that im
dayPC
plement the ma jor standards One hardware
solution which lo oks promising is the MPACT
Fxtv is a recent application that uses the video
chipset
acquisition device to write digitized video data di
rectly to the linear frame bu er of the video display
The video interface functions well and has b een
device Both the meteor and the bktr drivers sup
extended to work with the Bro oktree
p ort direct PCI to PCI bus transfers Because this
video capture chip however an extensible generic
application do esn t require real time compression
multimedia kernel level interface with integrated
and it utilizes the PCI to PCI transfer capabilities
and synchronized audio video compression and
it can maintain full sp eed full motion video on the
decompression will be required for advanced ap
display device without a ecting available memory
plications Some thought has b een given to
bandwidth or pro cessor p ower
this interface and it still under discussion on the
ia freebsd org mailing list freebsd multimed
The video driver interface do es not currently imple
ment the select system call For consistency and
synchronization purp oses this call should b e added
to the video driver
Acknowledgments
This work has been partly supp orted by the Min
Conclusions
istero dell Universita e della Ricerca Scienti ca
e Tecnologica MURST in the framework of the
Pro ject Design Metho dologies and To ols of High
Wehave describ ed software interfaces for audio and
Performance Systems for Distributed Applications
video acquisition devices to improve supp ort for
It has also b een partially supp orted by the Electri
multimedia applications In de ning these inter
cal Engineering and Computer Science Department
faces wehave tried to pursue the following goals
at the University of Wisconsin Milwaukee
lo ok at the requirements of applications rather
The authors wish to thank Mark Tinguely co
than trying to extend an existing software in
author of the meteor device driver and Amancio
terface
Hasty who wrote the Bt device driver and has
provided invaluable supp ort to manyof the multi
The meteor driver has also been p orted to the Linux
op erating system media drivers and applications There are many
Steve McCanne and Van Jacobson vic A users on the freebsd multimedia freebsd org
exible framework for packet video in Pro c mailing list who have also provided invaluable sup
of ACM Multimedia Nov port
Larry McVoy and Carl Staelin lmb ench
Portable to ols for p erformance analysis
Usenix pro ceedings January
References
John D McCalpin Memory bandwidth and
machine balance in current high p erformance
The FreeBSD op erating system home page
computers IEEE Technical Conference on
http www freebsd org
Computer Architecture newsletter December
P Bahl The J Family of Video and Au
dio Adapters Software Architecture Digital
V Jacobson S McCanne The
Technical Journal vo n pp
LBL audio to ol vat Manual page
http www nrg ee lbl gov vat
The Op en Sound System OSS Web page
http www front tech com
V Hardman M A Sasse M Handley
A Watson Reliable audio for use over
Microsoft Corp Do cumentation on
the Internet INET conference
the DirectSound SDK available at
http www microsoft com DirectX
V Hardman I Kouvelas M A Sasse
A Watson A packet loss Robust Audio
T M Levergood A C Payne et al Au
Tool for use over the Mb one Research
dioFile Network Transparent System for Dis
Note RN Dept of Computer Science
tributed Audio Applications USENIX Sum
University College London
mer Conference June
Luigi Rizzo s FreeBSD home page
Renda The Network Audio J Fulton G
http www iet unipi it luigi
System th Annual X Technical Conference
FreeBSD html
in The X Resource Issue Nine January
Randall Hopp er s FreeBSD
X TV home page
Flight Training Handb o ok U S Depart
http multiverse com rhh fxtv
mentofTransp ortation Federal Aviation Ad
L A Rowe Video Compression for Desk
ministration pp AC A
top Applications Informationstechnik und
Color space digital co ding and sampling
Technische Informatik Vol No pp
schemes for video signals Desktop Video
August
Data Handb o ok Philips Semiconductors
The MPACT home page Chromatic Re
Data Handb o ok IC pp
search http www mpact com
CCIR Consultative Committee on Interna
tional Radio Recommendation En
co ding Parameters of Digital Television for
Studios
The FOURCC Four Character Co de home
page http www webartz com fourcc
Bt Single Chip Video Capture for PCI
Data Sheet Bro oktree Corp oration August
k Exp eriences with real time Ron Frederic
software video compression in Sixth Interna
tional Workshop on Packet Video July