Java Media Frameworks (JMF)

Die bunte Welt und die dunkle Seite des Java Media Frameworks (JMF)

[email protected] © 10/2004 [email protected] Java Media Framework (JMF) 1 Agenda

● Die bunte Welt von JMF ● Architektur ● The Dark Side ● The Bright Side ● Dokumentation

TV is chewing gum for the eyes. -- Frank Lloyd Wright

● JMF = Java Media Framework ● Umgang mit Video- und Audio-Daten ● API für Audio-/Video-Player ● Streaming-API

● http://java.sun.com/products/java-media/jmf/index.jsp – akt. Version: 2.1.1e (v. 2003) ● Framework: lib – JARs und Nativ Libs ● Demos: bin – JMStudio, JMF-Registry ● Plugin für IE / Mozilla – Linux: Handarbeit notwendig

● Audio: – G723, GSM, WAV, ULAW – MP3: ab JMF 2.1.1b wieder entfernt (aus Lizenz-Gründen) ● Video: – CINEPAK, H263, MJPG, MPEG, MOV (QuickTime) – streamingfähig: ● QuickTime, MPEG, WMF – nicht streamingfähig ● AVI

● Audio-/Video-Player – Media lokal / über URL adressierbar ● auch als Streamer einsetzbar ● Achtung: läuft nicht unter JDK 1.5 – NullPointerException – ebenfalls anfällig: Sound SPI (MP3SPI v. javazoom) ● Doku-Ersatz bzw. -Ergänzung

● s. http://java.sun.com/products/java-media/jmf/ ● aber: – teilweise Mängel in der Programmierung – teilweise veraltet ● SessionManager durch RTPManager abgelöst

● Beispiel für einen einfachen Video-Player: MediaPlayer1 = new javax.media.bean.playerbean.MediaPlayer(); MediaPlayer1.setMediaLocation("file://jvideo/media/Sample1.mov"); MediaPlayer1.start();

● aus http://java.sun.com/products/java-media/jmf/2.1.1/playerbean.html

© 10/2004 [email protected] Java Media Framework (JMF) 10 Buffer interface interface interface java.util.EventObject Error CaptureDeviceManager interface Exception interface PackageManager Control Clock Duration MediaEvent MediaError TimeBase MediaException ControllerListener +FLAG_EOM:int -cdm:CaptureDeviceManager -pm:PackageManager +FLAG_DISCARD:int +RESET:Time +DURATION_UNBOUNDED:Time -mGetDeviceList:Method -mGetProtocolPrefixList:Method +FLAG_SILENCE:int +DURATION_UNKNOWN:Time -mGetDevice:Method -mSetProtocolPrefixList:Method +MediaEvent +MediaError +MediaException +controllerUpdate:void +FLAG_SID:int +syncStart:void -mAddDevice:Method -mComm itProtocolPrefixList:Method +MediaError +MediaException +FLAG_KEY_FRAME:int -mRemoveDevice:Method -mGetContentPrefixList:Method controlComponent:Component +stop:void time:Time +FLAG_NO_DROP:int -mComm it:Method -mSetContentPrefixList:Method +mapToTimeBase:Tim e nanoseconds :long +FLAG_NO_WAIT:int +setRate:float duration:Time -mComm itContentPrefixList:Method +FLAG_NO_SYNC:int -runMethod:Object -sigNone:Clas s [] +FLAG_SYSTEM_TIME:int +getDevice:CaptureDeviceInfo -sigVector:Class [] timeBas e:TimeBas e java.util.EventListener +FLAG_RELATIVE_TIME:int +getDeviceLis t:Vector -protoPrefixList:Vector s topTim e:Time NotPrefetchedError ClockStartedError NotRealizedError StopTimeSetError NotConfiguredError IncompatibleSourceException ResourceUnavailableException UnsupportedPlugInException BadHeaderException NoDataSinkException IncompatibleTimeBaseExcep... CannotRealizeException NoPlayerException ClockStoppedException NoDataSourceException ControllerAdapter +FLAG_FLUSH:int mediaTime:Tim e +addDevice:boolean +FLAG_SYSTEM_MARKER:int mediaNanos econds :long +removeDevice:boolean SystemTimeBase Manager -runMethod:Object +FLAG_RTP_MARKER:int s yncTime:Time +commit:void +commitProtocolPrefixList:void +FLAG_RTP_TIME:int rate:float +NotPrefetchedError +ClockStartedError +NotRealizedError +StopTim eSetError +NotConfiguredError offset:long -VERSION:String +IncompatibleSourceException +ResourceUnavailableException +UnsupportedPlugInException +BadHeaderException +NoDataSinkException +IncompatibleTimeBas eException +CannotRealizeException +NoPlayerException +ClockStoppedException +NoDataSourceException +cachingControl:void +commitContentPrefixList:void +FLAG_BUF_OVERFLOWN:int +ClockStartedError +MAX_SECURITY:int +IncompatibleSourceException +UnsupportedPlugInException +BadHeaderException +NoDataSinkException +IncompatibleTimeBas eException +CannotRealizeException +NoPlayerException +ClockStoppedException +NoDataSourceException +controllerError:void getDeclaredMethod:Method +FLAG_BUF_UNDERFLOWN:int +CACHING:int +dataLos tError:void +FLAG_LIVE_DATA:int +LIGHTWEIGHT_RENDERER:int +dataStarved:void protocolPrefixList:Vector +TIME_UNKNOWN:long time:Time +PLUGIN_PLAYER:int +internalError:void contentPrefixList:Vector +SEQUENCE_UNKNOWN:long nanoseconds :long -num berOfHints:int +resourceUnavailable:void -sysTimeBase:Sys temTimeBase +durationUpdate:void +copy:void +UNKNOWN_CONTENT_NAME:String NoProcessorException +mediaTimeSet:void +copy:void interface interface interface interface interface java.io.Serializable interface ControllerEvent GainChangeEvent -jdkInit:boolean +rateChange:void +clone:Object Controls MediaHandler GainControl Controller CachingControl Time Track -forNam e3Args M:Method +stopTim eChange:void eventSrc:Controller eventSrc:GainControl +trans ition:void -getSys temClass LoaderM:Method +NoProces s orException format:Format +LATENCY_UNKNOWN:Time +LENGTH_UNKNOWN:long +ONE_SECOND:long +TIME_UNKNOWN:Time newMute:boolean +prefetchCom plete:void -systemClass Loader:Class Loader +NoProces s orException interface flags:int +Unrealized:int +TIME_UNKNOWN:Time +FRAME_UNKNOWN:int +ControllerEvent newDB:float +realizeComplete:void +getControl:Object +setDB:float -getContextClass LoaderM:Method DownloadProgressListener EOM:boolean +Realizing:int -NANO_TO_SEC:double +toString:String newLevel:float +start:void +setLevel:float -fileSeparator:String dis card:boolean +Realized:int +readFram e:void +stop:void controls:Object[] s ource:DataSource +addGainChangeListener:void -hintTable:Hashtable data:Object +Prefetching:int downloading:boolean +Time +mapTimeToFrame:int +GainChangeEvent +deallocate:void +removeGainChangeListener:void s ourceController:Controller DONE:int header:Object +Prefetched:int contentLength:long +Time +mapFram eToTime:Time s ource:Object SUCCESS:int +endOfMedia:void +downloadUpdate:void length:int +Started:int contentProgress :long #secondsToNanos econds:long source:Object +restarting:void offset:int mute:boolean progres sBarComponent:Component +stopAtTim e:void format:Format sourceGainControl:GainControl -Manager DB:float +stopByRequest:void tim eStam p:long +realize:void controlComponent:Component nanoseconds:long enabled:boolean DB:float +getVersion:String level:float +audioDeviceUnavailable:void duration:long +prefetch:void s econds:double startTim e:Time level:float +createPlayer:Player +configureComplete:void interface interface sequenceNumber:long +deallocate:void trackLis tener:TrackListener mute:boolean +createPlayer:Player +controllerClosed:void Prefetchable Drainable +close:void +createPlayer:Player +sizeChange:void +getControl:Control +createRealizedPlayer:Player +connectionError:void +addControllerListener:void +createRealizedPlayer:Player +formatChange:void +removeControllerListener:void +createRealizedPlayer:Player +drain:void +replaceURL:void +createProces sor:Process or +showDocument:void PlugInManager java.lang.Cloneable s tate:int +createProces sor:Process or prefetched:boolean +controllerUpdate:void java.io.Serializable targetState:int +createProces sor:Process or -pim:PlugInManager Format s tartLatency:Time +createRealizedProcessor:Process or +DEMULTIPLEXER:int controls:Control[] +createDataSource:DataSource +CODEC:int +NOT_SPECIFIED:int +createDataSource:DataSource interface bean +EFFECT:int +TRUE:int +createMergingDataSource:DataSource TrackListener playerbean +RENDERER:int +FALSE:int -reflectMDS:DataSource MCA +MULTIPLEXER:int +intArray:Clas s -reflectDS:DataSource -mGetPlugInList:Method +shortArray:Class +createCloneableDataSource:DataSource -mSetPlugInList:Method +byteArray:Class sync:boolean[] +readHasBlocked:void interface interface interface interface interface CachingControlEvent MediaTimeSetEvent StopTimeChangeEvent DurationUpdateEvent RateChangeEvent ControllerClosedEvent AudioDeviceUnavailableEvent TransitionEvent +getSystemTimeBase:Tim eBase -mCommit:Method 0..* +formatArray:Class state:int PlugIn DataSink M ediaProxy Player ExtendedCachingControl createPlayerForContent:Player -mAddPlugIn:Method #clz:Clas s createPlayerForSource:Player -mRemovePlugIn:Method -encodingCode:long control:CachingControl +MCA +BUFFER_PROCESSED_OK:int progres s:long createProcess orForContent:Proces sor -mGetSupportedInputFormats:Method +MediaTimeSetEvent +StopTimeChangeEvent +DurationUpdateEvent +RateChangeEvent +ControllerClos edEvent +AudioDeviceUnavailableEvent +TransitionEvent createProcess orForSource:Processor -succeed:void +BUFFER_PROCESSED_FAILED:int interface interface -mGetSupportedOutputFormats:Method +Format +start:void +start:void +pauseDownload:void +toString:String +toString:String +toString:String +toString:String +ControllerClos edEvent +toString:String +createDataSink:DataSink -fail:void +INPUT_BUFFER_NOT_CONSUMED:int +CachingControlEvent GainChangeListener Owned -emptyFormat:Format [] +Format +stop:void +addController:void +resumeDownload:void stateName:String +getCacheDirectory:String +configureComplete:void +OUTPUT_BUFFER_NOT_FILLED:int +toString:String +equals:boolean +open:void dataSource:DataSource +rem oveController:void +addDownloadProgressListener:void +realizeComplete:void +PLUGIN_TERMINATED:int mediaTime:Time s topTime:Time duration:Time rate:float mess age:String +setHint:void runMethod:Object +matches:boolean +close:void +removeDownloadProgressListener:void previousState:int +getHint:Object +controllerError:void cachingControl:CachingControl +getPlugInLis t:Vector +intersects:Format +addDataSinkListener:void visualComponent:Com ponent currentState:int -blockingCall:void +deallocate:void +gainChange:void +open:void contentProgress:long +controllerClosed:void +setPlugInLis t:void +is SameEncoding:boolean +removeDataSinkListener:void gainControl:GainControl bufferSize:Time targetState:int +getDataSourceLis t:Vector +close:void +commit:void +is SameEncoding:boolean controlPanelComponent:Component startOffset:long +getHandlerClassList:Vector owner:Object +reset:void +addPlugIn:boolean -getEncodingCode:long outputLocator:MediaLocator endOffset:long +getProcessorClassList:Vector +removePlugIn:boolean +relax:Format contentType:String buildClassList:Vector nam e:String +getSupportedInputFormats:Format [] +clone:Object getContentPrefixList:Vector FormatChangeEvent +getSupportedOutputFormats:Format [] #copy:void ArchitektgetProtocolPrefixList:Vector uSizeChangeEvent r +toString:String -getNTypesOfCaptureDevices:int rtp pm protocol renderer control util datasink cdm format pim -checkIfJDK12:boolean encoding:String event +PackageManager +PushDataSource +VideoRenderer +FormatControl +ImageToBuffer +DataSinkEvent +CaptureDeviceManager +VideoFormat PlugInInfo getClas sForName:Clas s +SizeChangeEvent dataType:Clas s rtcp +PullBufferDataSource +VisualContainer +MonitorControl +BufferToIm age +EndOfStreamEvent +H261Format +PlugInManager interface interface interface interface interface interface StopEvent StartEvent DataLostErrorEvent ControllerErrorEvent RealizeCompleteEvent PrefetchCompleteEvent ConfigureCompleteEvent +OutputDataStream +SourceCloneable +StreamWriterControl +DataSinkErrorEvent +H263Format Lis tInfo 0..* 0..* Multiplexer Codec Renderer DataSinkProxy Demultiplexer Processor width:int +Glob alReceptionStats +Controls +RtspControl +DataSinkListener +RGBFormat PlugInVector height:int +RTPStream +RateConfiguration +PortControl +JPEGFormat ClassNam eInfo +Configuring:int scale:float +DataSource +FrameGrabb ingControl +YUVFormat +StopEvent +StartEvent +DataLos tErrorEvent +ControllerErrorEvent +RealizeCompleteEvent +PrefetchCompleteEvent +ConfigureCompleteEvent +SessionManager +Configured:int +FileTypeDes criptor +TrackControl +UnsupportedFormatException +getSupportedOutputContentDescriptors+g:CeotnteSupntpDoertscriedOutpptor[]utFormats:Format [] +setInputFormat:Format +getContentType:String +start:void +toString:String +toString:String +DataLos tErrorEvent +ControllerErrorEvent +LocalParticipant +InputSourceStream +FrameProcessingControl +IndexedColorFormat +setNumTracks:int +setInputFormat:Format +start:void +stop:void +toString:String +EncryptionInfo +configure:void +URLDataSource +KeyFrameControl +AudioFormat Serializable ProcessorModel +setInputFormat:Format +setOutputFormat:Format +stop:void +setPosition:Time mediaTime:Time mediaTime:Time +Rem oteListener +setContentDescriptor:ContentDescriptor +CachedStream +H261Control +FormatChangeEvent CaptureDeviceInfo +process:int +process:int +process:int +InvalidSessionAddressException timeBaseTime:Time +SourceStream +SilenceSuppressionControl -formats:Format [] +setContentDescriptor:ContentDescriptor supportedInputContentDes criptors:ContentDescriptor [] +DataChannel trackControls :TrackControl[] +CaptureDevice +FramePositioningControl -outputContentDescriptor:ContentDes criptor supportedInputFormats :Format [] supportedInputFormats:Format [] tracks:Track[] +TransmissionStats supportedContentDescriptors:ContentDescriptor[] +SSRCInUseException +SourceTransferHandler +MpegAudioControl +CaptureDeviceInfo s upportedInputFormats:Format [] positionable:boolean +ContentDescriptor +PacketSizeControl +Proces sorModel contentDescriptor:ContentDes criptor +RTPControl +CaptureDeviceInfo dataOutput:DataSource randomAcces s:boolean +BufferTransferHandler +BitRateControl +Proces sorModel dataOutput:DataSource +SessionListener +equals:boolean mediaTime:Time +PushBufferDataSource +BufferControl +Proces sorModel +Rem oteParticipant +toString:String duration:Time +PushSourceStream +H263Control +Proces sorModel +RTPConnector +PushBufferStream +FrameRateControl +getTrackCount:int +ReceiveStreamListener formats:Format [] +PullBufferStream +QualityControl +getOutputTrackFormat:Format +ReceptionStats locator:MediaLocator +Seekable +is FormatAcceptable:boolean +RTPPus hDataSource name:String +Glob alTransmissionStats +RateConfigureab le interface StopAtTimeEvent StopByRequestEvent EndOfMediaEvent RestartingEvent DataStarvedEvent DeallocateEvent InternalErrorEvent ResourceUnavailableEvent ConnectionErrorEvent +SendStream +Positionable contentDescriptor:ContentDescriptor Effect +RateRange inputDataSource:DataSource +Participant +RTPManager +PullSourceStream inputLocator:MediaLocator +PullDataSource +StopAtTim eEvent +StopByRequestEvent +EndOfMediaEvent +RestartingEvent +DataStarvedEvent +DeallocateEvent +InternalErrorEvent +Res ourceUnavailableEvent +ConnectionErrorEvent +Ses sionManagerException +InternalErrorEvent +Res ourceUnavailableEvent +ConnectionErrorEvent +Ses sionAddres s +RTPHeader +ReceiveStream +RTPSocket java.io.Serializab le +SendStreamListener MediaLocator

-url:URL -locatorString:String

+MediaLocator +MediaLocator +toString:String +toExternalForm:String

URL:URL protocol:String remainder:String javax.media.*

Referenzimplementierung JMF API von Sun und IBM

JMF Plugin API

Demulti- Multi- Effect Codec Renderer plexer plexer

Hardware (Capture Devices, etc.)

● RTSP (Realtime Streaming Protokoll RTSP – RFC 2326 – Verbindungs-Auf/Abbau, Steuerung Codecs (Komprimierung) ● RTP (Realtime Transfer Protokoll) RTP RTCP – RFC 1889 + 1890 UDP – Daten- (RTP) und Steuerungs-Pakete (RTCP) IP / IPv6 ● Session-Arten Problem: QoS / Bandbreite – Unicast -> „send and pray“ – Multicast (224-239.x.x.x) – Broadcast (x.x.x.255)

● Adressbereich: 224-239.x.x.x, RFC 1112 ● Vorteil: – Daten werden nur 1x übertragen (Datenvolumen) ● Vorraussetzungen: – multicastfähiges Betriebssystem – Multicast-fähige Router – Multicast-Adresse muss bekannt sein

Prozesso Player r Netzwerk

RTP RTP Manager Manager

Sender Empfänger

RTPManager ersetzt SessionManager (alt)

DataSource Prozesso r

DataSource Player Bildschirm

Controller

DataSink Datei

Data Send RTPManager Processor Source Stream

RTP Internet RTCP

Data Receive Player Source Stream RTPManager

Unrealized realize /RealizeCompletEvent

Realizing Realized prefetch /PrefetchCompleteEvent deallocate Prefetching Prefetched start deallocate, setMediaTime Started

stop/StopEvent

● Zeitmodell in JMF ● Intramedia- – Clock Synchronisation – SystemTimeBase 1. TimeBase Synchronisation – TimeBase 2. Kontrolle eines Players – Time durch einen anderen – Duration 3. direkte Synchronisation ● Aktuelle Zeit – getMediaTime() – setMediaTime()

● problembehaftete Beispiele: – laufen nicht immer – verwenden undokumentierte Methoden

● z.B. dataSource.getControl("javax.media.rtp.RTPControl") – referenzierte Klassen fehlen, z.B. VFWCapture ● Sourcen unter SCSL, aber unvollständig – z.B. com.sun.media.rtp.RTPSessionMgr fehlt – aber in JMF.jar vorhanden ➔ Sourcen stimmen nicht mit JMF.jar überein

● NullPointExceptions are ● Mängel in der the frameworks best Programmierung friends!!! – Exceptions werden als ● unnötige Casts Rückgabewert verschleiert (C-Stil!) – Object als Rückgabewert – IOExceptions können ● dyn. Laden von verschiedenes bedeuten: Klassen ● File konnte nicht geladen werden – Übergabe als String! ● JMF konnte keine ● Threads “verhungern” Klasse nachladen manchmal ● Framework hatte gerade keine Lust

● Controller nie im falschen Status aufrufen – keine Exception, nur undefiniertes Verhalten ● JMF programmieren ist wie um Hand anhalten: – Blumen nicht vergessen (für Schwiegermama) – nichts Falsches im falschen Moment sagen – keine Reaktion kann bedeuten: ● alles ok, es fehlt nur noch der Gehaltsnachweis ● Ablehnung – am Ende bitte das Geschirr aufräumen

You must work hard what the only that existis: JMF guide, sun code examples and jmf studio source code.

After a lot of hours you will feel better!

Antwort auf eine Anfrage im JMF-Forum bzgl. Dokumentation

● als erster Einstieg geeignet, aber ● veraltet – RTP-Dokumentation – SessionManager durch RTPManager abgelöst ● unvollständig – MimeManager fehlt – RGBScaler

● Offline-Version ● besser: Online-Version (neuerer Stand) ● noch besser: – Sourcen herunterladen – Javadoc selber generieren (aus „Medienverarbeitung in Java“) – zusätzlich: Klassendiagramm (z.B. über Together)

● für alles, was nicht in der Doku steht: – MIME-Types – Codecs – ...

● „write once run test anywhere“ ● unterschiedliches Verhalten unter Windows / Linux – Beispiele (.au/.wav) laufen unter Linux, aber nicht unter Windows (und umgekehrt) – kein mitgeliefertes MP3 unter Linux – Broadcast geht nicht ● automatische JUnit-Tests

● leistungsfähige Architektur – an JavaSound angelehnt ● konkurrenzlos – einziges plattform-unabhängige Media-Framework – MMAPI (Mobile Media API) stark an JMF angelehnt ● Erweiterungen / Plugins für JMF erhältlich

● leistungsfähiges Logging (z.B. log4j) – normales Debuggen ist nicht möglich ● Thread-Debugger ● sauberes Exception-Handling ● automatische Tests – write once test anywhere ● Netzwerk-Monitor – tcpdump, ethereal, ...

● viele Beispiele ● API-Doc zum Nachschlagen ● JMStudio debuggen ● Log-Aspekte / AspectJ

● Fobs4JMF – FFMpeg-Wrapper – FFMpeg = Library zum Aufnahmen, Verarbeiten und Streamen von Media-Daten (http://ffmpeg.sf.net) – http://fobs.sf.net ● JLayer – MP3 Library – http://www.javazoom.net/projects.html

● NeburStream – Java streaming server over JMF (leider auf Spanisch) – http://sourceforge.net/projects/neburstream/ – baut auf http://vod.sf.net auf ● VASTE (Video And Slideshow Teaching Environment) – http://www.wire.tu-bs.de/OLDWEB/multimedia/ ● X-Smiles (XML-Browser + SMIL-Player) – http://www.xsmiles.org/ ● TimCam (WebCam-Anwendung, http://timcam.sf.net) ● einige Diplomarbeiten

● Synchronized Multimedia Integration Language – SMIL (sprich: smile :-) ● W3C-Standard ● Beschreibungssprache für Media-Daten/- Präsentationen – z.B. für Playlist – für Präsentationen

● kommerziell (C++) ● OpenSource (C, C++) – Darwin Streaming Server – VideoLAN ● freie Port d. Apple ● http://www.videolan.org QuickTime Servers ● RTSP? – Reals HELIX Server – LIVE.COM Streaming ● open multi-format Media platform for digital ● http://live.sf.net media creation

● Medienverarbeitung in Java – s. http://www.dpunkt.de/buch/3-89864-184-8.html – Grundlage für Vorlesung an der TU Wien – Beispiele unvollständig ● JMF API Guide – http://java.sun.com/products/java-media/jmf/2.1.1/specdownload.html – teilweise veraltet ● API-Dok – online: http://java.sun.com/products/java-media/jmf/2.1.1/apidocs/index.html – aktueller als Offline-Version

● MCRL > MCRL Media > Media Projects > Java Media Framework – http://pirobase.pironet-ndh.com/servlet/PB/menu/1330640/index.html ● JMF Home / FAQ – http://java.sun.com/products/java-media/jmf/index.jsp – http://java.sun.com/products/java-media/jmf/reference/faqs/index.html ● SMIL – http://www.informatik.fernuni-hagen.de/import/pi3/peter/smil.htm

agentes AG

Oliver Böhm

[email protected]

Räpplenstaße 17 70191 Stuttgart