Multilingual mark-up of text-audio synchronization at a word-by-word level, how HTML5 may assist in-browser solutions

Gavin Brelstaff (gjb@ crs4.it) CRS4, Sardinia, Italy

Francesca Chessa University of Sassari, Italy

Multilingual Web Workshop Rome March 2013

MLW Rome 2013 G.Brelstaff & F.Chessa 1 First, to the movies

MLW Rome 2013 G.Brelstaff & F.Chessa 2 Movies

MLW Rome 2013 G.Brelstaff & F.Chessa 3 HTML5 video

MLW Rome 2013 G.Brelstaff & F.Chessa 4 HTML5 audio

Simply supply the vtt or srt timed-text file and the browser does it all for you line by line.

MLW Rome 2013 G.Brelstaff & F.Chessa 5 Timed-text audio on the web:

http://commons.wikimedia.org/wiki/TimedText:GraziaDeledda.ogg.en.srt

MLW Rome 2013 G.Brelstaff & F.Chessa 6 Timed-text audio srt:

MLW Rome 2013 G.Brelstaff & F.Chessa 7 Speech to Text

digital spectrogram

Speech analysis credit: Carlo Schirru, Univ. Sassari MLW Rome 2013 Aspetti fonetico-fonologiciG.Brelstaff introduttivi all’analisi & F.Chessa strumentale sull’intonazione del sardo (2006) 8 Demo + med-text MLW Rome 2013 G.Brelstaff & F.Chessa 9 Multilingual markup - recap

A human marks up the equivalances bewteen bilingual texts at three different levels: word, phrase, idea.

word

Colour-coded idea equivalence phrase

Web-based alignment and presentation of semantic equivalence [XHTML + CSS + jQuery]

MLW Rome 2013 G.Brelstaff & F.Chessa 10 HTML under the hood

...
Astonished was I: Timed Text Markup
(TTML) by 31 Jan 2013 the hush over water ... MLW Rome 2013 G.Brelstaff & F.Chessa 11 Archive format: XML TEI

… Astonished was I: Add one TEI by milestone “anchor” the hush over per audio cue-point water ... Text Encoding Initiative P5, 2012

MLW Rome 2013 G.Brelstaff & F.Chessa 12 HTML5 audio tag

HTML code

No subtitles here Javascript audio play

var myAudio=$('#audio'); // jQuery selector myAudio.get(0).currentTime = 15.5 //secs myAudio.get(0).play(); // start HTML5 audio

Javascript text sync (scarry stuff instead)

setTimeout('switch_on (... )', start_ms ); // times in setTimeout('switch_off(... )', end_ms ); // milliseconds

See also: westonruter--audio-read-along on github

MLW Rome 2013 G.Brelstaff & F.Chessa 13 Cue-point mark-up tools? www.nikse.dk/subtitleedit

MLW Rome 2013 G.Brelstaff & F.Chessa 14 Cue-point mark-up tools? www.fon.hum.uva.nl/praat

MLW Rome 2013 G.Brelstaff & F.Chessa 15 Cue-point mark-up (visual interface)

Insert & nudge cue-points directly on the web-page while listening

MLW Rome 2013 G.Brelstaff & F.Chessa 16 Our aim: to activate poetic memory

Involve ear, tongue and eyes Nel mezzo del to cammin di nostra reinforce memory/ vita mi ritrovai per appreciation across una selva oscura ché la diritta via the language divide. era smarrita

MLW Rome 2013 G.Brelstaff & F.Chessa 17 "We preferred poems that make a powerful impact when they are heard aloud - not because they are theatrical, but because they dramatise experiences that surprise us into a new apprehension of ourselves and our capacity for imagining, thinking and marvelling."

Mr Gove said the project would ensure that more children would be captivated by great poetry and it would help "pass our cultural legacy on to the next generation".

MLW Rome 2013 G.Brelstaff & F.Chessa 18 Caesar’s Europe: poetic memory http://www.perseus.tufts.edu/hopper/text?doc=Perseus%3Atext%3A1999.02.0001%3Abook%3D6%3Achapter%3D14

The Druids … learn by heart a great number of verses; …. Nor do they regard it lawful to commit these to writing …

MLW Rome 2013 G.Brelstaff & F.Chessa 19 Poetic memory internal external

Internal: Immediately available to society (in cache) External: Appreciation, Available comprehension across on demand Juliane Stiller, Marlies Olensky MLW Dublin 2012 the language divide? (in digital archive)

MLW Rome 2013 G.Brelstaff & F.Chessa 20 • Information is not knowledge • knowledge is not wisdom • wisdom is not truth …

F.Zappa 1979

MLW Rome 2013 G.Brelstaff & F.Chessa 21 Poetic memory informs society

1562 Arthur Prose plot Lost on us Brooke (information)

1597 Able to William Poetic language inform Shakespeare (information plus) society

MLW Rome 2013 G.Brelstaff & F.Chessa 22 Back to the movies – an extreme social network

Learning by rote or Learning by heart?

MLW Rome 2013 G.Brelstaff & F.Chessa 23 http://www.youtube.com/watch?v=ZriW3CPU9G4&list=PLGGjdQw3TIx9Dk0CYaHS9R6LyI_HmtG9i

MLW Rome 2013 G.Brelstaff & F.Chessa 24 That’s all folks:

Gavin Brelstaff (gjb@ crs4.it) CRS4 09010 Pula (CA) – Sardinia, Italy

Francesca Chessa University of Sassari, Italy

MLW Rome 2013 G.Brelstaff & F.Chessa 25