Technical Report

Transcripción

Technical Report

Technical Report
Abstract: This technical report analyzes the state of the art of technology for musical real-time
interaction over computer/data networks (NMP, Networked Music Performances). It describes the
challenges that it introduces and the software implementations that make NMP possible at present. It
also presents a practical experience based on NMP where a group of musicians, located at geographical
distant places, interact over a network to perform as they would if they were located in the same room.
Document Id.:
CESGA-2014-003
Date:
01/05/14
Responsible:
Natalia Costas and
Ivelina Atanasova
Karagyulieva
Status:
FINAL
Technical Report CESGA-2014-003
Networked Music Performance
May 2014
Galicia Supercomputing Centre
Summary:
This technical report analyzes the state of the art of
technology for musical real-time interaction over
computer/data networks (NMP, Networked Music
Performances). It describes the challenges that it
introduces and the software implementations that make
NMP possible at present.
It also presents a practical experience based on NMP
where a group of musicians, located at geographical
distant places, interact over a network to perform as
they would if they were located in the same room.
Keywords:
Networked
Music
Performance,
Latency,
High
performance data network applications, RECETGA
(Galicia Science and Technology Network)
License:
Author: Ivelina Atanasova Karagyulieva
Author: Natalia Costas Lago
Copyright notice:
Copyright © CESGA, 2014.
See www.cesga.es for details on the copyright holder.
You are permitted to copy, modify and distribute copies of this document
under the terms of the CC BY-SA 3.0 license described under
http://creativecommons.org/licenses/by-sa/3.0/
Using this document in a way and/or for purposes not foreseen in the
previous license requires the prior written permission of the copyright
holders.
The information contained in this document represents the views of the
copyright holders as of the date such views are published.
THE INFORMATION CONTAINED IN THIS DOCUMENT IS PROVIDED BY THE
COPYRIGHT HOLDERS “AS IS” AND ANY EXPRESS OR IMPLIED
WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL CESGA BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
IN ANY WAY OUT OF THE USE OF THE INFORMATION CONTAINED IN THIS
DOCUMENT, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Table of Contents
Changes history ............................................................................... 5
Glossary........................................................................................... 6
Conventions ..................................................................................... 7
1.
Introduction ............................................................................ 2
2.
Real time distributed performance technical challenges ......... 3
2.1
Latency and jitter .................................................................... 3
2.2
Network requirements .............................................................. 5
2.2.1
Bandwidth ........................................................................... 5
2.2.2
Network traffic management for NMP ...................................... 6
2.2.3
Network latency.................................................................... 7
2.3
Servers .................................................................................. 7
2.4
Capture and display devices ...................................................... 8
2.5
Applications ............................................................................ 8
3.
NMP Software implementations ............................................ 10
3.1
LOLA. LOw LAtency audio visual streaming system [6] ................ 10
3.1.1
LOLA main features .............................................................. 10
3.1.1.1
3.1.2
3.2
LOLA latency .................................................................... 11
Software and hardware requirements [8] ................................ 11
Ultra-Videoconferencing .......................................................... 12
3.2.1
Ultra - videoconferencing main features .................................. 13
3.2.2
Ultra-videoconferencing latency ............................................. 13
3.2.3
Software and hardware requirements ..................................... 14
3.3
JackTrip ................................................................................ 14
3.3.1
JackTrip main features ......................................................... 15
3.3.2
JackTrip latency ................................................................... 15
3.3.3
Software and hardware requirements ..................................... 15
3.4
Other developments................................................................ 16
3.5
Latency measurements ........................................................... 16
4.
Galician Network Music Performance experience .................. 19
5.
Conclusion ............................................................................. 23
6.
References ............................................................................ 24
140412 - DO_COM_NMP Report _v0.9.docx
ENTIDAD CERTIFICADA
Act: 8/05/2014
UNE-EN-ISO 9001
Annex I. Audio interfaces .............................................................. 26
1.
MOTU 2408 mk3..................................................................... 26
2.
Focusrite Scarlett 2in2 ............................................................ 26
Annex II. Interfaces and latency ................................................... 27
1.
USB bandwidth ....................................................................... 27
Annex III. Software installation .................................................... 28
1.
LOLA setup ............................................................................ 28
2.
Ultra-video setup:................................................................... 29
3.
Jacktrip setup ........................................................................ 30
List of Figures
Figure 1. Comparison between delay time and subjective evaluation of
viewers (ref: www.itu.int)
4
Figure 2. Bandwidth requirements of different real time videoconference
systems (* approximate values)
5
Figure 3. The final configuration
19
Figure 4. FCC-City of culture physical path
20
Figure 5. Boys e Vacas distributed performance (organized by: Xaime
Fandiño, technical support: Javier Abreu, Natalia Costas, Manuel Fidalgo,
Ivelina Karagyulieva)
22
Figure 6. MOTU 2408 mk3 interface
26
Figure 7. Focuserite Scarlett 2i2 USB audio interface
26
Figure 8. Lola's main user interface
29
Figure 9. Jack Audio Connection Kit
32
List of Tables
Table 1. Bandwidth required for various video formats (25fps,
uncompressed, 24 bits/pixel, interlaced)
6
Table 2. Measurements of ping latencies between points located in research
networks
7
Table 3. Results from the latency measurements
17
Table 4. Transfer rate of the different USB versions
27
Changes history
Version Author
0.0
IAK
0.1
NCL
0.2
NCL
0.3
IAK
0.4
0.5
NCL
IAK
0.6
NCL
0.7
IAK
0.8
0.9
NCL
NCL
Description
First draft of the document for sections Introduction,
Technology Analysis, Solution, References, Annex I
(Motu and Focusrite tech. Info) and Annex II (USB
and interfaces relation to latency).
Draft’s first revision. Reorganize all sections, add
section 3.4 Latency measurements, complete
Conclusions and Experience description. Reorganize
annexes.
New complete revision of sections 2 and 3. Fixing
references.
Adding
copyright,
glossary
and
conventions.
Fixing sections 2 and 3, modification of the Annex I
and Annex III.
Review all subsections of section 2.
Fixing some of the figures.
General revision to document. Minor changes to all
sections.
Modification of Annex III. General revision of the
document.
Minor changes to Annex III and to format.
Update technical report reference identifier.
ENTIDAD CERTIFICADA
Act: 8/05/2014
UNE-EN-ISO 9001
Glossary
NMP
Network Music Performance
RECETGA
Red de Ciencia y Tecnología de Galicia (Galicia Science
and Technology Network.
CESGA
Galicia Supercomputing Centre
GARR
Italian Research & Education Network (NREN)
Geant
Pan European data network dedicated to the research
and education community
Internet2
High-performance backbone
network created
by
the Internet2 community dedicated to the research and
education community in United States. It was first
called Abilene Network.
NAT
Network Address Translation
IP
Internet Protocol
CCRMA
Centre for Computer Research in Music and Acoustics
LOLA
LOw LAtency Audio Visual Streaming System
JACK
System for handling real-time, low latency audio (and
MIDI).
Jacktrip
A system for high-quality audio network performance
over the Internet.
Ultravideoconferencing
System for high-bandwidth, high-quality, and lowlatency videoconference interaction
Conventions
This document uses several conventions to highlight certain words and
phrases and draw attention to specific pieces of information.
This icon indicates tips that could be useful for the reader.
This icon indicates important considerations that are easily
missed.
This icon indicates critical aspects that should not be
overlooked.
ENTIDAD CERTIFICADA
Act: 8/05/2014
UNE-EN-ISO 9001
Introduction
1. Introduction
A networked music performance (NMP) is a real-time interaction over
a computer/data network that enables musicians in different locations to
perform as if they were located in the same room.
While not intended to be a replacement for traditional live stage
performance, networked music performance supports musical interaction
when co-presence is not possible and allows for novel forms of music
expression.
In this work we describe the challenges that NMP introduces and also
several software implementations that make NMP possible.
We also present a practical experience based on NMP where a group
of musicians, located at geographical distant places, interact over a network
to perform as they would if they were located in the same room. Both parts
played in real-time and synchronously and the audience was located at one
of the sites and were able to listen to the resulting performance. Also low
latency video was used to allow the musicians to see each other.
This performance and tests were among locations connected to the
Galician Regional Science and Technology Network (RECETGA).
In the last section, conclusions and future work is presented.
ENTIDAD CERTIFICADA
Act: 26/04/2014
UNE-EN-ISO 9001
2
Real time distributed performance technical challenges
2. Real time distributed
technical challenges
performance
NMP is a challenging application where a number of factors influence
the audio and video quality. In this section the most relevant aspects are
analysed.
2.1 Latency and jitter
Internet, by its nature, is a network that relays on a best effort traffic
policy; this implies that there is no guarantee of quality of service end to
end to a communication specifying QoS parameters like traffic rate
(bandwidth average, peak rate, bursts duration, etc.), maximum latency
and jitter, etc. For this reason most of the music Internet tools and systems
available today require the use of pre-buffering mechanisms to avoid
running out of data and this fact makes them unsuitable for applications in
which real-time and quick interactivity is a requirement.
Research results [1] have indicated that in the case of distributed
musical rehearsals for optimal audio quality, jitter has to be kept at a
minimum and one-way latency has to be controlled at about 20ms. In [2] is
analyzed the influence in tempo of delay and it is specified that delays
≥35ms produce a deceleration in tempo that is not considered musically
acceptable. 35ms is comparable to the delay suffered by orchestra
performers separated by distances of 10m, who can maintain a steady
tempo (with a conductor).
Other important fact that affects the quality of the performance is the
perception of synchronization between audio and video. The video stream
will be delayed with respect to audio and this will also be detected by the
musicians and the audience depending on audio/video timing difference.
From [3] we can read some figures that illustrate the problem of
synchronization in broadcast systems, in particular in [4] it gives us the
detectability and unacceptability thresholds for viewer’s reception:
ENTIDAD CERTIFICADA
Act: 8/05/2014
UNE-EN-ISO 9001
3
Figure 1. Comparison between delay time and subjective evaluation of viewers (ref:
www.itu.int)
From this document (Rec. ITU-R BT.1359) we get some useful
thresholds that are relevant when dealing with audio and video
transmissions:
- When the latency between the audio and video signals is in the range
from -100ms to +25ms (negative means that audio is delayed with respect
to video, and positive value means that video is delayed with respect to
audio) the user does not perceive delay.
- When the latency between the audio and video signals is in the
range from -100ms to -125ms; and from +20ms to +45ms (negative
means that audio is delayed with respect to video, and positive value means
that video is delayed with respect to audio) the user detects the delay.
When the timing difference between audio and video is bigger than
the values given before then it is unacceptable for the user.
As already stated, audio and video streams from musicians located at
different places have to be synchronized to form consistent music
presentation. For this reason various components such as the clocks of the
computers, latencies from the soundcards, capture cards and their drivers,
network interface cards, and network components, rhythm adjustments
among different players, all raise difficulty for synchronization [5]. This
requires fine tuning of all end systems and networks in the path end to end
as all these sources of variability impose many challenges in order to deploy
NMP as a service.
The one way latency should not be higher than 25-35 ms end to
end between distant locations to be considered “real time” in
music. Tests in advance are recommended to verify the effect of
latency and jitter in the performance.
ENTIDAD CERTIFICADA
Act: 26/04/2014
UNE-EN-ISO 9001
4
2.2 Network requirements
2.2.1
Bandwidth
High definition audio streaming is required to make a networked
music performance as realistic as possible. Transmission of mono PCM (raw)
CD-quality audio (44.1 kHz, 16 bit/sample) requires a data rate of 0.7Mb/s.
When stereo/multi-channel or high definition sound (high sampling rate e.g.
48k/96k/192kHz or better quantized e.g. using 24 bit) is needed, the
network can be further stressed.
However, real stress to network is tested when also adding video
communications to the performance. This is due to the fact that real time
systems will usually transmit uncompressed video in order to avoid the
latency introduced by compression algorithms.
The following image compares bandwidth usage of two NMP systems:
LOLA [6] and Ultra-videoconferencing [7]. The columns depict the
bandwidth needs for different (uncompressed) video qualities used by each
tool. It is shown that LOLA black&white requires less than 100 Mbps while
Ultra-videoconferencing
full-frame
uncompressed
video
requires
approximately 150 Mbps. It is also shown that the bandwidth that LOLA
requires for coloured video is almost 500 Mbps while for HD-SDI video
Ultra-videoconferencing needs at least 1.5Gbps.
LOLA b/w, 640x480, 30fps,
10bpp *
1600
1500
1400
1300
1200
1100
1000
900
800
700
600
500
400
300
200
100
0
Ultra-videoconferencing fullframe video 720x576, 24fps,
15bpp *
LOLA colored, 640x480, 60 fps,
24bpp *
Ultra-videoconferencing HDSDI, 1920x1080, 30fps, 24bpp
*
Bandwidth Mbps
Figure 2. Bandwidth requirements of different real time videoconference systems
(* approximate values)
ENTIDAD CERTIFICADA
Act: 8/05/2014
UNE-EN-ISO 9001
5
Below is shown a comparison table of the bandwidth required to send
uncompressed video for different formats:
Video type
Resolution
Bandwidth
Uncompressed DVD
720x576
124Mbps
Uncompressed DV
720x576
124Mbps
Uncompressed HDV 720p
1280x720
553Mbps
Uncompressed HDV 1080p
1440x1080
933Mbps
Uncompressed HDV 1080i
1440x1080
466Mbps
Table 1. Bandwidth required for various video formats (25fps, uncompressed, 24
bits/pixel, interlaced)
2.2.2
Network traffic management for NMP
NMP applications have been deployed within research networks
(Geant, Internet2) before and, as we already stated, they are very sensitive
to latency, transmission errors, bandwidth restrictions, etc. For this reason
it is important to check several aspects related to network infrastructure:
- Analyze the path the traffic traverses as the whole path should be
adequate in terms of bandwidth, latency, transmission errors and jitter end
to end.
- Network issues might arise if the NMP traffic goes via network
equipment that is shared with other IP traffic (QoS, Bandwidth on Demand
or other automated traffic allocation mechanisms should be provided).
- Firewalls, traffic shapers and other network devices that process
traffic should be avoided as they generally introduce additional delay and
might drop packets/flows when they discover unknown protocols/traffic.
- Network Address Translation (NAT) can introduce problems even in
1:1 configuration (check with the network service support).
In summary, for NMP to be successful, the whole end to end network
path must be of very high quality and low occupation (or other traffic
allocation mechanism): low latency and jitter, error free and transparent
(no NAT if possible1).
1
At CESGA LOLA audio was tested behind 1:1 NAT successfully. In the same testlab
Ultra-videoconferencing video showed strange behavior (transmission dropped in
one way).
ENTIDAD CERTIFICADA
Act: 26/04/2014
UNE-EN-ISO 9001
6
Tools like IPERF2 can help to debug network issues and check
the quality of the network connection.
2.2.3
Network latency
Network latency depends on the distance between performers and
also on the network equipment and traffic processing that is done. We will
focus on the first at the second term has to be analyzed on each particular
case.
Below we provide some measurements of ping latencies between points
located in research networks. The latency value takes into account fibre
transmission time, network equipment and also processing on end systems
(2 servers).
Location 1
Location 2
Distance
Latency
(one way)
Santiago de
Compostela
Vigo, Spain
100km
1 ms
Santiago de
Compostela
Madrid, Spain
600 km
12 ms
Santiago de
Compostela
Barcelona, Spain
1100 km
20 ms
Santiago de
Compostela
Geneva, Swiss
1500 km
25 ms
> 6000 km
Transoceani
c cable
65 ms
Santiago de
Compostela
Quebec, Canada
Table 2. Measurements of ping latencies between points located in research
networks
2.3 Servers
2
IPERF sourceforge website: http://sourceforge.net/projects/iperf/
ENTIDAD CERTIFICADA
Act: 8/05/2014
UNE-EN-ISO 9001
7
Some aspects need to be taken into account when deploying NMP.
Capacity of servers should be carefully evaluated particularly when
transmitting not only audio but also video streams. In particular:
-
RAM and CPU should be enough to run seamlessly the audio/video
tools
- They should provide slots enough to accommodate the required
audio/video cards.
- Gigabit Ethernet connectivity is also a must when planning for video
communications.
- Input/output interfaces for audio/video/network should be carefully
selected
o The interfaces used to connect to the capture and projection systems
are relevant, some type of interfaces like HDMI should be avoided [8]
because of latency issues, VGA, DVI-D or RGB connectors should be used
instead. Others, like USB v1, should be avoided as they cannot cope with
the necessary bandwidth.
o Professional audio interfaces allow the end user to control the sample
rate and also the size of the buffers that are used. This last parameter can
be reduced to minimize latency. It has to be handled with care as very low
buffers can produce distortions in sound or require very high CPU
processing.
2.4 Capture and display devices
The capture and projection systems should also be carefully reviewed:
- Plasmas and LCD/LED TVs can add a big amount of latency (up to
200ms in the models tested at CESGA). Some of they can be configured in
gaming mode to avoid the processing that is done for image enhancement
and thus reducing the latency that is added.
- Additional devices, like video splitters, cable extenders, etc. might
add further latency and also affect the final result.
- Video cameras: frame rate, resolution, video interfaces
2.5 Applications
In the previous section we talked about the latency introduced by the
hardware, but this is not the only latency that eventually impacts the sound
by the time it reaches our ears, as software can also introduce latency.
Some plugins cause latency. Some computers have more, or less, latency
than others. Generally the better a computer is, the less latency that it will
ENTIDAD CERTIFICADA
Act: 26/04/2014
UNE-EN-ISO 9001
8
incur on its own, but this depends on the condition of the computer, and its
load at the time. This is another reason to keep the server correctly tuned,
with the fewer processes running that can interfere with NMP software.
The last important element in the chain is the software that is
actually used for NMP. There are several software implementations for NMP
like LOLA, Ultra-videoconferencing or Jacktrip that will be explained in detail
in Section 3, “NMP Software implementations“.
What all these implementations have in common is that they try to
reduce to a minimum the internal buffers and processes that are required
for capture, audio/video signal processing and transmission/reception in
order to be fast and reduce the latency to a minimum.
ENTIDAD CERTIFICADA
Act: 8/05/2014
UNE-EN-ISO 9001
9
NMP Software implementations
3. NMP Software implementations
During the latest years several research institutions have developed
software that allows NMP and were able to conduct real-time distributed
multimedia performances in education and research networks. In this
section a summary of the most important relevant technologies and
projects are reviewed.
3.1 LOLA - LOw LAtency audio visual streaming
system [8]
Lola is a development of the “Conservatorio di Musica G. Tartini”3,
Trieste, Italy, in collaboration with GARR4, the Italian Academic and
Research Network.
LOLA’s goal is to provide a tool for real-time audio/video “natural”
distance human interaction. A special attention is paid to the optimization of
the signal processing and transmission in order to keep the system latency
as low as possible, below the human delay perception threshold.
The system is based on high performance audio/video acquisition
hardware, and on the integration and optimization of audio/video stream
acquisition, presentation and transmission.
3.1.1
LOLA main features
LOLA is an audio/video
communication point to point.
system
that
allows
bidirectional
Each LOLA node is composed by one windows server with LOLA
software installed. LOLA nodes include specific hardware for audio
capture/playback and video capture and projection systems.
The main features of LOLA are:




It allows point to point bidirectional communication
It supports multichannel audio
It supports audio devices that use ASIO interface
It supports standard definition video based
manufacturer grabbers.
Conservatorio
di
musica
Giuseppe
http://www.conservatorio.trieste.it/
4
GARR consortium: http://www.garr.it/b/eng
3
ENTIDAD CERTIFICADA
on
Tartini
Bitflow
(Trieste):
Act: 26/04/2014
UNE-EN-ISO 9001
10
3.1.1.1 LOLA latency
As we can read in LOLA technical specifications, the software itself
introduces no more than 5ms to the audio and video signal.
3.1.2
Software and hardware requirements [8]
Below the main software and hardware requirements are described:
Operating system
To date, LOLA has been tested on both MS Windows XP and MS
Windows 7 (32 bits and 64 bits versions). As Windows XP is being phased
out, and Windows 7 offer better performance for LOLA, it is better Windows
7 to be used as the base operating system.
Audio input/output hardware requirements
Low latency audio performance requires that small in/out audio
buffers can be used, that no bottlenecks are present in the data transfer
between the converters and the system, and that robust and efficient
software drivers are available for the OS in use. LOLA project does not
recommend the usage of USB and FireWire (IEEE 1394) external devices as
they do not offer such a fast data acquisition processing. For this reason
LOLA uses a PCI/PCIe internal device and a reliable ASIO driver.
Video hardware requirements
Fast video acquisition and streaming relies on a family of video
grabbers by BitFlow Inc. The BitFlow video grabbers currently used by LOLA
are the analogue ones, like LOLA: Low Latency Audio Visual Streaming
System the BitFlow ALTA-AN1 grabber with SDK 5.60 drivers. It supports
both analogue b/w and colour cameras.
Cameras and projection systems
LOLA technical specification details black&white and also colour
cameras that had been tested successfully for NMP and that are compatible
with the video grabbers that LOLA supports [8].
Also low response time LED/LCD monitors are recommended.
Server requirements
ENTIDAD CERTIFICADA
Act: 8/05/2014
UNE-EN-ISO 9001
11
LOLA gives recommendations for minimum server configuration. The
most relevant features are the following5:
- Intel dual/quad core based on i5 or better i7 processors
- 4GB RAM minimum
- Mid/high performance motherboard with PCI Express bus and DDR3
RAM
- 1Gbit LAN port
- Integrated graphic adapter should be enough, however a PCI-Express
Video Adapter (e.g., Nvidia GeForce cards) is strongly recommended,
especially for colour video processing and for incoming GPU-accelerated
features.
- Enough PCIe free slots to host the audio, video display card and video
capture card.
Network requirements
LOLA’s bandwidth requirements come mainly from the bandwidth
usage of the bidirectional video stream.
LOLA requires at least 100Mbps in minimal uncompressed
configuration (640x480, black&white, 30 frames per second) and up to
500Mbps in full uncompressed configuration (640x480, colour, 60 frames
per second), and generates a very high Packet Per Second (pps) rate, as it
uses 1K data packets, e.g. from 100 up to 500 Kpps. Thus the minimal endto-end connectivity must be at least 1Gbps.
Port usage
LOLA uses by default ports TCP 7000, UDP 19788 and UDP 19798 for
service communication and audio/video streaming. It is thus required that
no restrictions apply for these ports (e.g. due to network or Windows
firewall settings or local network administration policies).
3.2 Ultra-Videoconferencing
Ultra-Videoconferencing is a development of the McGill University in
Montreal, Quebec, Canada. Its main focus is to develop a high quality and
low latency tool for videoconference interaction, such as that is required for
5
For a complete list of requirements please check LOLA technical specifications [8].
ENTIDAD CERTIFICADA
Act: 26/04/2014
UNE-EN-ISO 9001
12
high fidelity instruction, distributed musical performance and vibrosensory
data6.
Ultra-videoconferencing has been used for a range of demanding
applications including live concert streaming (1999), remote mixing (2000),
collaborative performance (2001), distance masters classes over SDI
(2002), and remote video interpreting of sign language using three
simultaneous DV streams (2003).
Ultra-Videoconferencing tool supports high-resolution multichannel
audio and video either from frame grabbers using v4l or v4l2 interfaces
(e.g., Bt878 chipset), digital video (DV) cameras, DCAM (raw,
uncompressed video) 1394 devices, and standard or high-definition Serial
Digital Interface (SDI). They are also working on mosaicing of video
displays with minimal latency and moving toward super-resolution
techniques to create video displays beyond the quality of high-definition.
3.2.1
Ultra - videoconferencing main features
The main features of Ultra-videoconferencing are:



It allows multipoint bidirectional audio and video communication
It supports multichannel audio
It supports general purpose video grabbers (standard and high
definition video)
It allows having multiple clients (multipoint communication) but not
by means of multicast technology (this implies that the streams have to be
sent to a central server). Multicast support is being implemented and it is
on its early stage nowadays (not accessible for the command line).
3.2.2
Ultra-videoconferencing latency
The measured one way end-to-end video latency is at approximately
75ms for analogue video, 100ms for DV, and 120ms for JPEG. Audio latency
can be lower than 20ms when using professional sound cards, as these
allow for minimal buffering.
6
Vibrosensory data is associated with the vibrotactile or kinematic
(motion platform) actuators to convey either mechanical displacement or vibration
to the user. Ultra-Videoconferencing has been used for transmission of additional
channels of such haptic information.
ENTIDAD CERTIFICADA
Act: 8/05/2014
UNE-EN-ISO 9001
13
3.2.3
Software and hardware requirements
Operating system
To date, Ultra-video works under Linux (release versions have been
tested under Ubuntu and Fedora distributions).
Regarding to audio it handles OSS and multichannel ALSA audio up to
24 bit, 96 kHz sampling.
Video hardware requirements
Regarding to video it supports high-resolution video from frame
grabbers using v4l or v4l2 interfaces (e.g., Bt878 chipset), digital video
(DV) cameras (750x576), DCAM (raw, uncompressed video (640x480)),
1394 devices, and standard ([email protected]) or high-definition (1080i60,
1080p30) Serial Digital Interface (SDI). Video can also be passed through a
JPEG codec to reduce bandwidth and the video display window may be
resized up to full screen.
Ultra videoconferencing has been tested with AJA Xena HS HD-SDI
video capture cards.
Full-frame uncompressed analogue video requires a minimum of 148
Mbps in the steady-state. PCM audio with default sampling (44.1 kHz, 16
bit) requires approximately 706 kbps per channel. DV camera output (audio
and video) consumes approximately 25 Mbps; JPEG-encoded full-frame
video is similar, dc1394 video at full-frame requires anywhere from 148-270
Mbps depending on colour space employed.
Port usage
Ultra-Videoconferencing uses UDP ports 8000-8010 and the helper
“uv” script uses TCP port 8000 on the server to accept incoming requests
from clients. As well as LOLA, Ultra-video should not run behind a firewall
and it should use public IP addresses (no NAT should be used).
3.3 JackTrip
JackTrip is a Linux and Mac OS X-based system used for multimachine network performance over the Internet. It supports any number of
channels (as many as the computer/network can handle) of bidirectional,
ENTIDAD CERTIFICADA
Act: 26/04/2014
UNE-EN-ISO 9001
14
high quality, uncompressed audio signal streaming. The technology being
used builds on early work by research groups at McGill University and
Stanford University.
It can be used between any combination of Linux and Mac OS X (i.e.,
one end using Linux can connect to the other using Mac OS X).
Currently, JackTrip uses JACK7 as its host audio server. JACK has
several advantages: it runs on Linux and Mac OS X, it has the ability to
make audio connections between many different audio clients on the same
host, and its current implementation takes advantage of multi-processor
machines.
3.3.1
JackTrip main features
The main features of JackTrip are:



3.3.2
It allows multipoint bidirectional audio (only) communication
It supports multichannel audio (as many as the computer/network
can handle)
It relays on JACK usage
JackTrip latency
No technical details for the latency introduced by JackTrip were
found.
3.3.3
Software and hardware requirements
Below the main software and hardware requirements are described:
Operating system
JackTrip supports Linux and Mac OSX operating systems.
7
JACK is system for handling real-time, low latency audio (and MIDI). It can connect a
number of different applications to an audio device, as well as allowing them to share audio
themselves. JACK also has support for distributing audio processing across a network. [
http://jackaudio.org/ ]
Act: 8/05/2014
ENTIDAD CERTIFICADA
UNE-EN-ISO 9001
15
16
As stated before JackTrip supports any number of channels (as many
as the computer/network can handle) of bidirectional, high quality,
uncompressed audio signal steaming.
JackTrip bandwidth requirement is 1-2Mbps per audio channel plus
additional bandwidth to accommodate network fluctuation.
Port usage
Each machine running JackTrip has to have the same UDP port(s)
open to run JackTrip. The default port is UDP 4464 for a two-way
connection. For projects with more than two locations, more open ports will
be necessary. Recommended ports are offset by 10, so use UDP 4474 for a
third location, UDP 4484 for a fourth location, etc. It is also recommended
to have backup ports available in case. For any project, the
recommendation is to have UDP ports 4444-4544 open.
3.4 Other developments
Other developments that are being used for NMP were found but
were not tested because of the short timeframe in which this project had to
be completed. Most of them are oriented to interactive applications
(cinema, broadcast, e-medicine, etc.) but not mainly focused on music
performance (as LOLA or JackTrip).
Among those are the following: Ultragrid (2 channel, up to 8K
resolution uncompressed and compressed to various formats, 83 ms
latency), iHDTV, DVTS and Cinegrid.
3.5 Latency measurements
During the tests some latency measurements were done, in the table
below the results are shown:
Type of test
Particular test conditions
Average
round trip
latency (ms)
LOLA audio test
between CESGA
(network 44) and
CESGA (net. 36)
1 end behind firewall, 2 hops
routing distance, integrated
audio, all traffic routed at
CESGA
18.00
LOLA audio test
between CESGA
(network 44) and
audio, all traffic routed at
ENTIDAD CERTIFICADA
43,29
Comments
Latency increased
due
to
LOLA
buffering to many
Act: 26/04/2014
UNE-EN-ISO 9001
CESGA
36)
(network
CESGA, 1 end with LOLA
buffer 20, 1 end with buffer 0.
LOLA audio test
between CESGA
(network 44) and
CESGA (network
32)
audio, all traffic routed to
Lugo
(5-6
ms
additional
network latency)
LOLA audio test
between CESGA
(network 44) and
CESGA (network
32)
outside
Galicia
(23
ms
additional RTT latency)
Jacktrip
audio
test
between
CESGA (network
44) and CESGA
(network 32)
Lugo
(5-6
ms
additional
network latency)
LOLA audio test
between City of
Culture (network
37) and CESGA
(network 34)
Both sites behind firewall, 2/3
hops routing distance locally,
unknown routing hops at CdC,
integrated
audio,
1
way
routed to Vigo
LOLA audio test
between City of
Culture (network
37) and CESGA
(network 36)
Only CdC behind firewall, 1/2
hops routing distance locally,
unknown routing hops at CdC,
integrated
audio,
1
way
routed to Vigo
LOLA audio test
between City of
Culture (network
37) and F.C.C
(network 81)
Only CdC behind firewall, 1
hop
routing
distance,
integrated audio, all traffic
routed at CESGA
43,20
24,44
43,50
43,25
15,17
samples
Latency increased
due
to
longer
network
path
(gateway in Lugo)
Latency increased
due
to
longer
network path (to
reach
gateway
traverses a 23 ms
RTT path)
Latency decreased
using
Jacktrip
instead of LOLA.
93.60
Latency increased
due to placing one
end
at
CdC
(firewalls,
traffic
shapers, ...)
68.00
Reduced
latency
with respect to
previous
case
eliminating
firewall at CESGA
end.
40.11
Reduced
latency
with respect to
previous
case
optimizing
routing.
(1) Average roundtrip latency (maximum tolerable 50ms.)
(2) In all tests (but Jacktrip one) Ultra-videoconferencing traffic was also
traversing the devices (50 Mbps)
Table 3. Results from the latency measurements
From these tests we appreciated variations with the following sources of
latency:
-
Audio buffers (LOLA/JackTrip and audio interfaces)
Network path latency
ENTIDAD CERTIFICADA
17
Act: 8/05/2014
UNE-EN-ISO 9001
-
Network devices latency (firewall/traffic shapers mainly, but also
other types of devices)
As it is shown, it is quite easy to reach the tolerable maximum value to
latency that is 40-50ms.
ENTIDAD CERTIFICADA
Act: 26/04/2014
UNE-EN-ISO 9001
18
Galician Network Music Performance experience
4. Galician Network
experience
Music
Performance
Our aim at this stage was to split a group of musicians in two
different locations and see if they were able to perform as well as if they
were located in the same room. These musicians would perform during the
event Galicia Connection’14 held in Santiago the Compostela.
Similar experiences were conducted before by research groups
involved in the development of the tools already described. We were
interested about analyzing the possibilities we had using commodity
hardware commonly available at our premises that was typically used for
videoconference and streaming purposes.
Below is depicted the configuration that we finally used:
Figure 3. The final configuration
We did a bidirectional audio and video connection between Cidade da
Cultura and Facultade de Ciencias da Comunicación with the following
characteristics:
19
- The main setup consisted on two computers at each end. Each end
had one computer dedicated to LOLA, with an audio capture interface
connected to the venues sound system and the network and other
dedicated to Ultra-videoconferencing software with video capture interface
connected to a PAL camera, to the projection system and to the network.
- LOLA’s computers were equipped with external sound cards
(Focusrite Scarlett 2i2 and MOTU 2408 mk38, with Windows 7 OS and LOLA
version 1.2 (latest version is 1.3). Buffer settings for both audio interfaces
ware set to minimum (1ms in the Focusrite and 16 samples at a sample
rate of 44.1 samples per second at MOTU 2408).
- Ultra-videoconferencing servers were used as LOLA’s video
capabilities require specific Bitflow video grabbers that were not available.
These servers were connected to projectors by means of VGA interfaces.
Each server was dual core with 2GB of RAM with Linux Ubuntu 12.04
installed on it.
- In order to achieve further distance between both places the traffic
was routed first to Vigo (a city located 98km far from Santiago de
Compostela). With this configuration the scenario was exactly the same as if
both locations were separated 200km from each other.
Figure 4. FCC-City of culture physical path
8
See Annex I for more information.
ENTIDAD CERTIFICADA
Act: 26/04/2014
UNE-EN-ISO 9001
20
In this case the musicians were able to perform perfectly as the audio
latency was kept under reasonable threshold.
The video latency was acceptable but still was noticeable the delay
between audio and video signals.
From this experience some hints are considered relevant:
- Previous test using video by means of H.323 videoconference devices
(based on Lifesize Express 220 platform) with HDMI interfaces showed that
delay from video with respect to audio was unacceptable. A measurement
was done showing 400ms one way delay of video.
- LOLA supports any device that used ASIO interfaces. At present most
of the general purpose audio cards work under ASIO4ALL software and, for
this reason, can be used with LOLA. We tested Soundmax integrated cards,
Soundblaster Live 5.1, Intel IDT HD Audio and Realtek integrated sound
cards. Although in some conditions they are workable (preliminary tests
were conducted successfully), some random problems arise that produce
glitches and bad effects on sound. Also the integration of unbalanced inputs
of these cards into the more professional venues sound system was
cumbersome. Changing into more quality sound cards with ASIO support
resulted in a cleaner sound and easier setup at both sites.
- Low latency PCI audio cards are preferred. In case USB is chosen
(check the latency introduced) we should take into account to connect by
means of USB2 (minimum) as the usage of USB version 1 interfaces
produced distorted audio.
- Although audio does not require much CPU consumption attention has
to be put that no processes or software interferes LOLA. CPU pikes can
produce glitches on sound. It is important to disable and avoid using any
other software that might conflict with LOLA usage.
- Due to City of Culture networking restrictions firewalls could not be
skipped on their side but time had to be invested to be sure that no traffic
was dropped. 1:1 Network Address Translation (NAT) had also a bad
influence in Ultra-video outgoing stream (from City of Culture), being the
connection dropped after the first 15-20 seconds, therefore private IP
addresses had to be routed in this case eliminating the need of NAT
translation.
- The resources of Ultra-videoconferencing showed not enough and
flickering appeared, so the video had to be sent cropped to 420x320px.
- Due to the City of Culture particular venue conditions and
conferences agenda a VGA splitter and a VGA matrix had to be added
introducing additional delay to video signal. Neither the effect of those
devices could be measured nor the delay introduced by the projector itself
because of the time constraints of the project.
ENTIDAD CERTIFICADA
Act: 8/05/2014
UNE-EN-ISO 9001
21
- Increasing the buffer size of the audio devices did not alleviate
glitches or distortions in all the tests we did but, on contrary, it proved to
worsen the quality of the audio (for example with Scarlett Focusrite 2i2
increasing the buffering from 1ms to 5ms to 10ms).
To summarize, the musicians were able to perform successfully as
they do not need video communications to synchronize however, they need
to hear each other within a reasonable latency threshold.
Below is shown a picture of the performance as it was done for the
Galician Connection’14 during the Xornada Educa conferences.
Figure 5. Boys e Vacas distributed performance (organized by: Xaime Fandiño,
technical support: Javier Abreu, Natalia Costas, Manuel Fidalgo, Ivelina
Karagyulieva)
ENTIDAD CERTIFICADA
Act: 26/04/2014
UNE-EN-ISO 9001
22
Conclusion
5. Conclusion
This present work is part of the analysis of the state of the art of
technology that is being developed to facilitate distributed performances
and ultra-low latency videoconferences within research networks worldwide.
These new tools will allow innovative use cases for distance music learning
and distributed performances.
In this document we described the technical efforts towards the
deployment of an environment for Network Music Performance, in which
real-time musicians interact from different locations to perform as if they
were in the same room.
All these efforts in this case had, as an objective, being able to do an
audio-visual performance within Galician Connection’14 event, this meant
that the deployment had to be made under strict time constraints (2
months) and that the difficulties had to be solved to deploy a research
infrastructure inside a production network as it is the one at City of Culture.
The performance resulted a success and further work has to be done
in order to cover more challenging scenarios and make the system easier to
deploy and adequate to be handled for nontechnical staff or with minimal
technical support.
The software that was tested worked seamlessly but it was not used
at their full capability. Further work is being done in the following areas:
 Analyze the usage of NMP in more challenging scenarios (longer
distances, multipoint performances, multichannel audio, increased video
quality).
 Analyze the complete hardware setup that is required for NMP.
 Research has been done in order to analyse the effects of latency in
performances. Although 25ms has been set a threshold for real time
performances some studies propose music compositional strategies to
maintain the perception of synchrony [9] beyond that limit.
 Evaluate/review NMP software capabilities for deployment on
production scenarios.
23
References
24
6. References
[1]
M. D. U. N. L. W. Xiaoyuan Gu, “A new networked music performance
system,” 3 12 2004. [Online]. Available: https://www.ibr.cs.tubs.de/projects/nmp/paper/nime04_networked_music_performance.pdf.
[Accessed 28 04 14].
[2]
C. C. G. L. S. T. M.Gurevich, “Simulation of Networked Ensemble
Performance with Varying Time Delays: Characterization of Ensemble
Accuracy,”
2004.
[Online].
Available:
https://ccrma.stanford.edu/~cc/pub/pdf/simNetEnsPerf.pdf. [Accessed 20
04 2014].
[3]
G. J. A. G. Patrick Waddell, “Audio/Video synchronization. Standards and
solutions. A status report.,” 300 1 2010. [Online]. Available:
http://www.atsc.org/cms/pdf/audio_seminar/12%20-%20JONES%20%20Audio%20and%20Video%20synchronization-Status.pdf.
[Accessed
12 4 2014].
[4]
ITU-R BT.1359, “Relative timing of sound and vision for broadcasting,”
1998. [Online]. Available: https://www.itu.int/dms_pubrec/itu-r/rec/bt/RREC-BT.1359-0-199802-S!!PDF-E.pdf. [Accessed 12 04 2014].
[5]
M. D. U. N. L. W. X. Gu, “A new networked music performance system,” 3
12
2004.
[Online].
Available:
https://www.ibr.cs.tubs.de/projects/nmp/paper/nime04_networked_music_performance.pdf.
[Accessed 12 04 14].
[6]
I. Conservatorio di musica G. Tartini – Trieste, “LOLA Low Latency Audio
Visual Streaming
System,” 30 1
2010. [Online]. Available:
http://www.conservatorio.trieste.it/artistica/lolaproject/documents/Lola_doc_v1.3.0.pdf. [Accessed 12 04 2014].
[7]
“McGill Ultravideoconferencing System,”
http://srl.mcgill.ca/projects/rtnm/.
[8]
1. d. m. G. Tartini, “LOLA Low Latency Audio Visual Streaming System
Installation
&
User's
Manual,”
2014.
[Online].
Available:
http://www.conservatorio.trieste.it/artistica/lolaproject/documents/Lola_doc_v1.3.0.pdf.
[9]
S. Weaver, “Latency: Music Composition and Technology Solutions for
Perception of Synchrony in “ResoNations 2010: An International Telematic
2002.
[Online].
Available:
References
25
Music Concert for Peace”,” 16 05 2011. [Online]. Available:
http://sarahweaver.org/files/weaver_masters_thesis_final.pdf. [Accessed
26 04 2014].
[10]
M. G. G. L. S. T. Chris Chafe, “Effect of Time Delay on Ensemble
Accuracy,”
3
4
2004.
[Online].
Available:
https://ccrma.stanford.edu/~cc/pub/pdf/ensAcc.pdf. [Accessed 18 04
2014].
[11]
P. Davis, “How do I use JACK over a network?,” [Online]. Available:
http://jackaudio.org/netjack.
[12]
“Jacktrip
Manual,”
[Online].
Available:
https://sites.google.com/site/jacktripdocumentation/the-team.
[13]
J.-P. Caceres and C. Chafe, “JackTrip: Under the Hood of an Engine for
Network
Audio,”
2010.
[Online].
Available:
https://ccrma.stanford.edu/~jcaceres/publications/PDFs/journals/2010caceres_chafe-JNMR-jacktrip.pdf.
[14]
“2408mk3
Overview,”
[Online].
http://www.motu.com/products/pciaudio/2408.
[15]
“USB,” [Online]. Available: http://en.wikipedia.org/wiki/USB.
[16]
“Focusrite Scarlett 2i2 USB Audio Interface,” [Online]. Available:
http://www.musiciansfriend.com/pro-audio/focusrite-scarlett-2i2-usbaudio-interface#productDetail.
[17]
“HD-CCTV,”
[Online].
Available:
http://www.webgateinc.com/wgi/eng/index.php?svc_name=content&idx=
1&amCode=C032&.
[18]
“QjackCtl JACK Audio Connection Kit - Qt GUI Interface,” [Online].
Available: http://qjackctl.sourceforge.net/.
ENTIDAD CERTIFICADA
Act: 8/05/2014
UNE-EN-ISO 9001
Available:
Annex I. Audio interfaces
26
Annex I. Audio interfaces
1. MOTU 2408 mk3
The 2408mk3 provides 8 channels of 96 kHz analogue recording and playback,
combined with 24 channels digital I/O. Connect up to four interfaces to the included
PCI-424 card for an expanded system capable of 96 simultaneous active input and
output connections at 96 kHz.
The 2408mk3 provides cross-platform compatibility for Mac and Windows.
Figure 6. MOTU 2408 mk3 interface
2. Focusrite Scarlett 2in2
This audio interface handles 24-bit resolution at sample rates of up to 96 KHz.
It is USB powered.
It provides two microphone inputs and two line outputs. Also it provides a
monitor output that is not routed through the computer.
Figure 7. Focuserite Scarlett 2i2 USB audio interface
ENTIDAD CERTIFICADA
Act: 26/04/2014
UNE-EN-ISO 9001
Annex II. Interfaces and latency
27
Annex II. Interfaces and latency
1. USB bandwidth
In the following table it is shown the transfer rate of the different USB
interfaces. USB 1.x is not adequate as interface for NMP performances due to its low
bandwidth support.
USB type
bandwidth
Other features
USB 1.x
1.5Mbps
(low They are usually coloured white
bandwidth)
and
12 (but some white USB ports are
Mbps (full bandwidth)
USB 2.0 compatible
USB 2.0
(Hi-Speed)
480 Mbps
They are usually coloured black
USB 3.0
(SuperSpeed)
4 Gbps
They are usually coloured blue
USB 3.1
(Superspeed+)
> 7 Gbps
Table 4. Transfer rate of the different USB versions
ENTIDAD CERTIFICADA
Act: 8/05/2014
UNE-EN-ISO 9001
Annex III. Software installation
28
1. LOLA setup
These are some hints about LOLA setup, for complete installation guide please
check LOLA technical documentation [8].
System setup
To install LOLA the following steps should be followed:
 WinPcap (version 4.1.2 or later) - LOLA relies on the WinPcap windows packet
capture library, which has to be installed on the PC.
 BitFlow (SDK version 5.60) - In order to support 64 bits Windows version the
minimal SDK version must be 5.60. You do not need a license code for the BitFlow
SDK, install it as “drivers only”, when prompted for the serial number.
 Visual C++ 2010 redistributable
 LOLA package
Audio hardware setup
When LOLA is launched, it will check for all available audio ASIO devices (if
none is available, the program will provide a warning and run without audio
support). It will be possible to select the desired ASIO audio device from the Audio
Video Setup dialog window, available in the Tools menu.
TCP/UDP ports used by LOLA
LOLA will use by default ports TCP 7000, UDP 19788 and UDP 19798 for
service communication and audio/video streaming. It is thus required that no
restrictions apply for these ports (e.g. due to network or Windows firewall settings or
local network administration policies).
Running LOLA
The LOLA user interface is intended to be as simple as possible: a remote
address to call (IP address), a “Check” button to make a quick and simple preconnection test, a “Connect” and a “Disconnect” button.
Interesting features are:
ENTIDAD CERTIFICADA
Act: 26/04/2014
UNE-EN-ISO 9001
29
- Local Audio test: This feature allows listening to audio that is LOCAL to the
current LOLA instance. This option is useful to debug local audio issues.
- Bounce – back Mode: This feature is useful to debug network issues. The
audio is routed to the remote site and sent back, so the complete path is tested.
- Chat Window: Since version 1.3 LOLA has a chat integrated that simplifies
communications between sites.
Figure 8. Lola's main user interface
2. Ultra-video setup:
Installing Ultra-videoconferencing
To successfully install Ultra-videoconferencing the exact release of Linux has
to be deployed. In our case we installed Ubuntu 12.04.
1. Download
Ultra-videoconferencing
http://ultravideo.mcgill.edu/download.html
ENTIDAD CERTIFICADA
Act: 8/05/2014
UNE-EN-ISO 9001
from
30
2. Unpack the tarball with tar -xzf uv-<version>.tar.gz.
3. To help install the necessary support packages for various video devices and
to configure appropriate device permissions and network buffers, there is provided a
"Brontify" script that should be invoked with root privileges as follows:
sudo ./Brontify
Running Ultra-videoconferencing
This is a short collection of basic commands needed to use uv in each case:

One direction, audio and video:
server:
uv -s -x
client:
uv -s -x IP_Server
(-s sends sound; -x sends video)

Two directions, audio and video:
server:
uv -s -v
client:
uv -s -v IP_Server
(-a requests bidirectional audio; -x bidirectional video)

Two directions, specific size, sender and receiver have a PAL camera:
server:
uv -v -N p -w 427 -h 320 -W 427 -H 320
client:
uv -v -N p -w 427 -h 320 -W 427 -H 320 IP_Server
(-v bidirectional video; -N p says to use the European
PAL video norm in place of NTSC; -w, -h - selecting quarter-frame size)

Killing the processes:
killall uv
killall bronto
3. Jacktrip setup
Installing Jacktrip
ENTIDAD CERTIFICADA
Act: 26/04/2014
UNE-EN-ISO 9001
31
JackTrip requires installation of JACK Audio Plugin prior to the actual
installation of JackTrip.
As explained before, Jack is a cross platform audio routing infrastructure that
allows to route audio between different simultaneously running computer
applications. QjackCtl is a simple application to control the JACK sound server
daemon, specific for the Linux Audio Desktop infrastructure. JackTrip requires Jack
to run and expands the audio routing capabilities of Jack to include network audio
streaming.
1. Install JACK audio
-
Download and install JACK software. The package can be downloaded
from http://www.jackaudio.org
2. Install QjackCtl
- Download QjackCtl
- The installation procedure follows the standard for source distributions.
In the extracted source directory, just do:
./configure [--prefix=prefix]
make
-
and optionally as root:
make install
3. Install JackTrip
- Download
the
latest
version
of
JackTrip
on
<http://code.google.com/p/jacktrip/downloads/list>
- Choose the universal binary (unless you will be working on JackTrip
software development).
- The rest of the installation takes place in the Terminal using shell
commands
For
more
installation
details
https://sites.google.com/site/jacktripdocumentation/calendar
Running Jacktrip
a) Run Jack audio daemon
jackd -R -d alsa
b) Run Jack audio connection
qjackctl
ENTIDAD CERTIFICADA
Act: 8/05/2014
UNE-EN-ISO 9001
check:
32
c) Run jacktrip
a. as a server
./jacktrip -s
b. as a client
./jacktrip -c IP_Server
d) Verify on Jack audio connection kit that the connections are set right, as it is
displayed in the picture:
Figure 9. Jack Audio Connection Kit
ENTIDAD CERTIFICADA
Act: 26/04/2014
UNE-EN-ISO 9001

Technical Report

Transcripción

Documentos relacionados

LED TV

Table of Contents

ELT_Teacher_111-144

Sample - Latin Pulse Music

How to use these symbols

THE CALABOZO: VIRTUAL RECONSTRUCTION OF A PRISON

catálogo 2015 elt e penguin readers - Portal ELT Brasil

SIP Paging Adapter Operations Guide

Milestones - National Geographic Learning

The Science of Sound - Katz Marketing Solutions | Radio Advertising

The Mexican Film Bulletin

Explore Our World Series Walkthrough

ELT Catalogue - Cambridge University Press