How to stream video and audio from a Raspberry Pi with no latency
I have finally finished my last exams, so now I have more time to focus on some of my own projects. It has been a while since our Kickstarter campaign was successfully funded, but we are still working on making the experience better for the final users.
After the campaign ended we sent out a survey to all our backers with several questions about there address, profession and so on, but we also asked them if they had any suggestions for improvements or extra features they would like to see added to the Balanduino. A lot of people asked if we could enable wireless streaming for it.
I was personally very excited about that since I have been playing with the thought for quite a while, so when the official camera module for the Raspberry Pi became available I bought it straight away.
If you do not have much experience with the Raspberry Pi I recommend reading Thomas’s blog post he wrote a while ago: http://blog.tkjelectronics.dk/2012/09/raspberry-pi-gpio-control/ and also to viset the official homepage: http://www.raspberrypi.org/.
I really have not used my Raspberry Pi that much – it has been laying on my desk for quite some time, as I did not have any real use for it until now. I have tried some different approaches, but ended up using a something called gstreamer 1.0. Note that there is currently no official binaries for Windows, so you will have to compile them yourself, but it works great on both Mac OS X and Linux.
I am running Arch Linux on my Raspberry Pi since I do not need the desktop environment that comes with the official Raspbian “wheezy” image, but it should not make much of a difference.
I will not go into much detail on how to install gstreamer 1.0 on your computer, but instead I will refer to the page I originally got the commands from: http://pi.gbaman.info/?p=150.
Also check out the original forum post: http://www.raspberrypi.org/phpBB3/viewtopic.php?uid=56951&f=43&t=44987&start=0.
Video
To stream video from my Raspberry Pi I run the following command:
gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=96 \
! gdppay ! tcpserversink host=serverIp port=5000
And to receive the video on my Mac I run the following command in the terminal:
! gdpdepay ! rtph264depay ! avdec_h264 ! videoconvert ! autovideosink sync=false
Be sure to replace serverIp with the ip of your Raspberry Pi. You can get it by running the following command on the Raspberry Pi:
The video quality is really great and there is basically no latency at all.
Audio
To stream audio turned out to be a little harder to get working. There is no audio input on the Raspberry Pi, so I used an old Icemat USB Sound card I had laying around.
First of all type the following command into your Raspberry Pi:
It should print out something like this:
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 004: ID 0d8c:000c C-Media Electronics, Inc. Audio Adapter
Note the last device is my USB sound card.
Btw I recommend using ssh to login to your device. In fact I never use the HDMI output, as I prefer using my laptop. For more information check out this guide by Adafruit: http://learn.adafruit.com/adafruits-raspberry-pi-lesson-6-using-ssh/.
If your audio device is not working properly, then I recommend googling your USB sound card, as somebody else have properly already written a guide on how to get it working.
After that you will need to install the following packages: ‘alsa-utils’, ‘alsa-firmware’, and ‘alsa-plugins’.
Since I run Arch Linux I do this by running:
After that reboot your device:
If you run Raspbian “wheezy”:
After that run the following command:
This should print out a list of your sound cards like so:
bcm2835 ALSA
1 [Set ]: USB-Audio - C-Media USB Headphone Set
C-Media USB Headphone Set at usb-bcm2708_usb-1.2, full speed
Notice that the name of my device is ‘Set’ – we will use this later.
You can also see the name by running the following command:
To get even more information run:
Now we want to set this sound card as the default. This is done by the following command:
Be sure to replace ‘cardname’ with the name of your sound card. In my case I replace it with ‘Set’ like so:
Also add the following:
type hw
card Set
device 0
}
ctl.!default {
type hw
card Set
device 0
}
To: /etc/asound.conf:
Now reboot once more.
Now it is time to test if it is actually working. Run the following command:
You should hear noise in your headphones.
Okay now try to run the following command:
You should hear a high pitch tone.
Now for something a bit more interesting. Try to run the following command:
Be sure yo replace ‘Set’ with the name of your sound card.
Now the input from the microphone should be played out the audio output of the USB sound card.
To adjust your setting – like volume, microphone level etc. Use the following command:
And then to store your settings:
To make a 10 second audio recording at a samplerate of 16000 bit/s via the microphone run:
To play it back:
Now to the fun part!
To stream the audio run:
! mulawenc ! rtppcmupay ! udpsink host=clientIp port=5001
Replace the clientIp with your computers ip address. Normally I just set the end to 255, so it can be received on every computer on the network, for instance I set mine to: 10.0.0.255.
To receive the audio on your computer:
! queue ! rtppcmudepay ! mulawdec ! audioconvert ! autoaudiosink sync=false
If you experience dropouts you might want to add ‘num-buffers=1000’ just after ‘device=plughw:Set’ at the server side, but in my experience it actually works better by not using it.
In the end I decided to create two scripts. One at the streaming end (the Raspberry Pi) and another and the receiver – my computer. This will allow me to stream the video and audio simultaneously from the Raspberry Pi to my computer.
The streaming script:
serverIp=$(ifconfig | grep -E 'inet.[0-9]' | grep -v '127.0.0.1' | awk '{ print $2}')
clientIp=$(echo $serverIp | cut -d '.' -f 1-3).255 # Send to all
gst-launch-1.0 -v alsasrc device=plughw:Set \
! mulawenc ! rtppcmupay ! udpsink host=$clientIp port=5001 &
raspivid -t 999999 -w 1080 -h 720 -fps 25 -hf -b 2000000 -o - | \
gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=96 \
! gdppay ! tcpserversink host=$serverIp port=5000
kill $!
And the receiving script:
serverIp=rpi.local
gst-launch-1.0 -v udpsrc port=5001 caps="application/x-rtp" \
! queue ! rtppcmudepay ! mulawdec ! audioconvert ! autoaudiosink sync=false &
gst-launch-1.0 -v tcpclientsrc host=$serverIp port=5000 \
! gdpdepay ! rtph264depay ! avdec_h264 ! videoconvert ! autovideosink sync=false
kill $!
Make sure to make the script executable:
Note that I have setup Bonjour on the Raspberry Pi so I can simply refer to it as hostname.local – in my case rpi.local. More information can be found at the following site: http://gettingstartedwithraspberrypi.tumblr.com/post/24398167109/file-sharing-with-afp-and-auto-discovery-with-bonjour.
I am very satisfied with the performance. There is almost no latency at all both via a wired or a wireless connection.
Fell free to post any questions or comments below and I will answer as quickly as possible! 🙂
Hi,
thank you very much for the bash scripts. I tried them out, they work very well.
But could you could help me?
I’m trying to adjust your code for streaming video (rPi Camera module) and Audio (USB soundcard – hw:1,0) to utream or another of the online services.
I tried ffmpeg and the delay is just too much and I can’t sync the audio, it moves in and out an in the end just lags behind more and more.
So I looked at gstreamer and came across your post.
Is there any advice you could give me?
Many thanks
@AdskiRemote
Sorry I can’t help you with that. I think you would need to convert the stream on a computer and then send that stream to ustream. You might be able to set up a server to do that for you.
The problem is that the Raspberrry Pi doesn’t have enough processing power to convert the stream and send it without major delay, as you saw with ffmpeg.
Hi,
thank you for the useful guide.
I need to clarify can gstreamer server in the Raspberry Pi stream out a video protocol that can be receive from another client application natively like using GUI VLC, GUI XBMC, console mplayer, console omxplayer etc.
Does the gstreamer server support multiple client connection ?
Thank you
@mybox
I have not figured out to do something like that without having a huge delay on the video feed. If you figure out to do so using gstreamer, please let me know.
I believe so, simply just run the receiver script on another computer and see how it works.
I have tried using GUI VLC open network stream (Pi IP Addr is 192.168.0.136):-
rtmp://192.168.0.136:5000
rtsp://192.168.0.136:5000
tcp://192.168.0.136:5000
rtp://192.168.0.136:5000
The script running at Pi is:-
raspivid -t 999999 -h 720 -w 1080 -fps 25 -hf -b 2000000 -o – | gst-launch-1.0 -v fdsrc ! h264parse ! rtph264pay config-interval=1 pt=96 ! gdppay ! tcpserversink host=192.168.0.136 port=5000
There is error message display on VLC :- Your input can’t be opened:
The network connection from my PC where VLC is running to the Pi is fine
Any idea is it the parameter given to VLC is wrong ?
Pls also let me know the script to run in GUI XBMC, console mplayer & console omxplayer, or any others GUI or console video player application, I will try that out.
Thanks
When I run the script on another Pi:-
gst-launch-1.0 -v tcpclientsrc host=192.168.0.136 port=5000 ! gdpdepay ! rtph264depay ! avdec_h264 ! videoconvert ! autovideosink sync=false
It gives below error:-
WARNING: erroneous pipeline: no element “avdec_h264”
Any advise ?
@mybox
You can’t open the stream in VLC. You need to have gstreamer installed on your computer and then run the receiver script I have provided. gstreamer is only available on Mac OS X and Linux, so you can’t use it in Windows.
In order to see the stream VLC you properly have to convert the raw stream from gstreamer on a your computer or server and then watch the outgoing stream in VLC.
I haven’t got two Pi’s, so I can’t test that, sorry.
Thank you for this great guide.
Would it be possible to save fullHD video and at the same time stream low quality video (320×240)? I managed to split video in two pipelines using gstreamer tee, but I can’t figure it out, how to properly reduce resolution. If I understand it correctly the video should be decoded, resized with videoconvert and encoded before using stream command.
Thanks
@Ronin.si
Sorry, but I can’t help you with that. Please post your solution if you figure it out!
Hey can you please contact me..
I think I have a use for your set up mentioned above but for audio only..
Hope to hear from you soon
@simon
Okay. Please elaborate.
Hey
Basically I would like to make a wireless microphone..
Could you please email me, and I can give you the full idea?
Thanks
Simon
@simon
Okay. Check you inbox. I have just send you one.
I have the same problem as @mybox: WARNING: erroneous pipeline: no element â??avdec_h264â?³
avdev_h264 doesn’t seem to be a gstreamer plugin according to http://gstreamer.freedesktop.org/documentation/plugins.html so I’m very confused!
Oh `sudo pacman -S gst-libav` solved the problem on Archlinux. Thanks!
@Kai Hendry (@kaihendry)
Thanks for posting your solution 😉
Hello.
I have tried your code but i have error.
—————————————-error mesages——————————–
pi@raspberrypi:~$ gst-launch-1.0 -v alsasrc device=plughw:1 ! mulawenc ! rtppcmupay ! udpsink host=192.168.10.255 port=5001
Setting pipeline to PAUSED …
ERROR: Pipeline doesn’t want to pause.
ERROR: from element /GstPipeline:pipeline0/GstUDPSink:udpsink0: GStreamer encountered a general resource error.
Additional debug info:
gstmultiudpsink.c(959): gst_multiudpsink_start (): /GstPipeline:pipeline0/GstUDPSink:udpsink0:
Failed to bind socket: Unable to create socket: Address family not supported by protocol
Setting pipeline to NULL …
Freeing pipeline …
—————————–end——————————
Please Help me…
@David
I don’t know the answer to that, sorry!
Hi there.
I need for a scool project one kit based on raspberry-pi with:
one Micro Load Cell (0-5kg), one webcam, one web interface where I can see the streaming of 1 or 2 webcamera, and values of applied force(compression force) on sensor distinct field, at intervals of 3 seconds.
Can you help me with this project?
@darius
Check out this blog post: http://wolfpaulus.com/jounal/embedded/raspberrypi_webcam. I will get you started with creating a webpage where you can watch the camera feed 😉
Hello. Your tutorial helped me a lot, I was trying to do some streaming but with a webcam and I used your tutorial for the audio, its all working well but I have a problem. ctrl+c only kills one of the processes, i wold like to know how to kill both.
@RicS
You need to kill the process. See the following site: http://www.cyberciti.biz/faq/kill-process-in-linux-or-terminate-a-process-in-unix-or-linux-systems/.
Hi there,
I am a newbie and trying to do your commands, step by step, but in my Archlinux, there is i am typing “asoundconf list”, receiving the following error:
-bash: asoundconf: command not found
so what can i do?
@Vahik
You need to install it from AUR: https://aur.archlinux.org/packages/asoundconf/. I recommend install Yaourt: https://wiki.archlinux.org/index.php/Yaourt. This will allow you to install it just like you would with pacman.
hi
nice blog
sorry for newbie
can u help me i want to do the other way round
my pc as a server for audio and raspberry as a receiver
thanks
@siraya
It is very similar. I would recommend using gstreamer as well. Do you only want to stream audio or video as well?
so waht does the “kill $!” does?
maybe i create another script with the pkill comand
@RicS
It kills the last process executed in the background. See: http://stackoverflow.com/a/1624713/2175837.
Okay you can do that as well.
@Ronin.si
can you tell me please how did you manage to see video stream at the same time with local storage? i also tried with tee, but didn’t succeed with the streaming part, i receive only some raw data, in chrome, a file starts downloading, raw data i guess. thanks!
@ciompoco
What do you mean? What are you using Chrome for anyway? You need to install gstreamer: http://gstreamer.freedesktop.org/download/ and then run the command that I have provided.
also, i don’t have a stream even with the command mentioned here. chrome return “no data received”, but if i refresh the page several times, a file start to download and stop after a short period of time. i can’t see video at all. i would really apreciate some help here, i’m stucked.
can’t i see video stream on a web page with gstreamer?
@ciompoco
No you can’t use gstreamer in Chrome. You would need to write some kind of application that uses gstreamer to receive the stream. This post only shows how to use gstreamer from the command line and not implement it in any application, as I don’t need that, so I haven’t looked into it.
ok, i appreciate your support!
@ciompoco
You are welcome. If you are interrested in writing your own application using gstreamer you should check out there tutorials: http://docs.gstreamer.com/display/GstSDK/Tutorials 😉
Lauszus,
Everything worked really well, thanks to your clear instructions. Very creative and much obliged.
raspberry:
gst-launch-1.0 -v tcpclientsrc host=xxx.xxx.xxx.xxx port=5000 ! gdpdepay ! rtph264depay ! avdec_h264 ! videoconvert ! autovideosink sync=false
os x:
gst-launch-1.0 -v tcpclientsrc host=10.0.1.16 port=5000 ! gdpdepay ! rtph264depay ! avdec_h264 ! videoconvert ! autovideosink sync=false
@pimlico
Good to hear that you found it useful 🙂
Hi, I am trying to stream audio from a raspberry pi to another, using your instructions. On the receiving end I get the message
Cannot connect to server socker err= No such file or directory
Cannot connect to server request channel
jack server is not running or cannot be started
Do you have any idea why this happens? also, is there any documentation to understand the commands you use?
@carlos
Are you sure that you are using the right IP address?
Documentation is available here: http://gstreamer.freedesktop.org/documentation/.
Hi.
Tested your camera streaming code for and working perfectly.
What i want to do is to use UDP instead of TCP but seem that i can’t find how to do this.. 🙁
Can you please guide me on how to use UDP ?
Thank you.
@NikTheGreek
No I can not, sorry. I haven’t used gstreamer for a while, but it is very similar to the approach above.
Hi, i`m doing your method step by step but i got the error bash:asoundconf:command not found.
I saw that someone earlier had the same problem but i`m running wheezy instead of arch linie.
So what can i do?
@JoostDutch
Have you remembered to install all the packages needed?
I tried streaming from pi to pi and from pi to windows but I was not able to get the latency as low as you described (I have a latency of almost half a second).
You mentioned that you are using Arch Linux instead of Raspbian. Could that be the reason for the lower latency?
@Alexander B.
I can’t say for sure, but it would be interesting to see. Please post your findings if you ever try it 😉
Thanx, nice tutorial.
[@Alexander B.] You probably should take care about your input buffering. The receiving application can have a large input buffer. Also the latency (in H.264-based apps) depends on the h.264 profile and level used by the encoder. The encoder can delay output because it can encode (when configured so) the next several frames using a reference frame.
I’m using gstreamer-0.10 in Ubuntu and this receiving code worked for me.
gst-launch -v tcpclientsrc host=192.168.1.107 port=5000 ! gdpdepay ! rtph264depay ! ffdec_h264 ! ffmpegcolorspace ! autovideosink sync=false
Transmitter code on the Pi is the one from this blog.
@Fred
Thanks for sharing 😉
Hi,
About Your solution,,,
Can I receive stream using Safari,Chrome,FireFox,Explorer on Windows?
And using Android Phone?
How about the IP CAM VIEWER of android App? Can I?
Thanks,
SB,YIM
@SB YIM
No you can not view it in a browser, but you will be able to write a dedicated Android app that. Please take a look at the following site for more information: http://www.gstreamer.com/.
Hello, Thank you very much for the scripts.
Can you you tell me how to save the stream locally (video + audio) in an unique avi file or in other extension ?
Hello,
With your streaming video script i have an error :
ERROR: from element /GstPipeline:pipeline0/GstH264Parse:h264parse0: No valid frames found before end of stream
Additional debug info:
gstbaseparse.c(1066): gst_base_parse_sink_default (): /GstPipeline:pipeline0/GstH264Parse:h264parse0
ERROR: pipeline doesn’t want to preroll.
Do you have an idea why ?
Thank you
@Damien
I haven’t used this for a while and haven’t got time to look into it right now, sorry.
For the preroll error, it was just a permission issue : just do that :
sudo chmod a+rw /dev/vchiq
and the stream start correctly, received is ok too.
Thank you !
@Lauszus
How do real-time video or audio is sent in real time to your Android phone?
@kimSanghuck
Please see the following link: http://docs.gstreamer.com/display/GstSDK/Android+tutorials.
?Would be perfect combined with the free open source software Datahrei/Restreamer to get a 24 FPS HLS streaming camera for your website. https://datarhei.github.io/restreamer
@Kristian Sloth Lauszus
Hey Lauszus,
I used these instructions about a year ago and they worked without a hitch! Well… for a linux noob like me it took some tinkering (and i was using Debian not Arch) but I got my audio streaming!
I decided to revisit this project but things really aren’t working this time. Doing
sudo apt-get install alsa-firmware alsa-plugins
just gives back:
E: Unable to locate package alsa-firmware
E: Unable to locate package alsa-plugins
And asoundconf returns:
asoundconf: command not found
Some initial googling makes it seem like it’s deprecated or not in use anymore for Pulse?
Anyway I’d really appreciate any guidance you can give. I really can’t seem to find anyone else or guides that provide latency free audio streaming.
Cheers
@funkedelic_bob
Unfortunately I have not played with this for a long time, so unfortunately I can’t be of much help.
Is there an possibilizy to change the sample rate of the audio with this method(gst-launch -1.0 -v alsasrc device=plughw:1,0 rate= 16000… for example) I have the problem, that if i want to save and stream at the same time the raspberry can’t read the audio input fast enought with sample rate on 44100. Thanks for help.
Hi there, getting this massage and couldn’t find solution on the web,
any idea? Thanks
** (gst-launch-1.0:8492): CRITICAL **: gst_gl_window_get_context: assertion ‘GST_IS_GL_WINDOW (window)’ failed
Caught SIGSEGV
exec gdb failed: No such file or directory
Spinning. Please run ‘gdb gst-launch-1.0 8492’ to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.