gphoto2: preview at countdown from DSLR

[florianmueller]

Is your feature request related to a problem? Please describe.
First, thank you so much for this project, it is the best photo booth project out there, I tried a lot.
I am missing a preview for people in front of the photo box before or while the image is taken.
I know you can utilize device cams, but I would only have the DSLR live preview feed available.

Describe the solution you'd like
A way to pipe out the cameras gphoto2 --capture-movie stream directly into the webinterface background.

Describe alternatives you've considered
I tried setup a seperate http stream on port 8080 and use it in the user interface settings as background URL, but it does not work. It seems chromium has problems with disyplaing mjpeg streams directly as a background.

Additional context
Thank you very much for any suggestion and help you could provide me on that topic.

[couz74]

Hi,
After multiples test I got this working:

In core.js

var pid;
    public.startVideo = function () {
    	  const data = {
            play: "true"
        };
        if (!navigator.mediaDevices) {
            return;
        }
    	 jQuery.post('api/takeVideo.php', data).done(function (result) {
    	  	   console.log('Start webcam',result);
    	  	   pid=result.pid;
        		const getMedia = (navigator.mediaDevices.getUserMedia || 
 navigator.mediaDevices.webkitGetUserMedia || navigator.mediaDevices.mozGetUserMedia || false);

        		if (!getMedia) {
        	   	console.log('No user media');
            	return;
        		}`

        		if (config.previewCamFlipHorizontal) {
            	$('#video--view').addClass('flip-horizontal');
        		}

        		getMedia.call(navigator.mediaDevices, webcamConstraints)
            	.then(function (stream) {
            	 	console.log('Success getting user media')
                	$('#video--view').show();
                	videoView.srcObject = stream;
                	public.stream = stream;
            	})
            	.catch(function (error) {
                console.log('Could not get user media: ', error)
            });
        }).fail(function (xhr, status, result) {
           console.log('Could not start webcam',result)
        });
    }
    public.stopVideoAndTakePic = function (data) {
        if (public.stream) {
        	 const dataVideo = {
            play: "false",
            pid: pid
            };
        
    	  	jQuery.post('api/takeVideo.php', dataVideo).done(function (result) {
            console.log('Stop webcam',result)            
            const track = public.stream.getTracks()[0];
            track.stop();
            $('#video--view').hide();
            public.callTakePicApi(data);
        	}).fail(function (xhr, status, result) {
           console.log('Could not stop webcam',result)
        	});
        }
    }
// take Picture
    public.takePic = function (photoStyle) {
        if (config.dev) {
            console.log('Take Picture:' + photoStyle);
        }
 	
        const data = {
            filter: imgFilter,
            style: photoStyle,
            canvasimg: videoSensor.toDataURL('image/jpeg')
        };	

        if (photoStyle === 'collage') {
            data.file = currentCollageFile;
            data.collageNumber = nextCollageNumber;
        }
        
        if (config.previewFromCam) {
            if (config.previewCamTakesPic && !config.dev) {
                videoSensor.width = videoView.videoWidth;
                videoSensor.height = videoView.videoHeight;
                videoSensor.getContext('2d').drawImage(videoView, 0, 0);
            }
            public.stopVideoAndTakePic(data);
        }else {
          public.callTakePicApi(data);
        }
     }

    public.callTakePicApi =function (data) {
    	console.log(data);
      jQuery.post('api/takePic.php', data).done(function (result) {
            console.log('took picture', result);
            $('.cheese').empty();
            if (config.previewCamFlipHorizontal) {
                $('#video--view').removeClass('flip-horizontal');
            }

            // reset filter (selection) after picture was taken
            imgFilter = config.default_imagefilter;
            $('#mySidenav .activeSidenavBtn').removeClass('activeSidenavBtn');
            $('#' + imgFilter).addClass('activeSidenavBtn');

            if (result.error) {
                public.errorPic(result);
            } else if (result.success === 'collage' && (result.current + 1) < result.limit) {
                currentCollageFile = result.file;
                nextCollageNumber = result.current + 1;

                $('.spinner').hide();
                $('.loading').empty();
                $('#video--sensor').hide();

                if (config.continuous_collage) {
                    setTimeout(() => {
                        public.thrill('collage');
                    }, 1000);
                } else {
                    $('<a class="btn" href="#">' + L10N.nextPhoto + '</a>').appendTo('.loading').click((ev) => {
                        ev.preventDefault();

                        public.thrill('collage');
                    });
                    $('.loading').append($('<a class="btn" style="margin-left:2px" href="./">').text(L10N.abort));
                }
            } else {
                currentCollageFile = '';
                nextCollageNumber = 0;

                public.processPic(data.photoStyle, result);
            }

        }).fail(function (xhr, status, result) {
            public.errorPic(result);
        });
    }    

in takeVideo.php

<?php
header('Content-Type: application/json');

require_once('../lib/config.php');

function isRunning($pid){
    try{
        $result = shell_exec(sprintf("ps %d", $pid));
        if( count(preg_split("/\n/", $result)) > 2){
            return true;
        }
    }catch(Exception $e){}

    return false;
}

if ($_POST['play'] === "true" ) {
  $pid = exec('gphoto2 --stdout --capture-movie | ffmpeg -i - -vcodec rawvideo -pix_fmt yuv420p -threads 0 -f v4l2 /dev/video0 > /dev/null 2>&1 & echo $!', $out);      
  sleep(3);  
  die(json_encode([
     'isRunning' => isRunning($pid),
      'pid' => $pid - 1
    ]));
}elseif($_POST['play'] === "false") { 
    exec('kill -15 '.$_POST['pid']);
    die(json_encode([
     'isRunning' => isRunning($_POST['pid']),
      'pid' => $_POST['pid']
    ]));
}

For it to work on startup I had to edit /etc/rc.local:
modprobe v4l2loopback exclusive_caps=1 card_label="GPhoto2 Webcam"
rmmod bcm2835-isp

rmmod bcm2835-isp was needed as chromium was taking this device instead of the V4l2 one.

How it work
We use v4l2loopback to create a virtual device and gphoto2 --stdout --capture-movie | ffmpeg -i - -vcodec rawvideo -pix_fmt yuv420p -threads 0 -f v4l2 /dev/video0 to send the photo preview to it.
On "take pic" button we start the process and end to before the picture is taken to free gphoto2.

It is a ugly draft, I hope I will improve it and post the update here.

[andi34]

@couz74 great! maybe you like to commit your changes and push them to GitHub?

[couz74]

It's still not clean at all, I will try to do something nicer and push them

[couz74]

I pushed the draft here https://github.com/couz74/photobooth.
There is still two issue:

• Sometime I need to run sudo gphoto2 --stdout --capture-movie | ffmpeg -i - -vcodec rawvideo -pix_fmt yuv420p -threads 0 -f v4l2 /dev/video0 first and load chrome a first time with a webpage asking for the camera. If I don't do so sometime chrome doesn't detect the V4l2 camera launch from php.
• On the takeVideo.php I need to use a sleep() method in order to allow the stream to launch, still need to figure a way to have something cleaner

[florianmueller]

Thanks a lot for your solution! I used a similar very unstable solution outside your app, where I used a bash script as the "take picture" command that essential stoped the gphoto2 capture-movie, takes a picture and starts the capture video again. The issue was that running the --capture movie command as a service, crashes the live view quite quickly. So really appreciate your integrated approach into the application itself.

I tried running your solution, but having issues to make the livestream show up.
Where do you enable it in the admin interface? Is it "Use device cam"? I don't see you using separate http service to stream the live view from /dev/video0 to a localhost port...like motion would do for example.

EDIT: it might be that on my pi the rmmod bcm2835-isp fails with rmmod: ERROR: Module bcm2835_isp is not currently loaded
May that mean the pi is not recognizing the /dev/video0 live view as a webcam because the kernel module is not loaded for webcams?

[couz74]

Hi,
In the admin panel the setting "See preview by device cam" must indeed be set.
Then there are three things:

• modprobe v4l2loopback exclusive_caps=1 card_label="GPhoto2 Webcam" create the virtual webcam and sudo gphoto2 --stdout --capture-movie | ffmpeg -i - -vcodec rawvideo -pix_fmt yuv420p -threads 0 -f v4l2 /dev/video0 in the takeVideo.php send the outpout of the Gphoto command to the /dev/video0 device (you can use v42l-ctl --list-devices to check which /dev/* is the correct one).
• The second thing is the issue with Chromium (did not test firefox), if there is another webcam like
  bcm2835-isp in my side it take it by default instead of the Gphoto one... therefore I always do rmmod bcm2835-isp.
• Finaly Chromium sometimes also doesn't detect the Gphoto webcam if it wasn't loaded first with a Gphoto Webcam having the Ffmjpg feed . To do that I send the command in a shell sudo gphoto2 --stdout --capture-movie | ffmpeg -i - -vcodec rawvideo -pix_fmt yuv420p -threads 0 -f v4l2 /dev/video0 then I start Chromium and check if a testing webcam website work.

As you can see there is still some issue with the solution XD

[florianmueller]

Great stuff! So theoretically that worked for me as well. Thanks! But somehow I only get a static and not up to date image as live view when taking a picture. Chrome mostly recognize the camera once access is allowed.
Some strange observation I made with the live view performance: running the gphoto livefeed on a test website like webcamtests.com I get a super smooth framerate. Like something around 22 to 25 fps. But when using it on the application or on a motion server, the framerate drops to 9fps or lower. Probably the reason the preview as device cam looks like a single stand image thats not really updating.

I think for the interface, having the live view constantly running as a background image, like discussed in another issue here is the most user friendly way of taking your picture.

So as a workaround for now, I would use motion and enter the http stream as a background address in interfaces. But in order for it to work, would it be possible to change the script in core js to just start and stop the takeVideo.pho, but not utilizing it, so motion is free to access the feed from /dev/video0 everytime takeVideo.php starts/stops?

Regarding the bmc2835 blocking gphoto, I could not test it as I only have a DSLR connected and the camera port deactivated.

Thanks a lot for your support and great work with this photobooth application :)

[andi34]

I think for the interface, having the live view constantly running as a background image, like discussed in another issue here is the most user friendly way of taking your picture.

https://github.com/andi34/photobooth/pull/58/files

That would be a way to use a stream as background. (Please note that my personal fork is quite ahead. some changes and commits might need to be adjusted to work here, e.g. for L10N translation library to work, but most I've added here too https://github.com/andi34/photobooth/tree/ipad2 )

[couz74]

After rebasing on your repo @andi34, I also thought about letting the stream on the background. The only issue for me is that after a test my camera battery only last 1h in this situation. I will try to do so with another camera.

[andi34]

Well, in such case you shouldn't use a battery but instead a power adapter.

Also preview by device cam should still be possible optional.

I'd so add an option for gphoto2 --stdout --capture-movie | ffmpeg -i - -vcodec rawvideo -pix_fmt yuv420p -threads 0 -f v4l2 /dev/video0 in case user like to adjust the command via admin panel.
Maybe:

$config['gphoto_preview'] = true; // true/false
$config['gphoto_preview']['cmd'] = "gphoto2 --stdout --capture-movie | ffmpeg -i - -vcodec rawvideo -pix_fmt yuv420p -threads 0 -f v4l2 /dev/video0";

[andi34]

For a Quick test of the camera:
https://www.aaronbenjamin.design/camera-app/part-2
(Source: https://github.com/abenjamin765/camera-app )

[andi34]

@florianmueller @couz74 andi34#83 maybe you like to test that? Currently only have a cam only without the possibility to test gphoto video.