Convert video to playable format if ffmpeg installed by freddyaboulton · Pull Request #2003 · gradio-app/gradio

freddyaboulton · 2022-08-11T21:37:39Z

Description

Fixes #1970
Fixes #1990
Fixes #1972

In summary, I added video_is_playable util function to determine if a video can be played in the browser and convert_video_to_playable_mp4 in the event that the video is not playable. The rest are just unit tests.

I also added white_noise_vid_not_playable which is a reproducer of the original problem listed in one of the issues.

Checklist:

I have performed a self-review of my own code
My code follows the style guidelines of this project
I have commented my code in hard-to-understand areas
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

freddyaboulton · 2022-08-11T21:42:43Z

            assert y.dtype == x.dtype


+def test_video_has_playable_codecs(test_file_dir):


I added some files so that we can test positive and negative cases. I know increasing the repo size is a concern but I think testing the logic is worth adding the couple hundred kb

abidlabs · 2022-08-15T19:34:35Z

+def video_is_playable(video_filepath: str) -> bool:
+    """Determines if a video is playable in the browser.
+
+    A video is playable if it has a playable container and codec.


This is true all across major browsers?

I believe so! Will confirm

abidlabs · 2022-08-15T19:35:42Z

+            (".webm", "vp9"),
+        ]
+    # If anything goes wrong, assume the video can be played to not convert downstream
+    except Exception:


Nit: can we narrow the scope of this Exception?

I got a

video_codec= output["streams"][0]["codec_name"] IndexError: list index out of range

The narrowed scope of the exception broke my application, it worked when I reverted back to "except Exception"

Ah yikes that's good to know. I think it would be still be better to have a scope that includes IndexError as well, but WDYT @freddyaboulton?

@lazarusking sorry to hear that! I originally used a wide exception because analyzing video streams with ffmpeg felt really unpredictable.

Can you provide a reproducer so we can dig in further? I’m curious why ffmpeg can’t find any video streams in your file, yet the video can be played in the browser if comment this out.

My ffmpeg demo
It basically takes a file and converts it to the chosen format, allowing you the option to play it as video, audio or just as a file.
In the older gradio version, playing an audio as a video still works(when you toggle to video display option) but the new one doesn't find any video stream for such a case and just breaks.
It didn't account for "if anything goes wrong in this case".

Thanks - I can put up a PR to also catch IndexError and KeyError!

abidlabs · 2022-08-15T19:36:52Z

+            (".webm", "vp9"),
+        ]
+    # If anything goes wrong, assume the video can be played to not convert downstream
+    except Exception:


Nit: can we narrow the scope of this Exception?

abidlabs · 2022-08-15T19:38:03Z

+    """
+    try:
+        container = pathlib.Path(video_filepath).suffix.lower()
+        output = subprocess.run(


Instead of using subprocess, could we use ffmpeg.probe for cleaner code and better error handling: https://kkroening.github.io/ffmpeg-python/#ffmpeg.probe

abidlabs · 2022-08-15T19:40:29Z

+                global_options="-y -loglevel quiet",
+            )
+            ff.run()
+    except:


Same nit: is it possible to narrow the scope to ffmpeg.Error?

Yea good call!

abidlabs · 2022-08-15T19:55:44Z

I tried running the white_noise_vid_not_playable demo, and it isn't working for me. I see a completely black output:

Note that downloading the file works fine.

No Python logs except for the warning: UserWarning: Video does not have browser-compatible container or codec. Converting to mp4

On the JS side, I see: "Uncaught (in promise) DOMException: The element has no supported sources."

Tested on Chrome and Edge browsers on Windows.

freddyaboulton · 2022-08-15T20:01:42Z

@abidlabs Thanks for reviewing/running the demo! Let me dig into why it didn't work for you

abidlabs · 2022-08-15T21:58:51Z

Working great now @freddyaboulton!

github-actions · 2022-08-16T16:19:37Z

All the demos for this PR have been deployed at https://huggingface.co/spaces/gradio-pr-deploys/pr-2003-all-demos

lukasfolle · 2022-12-08T09:22:23Z

In my hf space, the video plays nicely on Chrome and Firefox, but not on Safari. I use the .ogg video container and the theora codec.

Normally, I would expect a conversion to mp4, but this is not happening..
Any ideas @freddyaboulton @abidlabs?

freddyaboulton · 2022-12-08T18:24:00Z

Hi @lukasfolle ! Thanks for asking. Very cool space and I am also unable to see the video on my mac when using safari.

The conversion to mp4 may not be happening because .ogg is a playable video container but it has spotty support on safari: https://caniuse.com/?search=ogg

Maybe you can "force" the mp4 conversion by giving your file an mp4 extension and using a different video codec?

I think just changing the extension here to .mp4 should be enough?

https://huggingface.co/spaces/lfolle/LDM-SyntheticChestX-Ray/blob/main/create_video.py#L19

lukasfolle · 2022-12-09T08:22:50Z

Hi @freddyaboulton,

Thanks for your reply. I now switched to mp4 (h264) and it seems to only work on edge, firefox now shows a black window for me. Running locally it works for me..
Any idea whats going wrong here?

freddyaboulton force-pushed the convert-video-to-playable-codecs branch from e55c4a8 to 459d485 Compare August 11, 2022 21:41

freddyaboulton commented Aug 11, 2022

View reviewed changes

Comment thread gradio/components.py

freddyaboulton commented Aug 11, 2022

View reviewed changes

Comment thread gradio/components.py

freddyaboulton marked this pull request as ready for review August 12, 2022 15:48

freddyaboulton requested review from abidlabs, aliabd, aliabid94, dawoodkhan82 and omerXfaruq August 12, 2022 15:48

abidlabs reviewed Aug 15, 2022

View reviewed changes

Comment thread gradio/processing_utils.py

abidlabs approved these changes Aug 15, 2022

View reviewed changes

freddyaboulton added 8 commits August 16, 2022 12:13

Add implementation + tests

23c64f7

Add comments'

da7c78b

Add warning

1ee9d8e

Update docs

f229104

Refactor logic

347dc23

3.7 fix attempt

a73e2b9

Fix 3.7 take 2

4c55a33

Return original video if the conversion fails

f3db84f

freddyaboulton force-pushed the convert-video-to-playable-codecs branch from 0247be3 to f3db84f Compare August 16, 2022 16:17

freddyaboulton added 3 commits August 16, 2022 12:52

Use ffmprobe + restrict scope of exception

b6afcdd

Fix for windows

f1caf4a

Fix tests

09ce25e

freddyaboulton merged commit 75b5180 into main Aug 16, 2022

freddyaboulton deleted the convert-video-to-playable-codecs branch August 16, 2022 17:21

freddyaboulton mentioned this pull request Aug 29, 2022

Catch IndexError, KeyError in video_is_playable #2113

Merged

6 tasks

bhack mentioned this pull request Apr 24, 2023

run app.py error z-x-yang/Segment-and-Track-Anything#17

Closed

		assert y.dtype == x.dtype


		def test_video_has_playable_codecs(test_file_dir):

Conversation

freddyaboulton commented Aug 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lazarusking Aug 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abidlabs Aug 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

abidlabs commented Aug 15, 2022

Uh oh!

freddyaboulton commented Aug 15, 2022

Uh oh!

abidlabs commented Aug 15, 2022

Uh oh!

github-actions Bot commented Aug 16, 2022

Uh oh!

lukasfolle commented Dec 8, 2022

Uh oh!

freddyaboulton commented Dec 8, 2022

Uh oh!

lukasfolle commented Dec 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

freddyaboulton commented Aug 11, 2022 •

edited

Loading

lazarusking Aug 27, 2022 •

edited

Loading

abidlabs Aug 15, 2022 •

edited

Loading

lukasfolle commented Dec 9, 2022 •

edited

Loading