Add Markdown support in chatbots

[abidlabs]

Adds support for Markdown in the chatbot component. Closes: #2498

Demo code 1 (supply your own image instead of lion.jpg):

import gradio as gr

with gr.Blocks(css="#chatbot .overflow-y-auto{height:700px!important}") as demo:
    gr.Chatbot([("hi", "hello **abubakar**"), ("![](/file=lion.jpg)", "cool pic")], elem_id="chatbot")
    
demo.launch()

Looks like this:

Demo code 2 (this has been added in the demo/ directory):

import gradio as gr

def add_text(state, text):
    state = state + [text] + [text + "?"]
    return state, state

def add_image(state, image):
    state = state + [f"!()[/file={image}]"] + ["Cool pic!"]
    return state, state


with gr.Blocks() as demo:
    chatbot = gr.Chatbot()
    state = gr.State([])
    
    with gr.Row():
        with gr.Column(scale=0.85):
            txt = gr.Textbox(show_label=False, placeholder="Enter text and press enter, or upload an image").style(container=False)
        with gr.Column(scale=0.15, min_width=0):
            btn = gr.UploadButton("🖼️", file_types=["image"])
            
    txt.submit(add_text, [state, txt], [state, chatbot])
    btn.upload(add_image, [state, btn], [state, chatbot])
            
demo.launch()

[github-actions]

The demo notebooks don't match the run.py files. Please run this command from the root of the repo and then commit the changes:

pip install nbformat && cd demo && python generate_notebooks.py

[github-actions]

All the demos for this PR have been deployed at https://huggingface.co/spaces/gradio-pr-deploys/pr-2731-all-demos

[github-actions]

The demo notebooks don't match the run.py files. Please run this command from the root of the repo and then commit the changes:

pip install nbformat && cd demo && python generate_notebooks.py

[abidlabs]

The demo notebooks don't match the run.py files. Please run this command from the root of the repo and then commit the changes:

Thanks @aliabd!

[abidlabs]

Fixed the UploadButton component to save files in the right location

[aliabid94]

LGTM

[pngwn]

Why markdown support over just supporting certain media discreetly? Does supporting all markdown cause any issues?

I also don't think forcing user who want to use images (other other media) in the chatbot to use markdown makes for the best experience. Markdown syntax for some thing is kinda weird, images is definitely one of those cases.

[abidlabs]

Why markdown support over just supporting certain media discreetly? Does supporting all markdown cause any issues?

Because we'd need to add support for each of those media types individually and come up with an appropriate syntax to include them whereas Markdown already does that. I haven't seen any issues when I tried using Markdown, but are there specific things you're concerned about?

Markdown syntax for some thing is kinda weird, images is definitely one of those cases

A little bit yes, but everything is well-documented which is a big plus over having to figure out gradio-specific documentation

[pngwn]

A little bit yes, but everything is well-documented which is a big plus over having to figure out gradio-specific documentation

A path and a file extension could determine how we handle the media, markdown has to be learned too and isn't as ubiquitous and some people say.

I haven't seen any issues when I tried using Markdown, but are there specific things you're concerned about?

Markdown supports pretty much all HTML which could easily break the rendering. Additionally, certain block level constructs could cause issues.

Additionally, we should also round the corners of images/ videos to ensure they match the style of the chatbox.

[abidlabs]

A path and a file extension could determine how we handle the media, markdown has to be learned too and isn't as ubiquitous and some people say.

But keep in mind that the Chatbot component takes in a set of strings, so within the string, we still way of designating that this particular string denotes a file as opposed to just a string that happens to include a path -- we need some sort of syntax, and Markdown offers an existing standard as opposed to us coming up with our own that we have to maintain and document.

Markdown supports pretty much all HTML which could easily break the rendering. Additionally, certain block level constructs could cause issues.

Let me investigate this more

Additionally, we should also round the corners of images/ videos to ensure they match the style of the chatbox.

Sure, I'll try to do this but may need some help on the css side!

[abidlabs]

Discussed with @pngwn and we'll go with Markdown, but clearly document that only a subset of Markdown is supported (note that audio and videos are not currently supported).

Also fixed the images to have a rounded border radius.

[abidlabs]

Here's how the rounded image looks:

[abidlabs]

Thanks for the reviews @aliabid94 and @pngwn!

[johnPertoft]

Cool change! But it broke my app unfortunately. Is it on the user now to parse the html/markdown output if we want to access the chat history or is there some option to do this?

[abidlabs]

How did it break your app @johnPertoft? Can you open a new issue with more details and a code example so that we can reproduce?

[johnPertoft]

It's really not a big deal but

In gradio<3.12 the history passed in looks like

[('hello!', " don't know")]

In gradio>=3.12 the history passed in looks like

[('<p>hello!</p>\n', "<p>I don't know</p>\n")]

So the app "broke" in the sense that it now requires the user to remove the html/markdown added by this change if depending on the chat history. It's a very small change though so I'm not sure it warrants any action taken. I'll open an issue and you can decide:)

[pngwn]

This feels like a bug to me. The history should contain the source you passed in, not the post-processed data. In this case it might be simple enough to work with the markup but some markdown transforms are very complex and will be impossible to do anything with other than appending to the history.

[falcondai]

An ergonomic/QoL issue with the multimedia messages: unlike an Image or Video component, the preprocess method does not copy the inputed file into a private temp directory. I am using TempFileManager to copy the media files before passing them into a Chatbot component. Feels awkward.

Happy to contribute a patch to preprocess in a manner similar to that of Image.

[abidlabs]

Hi @falcondai good catch. Please do open a PR and I'm happy to review

[Simon1V]

With a jpg file called lion.jpg in the same directory, I only get this^^
Using the latest version of gradio

[abidlabs]

What's your full code @Simon1V?

[Simon1V]

What's your full code @Simon1V?

The original one, just copy & pasted

I just tried with another 350x350 jpg, also named lion.jpg to no avail
I got

• main.py with that code
• lion.jpg

in the same folder

[abidlabs]

Now you should just be able to do this:

import gradio as gr

with gr.Blocks(css="#chatbot .overflow-y-auto{height:700px!important}") as demo:
    gr.Chatbot([("hi", "hello **abubakar**"), (("lion.jpg", ), "cool pic")], elem_id="chatbot")
    
demo.launch()

[Simon1V]

Thank you, this worked : )
(Is there also a way to send images and text simultaneously? For multimodal input)

[keranrong]

Thank you, this worked : ) (Is there also a way to send images and text simultaneously? For multimodal input)

+1 And is there a way of make the image inline with the text please? (such as we send a images with a comment followed inline)

[semirke]

Hi there,

is it possible to insert an image between the lines? (So like when the query engine responds with text containing markdown.)
For eg.: "As you can see on the image , all fields has to be filled."

I tried with "" and fun part it sort of works, however, the file is not moved to tmp and so the link is malformed and the img url points to the root.

Thanks,
Semirke