Add cancel() method to interrupt a stream

[simonchatts]

Fixes #599.

Thanks for all your work on this project!

[tk-master]

please accept this pr @abetlen

[tk-master]

Actually.. I found an issue with this method.. this will only cancel after a token is generated but if the llm is slow or gets stuck processing the prompt, this doesn't cancel it..

We need a better method.

[tk-master]

I'm coming back to this because I need to figure out a better method to interrupt the generation programmatically..

For a console-based scenario it's pretty easy in python, all I have to do is surround the code with try except KeyboardInterrupt: .. then I can just press ctrl+c at any point to gracefully interrupt the llm..

But.. if I'm using a front-end user interface, I haven't managed to make it work properly let's say with a button "Stop generating" that can call a python function.. because of the issue I mentioned in the previous post..

@abetlen sorry to bother again but do you have any suggestions/ideas on how to accomplish this?

[woheller69]

Why not add it now and improve if there is a better solution. For now this would work in most cases.

[woheller69]

has anyone found a reasonable solution for this? Or am I the only one not willing to wait until the model finishes without killing the job and losing context?

[jewser]

Any chance this gets merged for now?

[madprops]

It indeed blocks until the first token is produced, but cancelling it after that is trivial. The other similar issue is cancelling a model that is loading.

[woheller69]

gpt4all python bindings offer a similar way which allows stopping with the next token

[ekcrisp]

+1 can we merge this?

[kingbri1]

Take a look at ggml-org/llama.cpp#10509 which should permanently solve this problem on lcpp's side