2024-09-10

Last modified: 2024-09-10 15:16:02

Encyclopedia Mechanica

I managed to get streaming responses working end-to-end. That is:

• ask for streaming output from OpenAI
• consume streaming output using LWP::UserAgent with :content_cb in a child process
• send streaming output out of Mojolicious
• consume streaming output in JavaScript

That was a lot of work, but it works now... except it only works on my desktop and not on the actual server. Frustrating.

I'm not sure what is going wrong when I try to use it on the server. It just seems to block forever and never give any output.

The problem could either be at the level of consuming the OpenAI output, or at the level of sending it out of Mojolicious. I have a test script for testing the OpenAI output, so let's try that on the server and see if that one works.

It doesn't! It just exits instead of starting the IO loop.

If I put AnyEvent->condvar->recv after the bit that launches the IO loop, then it works on the server as well.

If I make it use Mojolicious::Lite and app->start then it stops working again, just no output.

It looks like the child process is writing to the pipe OK, but the parent process is not managing to read from it.

The parent process tries to read the pipe using an AnyEvent::Handle, so perhaps Mojo is not using AnyEvent? How do I find out?

Bingo. ref(Mojo::IOLoop->singleton->reactor) is Mojo::Reactor::Poll. OK, so how do I make it use AnyEvent?

EV was not installed, maybe that's the problem. Woo, sorted now. Streaming mode works on https://incoherency.co.uk/encmech/

It seems a bit slower than when it was local, is there a larger buffer in nginx or something? Maybe I don't care.

proxy_buffering off in the nginx config did the trick. Great success.