Allow configuration that output Vespa ai logs to stdout together with file system

[b1102]

Our kubernetes environment configured in the way that everything output to the stdout of a pod, goes to Kibana logging warehouse.
If Vespa ai allows configuration that outputs logs to stdout, than no any additional configuration would require for getting logs.

Right now there is a workaround to fetch logs, by adding logging sidecar, but without it a configuration will be much simpler.

[bnsd55]

+1.
We also need it, we are also shifting logs from stdout to ES.

[arnej27959]

can't you just run "tail -F /opt/vespa/logs/vespa/vespa.log" ?

[bnsd55]

We can, I've created a sidecar container that run "tail -F /opt/vespa/logs/vespa/vespa.log"
But it don't think this should be a solution...

[arnej27959]

% cat Dockerfile

FROM docker.io/vespaengine/vespa:latest
RUN perl -pi -e 's{sleep infinity}{tail -F /opt/vespa/logs/vespa/vespa.log}' /usr/local/bin/start-container.sh

we go to great lengths to ensure all our logs end up in vespa.log, this looks like a simple solution.

[ean]

@bnsd55 support for sending all vespa log entries to the container's stdout is in progress here: vespa-engine/docker-image#34

Here is a sample of what it will look like with the new JSON output format enabled:

{"timestamp":"2022-10-07T12:12:25.46088192Z","host":"vespa-container.local","pid":"303/1","service":"configserver","component":"Container.org.eclipse.jetty.server.Server","level":"info","messages":["Started @6373ms"]}
{"timestamp":"2022-10-07T12:12:25.461883904Z","host":"vespa-container.local","pid":"303/1","service":"configserver","component":"Container.com.yahoo.container.jdisc.ConfiguredApplication","level":"info","messages":["Switching to the latest deployed set of configurations and components. Application config generation: 0"]}
{"timestamp":"2022-10-07T12:12:27.300350976Z","host":"vespa-container.local","pid":"626/33","service":"configproxy","component":"configproxy.com.yahoo.vespa.config.JRTConnection","level":"info","messages":["Connecting to tcp/vespa-container.local:19070"]}
{"timestamp":"2022-10-07T12:12:27.323859968Z","host":"vespa-container.local","pid":"626/32","service":"configproxy","component":"configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester","level":"warning","messages":["Request callback failed: APPLICATION_NOT_LOADED. Connection spec: tcp/vespa-container.local:19070, error message: Failed request (No application exists) from Connection { Socket[addr=/172.17.0.2,port=38244,localport=19070] }"]}

The JSON object is not yet stable, and will likely change in the future. We hope to come up with a format that will be as much a as possible a drop-in for the major cloud providers managed kuberenetes clusters log parsing.

If you have any input on the format, please let me know!

[arnej27959]

after adding "/opt/vespa/bin/vespa-logfmt --follow" to default startup script, this should be OK, if not we need a more specific description of what is needed / wanted.