Skip to main content
The interactive shell mode provides a convenient way to test and interact with your deployed model directly from the command line. The shell maintains conversation history and supports special commands for session management.

Launching the Shell

1

Start in shell mode

Add the --shell flag to launch the interactive shell instead of the API server:
2

Wait for initialization

The model will load and compile kernels. Once ready, you’ll see the prompt:
3

Start chatting

Type your message and press Enter to get a response from the model.

Example Interaction

Once loaded:
This function calls itself with n-1 until it reaches the base case (n=1). $
/reset/reset
$ /exit Exiting shell…

Features

Auto-completion

The shell provides auto-completion for available commands. Start typing / and press Tab to see suggestions.

Conversation History

The shell automatically maintains conversation context, allowing for multi-turn dialogues. All previous messages are sent with each request to preserve context:

Streaming Output

Responses are streamed token-by-token for a more interactive experience, similar to ChatGPT.

Use Cases

Model Testing

Quickly test model behavior and quality before deploying to production

Prompt Engineering

Experiment with different prompts and conversation flows

Debugging

Verify model outputs and troubleshoot generation issues

Demonstrations

Show model capabilities in live demos and presentations
Shell mode is not recommended for production use. For serving applications, use the Online Serving mode instead.

Limitations

  • Shell mode does not support dummy weights (testing mode)
  • History is stored in memory only and lost when the process exits
  • No support for multi-user scenarios
  • Single conversation thread only