Skip to content

Commit

Permalink
Sending in prompts with the wrong type hangs the server. This is a ch…
Browse files Browse the repository at this point in the history
…eck to make sure it's a list
  • Loading branch information
rprenger committed Oct 27, 2022
1 parent f4a8b1d commit 2fdd54e
Showing 1 changed file with 7 additions and 1 deletion.
8 changes: 7 additions & 1 deletion megatron/text_generation_server.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,15 @@ def put(self):
return "sentences is no longer used. Replace with prompts", 400

prompts = request.get_json()["prompts"]
if not isinstance(prompts, list):
return "prompts is not a list of strings", 400

if len(prompts) == 0:
return "prompts is empty", 400

if len(prompts) > 128:
return "Maximum number of prompts is 128", 400

tokens_to_generate = 64 # Choosing hopefully sane default. Full sequence is slow
if "tokens_to_generate" in request.get_json():
tokens_to_generate = request.get_json()["tokens_to_generate"]
Expand Down

0 comments on commit 2fdd54e

Please sign in to comment.