Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]: Base python client example fails (paddle model error) #371

Open
joaopcm1996 opened this issue Jan 23, 2025 · 1 comment
Open

[BUG]: Base python client example fails (paddle model error) #371

joaopcm1996 opened this issue Jan 23, 2025 · 1 comment
Labels
bug Something isn't working

Comments

@joaopcm1996
Copy link

Version

24.10.1

Which installation method(s) does this occur on?

Docker

Describe the bug.

Deployed nvingest with the docker compose from main branch, on H100's.

Tried to run the base python client example, with the default document in the notebook.

Immediately got an error in the handling of paddle NIM output.
Error logs extracted from nvingest runtime container.

Minimum reproducible example

https://github.com/NVIDIA/nv-ingest/blob/main/client/client_examples/examples/python_client_usage.ipynb

Relevant log output

INFO:     172.18.0.7:38424 - "GET /v1/health/ready HTTP/1.1" 200 OK
2025-01-22 11:04:55,866 - ERROR - Inference failed for model paddle: 'int' object has no attribute 'decode'
Unhandled error calling image inference model: Inference failed for model paddle: 'int' object has no attribute 'decode'
Traceback (most recent call last):
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 120, in _call_image_inference_grpc_client
    return " ".join([output[0].decode("utf-8") for output in result.as_numpy("output")])
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 120, in <listcomp>
    return " ".join([output[0].decode("utf-8") for output in result.as_numpy("output")])
AttributeError: 'int' object has no attribute 'decode'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/stages/nim/table_extraction.py", line 78, in _update_metadata
    paddle_result = call_image_inference_model(paddle_client, "paddle", image_array, trace_info=trace_info)
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/tracing/tagging.py", line 185, in wrapper_inject_trace_info
    result = func(*args, **kwargs)
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 103, in call_image_inference_model
    response = _call_image_inference_grpc_client(client, model_name, image_data)
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 124, in _call_image_inference_grpc_client
    raise RuntimeError(err_msg)
RuntimeError: Inference failed for model paddle: 'int' object has no attribute 'decode'
2025-01-22 11:04:55,877 - ERROR - Future result failure - Inference failed for model paddle: 'int' object has no attribute 'decode'

Error occurred while extracting table data.
Traceback (most recent call last):
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 120, in _call_image_inference_grpc_client
    return " ".join([output[0].decode("utf-8") for output in result.as_numpy("output")])
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 120, in <listcomp>
    return " ".join([output[0].decode("utf-8") for output in result.as_numpy("output")])
AttributeError: 'int' object has no attribute 'decode'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/stages/nim/table_extraction.py", line 133, in _extract_table_data
    df["metadata"] = df.apply(_update_metadata, axis=1, args=(paddle_client, paddle_version, trace_info))
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/pandas/core/frame.py", line 9568, in apply
    return op.apply().__finalize__(self, method="apply")
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/pandas/core/apply.py", line 764, in apply
    return self.apply_standard()
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/pandas/core/apply.py", line 891, in apply_standard
    results, res_index = self.apply_series_generator()
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/pandas/core/apply.py", line 907, in apply_series_generator
    results[i] = self.f(v)
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/pandas/core/apply.py", line 142, in f
    return func(x, *args, **kwargs)
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/stages/nim/table_extraction.py", line 78, in _update_metadata
    paddle_result = call_image_inference_model(paddle_client, "paddle", image_array, trace_info=trace_info)
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/tracing/tagging.py", line 185, in wrapper_inject_trace_info
    result = func(*args, **kwargs)
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 103, in call_image_inference_model
    response = _call_image_inference_grpc_client(client, model_name, image_data)
  File "/opt/conda/envs/nv_ingest/lib/python3.10/site-packages/nv_ingest/util/nim/helpers.py", line 124, in _call_image_inference_grpc_client
    raise RuntimeError(err_msg)
RuntimeError: Inference failed for model paddle: 'int' object has no attribute 'decode'

Other/Misc.

No response

@joaopcm1996 joaopcm1996 added the bug Something isn't working label Jan 23, 2025
@mupeifeiyi
Copy link

I have the same problem, have you solved it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants