Skip to content

Commit

Permalink
More fixes for OmniDoc, README updates
Browse files Browse the repository at this point in the history 
Signed-off-by: Christoph Auer <[email protected]>
  • Loading branch information
@cau-git
cau-git committed Jan 7, 2025
1 parent 7d111c1 commit e2776f3
Show file tree
Hide file tree
Showing 3 changed files with 45 additions and 44 deletions.
87 changes: 44 additions & 43 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -61,18 +61,19 @@ poetry run evaluate -t create -m layout -b DPBench -i <location-of-dpbench> -o .
poetry run evaluate -t evaluate -m layout -b DPBench -i ./benchmarks/dpbench-layout -o ./benchmarks/dpbench-layout
```

| id | label | MaP[0.5:0.95] |
| -- | -------------- | ------------- |
| 0 | page_header | 0.151 |
| 1 | text | 0.678 |
| 2 | section_header | 0.443 |
| 3 | footnote | 0.221 |
| 4 | picture | 0.761 |
| 5 | caption | 0.458 |
| 6 | page_footer | 0.344 |
| 7 | document_index | 0.755 |
| 8 | formula | 0.066 |
| 9 | table | 0.891 |
| label | Class mAP[0.5:0.95] |
|----------------|-----------------------|
| table | 89.08 |
| picture | 76.1 |
| document_index | 75.52 |
| text | 67.8 |
| caption | 45.8 |
| section_header | 44.26 |
| page_footer | 34.42 |
| list_item | 29.04 |
| footnote | 22.08 |
| page_header | 15.11 |
| formula | 6.62 |
</details>

<details>
Expand Down Expand Up @@ -132,17 +133,17 @@ poetry run evaluate -t evaluate -m layout -b OmniDocBench -i ./benchmarks/omnido
poetry run evaluate -t visualize -m tableformer -b OmniDocBench -i ./benchmarks/OmniDocBench-dataset/layout -o ./benchmarks/OmniDocBench-dataset/layout
```

| index | label | Class mAP[0.5:0.95] |
|---------|----------------|-----------------------|
| 00 | picture | 0.293 |
| 01 | section_header | 0.131 |
| 02 | text | 0.24 |
| 03 | footnote | 0.025 |
| 04 | caption | 0.107 |
| 05 | table | 0.693 |
| 06 | page_header | 0.1 |
| 07 | page_footer | 0.161 |
| 08 | formula | 0.038 |
| label | Class mAP[0.5:0.95] |
|----------------|-----------------------|
| table | 69.32 |
| picture | 29.29 |
| text | 23.99 |
| page_footer | 16.14 |
| section_header | 13.09 |
| caption | 10.74 |
| page_header | 10.02 |
| formula | 3.83 |
| footnote | 2.48 |
</details>

<details>
Expand Down Expand Up @@ -221,26 +222,26 @@ The final result (struct only here) can be visualised as,

| x0<=TEDS | TEDS<=x1 | prob [%] | acc [%] | 1-acc [%] | total |
|------------|------------|------------|-----------|-------------|---------|
| 0 | 0.05 | 0 | 0 | 100 | 0 |
| 0.05 | 0.1 | 0.01 | 0 | 100 | 1 |
| 0.1 | 0.15 | 0.01 | 0.01 | 99.99 | 1 |
| 0.15 | 0.2 | 0.02 | 0.02 | 99.98 | 2 |
| 0.2 | 0.25 | 0 | 0.04 | 99.96 | 0 |
| 0.25 | 0.3 | 0 | 0.04 | 99.96 | 0 |
| 0.3 | 0.35 | 0 | 0.04 | 99.96 | 0 |
| 0.35 | 0.4 | 0 | 0.04 | 99.96 | 0 |
| 0.4 | 0.45 | 0.02 | 0.04 | 99.96 | 2 |
| 0.45 | 0.5 | 0.1 | 0.06 | 99.94 | 10 |
| 0.5 | 0.55 | 0.1 | 0.15 | 99.85 | 10 |
| 0.55 | 0.6 | 0.24 | 0.25 | 99.75 | 25 |
| 0.6 | 0.65 | 0.47 | 0.49 | 99.51 | 49 |
| 0.65 | 0.7 | 1.04 | 0.96 | 99.04 | 108 |
| 0.7 | 0.75 | 2.44 | 2 | 98 | 254 |
| 0.75 | 0.8 | 4.65 | 4.44 | 95.56 | 483 |
| 0.8 | 0.85 | 13.71 | 9.09 | 90.91 | 1425 |
| 0.85 | 0.9 | 21.2 | 22.8 | 77.2 | 2204 |
| 0.9 | 0.95 | 28.48 | 43.99 | 56.01 | 2961 |
| 0.95 | 1 | 27.53 | 72.47 | 27.53 | 2862 |
| 0 | 0.05 | 0 | 0 | 100 | 0 |
| 0.05 | 0.1 | 0 | 0 | 100 | 0 |
| 0.1 | 0.15 | 0 | 0 | 100 | 0 |
| 0.15 | 0.2 | 0.2 | 0 | 100 | 2 |
| 0.2 | 0.25 | 0 | 0.2 | 99.8 | 0 |
| 0.25 | 0.3 | 0 | 0.2 | 99.8 | 0 |
| 0.3 | 0.35 | 0 | 0.2 | 99.8 | 0 |
| 0.35 | 0.4 | 0 | 0.2 | 99.8 | 0 |
| 0.4 | 0.45 | 0 | 0.2 | 99.8 | 0 |
| 0.45 | 0.5 | 0 | 0.2 | 99.8 | 0 |
| 0.5 | 0.55 | 0.3 | 0.2 | 99.8 | 3 |
| 0.55 | 0.6 | 0.5 | 0.5 | 99.5 | 5 |
| 0.6 | 0.65 | 0.7 | 1 | 99 | 7 |
| 0.65 | 0.7 | 0.6 | 1.7 | 98.3 | 6 |
| 0.7 | 0.75 | 1.5 | 2.3 | 97.7 | 15 |
| 0.75 | 0.8 | 3.3 | 3.8 | 96.2 | 33 |
| 0.8 | 0.85 | 15.3 | 7.1 | 92.9 | 153 |
| 0.85 | 0.9 | 19 | 22.4 | 77.6 | 190 |
| 0.9 | 0.95 | 30.7 | 41.4 | 58.6 | 307 |
| 0.95 | 1 | 27.9 | 72.1 | 27.9 | 279 |
</details>

### Pub1M
Expand Down
2 changes: 1 addition & 1 deletion docling_eval/benchmarks/omnidocbench/create.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -387,7 +387,7 @@ def create_omnidocbench_tableformer_dataset(
):

jpg_path = page_tuple[0]
pdf_path = page_tuple[1]
pdf_path = Path(page_tuple[1])

# logging.info(f"file: {pdf_path}")
if not os.path.basename(jpg_path) in gt:
Expand Down
Binary file modified docs/evaluations/evaluation_DPBench_tableformer.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit e2776f3

Please sign in to comment.