Convert Hyrax Results to a Pandas DataFrame#
After running inference with Hyrax, results are saved in the Lance format — a columnar format optimized for ML workloads.
This notebook shows two ways to load those results into a familiar Pandas DataFrame:
LanceDB — provides a SQL-like query interface for easy filtering and selection.
PyLance — provides direct, low-level access to the data on disk.
Note: Neither approach requires Hyrax to be installed. Only
lancedborlanceis needed.
The example below uses results produced by the Getting Started notebook.
Setup#
Set result_directory to the path of your Hyrax output directory. Here we use the location of the saved predictions from the Getting Started notebook.
[1]:
from pathlib import Path
result_directory = Path("./example_results/getting_started_results")
Option 1: LanceDB#
Connect to the results directory using lancedb, open the "results" table, and convert it to a Pandas DataFrame.
[2]:
import lancedb
lance_dir = result_directory / "lance_db"
db = lancedb.connect(str(lance_dir))
table = db.open_table("results")
df = table.to_pandas()
df.head()
[2]:
| object_id | data | |
|---|---|---|
| 0 | 00000 | [0.096435, -2.635337, 1.734471, 2.033914, 0.13... |
| 1 | 00001 | [4.793885, 6.458918, 0.205107, -2.394825, -1.5... |
| 2 | 00002 | [2.774884, 3.678134, 0.251015, -0.957243, -1.8... |
| 3 | 00003 | [3.894425, 1.825525, 0.857038, -1.040612, -0.5... |
| 4 | 00004 | [-2.83718, -2.558729, 2.639043, 2.211744, 3.59... |
Option 2: Lance#
The lance library gives you direct access to the dataset file on disk. Note that the path points to the specific .lance dataset file inside lance_db/, rather than the lance_db/ directory itself.
[3]:
import lance
lance_dir = result_directory / "lance_db" / "results.lance"
ds = lance.dataset(lance_dir)
table = ds.to_table()
df = table.to_pandas()
df.head()
[3]:
| object_id | data | |
|---|---|---|
| 0 | 00000 | [0.096435, -2.635337, 1.734471, 2.033914, 0.13... |
| 1 | 00001 | [4.793885, 6.458918, 0.205107, -2.394825, -1.5... |
| 2 | 00002 | [2.774884, 3.678134, 0.251015, -0.957243, -1.8... |
| 3 | 00003 | [3.894425, 1.825525, 0.857038, -1.040612, -0.5... |
| 4 | 00004 | [-2.83718, -2.558729, 2.639043, 2.211744, 3.59... |
Both approaches produce a standard Pandas DataFrame, making it easy to share results and explore them without any dependency on Hyrax — only lancedb or lance is required.