Run Ray Data Pipelines from pipelines-components

The next Alauda AI Platform release packages reusable components from the mirrored pipelines-components project for Kubeflow Pipelines 2.16.1. AutoGluon pipelines can be preloaded by the operator. The Ray Data group is experimental and is not preloaded. Compile it the same way as any other custom asset: clone the mirror Alauda supplies, install kfp-components editable, and upload the YAML. See Use Reusable Kubeflow Pipeline Components.

Ray Data itself never appears as a KFP task. The parse_and_chunk component submits a KubeRay RayJob. The job builds a ray.data dataset of PDF paths, runs Docling inside an actor pool, and writes one JSONL object per file to S3-compatible storage.

PDFs on an RWX PVC
        │  parse_and_chunk  (RayJob + Ray Data actor pool)
        ▼
JSONL chunks in S3  (source_file, chunk_index, text)
        │  ingest_to_milvus  (optional embedding InferenceService)
        ▼
Milvus collection
        │  download_model + model_deployment  (parallel with ingest)
        ▼
LLM InferenceService

A compile-and-upload notebook lives next to this page:

For a few dozen PDFs, convert them in a Workbench instead: Prepare Fine-Tuning Data with Docling and SDG Hub. For RAG search over chunking and retrieval settings, use Optimize a RAG Application with AutoRAG Pipelines.

Prerequisites

RequirementDetails
Kubeflow Pipelines 2.16.1Alauda kfp-operator v26.3.1 or DSPO v2.15.2. See Use Reusable Kubeflow Pipeline Components.
KubeRay OperatorCluster plugin. See Install KubeRay Operator. The pipeline ServiceAccount must be able to create rayjobs.ray.io in the run namespace.
WorkbenchStandard Data Science (Python 3.12) so you can compile with kfp==2.16.1. See Create Workbench.
RWX PVCHolds the input PDFs. parse_and_chunk mounts it read-only on the Ray head and every worker.
S3-compatible bucketIntermediate chunk JSONL. Create a Secret with keys access_key and secret_key (not AWS_ACCESS_KEY_ID).
Ray + Docling imageray_image must already contain Ray and Docling. Mirror it into the cluster registry.
Optional, for the full RAG DAGA reachable Milvus 2.4+ service, a Hugging Face token Secret (token), a model-cache PVC, and a Hardware Profile in the namespace you pass as llm_hardware_profile_namespace.

The component runtime image defaults to an ODH CPU datascience image. Replace it with the Alauda-mirrored runtime for your cluster, or set RELATED_IMAGE_RAG_BASE_RUNTIME when you compile.

Kueue: leave bypass_kueue=false (the default) when the namespace has a Local Queue. Set bypass_kueue=true only on clusters without Kueue.

Prepare input storage

Create the PDF PVC and the chunk-store Secret in the pipeline namespace:

NS=my-pipeline-namespace

kubectl -n "$NS" apply -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 50Gi
EOF

kubectl -n "$NS" create secret generic minio-secret \
  --from-literal=access_key='<access-key>' \
  --from-literal=secret_key='<secret-key>'

Copy PDFs under input/pdfs/ on that PVC (the default input_path). A Workbench that mounts data-pvc can upload files from JupyterLab. Nested directories are included; num_files=0 processes every *.pdf.

Compile from a Workbench

The kfp-components package is not published to PyPI. Clone the Alauda mirror and install it in a Python 3.11 or later virtual environment. Pin the SDK to the version shipped by Alauda AI Platform. Replace <pipelines-components-repo-url> with the repository address supplied by Alauda for the environment:

git clone <pipelines-components-repo-url>
cd pipelines-components

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install 'kfp==2.16.1' 'kfp-kubernetes==2.16.1'
python -m pip install --no-deps -e .

Compile only parse_and_chunk when you need JSONL chunks and will embed them yourself:

from kfp import compiler
from kfp_components.components.data_processing.parse_and_chunk import parse_and_chunk

compiler.Compiler().compile(
    parse_and_chunk,
    package_path="parse_and_chunk.yaml",
)

Compile the full RAG DAG when you also want Milvus ingest and an LLM InferenceService. Override ray_image and namespace at run time, not in this compile step:

from kfp import compiler
from kfp_components.pipelines.data_processing.ray_data.pdf_documents_processing_rag_pipeline import (
    rag_multistep_pipeline,
)

compiler.Compiler().compile(
    rag_multistep_pipeline,
    package_path="rag_multistep_pipeline.yaml",
)

Upload the YAML from Kubeflow Pipelines > Pipelines > Upload pipeline, or submit it with the KFP SDK from the same Workbench.

Run parse-and-chunk only

Create a run of parse_and_chunk and set:

InputExampleNotes
pvc_namedata-pvcRWX claim with the PDFs
pvc_mount_path/mnt/dataMust match the RayJob volume mount
input_pathinput/pdfsRelative to the mount
ray_image<registry>/mlops/docling-ray:<tag>Ray + Docling, already in the cluster
namespacemy-pipeline-namespaceSame namespace as the run, PVC, and Secret
s3_endpointhttp://seaweedfs.mlops-demo-e2e:8333Include the scheme
s3_bucketrag-chunksCreated on first write if missing
s3_prefixchunksOne <stem>.jsonl per PDF
s3_secret_nameminio-secretKeys: access_key, secret_key
num_workers2Ray worker pods; actors must fit on these pods
cpus_per_actor4Docling worker CPUs; keep OCR off (the job sets do_ocr=False)
min_actors / max_actors2 / 4Ray Data ActorPoolStrategy
batch_size4PDFs per map_batches call
num_files00 means all PDFs
bypass_kueuefalsetrue only when the namespace has no Local Queue

The component waits until the Ray cluster reports readyWorkerReplicas == num_workers, then until the RayJob jobStatus is SUCCEEDED (up to four hours). Output is the URI s3://<bucket>/<prefix>.

Each JSONL line is:

{"source_file": "policy.pdf", "chunk_index": 0, "text": "..."}

tokenizer should match the embedding model you will use later. The default is sentence-transformers/all-MiniLM-L6-v2. For the full RAG pipeline, pass the same string as embedding_model.

Do not schedule Docling actors on the Ray head. Size cpus_per_actor × max_actors so the pool fits on the worker group; a hung converter is killed inside a subprocess after timeout_seconds.

Run the full RAG pipeline

The compiled rag_multistep_pipeline has two independent chains:

  1. parse_and_chunk → ingest_to_milvus (optional deploy_embedding_model first)
  2. download_model → model_deployment

Leave deploy_embedding=false and embedding_endpoint="" to embed with a local sentence-transformers model inside the ingest task. Set deploy_embedding=true only when you want a dedicated embedding InferenceService.

Additional run inputs for the model chain:

InputExampleNotes
hf_secret_namehf-token-secretKey must be token
llm_model_namea model id your cluster is allowed to pullCached on model_cache_pvc
model_cache_pvcmodel-cache-pvcCreated before the run
llm_hardware_profile_namegpu-profileMust exist
llm_hardware_profile_namespacethe namespace that holds the Hardware ProfileUse the Hardware Profile namespace on this cluster
milvus_hostmilvus.milvus.svc.cluster.localIn-cluster DNS, no scheme
drop_existingtrueRecreates the collection

ingest_to_milvus reads S3_ACCESS_KEY / S3_SECRET_KEY from the same minio-secret. The LLM deploy step does not wait for ingest; both chains can finish independently.

What Ray Data does inside the job

The entrypoint that parse_and_chunk injects into the RayJob is the Ray Data pipeline:

ds = ray.data.from_pandas(pd.DataFrame({"path": pdf_paths}))
ds = ds.repartition(max_actors * 10)
results = ds.map_batches(
    DoclingChunkProcessor,
    compute=ray.data.ActorPoolStrategy(min_size=min_actors, max_size=max_actors),
    batch_size=batch_size,
    batch_format="numpy",
    num_cpus=cpus_per_actor,
)

DoclingChunkProcessor is a Ray actor. Each actor owns a subprocess that runs DocumentConverter + HybridChunker and uploads JSONL. Timeouts restart that subprocess so one stuck PDF does not kill the actor. This is the same pattern the Workbench Docling guide points at for thousands of files; you do not write this script yourself when you use the component.

A CPU-only smoke of that actor-pool shape (no Docling, no PDF PVC) is assets/ray-data-pipelines/rayjob-smoke.yaml.

Troubleshooting

SymptomCause
RayJob stays Pending / SuspendedKueue Local Queue missing, or bypass_kueue still false.
Ray cluster did not become readyWorker CPU/memory requests exceed the namespace quota, or ray_image cannot be pulled.
No PDFs foundFiles are not under <pvc_mount_path>/<input_path>/**/*.pdf.
S3 403 / missing credentialsSecret keys must be access_key and secret_key. AutoGluon loaders use different key names.
Workbench OOM if you convert locallyMove the job here; one Docling actor wants several GB and multiple CPUs.
model_deployment cannot bind GPUsllm_hardware_profile_name / namespace do not match a live Hardware Profile.
Ingest fails with empty collectionparse_and_chunk wrote no objects; check the RayJob log for timeout / empty rows.

Next steps