Skip to main content
The Conformly analysis & evaluation pipeline is proprietary and not publicly released. This reference provides the software requirements, hardware compatibility, and performance characteristics needed to evaluate its suitability for a high-performance computing allocation, without disclosing source. Reference target: NVIDIA H100 accelerated partition (e.g. MareNostrum 5, BSC).

​
Overview

Conformly evaluates automotive engineering documents against compliance standards (ASPICE, ISO 26262, ISO/SAE 21434). The pipeline runs in stages: document parsing (Docling plus a vision pass for figures), retrieval of standard/clause context, and a LangGraph-orchestrated, per-process LLM evaluation that produces structured, evidence-anchored findings. For an HPC allocation the pipeline is reconfigured to call open-weight models served on the allocation in place of external APIs — a single provider-interface change. The workload is batch and episodic — open-weight model benchmark sweeps and LoRA fine-tune runs scored against a frozen evaluation corpus. It is not a latency-sensitive online service and maps cleanly to Slurm batch scheduling.

​
Software requirements

A single Apptainer image carries the full stack; SBATCH scripts request GPUs and launch container jobs. No compilation on the login node.

​
Hardware compatibility

Fallback to an A100 64 GB system is supported, with the FP8 path replaced by bf16/INT8 (more GPU memory / GPUs for the 70B case).

​
Performance characteristics

  • Profile. GPU-bound, batch-shaped: inference sweeps and fine-tune steps. Per-process compliance evaluation (8 ASPICE/ISO processes per document) is embarrassingly parallel across processes and documents.
  • Inference. Served through vLLM with continuous batching; large evaluators use tensor parallelism within a node. GPU utilisation is high during sweeps.
  • Fine-tuning. LoRA (PEFT) updates a small adapter rather than the full model, so memory and compute per run are modest relative to full fine-tuning, and it scales data-parallel across GPUs.
  • Scaling. Strong scaling within a node for tensor-parallel serving; near-linear data-parallel scaling for LoRA replicas. No multi-node communication at this scale.
  • Indicative resource use (to be confirmed empirically — measuring these is a project objective): benchmark sweep ≈ tens of GPU-hours; a 7–8B LoRA iteration ≈ low-hundreds of GPU-hours including hyper-parameter variation; ~70B FP8 inference benchmark ≈ hundreds of GPU-hours. H100 throughput means each GPU-hour does more work than on A100.
  • Determinism / evaluation. Quality is measured with a frozen, labelled corpus and a reproducible detection-metric harness (precision/recall/F1), with low-temperature decoding and inter-run stability tracked, so runs are comparable and the allocation’s output is auditable.

​
Execution model

  1. Build the Apptainer image once from the pinned definition file.
  2. Stage open-weight model weights from Hugging Face Hub to project storage.
  3. Submit SBATCH jobs requesting up to one accelerated node (4× H100): benchmark-sweep jobs and LoRA fine-tune jobs.
  4. Score outputs against the frozen evaluation corpus inside the same container.
  5. Push the resulting fine-tuned adapter/model artifact (our IP, not customer data) to a private Hugging Face Hub repository.
No customer data is processed on the allocation — the corpus is synthetic and self-authored.