Skip to content

The Other Half of Workflow Portability: Evidence-Backed HPC Site Profiles with Agentic Discovery

Oct 2026 · 0 citations · 27 references
Computer Science

TL;DR

The HPC site profile is proposed, a structured, evidence-backed document that preflights a workflow into an execution plan or an early, explainable failure and is presented as a case study of preflighting a real workflow.

Abstract

Moving a workflow developed and tested at one HPC site to another rarely succeeds without some amount of trial and error. Package managers rebuild software environments, containers ship whole filesystems, and workflow specifications such as backpacks package a workflow with its software, data, and resource requirements. These approaches address one half of workflow portability: what a workflow needs. But none describes how a given HPC site must be used, and that missing half is why even a portable workflow requires manual adjustment at each new site. That gap includes the site's resource shape, storage configuration, network permissions, and operating policies. This information may be explicit in the batch system, hidden in the prose of documentation, or buried deep within a router's configuration, making it difficult for an automated deployment tool to turn site knowledge into useful deployment decisions. We propose the HPC site profile, a structured, evidence-backed document that makes this knowledge actionable. We automatically construct it in three steps that mirror where the information lives: measuring the login node, extracting typed fields from documentation with a bounded language-model agent, and submitting pilot jobs for eligible unresolved fields. Every field is verified against its evidence or discarded, so a rule, not the model, decides what enters the profile. The profile then preflights a workflow into an execution plan or an early, explainable failure. We build profiles at Purdue Anvil, TACC Stampede3, and Notre Dame CRC and present a case study of preflighting a real workflow.

View source

Similar papers

Open access Sep 2026

Serverless Data Engineering: Innovations in Python-Driven ETL Automation on AWS

The central argument is that serverless ETL does not simplify data engineering; it redistributes engineering effort from infrastructure management to execution model design, and Lambda timeout decomposition, cold start management, Glue worker right-sizing, IAM role scoping, and Step Functions state machine cost design...

Rambabu Bolineni · 0 citations
Open access Aug 2026

HBA: A Heuristic Batch Advisor for Self-Tuning Bulk Data Ingestion in the OutSystems Low-Code Platform

Low code development platforms like OutSystems allow teams to build data-centric applications using visual models that the platform takes care of the database plumbing [1] [2]. We have demonstrated in an earlier study that the write path is row-wise and fails at scale; and in a previous study we introduced EAVS [25], a...

Anamika Garg, Sachin H. Patel · 0 citations
Preprint Aug 2026

Metis: Typed Runtime Mediation for Tool-Using Software Agents

Metis, a multi-provider runtime that converts provider streams into typed events before admitted calls reach external effects is presented, a multi-provider runtime that converts provider streams into typed events before admitted calls reach external effects.

Jun'an Yu · 0 citations
Aug 2026

Automating data reduction and archival workflows using Apache NiFi

Traditional data reduction pipelines are run on local or remote machines and depend on user interaction for execution. This approach can lead to reproducibility and discoverability issues, as processing decisions made by the user go undocumented and are made in a single instance leading to difficulty in reproduction. O...

James Coulson, Nuria P. F. Lorente, Brent Miszalski et al. · 0 citations

Related blog posts

Microsoft Research Blog Nov 12, 2026

Microsoft Research Focus – Edition #98

What's new in research at Microsoft? Host Anson Ho explores the latest, including AI-powered universal translation, a breakthrough in a 31-year-old mathematics challenge, and advances making AI coding agents faster and more efficient.  The post Microsoft Research Focus – Edition #98 appeared first on Microsoft Research.

Microsoft Research Blog Nov 5, 2026

Formally verifying agent-written full-stack web apps with PDD Kit

AI agents can generate code at a scale that’s hard for humans to review line-by-line. PDD Kit helps ensure the code is doing what the developer intended by creating formal specifications and testing against them. It converts developer intent into a Verus specification, proves AI-generated code meets it for every input, and uses tests where proofs aren't practical. Developers review intent, not every line. The post Formally verifying agent-written full-stack web apps with PDD Kit appeared first o…

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.