arXiv.itsnick.co

an independent field guide to 23 million papers from arXiv & OpenAlex — not affiliated with arXiv.org

Author

Tyler Procko

14 papers · 2022–2025

2022: 12023: 62024: 62025: 1
2022peak 6 in 20232025

Frequent co-authors

Papers

showing 1–14 of 14 RSS

Graph Retrieval-Augmented Generation for Large Language Models: A Survey

Tyler Procko, Omar Ochoa

Artificial Intelligence · · cited 50× · DOI ↗ · more like this ·

Large Language Models (LLMs) demonstrate general knowledge, but they suffer when specifically needed knowledge is not present in their training set. Two approaches to ameliorating this, without retraining, are 1) prompt engineering and 2) Retrieval-Augmented Generation (RAG). RAG is a form of prompt engineering, insofar as relevant lexical snippets retrieved from RAG corpora are vectorized and aggregated with prompts. However, RAG documents are often noisy, i.e., while relevant to a given prompt, they can contain much other information that obfuscates the desired snippet. If the purpose of pretraining a LLM on massive and general corpora is to engender a generally applicable model, RAG is not: it is a means of LLM optimization, and as such, RAG document selection must be precise, not general. For expert tasks, it is imperative that a RAG corpus be as noise-free as possible, in much the…

Scrum in the Classroom: An Implementation Guide

Sarah Reynolds, Alexis Caldwell, Tyler Procko +1 more

Information Systems · Scholarly Commons (Embry–Riddle Aeronautical University) · · cited 6× · DOI ↗ · more like this ·

Over the years, Agile approaches have been proven successful in industry settings, as documented in the literature. In response to this success, education professionals have developed ways to introduce Agile practices into engineering classrooms with similar success. These practices have been most popular in project-based courses because they enhance student learning and prepare students for using Agile practices in industry after graduation. The Scrum approach is one of the most popular Agile methods in industry and classroom adoption. Modified versions of Scrum are utilized within the classroom to align with student needs, familiarity with Scrum, and the materials presented within the class. As a result of this adaptation, many different Scrum-based implementations are found in classrooms. The popularity of this approach has led to numerous publications detailing individual…

An Exploration of Explainable Machine Learning Using Semantic Web Technology

Tyler Procko, Timothy Elvira, Omar Ochoa +1 more

Artificial Intelligence · · cited 6× · DOI ↗ · more like this ·

The behavior of a Machine Learning (ML) algorithm is generally accepted to be a black box, i.e., it cannot be opened and understood. This paper reports on an effort to provide explanation to ML algorithms by using semantic background knowledge. A preliminary paper was found as a project seed, its experiment of ML explanation using the DL-Learner tool recreated and semi-automated. DL-Learner is a framework for supervised ML using background knowledge. DL-Learner induces class relationships that hold true for a positive example set. The work presented in this paper is a novel, semi-automated framework for testing the use of DL-Learner in ML explanation. A scene classifier pipeline was created to obtain test data. For the chosen dataset input to the ML, 32 trials were conducted, and explanations produced. Furthermore, this paper reports on the use of DL-Learner as a tool and the lessons…

Leveraging Large Language Models on the Traditional Scientific Writing Workflow

Tyler Procko, Alexandra Davidoff, Timothy Elvira +1 more

Artificial Intelligence · · cited 3× · DOI ↗ · more like this ·

Technological advances in Natural Language Processing have brought forth language models capable of advanced response delivery. Scientific papers are traditionally written manually by human researchers, but with the advent of mainstream Large Language Models, e.g., OpenAI's ChatGPT, it is of increasing concern to scientists and academics that content in scientific papers may be generated by Artificial Intelligence (AI). Wishing to stop this is a losing attitude, as large-scale generative AI only becomes more powerful and accessible. Taking the more tenable position of cautious adaptation, this paper argues that there exists a taxonomy in the structure of scientific papers, and that language models can be used by scientific researchers to bootstrap scientific writing. Furthermore, AI can augment their own writing workflow to more efficiently traverse the academic publishing pipeline.…

GPT-4: A Stochastic Parrot or Ontological Craftsman? Discovering Implicit Knowledge Structures in Large Language Models

Tyler Procko, Timothy Elvira, Omar Ochoa

Artificial Intelligence · · cited 3× · DOI ↗ · more like this ·

Ontologies are representational artifacts that purport to accurately portray the aspect of reality under the purview of the ontologists laboring upon them. Ontologies exist in a spectrum of formality, from lexical thesauri to knowledge graphs, to collections of statements of first-order logic. The recent proliferation of Large Language Models (LLMs) has brought to bear interactive “knowledge bases” with general awareness of most things. As ontologists create ontologies from their understanding of reality; and as LLMs, presumably, possess some “understanding” of reality, embedded in their vector matrices corresponding to lexical terms from massive quantities of learned texts, a question is posed: what form of ontology can an LLM create when prompted about some novel facet of reality, without explicitly asking it for an ontology? I.e., will an LLM categorize things into bins, or a…

Leveraging Linked Data for Knowledge Management: A Proposal for the Aerospace Industry

Tyler Procko, Omar Ochoa

Artificial Intelligence · Journal of Aerospace Information Systems · · cited 2× · DOI ↗ · more like this ·

Within a given subculture or industry, knowledge proliferates insofar as those outside the sphere of knowledge lack an understanding of the terms, concepts, and ideas specific to the domain. Moreover, collaborative work within large industries suffers from a lack of understanding between collaborators and from the use of nonstructured legacy knowledge artifacts. The aerospace and defense industries are no exception; indeed, they present examples of some of the largest industries with extensive projects and a diverse array of applications. This paper presents a state-of-the-art literature review of knowledge management and knowledge collaboration systems in the aerospace and defense industries. Based on the review findings, a proposal for an extensible and queryable knowledge management system for aerospace industry workspaces using the Web Ontology Language of the Semantic Web is…

Automatic Generation of BFO-Compliant Aristotelian Definitions in OWL Ontologies with GPT

Tyler Procko, Omar Ochoa, Timothy Elvira

Artificial Intelligence · · cited 2× · DOI ↗ · more like this ·

Ontologies are representational artifacts that purport to accurately describe some aspect of reality, including the entities and the relations that hold between them. In computer science, ontologies are software artifacts containing the schematic structure for machine-readable knowledge, typically formed as a graph of subject-predicate-object triples, constrained through Description Logics. These resources and their relations are self-defining, i.e., some resource may be defined by considering all its stated relations. Resources are often attended with natural language annotations, that humans may read and interpret, such as labels and definitions. Many long-standing ontologies have useless lexical definitions that define resources cyclically, e.g., a FOAF: Person is simply defined as “A person”. In Aristotelian terms, the definition of a thing should be reducible, by using terms…

Impacts of Catastrophic Forgetting: From a Machine Learning Engineering Perspective

Timothy Elvira, Juan Ortiz Couder, Tyler Procko +1 more

Artificial Intelligence · · cited 2× · DOI ↗ · more like this ·

Software validation is the method of ensuring that a software system is built correctly, meets user needs, and functions according to defined requirements. In the current age, Machine Learning (ML) and Deep Learning (DL) are at the forefront of technological innovation. Software engineering practices are being adopted in the ML/DL domain to assist in defining and mapping software validation anomalies, e.g., catastrophic forgetting. A ML model can be engineered, trained, and used to fulfill a requirement specification based on its classification capabilities. Catastrophic forgetting can cause a model to forget a class fully or partially insofar as a requirement is no longer fulfilled. This paper posits that catastrophic forgetting is an issue of ML validation and presents the issues of catastrophic forgetting in the novel domain of ML engineering. Ideally, this paper will spark a…

Digital Rubber Duck: Leveraging Large Language Models for Extreme Programming

Timothy Elvira, Tyler Procko, Juan Ortiz Couder +1 more

Hardware and Architecture · · cited 2× · DOI ↗ · more like this ·

The recent prevalence of Large Language models (LLMs), e.g., GPT-3.5 and GPT-4, has brought about a new age of man-computer symbiosis, where LLMs are employed for a litany of creative, constructive, scientific, or otherwise content-generative tasks, e.g., as general chatbot assistants, writing editors, digital subject matter experts, programming consultants, and so on. Of interest to software engineers is the concept of “rubber duck debugging”, which is the act of expressing code, line-by-line, in natural language, to an inanimate object, e.g., a rubber duck, for the purpose of elucidating potential issues that can then be corrected. In this paper, we detail a workflow process that leverages the concept of rubber duck debugging, replacing the duck with a capable LLM, e.g., GPT-4. We call it Digital Rubber Duck Programming. Furthermore, the Extreme Programming (XP) method, an…

Exploring Testing Methods for Large Language Models

Timothy Elvira, Tyler Procko, Lynn Vonderhaar +1 more

Artificial Intelligence · · cited 1× · DOI ↗ · more like this ·

Large Language Models (LLMs) are extensive aggregations of human language, designed to understand and generate sophisticated text. LLMs are becoming ubiquitous in a range of applications, from social media to code generation. With their immense size, LLMs face scalability challenges, making testing methods particularly difficult to implement effectively. Traditional machine learning and software testing methods, derived and adapted for LLMs, test these models to a point; however, they still struggle to accurately capture the full complexity of model behavior. This paper aims to capture the current efforts and techniques in testing LLMs, specifically focusing on stress testing, mutation testing, regression testing, metamorphic testing, and adversarial testing. This survey focuses on how traditional testing methods must be adapted to fit the needs of LLMs. Furthermore, while this area is…

Semantic Science: Publication Beyond the PDF

Tyler Procko, Omar Ochoa

Information Systems · · cited 1× · DOI ↗ · more like this ·

Scientific papers have evolved since the establishment of the Royal Society in the mid-17<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">th</sup> century. Today, despite the interconnectedness of Internet discourse, scientific papers are released as relatively static, inflexible artifacts, to wit, as Portable Document Format (PDF) files. This fact is an egregious slight to the base principle of science: that it is to be mutable and evolutionary. Thus, the present paper proposes a means of writing scientific papers that allows for change and explicit connection to other papers, through the use of (1) Linked Data knowledge graphs, (2) the Findable-Accessible-Interoperable-Reusable (FAIR) data principles, (3) the Markdown language, and (4) OpenAI“s Large Language Model, the Generative Pre-trained Transformer, GPT -4. A handful of extant tools…

Validating Security Requirement Specifications through the use of a Knowledge Graph

Lynn Vonder Haar, Sarah Reynolds, Tyler Procko +1 more

Information Systems · · cited 1× · DOI ↗ · more like this ·

Turning a customer's product vision into implementable requirement specifications is notoriously difficult and time-consuming. Beyond that, security is an aspect of requirement specifications that is frequently underdeveloped. If developers had access to an automated system that could validate requirement specifications based on known loopholes and external expertise, the likelihood of injecting a security vulnerability into a software product would decrease. As depicted in this paper, the automated system takes a Software Requirement Specification (SRS) document as input. The system then potentially validates that all related security topics and vulnerabilities are covered within the security requirement specifications. Entity linking is used to parse the SRS to form queries for a security concepts and vulnerabilities knowledge graph. The query result can provide developers with areas…

Creating Robust Data Sets for AI by Leveraging Ontological Structures

Lynn Vonderhaar, Timothy Elvira, Tyler Procko +1 more

Artificial Intelligence · · cited 0× · DOI ↗ · more like this ·

Machine Learning (ML) models are essential across numerous fields, including those where safety is paramount, e.g., autonomous driving, which is the primary focus of this paper. In many areas, the opaque nature of ML systems may be a minor issue, but in safety-critical contexts, it creates challenges in establishing trust. To fully harness ML models in these high-stakes fields, a method is needed to enhance confidence in their reliability and accuracy without the need for human oversight on every decision. This research introduces a technique aimed at boosting trust in ML models by improving the robustness and thoroughness of the training datasets. Since ML models are shaped by the data they are trained on, ensuring the dataset's completeness can potentially enhance confidence in the model's performance. To achieve this, the paper suggests leveraging both a domain-specific ontology and…

Towards Robust Training Datasets for Machine Learning with Ontologies: A Case Study for Emergency Road Vehicle Detection

Lynn Vonderhaar, Timothy Elvira, Tyler Procko +1 more

cs.AI · · cited 0× · arXiv ↗ · more like this ·

Countless domains rely on Machine Learning (ML) models, including safety-critical domains, such as autonomous driving, which this paper focuses on. While the black box nature of ML is simply a nuisance in some domains, in safety-critical domains, this makes ML models difficult to trust. To fully utilize ML models in safety-critical domains, it would be beneficial to have a method to improve trust in model robustness and accuracy without human experts checking each decision. This research proposes a method to increase trust in ML models used in safety-critical domains by ensuring the robustness and completeness of the model's training dataset. Because ML models embody what they are trained with, ensuring the completeness of training datasets can help to increase the trust in the training of ML models. To this end, this paper proposes the use of a domain ontology and an image quality…