http://www.sei.cmu.edu/feeds/topic/Updates on changes and additions to the SEI Blog for posts matching Artificial Intelligence Engineeringen-usWed, 06 May 2026 00:00:00 -0400https://www.sei.cmu.edu/blog/the-elm-library-an-llm-evaluation-toolset/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesTo help teams meet the need for rigorous evaluation methods, researchers in SEI’s AI Division developed a library built on best practices for LLM evaluation and benchmarking.Violet Turri, Natalie Schieber, Charles Loughin, Tyler BrooksWed, 06 May 2026 00:00:00 -0400https://www.sei.cmu.edu/blog/the-elm-library-an-llm-evaluation-toolset/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updateshttps://www.sei.cmu.edu/blog/from-reality-to-virtual-reality-the-impact-of-3dgs-on-training-education-and-beyond/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesThis blog post describes cutting-edge method for creating digital models of the physical world called 3D Gaussian Splatting.Roxxanne White, Matt Walsh, Dominic Ross, Richard LaughlinWed, 25 Mar 2026 00:00:00 -0400https://www.sei.cmu.edu/blog/from-reality-to-virtual-reality-the-impact-of-3dgs-on-training-education-and-beyond/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updateshttps://www.sei.cmu.edu/blog/my-ai-system-worksbut-is-it-safe-to-use/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesThis blog post introduce System Theoretic Process Analysis (STPA), a hazard analysis technique uniquely suitable for dealing with the complexity of AI systems.David Schulker, Matt Walsh, Emil MathewTue, 09 Sep 2025 00:00:00 -0400https://www.sei.cmu.edu/blog/my-ai-system-worksbut-is-it-safe-to-use/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updateshttps://www.sei.cmu.edu/blog/artificial-intelligence-in-national-security-acquisition-and-integration/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesThis blog post highlights practitioner insights from a recent AI Acquisition workshop, including challenges in differentiating AI systems, guidance on when to use AI, and matching AI tools to mission needs.Paige Rishel, Carol Smith, Brigid O'Hearn, Rita CreelTue, 05 Aug 2025 00:00:00 -0400https://www.sei.cmu.edu/blog/artificial-intelligence-in-national-security-acquisition-and-integration/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updateshttps://www.sei.cmu.edu/blog/amplifying-ai-readiness-in-the-dod-workforce/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesThe SEI recently partnered with the Department of the Air Force Chief Data and AI Office to develop a strategy to identify and assess hidden workforce talent for data and AI work roles.Eric Keylor, Robert Beveridge, Jonathan FrederickMon, 23 Jun 2025 00:00:00 -0400https://www.sei.cmu.edu/blog/amplifying-ai-readiness-in-the-dod-workforce/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updateshttps://www.sei.cmu.edu/blog/out-of-distribution-detection-knowing-when-ai-doesnt-know/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesHow do we know when an AI system is operating outside its intended knowledge boundaries?Eric Heim, Cole FrankMon, 09 Jun 2025 00:00:00 -0400https://www.sei.cmu.edu/blog/out-of-distribution-detection-knowing-when-ai-doesnt-know/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updateshttps://www.sei.cmu.edu/blog/10-things-organizations-should-know-about-ai-workforce-development/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesThis post outlines 10 recommendations developed in response to work with our mission partners in the Department of Defense.Jonathan Frederick, Dominic Ross, Eric Keylor, Cole Frank, Intae NamMon, 28 Apr 2025 00:00:00 -0400https://www.sei.cmu.edu/blog/10-things-organizations-should-know-about-ai-workforce-development/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updateshttps://www.sei.cmu.edu/blog/dataops-towards-more-reliable-machine-learning-systems/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesDecisions based on ML models can have significant consequences, and managing the raw material—data—in ML systems is a challenge. This post explains DataOps, an area that focuses on the management and optimization of data throughout its lifecycle.Daniel DeCapriaMon, 21 Apr 2025 00:00:00 -0400https://www.sei.cmu.edu/blog/dataops-towards-more-reliable-machine-learning-systems/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesArtificial Intelligence EngineeringMachine Learninghttps://www.sei.cmu.edu/blog/evaluating-llms-for-text-summarization-introduction/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesDeploying LLMs without human supervision and evaluation can lead to significant errors. This post outlines the fundamentals of LLM evaluation for text summarization in high-stakes applications.Shannon Gallagher, Swati Rallapalli, Tyler BrooksMon, 07 Apr 2025 00:00:00 -0400https://www.sei.cmu.edu/blog/evaluating-llms-for-text-summarization-introduction/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesMachine Learninghttps://www.sei.cmu.edu/blog/the-essential-role-of-aisirt-in-flaw-and-vulnerability-management/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesThe SEI established the first Artificial Intelligence Security Incident Response Team (AISIRT) in 2023. This post discusses the role of AISIRT in coordinating flaws and vulnerabilities in AI systems.Lauren McIlvenny, Vijay SarvepalliWed, 26 Mar 2025 00:00:00 -0400https://www.sei.cmu.edu/blog/the-essential-role-of-aisirt-in-flaw-and-vulnerability-management/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesCERT/CC VulnerabilitiesCybersecurityAISIRThttps://www.sei.cmu.edu/blog/enhancing-machine-learning-assurance-with-portend/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesThis post introduces Portend, a new open source toolset that simulates data drift in machine learning models and identifies the proper metrics to detect drift in production environments.Jeffrey Hansen, Sebastián Echeverría, Lena Pons, Gabriel Moreno, Grace Lewis, Lihan ZhanMon, 24 Mar 2025 00:00:00 -0400https://www.sei.cmu.edu/blog/enhancing-machine-learning-assurance-with-portend/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesSoftware AssuranceMachine Learninghttps://www.sei.cmu.edu/blog/introducing-mlte-systems-approach-to-machine-learning-test-and-evaluation/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesMachine learning systems are notoriously difficult to test. This post introduces Machine Learning Test and Evaluation (MLTE), a new process and tool to mitigate this problem and create safer, more reliable systems.Alex Derr, Sebastián Echeverría, Katherine Maffey, Grace LewisMon, 17 Feb 2025 00:00:00 -0500https://www.sei.cmu.edu/blog/introducing-mlte-systems-approach-to-machine-learning-test-and-evaluation/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesTestingMachine Learninghttps://www.sei.cmu.edu/blog/the-myth-of-machine-learning-reproducibility-and-randomness-for-acquisitions-and-testing-evaluation-verification-and-validation/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesA reproducibility challenge faces machine learning (ML) systems today. This post explores configurations that increase reproducibility and provides recommendations for these challenges.Andrew Mellinger, Daniel Justice, Marissa Connor, Shannon Gallagher, Tyler BrooksMon, 13 Jan 2025 00:00:00 -0500https://www.sei.cmu.edu/blog/the-myth-of-machine-learning-reproducibility-and-randomness-for-acquisitions-and-testing-evaluation-verification-and-validation/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesAcquisition TransformationTestingMachine LearningVerificationhttps://www.sei.cmu.edu/blog/beyond-capable-accuracy-calibration-and-robustness-in-large-language-models/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesFor any organization seeking to responsibly harness the potential of large language models, we present a holistic approach to LLM evaluation that goes beyond accuracy.Matt Walsh, David Schulker, Shing-hon LauTue, 03 Dec 2024 00:00:00 -0500https://www.sei.cmu.edu/blog/beyond-capable-accuracy-calibration-and-robustness-in-large-language-models/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updateshttps://www.sei.cmu.edu/blog/genai-for-code-review-of-c-and-java/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesWould ChatGPT-3.5 and ChatGPT-4o correctly identify errors in noncompliant code and correctly recognize compliant code as error-free?David SchulkerMon, 18 Nov 2024 00:00:00 -0500https://www.sei.cmu.edu/blog/genai-for-code-review-of-c-and-java/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updateshttps://www.sei.cmu.edu/blog/introduction-to-mlops-bridging-machine-learning-and-operations/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesMachine learning operations (MLOps) has emerged as a critical discipline in artificial intelligence and data science. This post introduces MLOps and its applications.Daniel DeCapriaMon, 04 Nov 2024 00:00:00 -0500https://www.sei.cmu.edu/blog/introduction-to-mlops-bridging-machine-learning-and-operations/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesArtificial Intelligence EngineeringMachine LearningEdge Computinghttps://www.sei.cmu.edu/blog/measuring-ai-accuracy-with-the-ai-robustness-air-tool/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesUnderstanding your artificial intelligence (AI) system’s predictions can be challenging. In this post, SEI researchers discuss a new tool to help improve AI classifier performance.Michael Konrad, Nicholas Testa, Linda Parker Gates, Crisanne Nolan, David Shepard, Julie Cohen, Andrew Mellinger, Suzanne Miller, Melissa LudwickMon, 30 Sep 2024 00:00:00 -0400https://www.sei.cmu.edu/blog/measuring-ai-accuracy-with-the-ai-robustness-air-tool/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesMachine LearningArtificial Intelligencehttps://www.sei.cmu.edu/blog/weaknesses-and-vulnerabilities-in-modern-ai-ai-risk-cyber-risk-and-planning-for-test-and-evaluation/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesModern AI systems pose consequential, poorly understood risks. This blog post explores strategies for framing test and evaluation practices based on a holistic approach to AI risk.Bill ScherlisMon, 12 Aug 2024 00:00:00 -0400https://www.sei.cmu.edu/blog/weaknesses-and-vulnerabilities-in-modern-ai-ai-risk-cyber-risk-and-planning-for-test-and-evaluation/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updateshttps://www.sei.cmu.edu/blog/weaknesses-and-vulnerabilities-in-modern-ai-integrity-confidentiality-and-governance/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesIn the rush to develop AI, it is easy to overlook factors that increase risk. This post explores AI risk through the lens of confidentiality, governance, and integrity.Bill ScherlisMon, 05 Aug 2024 00:00:00 -0400https://www.sei.cmu.edu/blog/weaknesses-and-vulnerabilities-in-modern-ai-integrity-confidentiality-and-governance/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updateshttps://www.sei.cmu.edu/blog/weaknesses-and-vulnerabilities-in-modern-ai-why-security-and-safety-are-so-challenging/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updatesThis post explores concepts of security and safety for neural-network-based AI, including ML and generative AI, as well as AI-specific challenges in developing safe and secure systems.Bill ScherlisMon, 29 Jul 2024 00:00:00 -0400https://www.sei.cmu.edu/blog/weaknesses-and-vulnerabilities-in-modern-ai-why-security-and-safety-are-so-challenging/?utm_source=blog&utm_medium=rss&utm_campaign=my_site_updates