AI Auditing Engineer

Title: AI Auditing Engineer - Penn Center for Media, Technology, and Democracy - Penn Engineering

Location: Philadelphia United States

Hybrid or Remote Eligible

locations

Amy Gutmann Hall

time type

Part time

Job Description:

University Overview

The University of Pennsylvania, the largest private employer in Philadelphia, is a world-renowned leader in education, research, and innovation. This historic, Ivy League school consistently ranks among the top 10 universities in the annual U.S. News & World Report survey. Penn has 12 highly-regarded schools that provide opportunities for undergraduate, graduate and continuing education, all influenced by Penn's distinctive interdisciplinary approach to scholarship and learning. As an employer Penn has been ranked nationally on many occasions with the most recent award from Forbes who named Penn one of America's Best Large Employers in 2023.

Penn offers a unique working environment within the city of Philadelphia. The University is situated on a beautiful urban campus, with easy access to a range of educational, cultural, and recreational activities. With its historical significance and landmarks, lively cultural offerings, and wide variety of atmospheres, Philadelphia is the perfect place to call home for work and play.

Posted Job Title

AI Auditing Engineer - Penn Center for Media, Technology, and Democracy - Penn Engineering

Job Profile Title

Temporary Employee - Non-Exempt

Job Description Summary

The Penn Center for Media, Technology, and Democracy (Penn MEDIATED) works to advance the scientific understanding of the information ecosystem and leverage that research to strengthen the foundations of democracy.

Penn MEDIATED seeks a part-time AI Auditing Engineer to develop and productionize auditing and monitoring of large language models (LLMs). The AI Auditing Engineer will unify existing research prototypes from several codebases in different academic projects to create an interoperable set of tools that is generally useful for longitudinal monitoring of LLMs. The tooling will be used to automatically and longitudinally monitor rates of refusal or soft censorship, political lean, sourcing information, and other features of LLM responses on political topics.

We seek an AI Auditing Engineer experienced in developing efficient and scalable data pipelines in Python, designing and managing NoSQL databases, working with cloud computing platforms, and in working with LLM APIs. Experience with and knowledge with natural language processing methods for LLM evaluations is also beneficial.

The AI Auditing Engineer will work closely with Danaé Metaxa, Raj & Neera Singh Assistant Professor in the Department of Computer and Information Sciences. The position will be jointly supervised by Danaé Metaxa and Alex Engler, Executive Director of the Penn Center on Media, Technology, and Democracy. This role can be fully remote or hybrid. This is a temporary role, and may be either part-time or full-time, depending on the applicants, with the expected term during summer 2026 at between 20-40 hours per week.

Job Description

Develop and deploy an interoperable set of tools that is generally useful for longitudinal monitoring of LLMs.

Design and manage appropriate databases for LLM monitoring.

Advance research methods for LLM monitoring of political topics.

Qualifications

• Bachelor's degree in a relevant field of study

• Required proficiency in developing efficient and scalable data pipelines in Python.

• Required proficiency in designing and managing NoSQL databases.

• Required proficiency in working with cloud computing platforms.

• Preferred understanding or prior experience working with LLM APIs.

• Preferred understanding or experience with natural language processing methods for LLM evaluation.

• Preferred experience with data visualization in interactive dashboards.

• Interest in the political outputs and impacts of large language models (LLMs) is a bonus.

To Apply: Please Submit a Resume and Cover Letter

Job Location - City, State

Philadelphia, Pennsylvania

Department / School

School of Engineering and Applied Science

Pay Range

$7.25 - $22.00 Hourly Rate

Salary offers are made based on the candidate's qualifications, experience, skills, and education as they directly relate to the requirements of the position, and in alignment with salary ranges based on external market data for the job's level. Internal organization and peer data at Penn are also considered.

Equal Opportunity Statement

The University of Pennsylvania is an equal opportunity employer. Candidates are considered for employment without regard to race, color, sex, sexual orientation, religion, creed, national origin (including shared ancestry or ethnic characteristics), citizenship status, age, disability, veteran status or any class protected under applicable federal, state or local law.

Special Requirements

A background check may be required after a conditional job offer is made. Consideration of the background check will be tailored to the requirements of the job.

University Benefits

• Health and Life: Penn offers comprehensive medical, prescription, behavioral health, dental, vision, and life insurance benefits to protect you and your family's health and welfare.
• Retirement: Penn offers generous retirement plans to help you save for your future. Penn's Basic, Matching, and Supplemental retirement plans allow you to save for retirement on a pre-tax or Roth basis. Choose from a wide variety of investment options through TIAA and Vanguard.

Please note that benefit eligibility is determined/based on ACA guidelines.