Job Details

Question And Answer Review Analyst - Financial Data Evaluation

We are seeking a detail-oriented and analytical Question and Answer Review Analyst to support our LLM (Large Language Model) evaluation project focused on financial data extraction and reasoning. The role involves reviewing model-generated answers to financial questions, assessing their accuracy, clarity and completeness, and evaluating the fairness of automated review scores.

Key Responsibilities:

• Review financial QA tasks where models answer specific data-driven questions (e.g., revenue comparisons across fiscal quarters).
• Verify the correctness and clarity of questions and answers using provided source documents (e.g. earnings reports).
• Assess model-generated answers against a gold-standard (Golden Answer) and assign human correctness scores.
• Validate automated reviewer (AR) scores and determine if they are fair or need correction.
• Provide concise and well-reasoned justifications for your scoring decisions, highlighting accuracy, alignment with source data and answer clarity.
• Flag ambiguous or poorly framed questions and suggest improvements if needed.

Qualifications:

• Expert in finance, accounting or similar analytical field, CFA preferred.
• Strong reading comprehension and attention to detail.
• Experience working with financial reports or earnings documents is a plus.
• Familiarity with QA evaluation tasks, especially involving LLM outputs, is desirable.

Excellent written communication skills and ability to explain reasoning clearly and concisely.