Questions: Psychological Test Construction and Psychometric Validation

Question 1

A researcher develops a new anxiety questionnaire. It has excellent internal consistency (Cronbach's α = 0.94) and high test-retest reliability. Can they conclude the test is valid for measuring anxiety?

• A. Yes — high reliability on both indicators confirms the test is measuring anxiety accurately
• B. No — reliability is necessary but not sufficient for validity; the test could consistently measure something other than anxiety
• C. No — factor analysis must also be run before any validity claim is possible
• D. Yes — high internal consistency directly implies construct validity for psychological constructs

Answer: (B)

A test can be perfectly reliable while measuring the wrong construct entirely — this is the crucial distinction. A highly consistent measure of handwriting pressure is a very reliable test, but that doesn't make it a valid measure of anxiety. Reliability sets a ceiling on validity (an unreliable test cannot be valid) but does not guarantee it. Validity requires multiple sources of converging evidence: does the test relate to external criteria it should? Does it discriminate from constructs it should be unrelated to? Does its internal structure match the theoretical model of the construct?

Question 2

A depression scale was rigorously validated in a sample of Western university students. A clinical researcher wants to use the same scale in a rural East Asian elderly population. What is the primary psychometric concern?

• A. The scale may require re-validation in the new population because cultural context, symptom expression, and item relevance may differ
• B. No concern — psychometric validation generalizes universally once completed
• C. Only the scoring norms need to be updated; the item structure remains valid across populations
• D. The concern is purely linguistic; accurate translation resolves all validity issues

Answer: (A)

Validation evidence is tied to the population in which it was gathered. Depression may manifest differently across cultures — somatic symptoms (fatigue, pain) are more prominent in some populations, while cognitive symptoms dominate in others. Items written and validated for young Western students may have different factor structures, different criterion relationships, and different measurement equivalence in an elderly East Asian clinical sample. This is why the Standards for Educational and Psychological Testing treat validation as an ongoing argument, not a universal certification.

Question 3

Factor analysis alone provides sufficient evidence to conclude that a psychological test is valid.

• T. True
• F. False

Answer: False

Factor analysis examines the internal structure of a test — whether items cluster in patterns consistent with the construct specification. This is one important source of validity evidence, but it is not sufficient. A test can have a clean factor structure and still fail to predict any real-world criterion it should predict, or it can correlate highly with constructs it should be unrelated to. Validity evidence must be triangulated from multiple sources: internal structure, criterion relationships (predictive and concurrent), convergent evidence, discriminant evidence, and invariance across relevant populations.

Question 4

Validation of a psychological test is better understood as an ongoing process of accumulating evidence than as a one-time pass/fail certification.

• T. True
• F. False

Answer: True

The modern view, formalized in the Standards for Educational and Psychological Testing, treats validity as an argument that must be continually supported by evidence. A test validated in 1990 may use outdated construct definitions, may have been normed on populations no longer representative, or may have been validated only for purposes that differ from current applications. New evidence can strengthen, qualify, or even undermine prior validity claims. Treating validation as 'done' after initial publication is a major source of inappropriate test use.

Question 5

Why is construct specification the critical first step in test development, and what goes wrong when it is skipped or done superficially?

Model answer: Construct specification defines precisely what the test is intended to measure — which facets of the construct are in scope, which are excluded, and for which population. Without this, item writers have no consistent target, so different items may tap different constructs, producing a scale with poor internal coherence and ambiguous interpretation. More critically, validation becomes impossible: you cannot determine whether a test measures 'depression' if 'depression' was never specified. Superficial specification leads to tests that over-represent facets that are easy to ask about (cognitive symptoms) and under-represent facets that are harder (somatic or behavioral), creating construct underrepresentation — a form of invalidity.

This is analogous to architectural blueprints: you cannot evaluate whether a building was built correctly without a specification of what it was supposed to be. In test construction, the spec is called the 'construct map' or 'test blueprint.' Well-specified constructs produce item pools that proportionally cover all relevant facets, allow systematic item review, and generate hypotheses about convergent and discriminant relationships that can be tested during validation.