“Should you enroll in a cram school to prepare for pilot training exams?”
This has been a perplexing question for everyone entering the aviation industry with a psychology background. The reasons for this consideration are mostly:
- Enrolling in a cram school may increase the chance of being accepted into pilot training programs.
- However, for airlines, cram school training can distort the selection system, leading to a significant waste of training resources. More seriously, it may also affect safety.
In an environment where all companies claim “This test has nothing to do with cram schools,” should you still go to a cram school? The conclusion is: It is not recommended to attend just to pass the selection test. But if you have already attended, please answer honestly when faced with this question.
But why is this the conclusion?
Here are the insights from having conducted selection tests and recruitment, combined with practical challenges in aviation psychology, offered as a reference for future candidates.
Background: Gaining Necessary Information to Pass Tests
Why can attending a cram school help you pass the exam? Because you can obtain necessary information that allows applicants to pass the tests, referred to in English as “Gouge” (Kay, Thurston & Front, 2013).
By simply searching for keywords like “pilot training” and “exam questions” on Google (or “Aviation Job Knowledge Test Gouge” in English), one can access a vast amount of information regarding the examinations online. This type of information varies, but the majority is related to selection tests. Some have even commercialized the exact psychological tests used in selection, allowing you to buy the formal version and practice at home. Even without them, cram schools create their own simulation versions for you to practice at home.
However, for psychological assessments to be accurate, test confidentiality is important. Subjects who know the test’s focus, question items, scoring method, etc., can manipulate their responses to score higher. If a test fails to maintain confidentiality, resulting in manipulable outcomes, the results will be distorted. Thus, cram schools might gather test materials and content through various means, such as online resources, previous test takers, or recollections from those who passed. In such instances, the leaked exam questions’ inaccurate results pose a high threat to prediction.
Therefore, rather than wondering why candidates are keen on attending cram schools, it’s more apt to say this is a battle—
A battle over “gaining necessary information to pass the test,” with cram schools on one side and airlines on the other.
Candidates stand like grass caught in the crossfire between these two mighty camps, striving merely for survival and better training opportunities.
So, in this battle, how does everyone think?
The Effects of Cram School Training on Airline Training
Let’s first look at the services offered by pilot training cram schools:
- Collection and organization of various exam intelligence
- Training in basic aviation knowledge
- Assistance in passing psychological tests through practice (including paper-and-pencil, equipment, and Vienna tests)
- Support in preparing for interviews through practice
During the peak times of local cram schools, in addition to in-person classes, there were also claims of correspondence courses for practicing at home. These services are not unique to the domestic market; foreign services are also available. However, some focus more on personal growth (like group discussions, motivations for becoming a pilot, enhancing self-awareness, etc.). Yet, as previously discussed, what truly concerns us are those trainings that may affect test accuracy: What impact do these trainings have?
Wastage of Training Resources
This is arguably the most significant aspect affected. Those who pass psychological tests after acquiring tricks from cram schools, but lack the inherent cognitive ability, may struggle during future training, leading to dropouts which directly waste training resources. From having no experience to obtaining a CPL costs about 2.5 million, and if dropped out during Type Rating or IOE phases, the waste due to dropout costs is even higher.
This might be why different airlines adopt various strategies in response to cram schools. If drop-out costs are low, airlines might be more tolerant of cram schools; in other words, if failing doesn’t cause much loss to the airline, they might not care whether a candidate went through cram school and just let them prove themselves during training. But if training costs are high and cram schools do not compensate, airlines have every reason to address the impact of cram schools on selection systems.
Increased Training Risks
A more problematic situation arises when candidates who are not truly fit for training pass because of cram school preparation, leading to marginal training results. If a student is evidently untrainable, the airline has to arrange for their dropout to cut losses. However, if a student’s situation is borderline, lingering between trainable and untrainable, training must continue, placing a heavier burden on managers and instructor pilots. For smaller airlines, losing a trainee can directly cause staffing issues, and if the company allows continuation due to manpower considerations, it may increase future safety risks in fleet management.
Airline Strategies for Managing the Cram School Issue
Considering these potential impacts, can airlines possibly know “who has attended cram school”? This becomes a critical question. For candidates, rather than “should I attend a cram school for pilot training,” the question may shift to “if I go to a cram school, will the airline find out?” And for airlines, is there a way to know “who has undergone cram school training”? Or how should the cram school dilemma be addressed?
(If an airline has low training costs, they aren’t too bothered by dropouts, so the following explanations may not be applicable.)
Administrative Measures
Administrative measures are the most primitive management strategy. Imagine being in charge of psychological testing, how would you prevent questions from being leaked? You might consider seat spacing, using A and B versions, frequently changing questions, monitoring on PTT Aviation and Dcard Aviation forums, ensuring no prohibited items during exams for all candidates… There’s even a story of a supervisor joking about stationing someone at the entrance of a cram school to photograph everyone who enters, so they know which candidates attended. Even if in jest, these illustrate administrative tactics against cram school issues.
Admin strategies typically lead to one of two outcomes, almost without fail:
- Initially, the party making the exam always wins; employers who craft the questions hold a distinct advantage in this arena.
- However, this approach soon gets exploited. After a few iterations, new questions appear online. The testers are thus pressured to rapidly innovate with fresh content.
If we consider international exams like TOEFL/TOEIC, administrative efforts still play a role; for instance, test-takers are required to recite or write a statement against cheating, with penalties for violations. Yet, leaked TOEFL or TOEIC questions still exist, guiding test candidates through online resources. Despite this, these exams maintain their credibility without frequently altering questions (changing questions is arduous). Why do they retain their credibility?
Administrative measures shouldn’t overshadow original goals; the aim of using psychological tests is to find candidates capable of completing training. Ultimately, we must address the test itself to solve this issue effectively.
Anomaly Detection in Psychological Testing
Anomaly detection in psychological testing is crucial for protecting the testing methods; here we can only explain the concept. Statistical execution requires a deep understanding of how psychological tests are constructed. Relevant methodologies can be found in the Handbook of Test Security (2013), though it’s unlikely most readers will find it necessary.
1. Significant Differences Between Tests
When measuring an ability, mature psychological tests rarely use a single subtest. For example, the Wechsler Intelligence Scales, targeting a single ability (e.g., Verbal Comprehension), include at least two regular subtests (Similarities, General Knowledge) and an alternative subtest (Vocabulary). To calculate an ability score, scores from the two regular subtests are summed and cross-referenced for an index score. If either regular subtest is unavailable, an alternative can be used. Regardless, a good psychological test will not rely on a single subtest.
When interpreting results in layers, we focus on whether two tests measuring the same ability yield consistent results. When differences are too large, several hypotheses arise:
- An unexpected event during testing, such as misinterpreting a question, timing errors, marking mistakes, etc.
- The test taker might have some unique cognitive traits distinct from the general population, explaining the variance.
- If such differences manifest in large groups, it likely suggests a malfunctioning test.
The threshold for significant differences in Wechsler scales can be found in tables; with adequate sample size norms, determining this is straightforward for test creators. Thus, when testers observe frequent significant differences in groups despite controlled environments, it might signal a test breakdown.
2. Deviations in Test Norms (Mean, Standard Deviation, Data Distribution)
Norms are crucial descriptive features. For ability-based psychological tests, less interested in describing the entire population’s cognitive level, they focus more on correctly differentiating individuals. Thus, compared to the mean, the standard deviation of the entire test sample is prioritized. Simultaneously, the normalcy of the test distribution is important.
When a test is compromised, both the mean and standard deviation deviate. Mean deviations are straightforward; as leaked questions increase, correct scores rise. Over time, candidates appear increasingly smarter—a clearly unlikely trend signaling potential test breakdown. Changes in standard deviation and data distribution, like forming a new peak or moving rightward, indicate early signs of test deterioration.
3. Inability to Reach Expected Cut-off Numbers
Setting cut-off scores is another critical aspect. For example, if a test was initially designed to filter out 50% of candidates, but this proportion continually decreases over time, it could be a warning sign of a failing test.
Importantly, don’t alter cut-off scores lightly. Adjustments often arise as more candidates pass, prompting higher cut-offs to maintain original filtering ratios. However, this can initiate a vicious cycle, as adjusted scores might cater predominantly to those who underwent preparatory training, expanding practice and training prevalance without addressing the underlying issue.
4. Ensuring Quality in Test Design
The above were anomaly detection methods. When tests display anomalies, it’s crucial to distinguish what issues arise between test administration and evaluation.
Some test issues are fixable; for instance, remedy memory question leakage by frequently altering items or sequences, potentially identifying those trained based on memorized, previously shared answers non-existent in current tests. In contrast, fundamental issues, like shortcuts bypassing intended psychological processes, may necessitate complete test redesigns to achieve consistent measurement of intended constructs.
Assessing the Combination of Genuine Ability and Motivation
But in the face of cram school conflicts with airlines, what actions can candidates take? Quite frankly, they can do little. Moreover, presuming dishonesty due to exposure to key information by attending a cram school seems unfair. Candidates may inadvertently find themselves exposed to this environment without actively seeking the information.
In fact, scholars suggest that information-seeking demonstrates applicant motivation. Isn’t strong motivation a positive predictor of training success? As Kay et al. mention:
Kay, Thurston & Front (2013)
… The mere fact that the candidate is researching the test is considered by some to be an indication of motivation. On the other hand, pursuit of some types of gouge could indicate an integrity problem. …
Thus, is there a testing method to evaluate an authentic combination of ability and motivation without touching on integrity issues? This can be contemplated theoretically.
(Note: The following method essentially means letting you practice thoroughly before testing. To my knowledge, Taiwan has not adopted this approach. Candidates can skip this part.)
1. True Score Theory in Testing
According to the classical true score theory in psychological testing:
X = T + E, where
X = Measured outcome, the only known value
T = True score, the actual condition of the measured target
E = Measurement error
If we incorporate a motivation variable into this formula, it becomes:
X = T + M + E, where
M = Effect of individual practice, explainable as personal motivation
As we only know X and the proportion of E in X (defined by test reliability), real ability (T) and individual motivation (M) cannot be distinguished. If we accept this, namely that true ability and personal motivation can be measured together, then we can proceed.
2. Allow Candidates to Practice Before Testing
The approach entails: clearly informing all candidates about the testing format, allowing them to prepare thoroughly at home before testing. Fill out provided materials, master them, as the test content will not deviate from this range, eliminating special tricks to influence pass or failure since everyone has the chance to practice with provided resources.
Admittedly, in psychological testing, this concept contradicts norms. Typically, subtle experimental designs aim to isolate factors influencing results, while this method intentionally combines true scores with individual motivation to neutralize uneven test technique distribution across a population. Perhaps this implies that the threat of cram schools compromising test validity outweighs openly sharing test materials for practice.
This represents an alternative, feasible method.
So, Should You Attend a Cram School for Pilot Training Exams?
For applicants, whether or not to attend a cram school for pilot training exams, the discussions lead to these findings:
- Airlines more susceptible to high training resource wastage due to cram schools will be more attentive to handling these issues.
- Airlines might use administrative means to identify cram school attendees, but this method isn’t necessarily efficient.
- By observing test result anomalies and norm shifts, recruitment personnel familiar with psychological testing can effectively gauge a cram school’s impact on selection systems. They might even assess potential cram school training among candidates, inferring unreliable test outcomes.
Therefore:
- Consideration of an airline’s training cost should guide your decision on whether to attend a cram school.
- In my experience in psychology, having witnessed some failing beyond repair, I advise against attending merely to pass screening tests. However, if your intent is networking, gaining interview experience, or acquiring aviation knowledge, I personally see no harm.
- If deciding to apply, answer this question honestly if asked during the process.
Airlines’ Perspective on the Cram School Issue
For airlines, the best strategy surrounding this issue appears to be:
- The cram school industry is an age-old problem, inevitably persisting, so rejecting it serves little purpose.
- Airlines should highly demand their recruitment partners: regardless of cram school influence on candidates, any tests used during selection must ensure colleagues accurately measure the intended abilities. Aim for TOEFL-level testing quality: even if everyone attends a cram school, we should assess their true capabilities.
- If truly concerned about wasteful training resources, investing in workforce and psychological testing is a worthwhile choice.
Further Reading
Kay, Thurston & Front (2013). Commercial airline pilot and air traffic controller selection. In Aeromedical Psychology, 37-62. Ashgate.
Handbook of Test Security (2013).
