< Previous1992 four additional UN agencies joined the UN TRAIN-X Network: the United Nations established TRAIN-SEA-COAST; the United Nations Institute for Training and Research (UNITAR) established CC:TRAIN for Climate Change; the Universal Postal Union (UPU) established TRAINPOST for postal services; and the Food and Agriculture Organization (FAO) established TRAINFISH for fisheries. The TRAINAIR Programme was created in the early 1990s to enable ICAO to cover training in all fields of civil aviation through global implementation of the Standardized Training Package (STP) concept, detailed in the Training and Development Guide (TDG), ICAO DOC 9941. In 2012, the TRAINAIR Programme evolved into the TRAINAIR PLUS Programme (TPP) which expanded to include the bigger aviation training community, international institutions and universities. The TPP enhanced the competency- based methodology which is currently implemented in close to 80 States and used by more than 100 training centres and educational institutions. The TPP comprises a virtual library with close to 160 courses, including learning packages that generate career paths combining professional certification and academic degrees. Training activities conducted within the realm of TPP, ICAO technical cooperation projects, and aviation safety and security training provided by ICAO Bureaux and ORIGINS In the early 1970s the International Telecommunication Union (ITU) began pioneering TRAINTEL (Telecommunications) training programme to develop their training courses. Building on their experiences, with the intent to globally implement competency- based training methodology, the United Nations Organization (UNO) launched the Train-X Programme in the 1980s. Train-X was very effective in terms of improving and modernizing the training carried out by the cooperative training networks of several UN agencies, including, among others, TRAINMAR and TRAINTRADE for the United Nations Conference on Trade and Development (UNCTAD) and CODETEL for ITU. Since AVIATION TRAINING INTELLIGENCE ™ » TRAINAIR PLUS Methodology - Instructional System Design (ISD) 8www.ICAO.int APPLYING DATA-DRIVEN DECISION MAKINGtraining planning and implementation. Making data-driven decisions is already applied in several aviation fields, such as safety management, passenger distribution, and aviation cargo, as examples. Data-driven decision making or D3M, is a process-driven approach to decision making that addresses issues leading to poor decision making. With D3M, effective and informed decisions are based on the results of pertinent data that has been collected and analysed. Using valid and relevant data helps place the “problem” in the right context to determine a best-fit “solution”. D3M provides credible evidence to stakeholders and management regarding strengths, weaknesses, opportunities, limitations and risks. It mitigates bias, influence and human error. Applying D3M in conjunction with organizational analyses, and training needs and impact assessment activities, becomes an important element of support to States facing challenges in aviation training. As with any mathematical model, the more information that can be integrated in the model, the more useful it becomes for strategic and operational planning purposes, given that it reflects facts and figures from the field. Applying D3M to aviation training planning will lead to the development of targeted training portfolios that respond to identified human resource needs in aviation. Business cases can be developed to acquire the expertise, equipment or facilities necessary to operationalize the portfolio. AVIATION TRAINING INTELLIGENCE™ To effectively apply D3M a management system and methodology needs to be adopted. Existing volumes of diverse information, and vast amounts of aviation data need to be managed. At ICAO, data tools such as iImplement, iSTARS, the Safety Solution Center, CAA HR-toolkit, Environment data, Economic Development and Aviation Business Analysis tools, Data+, traffic forecasting, statistics analysis, are transformed through a technology-driven process into an aviation training data management environment. Applying the data-driven business process using the principle of analysing aviation and training data to provide actionable information for training decisions, provides executives, managers and other corporate end users with Aviation Training Intelligence™ (ATI™). This intelligence is used to make informed business decisions. Training organizations that invest in technology tools and methodologies to collect, analyse and manage their ATI™ will grow in alignment with aviation growth and training methodologies that are rapidly evolving. MANAGING AVIATION TRAINING INTELLIGENCE™ (MATI™) Managing aviation training intelligence is the real challenge of aviation training organizations today. Activities involve continuous improvement planning; implementation of technology systems to manage aviation big data; State aviation forecasts; predictive analytics; needs assessment data; trainee, impact and collaborative data; as well as data generated from new learning technologies such as machine learning, artificial intelligence, augmented reality, gamification and more. Managing ATI™ is the practice of mapping methods, tools, systems, applications, needs and solutions for individuals and groups. Regional Offices, laid the ground for the creation of the ICAO Civil Aviation Training Policy. In 2014, the Global Aviation Training (GAT) Office was established to harmonize standardization and management of training activities. TRAINING CHALLENGES In standardizing global training, several recurring challenges have been encountered. These include insufficient funding; lack of subject matter expertise and qualified instructors; poor specialized equipment; non-recognition of certificates between States (even in the same regions); and a lack of harmonization of curricula and licensing requirements. Scenarios like these make it difficult to implement the ICAO Civil Aviation Training Policy. The policy provides for a global training standardization model which facilitates the effective and harmonized implementation of training in aviation, reduces costs, increases quality and efficiency and generate synergies between two important ICAO initiatives: the Next Generation of Aviation Professionals (NGAP) and No Country Left Behind (NCLB). DATA-DRIVEN DECISIONS These challenges are predicated on the absence of suitable and effective links between aviation data and aviation » AVIATION TRAINING INTELLIGENCE™ 9 ICAO TRAINING REPORT | NO. 1 | 2018 APPLYING DATA-DRIVEN DECISION MAKINGAVIATION TRAINING INTELLIGENCE™ CONSISTS OF EIGHT ESSENTIAL COMPONENTS (EC): The essential components shown above are integrated on logical flows, sequences and feedback mechanisms, providing the data sources leading to data-driven decision making. INTEGRATED TRAINING MANAGEMENT SYSTEM (ITMS) AND MATI™ TOOLS ATI™ must be collected, structured, processed, analysed, integrated and evaluated. An integrated information structure is required for conducting up-to- date and needs-based aviation training activities. For ICAO, the automation of training and capacity-building processes is key to providing States, TPP Members, and the aviation community, efficient and cost-beneficial support and solutions for their training needs and endeavours. Strategic and operational planning, business implementation, course development and delivery, are managed according to diverse data using a set of tools and methodologies, to make optimal aviation training decisions. ICAO’s Aviation Training Intelligence™ management system will integrate TRAINAIR PLUS systems such as the TPeMS (TRAINAIR PLUS Electronic Management System); a web-based Training Needs Assessment tool, Training Evaluation Portal, Instructional System Design (ISD) tool, Aviation Training and Education Directory (AETD), Subject Master Expert roster, and a Learning » AVIATION TRAINING INTELLIGENCE™ CONSISTS OF EIGHT ESSENTIAL COMPONENTS (EC) » ESSENTIAL COMPONENT (EC: 4) 10www.ICAO.int APPLYING DATA-DRIVEN DECISION MAKINGManagement System (LMS) in a new, integrated network system architecture. The integrated management system stores, structures and automates training transactions and decisions actioned by a variety of global users including trainees, managers, course developers, instructors, validators, and administrators. MATI™ FOR TRAINING EFFECTIVENESS Excellent learning progression during a training course with positive feedback from trainees and instructors, appropriate facilities and equipment, can be a useless exercise and even worse, a waste of time and money, if there is no transfer of the acquired competencies to the workplace. The capability to measure the impact of training on professional performance is critical for personal and professional development, and organizational achievement. Evaluation of the training effectiveness of each course using the ICAO TRAINAIR PLUS Post- training Evaluation (PTE) methodology follows a scientific basis for measuring the impact of training programmes developed under the guidelines set out in the TDG/Doc 9941. Given the inherent nature of the methodology and its competency-based approach, MATI™ integrates an evaluation procedure aimed to enhance the quality of training developed, the instructional processes used, and provides intelligence on the impact of the training as part of the continuous improvement of the TRAINAIR PLUS Programme. The MATI™ PTE Weighted Impact Index™ (WIN™) equation is calculated by analysing the data collected after course deliveries (post three to six months), using the L3 PTE tool which is a standardized scale with 15 impact indicators related to six dimensions » INTEGRATED TRAINING MANAGEMENT SYSTEM 11 ICAO TRAINING REPORT | NO. 1 | 2018 APPLYING DATA-DRIVEN DECISION MAKING » Level 3 Post-training Evaluation (L3 PTE) Methodology of evaluation. The result obtained is a quantitative value of the real effectiveness representing the impact of the training on work performance. The complete implementation of the ICAO training portfolio will be evaluated and reported based on the WIN rate from each course. It will be the mechanism used for continuously revising curricula to calibrate and correct technical, practical and useful content, and the instructional interactivity of the training. MATI™ INTEGRATED PROCESSES AND IMPLEMENTATION Managing your ATI™ necessitates engagement in diverse business practices and processes that generate data and information that will be used for decision making to optimize the training portfolio of your organization. The integration of each essential component is supported by an integrated system architecture with the compatibility and interoperability of projects, workflows, social media management, collaborative management with business networks and communities of practice, business processes, communication and connectivity. MATI™ Essential Components (EC) are linked through the safety and air transport data collected (EC1, EC2), that are transformed into aviation training data (EC3) and used for training need analysis (EC4), identifying existing training solutions (EC5) or developing the missing courses (EC6), giving evaluation of its effectiveness (EC7) in the alignment with the State’s Aviation Master Plan and Human Resources and Capacity Improvement (HR/CI) planning and development (EC8). MATI™ integrates a structured plan of actions to implement comprehensive and consistent human 12www.ICAO.int APPLYING DATA-DRIVEN DECISION MAKINGDIEGO MARTINEZ Manager, TRAINAIR PLUS Programme ICAO Global Aviation Training (GAT) BONNIE MCEACHERN ISD Specialist ICAO Global Aviation Training (GAT) resources development strategies to meet the State’s strategic objectives, development goals, and the needs of the aviation sector. BUSINESS EFFICIENCY MODELS Aviation Data-driven Decision Making (AD3M) and Managing Aviation Training Intelligence™ (MATI™) are the new models for business efficiencies in a complex aviation training world. The ability to provide quality, standardized, competency-based training through a variety of media, that meets current and future needs of multiple stakeholders, requires multiple sources of data and information to be systematically collected, integrated and analysed by aviation training managers. A deep dive into Aviation Training Intelligence™ reveals that its global implementation will strongly support the generation of strategic partnerships among government, education, academia, and industry, as well as a strong alliances among enthusiastic students, competent instructional teams, and committed training managers, to attract and transform each trainee into new, competent aviation talent for personal, professional, and organizational success. These models will help training managers to continue to modernize their operations, better address the training challenges, and meet both the actual needs of the aviation sector, and the demands from industry. Managing Aviation Training Intelligence™ enables people and organizations to join the scenario where effective training goes beyond training solutions for performance, to improving safe, secure and efficient air transport connectivity as a key element to social, environmental and economic growth, supporting UN Sustainable Development Goals and 2030 Agenda. 13 ICAO TRAINING REPORT | NO. 1 | 2018 APPLYING DATA-DRIVEN DECISION MAKINGTraining needs are a moving target. Regardless of the industry, any given skill set has the potential to become irrelevant if it is not maintained and kept up-to-speed with the latest developments. This is even more painfully true in fields where compliance to standards and regulations is involved, or where technical skills drive business results. In the same way, as skill sets become obsolete, so too do the training programmes that nurture them. For professional training organizations, it is critical to remain up-to- date and relevant to industry needs. To achieve this, it is crucial to establish an instructional quality management system that helps organizations to continuously monitor developments in the subject area; to anticipate impacts on industry activities; and to regularly update training programmes and products based on intelligence. That, we can say, is the foundation. When that training hits the market, the next stage involves collecting input from all stakeholders in the learning ecosystem to measure training effectiveness and make informed decisions about what needs to be done to improve it. In this article, we will explore the sources and channels IATA Training uses to gather data on training effectiveness, and more interestingly, how we use it. DATA IS GOLD AND WE DIG IT Let’s imagine a development effort for a classroom or self- study course. It has followed the instructional design process by the book and has yielded a perfectly structured course with each component of the course package, from lesson plans to assessment instruments, all finely aligned to meet the course objectives. How can we tell that this course will deliver on the expectations? THE SMARTS ON TRAINING EFFECTIVENESS INTELLIGENCE 14www.ICAO.int ICAO PARTNER BRIEF – IATATo answer this question we need to wear many hats to consider things from every stakeholder’s perspective. Instructors, learners, and sponsors are our ears to the ground. Their evaluation of training effectiveness is the echo of how well we, as training suppliers, are doing. At IATA Training, and for many other training organizations, instructors and faculty are the first-line validators of training effectiveness. This is why, after each training they deliver, we ask instructors to complete a comprehensive report which collects their appreciation of variables in key areas that can impact the quality and effectiveness of training. These areas cover the end-to-end process from the instructor’s perspective before the training begins, all the way to learner assessment and grading. For instructors, the work starts well before class begins so it’s important that we get their feedback on the appreciation of the administration and logistical aspects of their experience. An instructor who receives the course material at the last minute, whose flight arrangements don’t allow for enough rest before the training, or who has to teach in below par facilities, may be affected by situations that negatively impact training effectiveness, and vice versa. Then of course, there is the course material. Whether the instructor developed the course or is delivering someone else’s creation, every training session is an opportunity to gauge, for example, whether the course outline and lesson plans offer adequate support to manage the training, or whether the course content, activities and assessment instruments are relevant, up-to-date, and well aligned with the learning objectives. Is the content clear? Are there topics to remove or add? Is the theory/practice balance adequate? Are the activities engaging? Is the course too long or too short for the scheduled time? Are the exams too easy or too difficult? These are questions only the instructors can answer and, here again the information collected highlights what works well, what doesn’t, and what can be improved. Now onto the next stakeholder in the training ecosystem: the learner. As the ultimate beneficiary of the training products we create, their voice speaks the clearest when it comes to giving feedback on training effectiveness. Indeed, for self-study training programmes, the learner is the only source with information on training effectiveness we can tap into. Even though students very rarely have the training professional’s critical eye on elements like course design or content validity, one thing they know better than anyone else is whether the training met their needs, gave them something they can use in the real world, and was good value for their time and money. To capture learner feedback, we use two different surveys: one for classroom courses and one for self-study courses. Of course, training quality and effectiveness in terms of course objectives, content and course material, course structure, and assessments are at the core of both surveys. Additionally, format-specific items are surveyed. This includes the instructor’s level of subject- matter expertise; level of preparedness for the class; and presentation and class management skills. On the self-study side, students are asked to provide input on the registration process to the course and exam, the ease-of-use of course package, the exam relevance, difficulty, and length, etc. Exam results are also systematically scrutinized for validity and integrity, thorough item analysis yielding yet another set of precious insight on training effectiveness. The next organism in our ecosystem, is the sponsor. Most times, they’re the ones picking up the bill so we have to make sure they’re satisfied with the results of the training! But exactly what affects their satisfaction level about the training they’ve sent their employees to? You got it! Return. Return on investment (ROI) – that is if they are able to really measure it, and the more qualitative and encompassing return on expectations (ROE). Sponsors want to see measurable on-the-job performance improvement, compliance to standards, behavioral, attitudinal, or cultural changes that drive tangible results for the business and link it to a training event or programme. Because they have that business results mindset, and because the cost of training comes out of their budget, sponsors can feedback valuable data to the training organization. At IATA Training, aside from surveys, we reach out to sponsors in person. This allows us to better capture some of the intangible data a pre- formatted questionnaire cannot. KNOWLEDGE IS POWER For data to become knowledge – and then power – it has to be analyzed, interpreted, communicated and actioned. With steady, massive streams of qualitative and quantitative data coming from different channels, several resources must dedicate their time to collecting that information, making sure the appropriate “ For the training organization, the students are not the only customers. From the instructors who deliver the training, to the sponsors who patron it, client experience happens on many fronts. ” 15 ICAO TRAINING REPORT | NO. 1 | 2018 ICAO PARTNER BRIEF – IATAcommunicated to stakeholders, actioned, and resolved. Depending on the root cause, the resolution process may include any actor in the training development and delivery value chain, and it may require a wide array of interventions. Issues related to the curriculum or the quality of course content or material, will trigger the course revision process and mobilize subject-matter experts, instructors, and the Learning & Development Specialist. In the same way, resolution of issues related to logistical, administrative, or technical considerations will be agreed, planned and actioned by the appropriate resources. INTELLIGENCE AND HONORING IT A commitment to quality training is essentially a commitment to training effectiveness as measured by each customer in the training ecosystem. For the training organization, the students are not the only customers. From the instructors who deliver the training, to the sponsors who patron it, client experience happens on many fronts. From the registration process, to the assessment of learning outcomes, these internal and external customers will interact with a variety of systems – instructional, administrative, and technological – which will all impact the results of their training experience. Creating channels for them to communicate their evaluation of the end-to-end experience we’ve created for them, constantly monitoring these channels, and making sure the intelligence we gather is used to concretely improve our offering to meet their needs, is the best way to stay relevant and effective. Because let’s face it, in this age of communications, anything else would be an insult to our customers’ intelligence. MATHIEU KHOURY Learning and Development Specialist IATA Training people are informed. Indeed, a significant benefit of the data crunching is the ability for multiple stakeholders to quickly gain visibility on key performance indicators and react in a timely manner. For the Product Managers who own the courses, the intelligence gathered helps them make informed decisions about variables impacting the intrinsic value of their products, how well these meet the needs of the learners, how their instructors are performing, etc. It also helps them identify strengths to capitalize on and areas for improvement. For the Quality Team, training intelligence means being notified of any rating below 3.5 on a scale of five that comes from any channel in our network, and it allows them to identify deficient questions in exams based on the item analysis reports. Whatever the quality issue, standard processes are in place to make sure it is investigated, documented, 16 ICAO PARTNER BRIEF – IATAIt would be fair to say that the aviation industry exists and continues to grow because of our customers, the traveling public. Aviation’s ability to deliver quality, safe and secure customer service experiences hinges on factors that include how well-trained, qualified and knowledgeable our customer experience staff is. It is our responsibility to provide them with the best training tools available. To accomplish this, human resources and training professionals need to understand and identify the key drivers in customer experience (CX), and increase the positive interactions (experiences) with customers. There are several approaches airports can take to train not only customer service staff, but all personnel who play a role in the airport community: concessions, airlines, security, janitorial, and customs and immigration. Training might include workshops, seminars and game activities. CX trainings are conducted to educate the entire airport community on the dynamic customer experience practices. Additionally, seminars and motivational speeches by CX experts can help staff gain perspective and get tips for enhancing their own performance. BENEFITS OF EFFECTIVELY TRAINING CUSTOMER SERVICE STAFF INCLUDE: SKILL ENHANCEMENT Training customer experience staff can yield remarkable results in terms of enhanced personal performance, increased non- aeronautical revenues and ultimately, satisfied customers. As staff are exposed to new and emerging customer experience trends (this changes rapidly), they will be able to apply new methodologies and approaches in their daily work. Moreover, training helps to develop different skillsets and grow networks TRAINING AND CUSTOMER EXPERIENCE 17 ICAO TRAINING REPORT | NO. 1 | 2018 ICAO PARTNER BRIEF - ACINext >