GLOSSARY

10% Window

An option under JTPA that allows an SDA or LWFDB to certify dislocated workers as eligible for services under Title IIA rather than Title III despite exceeding earnings criteria during specified pre-service quarters.

13th-Week Follow-Up

The method commonly used for gathering outcomes information from former JTPA participants via traditional survey techniques -- so named because the Secretary of Labor has stipulate through federal regulations that it be conducted on participants 13 weeks after they exit JTPA programs.. (See also Traditional Follow-Up and Participant Contact.)

501c(3)

The number of the form that must be filed by a non-profit entity seeking tax-exempt status and, hence, used as an adjective to describe such entities.

Accountability

The principles under which program administrators and service providers are held responsible for meeting their responsibilities and obligations.

Accountability, Fiscal

The cost principles used to ensure that program funds have been spent only for legal and authorized purposes; i.e., that funds were not embezzled and properties were not misappropriated.

Accountability, Managerial

The personnel principles for ensuring that program operations are efficient; i.e., that fiscal and human resources are deployed without waste.

Accountability, Program

The operational principles through which administrators and service providers are held responsible for achieving the results specified in the legislation or executive orders authorizing delivery of services to a specified customer group.

Adjustment

Computing an expected value for an outcome variable by using statistical techniques that take the effects of key independent and/or exogenous variables into account. (See also Baselining and Benchmarking.)

Adjustment Model

In the JTPA system, the use of a regression equation to set a different performance standard for each service delivery area according to expected outcomes given each SDA's unique customer mix and labor market conditions.

Admin Dollars

Administrative dollars -- a term commonly used by those managing publicly-funded employment and training programs (particularly JTPA) to denote funds specifically earmarked in a grant to the state or in subgrant recipients' budgets for administrative purposes. The distinction between administrative dollars and other kinds of funds in the budgets of employment and training programs is important because they cannot exceed a specified portion of the entire budget. Moreover, with recent moves toward devolution, legislators are likely to lower that percentage. Since follow-up commonly is considered an administrative function, availability of funds is impacted not only by the size of a state's various employment and training program grants but also by the allowable percentage for setting aside admin dollars.

ADP

Automated Data Processing: in the context of this Guide, a division within a company or a third party (outsource) provider of payroll processing and/or personnel record- keeping services; also referenced as Electronic Data Processing or EDP.

AEA

Adult Education Act of 1996: federal legislation which provides grants to the states to deliver adult education and literacy services.

AFDC

Aid to Families with Dependent Children; replaced by Temporary Assistance for Needy Families. (See also TANF.)

AJB

America's Job Bank: a publicly- supported electronic forum on the InterNet where employers across the nation can post job openings in a standardized, skills-based format. Taken together with Amer-ica's Talent Bank, the AJB represents a significant step toward standardizing the description of job openings to streamline job matching as well as automating and expanding the scope of the traditional (non-automated, state-specific) labor exchange function. (See also ATB.)

ALMIS

America's Labor Market Information System: a series of initiatives by the DoL/ETA to standardize the collection and analysis of labor market supply and demand information and to automate delivery of that information to all interested parties.

ALMIS-D

America's Labor Market Information System-Database: an enhanced and more comprehensive file structure that will contain the OLMID (as a substructure) with links to an occupationally oriented, skills-based structure in machine-readable form (called the O*NET) that is destined to replace hardcopies of the Dictionary of Occupational Titles. (See also OLMID and O*NET.)

Anomaly

A value for a measure or observations of a variable that is counter-intuitive or inconsistent with previous sets of values or observations.

Application

Programs and organizations for whom follow-up services are provided. All GED students, for example, would be one application; all released ex-offenders would be another application. Applications are a particularly important concept in forecasting follow-up expenses. Because follow-up by and large uses cost-effective record linkages, substantial increases in the number of seed records processed in each wave will not have a significant effect on total operating costs. The number of partner agencies served does not provide an adequate handle for cost-forecasting because serving one partner agency might entail a single application (e.g., JobCorps) while service to another might entail several applications (e.g., in serving the state's JTPA grant recipient, one for each Title.) In Texas, for example, the lead agency was asked to bypass the state education agency and deliver products directly to each independent school district. Thus, service to public education went from one to 1,040 "applications" and, while only one partner agency was involved, the cost of doing follow-up on secondary education completers and leavers increased proportionately with virtually no change in the number of records processed.

Application Layer

The interface the end-user sees; a layer atop the databases underlying an automation tool. For, example, consistency and articulation, system-building efforts in the employment and training arena rely on the same data sets to drive program planning, case management, and self-help career guidance. However, planners, case managers and individual job-seekers want data extracted and combined in different ways to meet their particular information needs and cognitive styles. Therefore, common elements in the data layer of the ALMIS-D, OLMIDS, or O*NET, for example, may be accessed through several different data layers. Similarly, both employers and educators need to tap the CIP-to-OES crosswalk to validate the way a follow-up entity might determine training-relatedness. The CIP-to-OES cross walk would be the data layer. Employers would enter through an OES-centered application layer while educators would enter through a CIP-centered application layer.

Archived Files

Data sets stored off-line or on a computer in compressed mode to conserve storage space; commonly aged files that are less likely to be needed. Multiple steps may be required to retrieve such files physically or to uncompress them electronically before needed information can be extracted.

ATB

America's Talent Bank: a publicly-supported electronic forum on the InterNet where job-seekers across the nation can post resumes in a standardized, skills-based format. Taken together with America's Job Bank, the ATB represents a significant step toward standardizing the description of job-seeker qualifications to streamline job matching as well as automating and expanding the scope of the traditional (non-automated, state-specific) labor exchange function. (See also AJB.)

AutoSOC

An automated table used to look up Standard Occupation Classification codes and titles.

Attentive Publics

Groups of citizens who pay special attention to specific policy issues because they are or perceive themselves to be affected directly by decisions being made.

Baselining

Using performance in a specified program year as the standard against which results in subsequent program years is judged. (See also Benchmarking.)

Benchmarking

Using officially-declared standards or the performance of entities with a comparable mission, cus-tomer base, etc., as the yardstick for assessing another program's performance. (See also Baselining.)

BLS

Bureau of Labor Statistics -- a division of the United States Department of Labor.

Boundary

In Systems Theory, the defining characteristics which separate a system from its external environ-ment or which separate one subsystem from another. (See also Domain.)

Buckley Amendment

Another name for the Family Education and Right to Privacy Act (FERPA); so named for its chief author who introduced the legislation as an amendment to the federal Data Privacy Act.

Buckley Agreement

A document specifying the obligations and responsibilities of two or more public entities that exchange individually-identifiable information under FERPA.

Capacity to Benefit

A determination, based on an assessment of prior employment experience, educational attainment, aptitude and work-related attitudes, that: a) a welfare-to-work client would not be likely to obtain a desired outcome unless provided employment and training services; and b) the client's barriers to employment are not so severe that he/she is likely to remain dependent on public assistance even after receiving services. As opposed creaming, this is an officially approved method of effectively targeting services to improve the return on investment. (See also Creaming.)

CBO

Community-Based Organization: in the context of follow-up and accountability, this term commonly refers to non-profit groups eligible to deliver education, training, social and charitable services under contract with a state agency or substate administrative entity.

CCSSO

Council of Chief State School Officers (elementary and secondary -- see also ESC and SHEEO).

CDR

Career Development Resources: an entity in each state responsible for facilitating analysis and delivery of education and training supply and employment demand information to individuals and public agencies. In Texas, the CDR serves as the central follow-up entity.

CEA

Central Education Agency: the entity at the state level responsible for administering public education (K-12); usually the grant recipient of federal education and training dollars on behalf of the state. (See also LEA.)

Central Follow-Up Entity

Agency or sub-agency unit that gathers outcomes information on behalf of several programs and service providers as opposed to each service provider or program collecting such data for itself. (See also Lead Agency.)

CETA

Comprehensive Employment and Training Act: replaced the Manpower Development and Training Act as the nation's largest workforce development program; in turn, CETA subsequently was replaced by the Job Training Partnership Act.

CIDS

Career Information Delivery System: an automated tool students, program participants, counselors and case managers can use to digest and understand occupational employment data and training program inventories to guide individuals in making informed choices. While there are a number of commercial programs on the market under various trade names, CIDS most commonly is used in reference to coordinated federal-state efforts through the NOICC-SOICC network to standardize, automate, and disseminate such a tool pursuant to §464(b)(2) of the Job Training Partnership Act. (Note: The Texas SOICC is now known as CDR - Career Development Resources)

CIM

Certificate of Initial Mastery: an educational credential based on demonstrable skills and competencies to a level of proficiency that is criterion-referenced and employer-validated rather than to a level set by an education and training provider or one that is norm-referenced.

CIP

Classification of Instructional Programs: a commonly used taxonomy for coding programs (fields of study) and courses offered by education and training providers.

Cohort

As used in this Guide, a group whose shared characteristic is the treatment, intervention or service received.

Cohort Coverage

The percentage of members in an exit cohort who are found through record linkages to be employed and for whom occupational employment details have been obtained through their employers' res-ponses to a follow-up survey. (See also Employer Response Rate.)

Committee of Practioners

A state-level body constituted under §115 of the federal Carl D. Perkins Act to assist a state in developing and implementing core performance measures and standards which, given their experiences at the field level, will be fair, meaningful, and actionable.

Completers

In education and training programs, those who finished a course of study and received a credential. In workforce development programs, those who receive a positive termination. (See also Exiters and Leavers.)

Comptroller General

The chief executive officer of the General Accounting Office at the federal level who is appointed by the President upon the advice and consent of the Senate but whose investigations are initiated upon request of Congress, Congressional committees, or individual members of Congress. States usually have a comparable figure known as the State Comptroller of Public Accounts.

Constructs

In empirical research, labels for properties that are indirectly observable. For example, "gravity" can't be seen but rather is a construct used to explain observations like the falling of objects toward the earth. (See also Indicator and Variable.)

Contingent Worker

Individuals who work on a project-to-project basis, as contract workers, or on lease from a temporary help agency; so named because their continued employment is subject to the peaks and declines or in the amount of work that needs to be done or the completion of particular projects; also means Workforce workers who are available -- by choice or otherwise -- for employment under those terms. (See also Core Workers.)

Conversion

In Systems Theory, the process of assessing inputs (demands and supports) and translating them into outputs (decisions and action).

Core Workers or Contingent Workforce

A firm's incumbent workers who perform on-going operations and whose tenure transcends indivdual projects and who enjoy relatively stable employment regardless of peaks and declines in the company's overall employment demand. (See also Contingent Workers.)

Cost-Consequences Analysis

A method for computing return-on-investment that compares the amount of funds invested in specific job preparation programs to the resulting self-sufficiency of former participants measured in terms of post-program earnings, reduced reliance on public assistance benefits, and increased pay-roll revenues. (See also ROI.)

COVE

Council On Vocational Education: a state-level advisory council dealing with career and technical education issues -- in states moving toward consolidated and integrated service delivery planning, the COVE's functions may now be performed by a Human Resource Investment Council.

Covered Employment

A type of job where earnings of incumbent workers must be reported to the state employment security agency under the applicable Unemployment Compensation Act; references also may be made to individuals in those jobs as covered workers. Conversely, those falling under exemptions to a state's Unemployment Compensation Act are known as "Non-covered employment" or "Non-covered workers."

CPM

Critical Path Method of project management developed by DuPont and Remington Rand in the 1950s to calculate total project duration based on individual task durations and dependencies, and to identify which tasks are most critical. (See also PERT and Gantt.)

Creaming

A pejorative term used to describe a form of gaming behavior that consists of serving the easiest-to-serve in order to improve an entity's performance measures without regard for the customer or clients' interest. As opposed to using capacity to benefit in screening and referring customers or clients to employment and training programs, creaming is discouraged by the way performance measures are mixed or is forbidden expressly by the way the application eligibility rules are moni-tored and enforced. (See also Capacity to Benefit and Gaming.)

Criterion-Referenced

The characteristic of an assessment tool whereby a passing score is based on standards that remain fixed regardless on how those taking being assessed performed. (See also Norm-Referenced and Venue-Neutral.)

Crosswalk

(n.) A table or matrix that facilitates efforts to look up codes for the variable arrayed along one axis to corresponding codes of or relationships with the variable arrayed along another axis -- as in the table used to translate DOT codes into their OES equivalents or the one used to determine the degree of training-relatedness between a field of study and post-exit occupational employment, CIP-to-OES crosswalk. (v.) The act of looking up information in a cross-referenced table. (Sometimes written as X-walk.)

CRS

Consumer Report System -- a DoL/ETA-funded multi-state initiative led by the CDR to develop an automated delivery mechanism to make service provider performance history infor-mation universally available in user-friendly formats to facilitate fair and meaningful comparisons and to promote informed choice in career decision-making and in selecting education and training options.

Customer

Anyone potentially having an interest in the products and services provided through the employment and training system -- especially those who want or need follow-up information. Includes but is not limited to participants, partner agencies, and stakeholders. (See also Participant, Partner Agency and Stakeholder.)

Datamation

An automated display of data in graphic format -- especially formats with images that move.

Decay Rate

The rate at which data in a management information system, on average, are rendered obsolete or out-of-date. In follow-up, the average time that elapses before contact information in former participants' files is rendered out-of-date because subjects have changed addresses and/or phone numbers.

Delphi Technique

A process developed by RAND Corp. in the 1950s to resolve conflicting assumptions among stakeholders in a large and diverse enterprise by going through multiple iterations of information exchange, brainstorming, and explanations to build consensus. (See also Focus Group.)

Demands

In Systems Theory, the expressed or implied expectations and needs of individuals or groups in the external environment which necessitate a response (action or decision) from a system.

Devolution

In the employment and training system, a process or trend that transfers authority over many aspects of service delivery from the federal government to the state grant recipients and/or local workforce development boards.

Diagnostics

As used in this Guide, measures or indicators used to identify problems or defects in strategies, processes or products before they are unveiled or released publicly; indicators for the internal rather than external uses by a central follow-up entity and its partner agencies. In MIS contect, tools and procedures used to identify problems in hardware, software, operating systems, etc. (See also Evaluation, Formative.)

Disintermediation

The process of removing persons or entities that stand between the producer and the ultimate consumer (as in direct-mail retailing); in the case of follow-up, the provision of information services directly to prospective customers of any component in the employment and training system. Example: the transition from CRS as a product installed on stand-alone computers in One-Stop centers for assisted-use to one that is mounted for use in self-help mode on the InterNet.

Dislocated Worker

An individual who has been laid off or terminated as the result of a mass lay-off or plant closure --especially those that result from adverse consequences of trade agreements or long-term economic trends. (See also TAA.)

Displaced Homemaker

An individual who worked without compensation in the home and who now is forced by economic circumstances to seek outside employment -- particularly those whose education and training has been rendered obsolete by the intervening years between the end of their formal schooling and their current job-search.

DoD

United States Department Of Defense: a source of outcomes (military service) typically not covered in a state's Unemployment Insurance wage records.

DoE

United States Department Of Education.

DoL

United States Department Of Labor.

Domain

The range of issues, subjects, problems or programs covered by a methodology, study or authority of a central follow-up entity -- as in "realm" or "bailiwick." (See also Boundaries.)

DOT

Dictionary of Occupational Titles -- soon to be rolled into the O*NET system.

Drop-Out

An individual who does not persist long enough to complete a program or service; whether withdrawal from a program is formal or de facto, drop-outs are distinguished from stop-outs insofar as the former express no intention of completing the program or service at a later date. (See Stop-Out.) A drop-out leaves a program voluntarily as apposed to having services terminated by the provider. (See Terminee.)

Economic Development

Efforts to increase employment opportunities by getting new businesses to relocate in a community or existing businesses to expand. Differs from job development in the sense that it seeks to in-crease the pool of available work rather than soliciting employers to post openings for jobs that already exist. (See also Job Development.)

ECS

Education Commission of the States: comprised largely of state governors and the chief public education and higher education administrators as well as researchers who specialize in assessing the effectiveness of education and training programs. (See also CCSSO and SHEEO.)

EDP

(v.) Electronic Data Processing: another acronym for automated data processing; (n.) the name often applied to the unit within an agency for its management information system. (See also ADP and MIS.)

Education and Training Delivery Subsystem

In this Guide, the actual efforts at the field level to impart knowledge, skills and abilities essential for individuals to achieve economic security through labor force participation; may include public and private for-profit providers of first chance and second chance programs at the basic education, secondary, and postsecondary levels in the classroom or on-the-job. (See also Employment and Training System.)

Emerging Occupation

A job whose mix of duties and tasks, and knowledge, skills and abilities is so unprecedented that it is not yet included in the most recent editions of prevailing taxonomies. Emerging occupations are noteworthy because they may require education and training service providers to respond by developing a new curriculum. (See also Evolving Occupations.)

Employer of Record

For a member of an exit cohort being studied, the employer whose address is obtained through linkages to the Unemployment Insurance wage records or documented through other outcomes resource databases such as DoD, OPM or USPS.

Employer Response Rate

In follow-up studies, the percentage of employers who respond to a survey to obtain occupational employment details. (See also Cohort Coverage.)

Employment and Training

An integrated and articulated set of federally-funded, state-administered programs designed to help citizens acquire the skills, knowledge, and abilities they need and to support them as they transition in and out of the workforce at various stages in their lives.

Employer-Generated Title

In survey research to obtain occupational-employment information, the practice of letting employers submit the lay titles they use in their payroll and personnel systems as opposed to forcing employers to shoehorn the job titles they use into an existing occupational coding taxonomy. (See also Standard Approach and Taxonomy Approach.)

Enhanced Quaterly UI Report

A proposed format for gathering labor market data from employers over and above the elements specifically required for administering UI benefits and computing employers' payroll tax contributions.

Enhanced Record

Information about the post-exit outcomes achieved by former program participants obtained via automated record linkages and electronically appended to those participants' original seed records; also known as output record.

Enrollment-Driven Performance

Arrangements for allocating funds to local providers based on the number of eligible persons who sign up to be served -- regardless of how well the service provider performs. (See also Performance-Driven.)

Enterprise Zone

A high poverty or economically depressed area within a community or state targeted for employment development through inducements for business and industry to locate therein and to hire, train, and retain economically disadvantages residents thereof.

Environment

In Systems Theory, the behavior of actors and conditions outside a system's boundaries.

ERIC/ACVE

Education Resource Information Center/Adult, Career, and Vocational Education: a national clearinghouse and archive of research materials and monographs; affiliated with the Education Department of the Ohio State University at Columbus.

Equity of Access

A construct related to the fairness of service delivery that focuses on inputs rather than outcomes; viz. performance measures in this genre are used to determine if the portion of some service-eligible subpopulation enrolled in a program is roughly equivalent to the proportionate presence of that subpopulation in the universe of persons residing in the service delivery area. (See also Service-Eligible, Special Pops, Target Population).

ES

Employment Service: usually a branch of a state's employment security agency responsible for matching job-seekers with job orders placed with the agency through its substate offices. (Some-times call the Job Service.)

ESL

English as a Second Language: a characteristic of individuals often defined as eligible for services because their lack of English is construed as a disadvantage in the world of education and training and/or in the world of work; also a mode of delivering language instruction to them.

ETA

Employment and Training Administration: a division within the United States Department of Labor.

Evaluation

Assessing the effectiveness and/or efficiency of a program, activity or service; comparing results to stated goals and objectives, a benchmark, a baseline, or an expected value.

Evaluation, Formative

Evaluation done for diagnostic purposes; e.g., to pinpoint problems and/or to identify "best practices" for the sake of driving program improvement. (See also Dignostics.)

Evaluation, Gateway

Assessment and testing at key articulation points to determine is an individual is qualified to receive a credential or to enter a program as in the Texas Assessment of Academic Skills (secondary exit test), the Texas Assessment of Postsecondary Skills (postsecondary entrance exam) or Florida's "rising junior" exam called College-Level Academic Skills Test (CLAST).

Evaluation, Summative

Analysis of program performance at the end of a fiscal or calendar period -- typically compiled for use by persons who are not engaged in program planning, program administration, or service delivery; e.g., as in consumer reports.

Evolving Occupation

A job whose title has not changed since the latest editions of prevailing taxonomies were issued despite significant changes in the knowledge, skills and abilities required to perform it. Evolving occu-pations are noteworthy because they may require revisions in the related training curriculum.

Exceptions Pending File

In the context of this Guide, former students or program participants for whom post-exit outcomes could not be documented through the record linkage efforts of a central follow-up entity. (See also Pending File and Resolution.)

Exceptions List

A printout or database containing the names and other pertinent information on subjects in an exit cohort for whom no successful outcomes could be documented through electronic record linkages. Such lists are provided to follow-up staff and/or the service pro-vider of record to facilitate supplemental follow-up through conventional techniques.

Exceptions Process

The method used by education and training service providers to document for themselves the post-exit outcomes achieved by their former students and program participants who could not be located through the central follow-up entity's record linkage efforts; may or may not include provisions for audit and verification by the central follow-up entity as a condition of adding the data obtained by the service provider to the outcomes database es-tablished through automated record linkages. Most often, this takes the form of traditional participant-contact surveys also known as supplemental follow-up.

Exempt Workers

Members of the labor force whose employment falls into a category not covered under their respective state's Unemployment Compensation Act; references also may be made to exempt employers as those who do not have to file quarterly reports because their workers are not covered by their respective state's Unemployment Compensation Act. (References also may be made herein to "non-covered" workers.)

Expectation Slippage

A situation where creeping escalation of specifications or demands results in efforts that deviate significantly from what was planned -- often resulting in endeavors coming in over budget and behind schedule.

Expected Value

A predicted outcome derived by using a statistical model that takes key independent and/or exogenous variables into account in adjusting mean scores obtained for a prior cohort.

Expert System

Application software that distills the knowledge possessed by a trained and experienced practitioner reduced to a sequence of questions that operate through Boolean logic to enable less experienced persons apply sound rules to arrive at appropriate decisions. Examples include automated medical diagnostics devised by specialists that can be used in the field by general practitioners and allied health workers or on-screen question-and-answer scenarios that relatively inexperienced persons in a support center can follow as they help customers install, use and recover from errors in computer hardware or software.

Extrapolation

A rigorous process of projecting trends into the future and building confidence intervals around them by adjusting for the effects that known change factors are likely to produce.

Feedback

In Systems Theory, the mechanisms through which reactions to a system's outputs are processed in the external environment to be translated into new stresses and demands requiring a system's response.

FERPA

Family Educational Rights and Privacy Act of 1974; also known as the "Buckley Amendments."

FETPIP

Florida Education and Training Placement Information Program: Florida's central follow-up entity.

File

A collection of records stored in a consistent format either in hardcopy or on magnetic media.

First Chance Programs

Education and training provided by primary, secondary and postsecondary institutions (public or private). While these service providers may deliver education and training under contract to participants in workforce development and welfare-to-work program, the co-authors find this a useful term to distinguish services originally designed to be taken in a traditional linear fashion as opposed to "second chance" programs designed primarily to remediate and/or "retool" non-traditional students. (See also Second Chance Programs.)

Focus Group

Panel of experts convened for a highly structured discussion of issues. (See Delphi Technique.)

Follow-Up

The collection and analysis of data on the outcomes achieved by a program's former participants.

Follow-Up, Additional

As used herein, part of the employer follow-up survey process; viz. persistence efforts by the surveyor to get employers to clarify their responses. (See also Persistence Effort.)

Follow-Up, Automated

Arrangements for using electronic record linkage techniques as the primary method of gathering outcomes data. Most states probably will use a hybrid system that couples record linkages (as the primary data gathering method) with one or more traditional surveys to fill in gaps with respect to locating additional subjects and/or gathering information on additional variables. Note that automated follow-up is not necessarily centralized or integrated, although those two features tend to go hand-in-hand with automation to realize economies of scale and comparability of data more fully.

Follow-Up, Auxillary

Record linkages to databases other than the state's Unemployment Insurance wage records and master enrollment files for public postsecondary institutions. A distinction is made between primary follow-up and auxiliary follow-up because the former usually is done for and entire cohort while the later may be done in some states only on an exceptions basis. (See also Follow-Up, Primary and Exceptions.)

Follow-Up, Centralized

Arrangements whereby a single entity in a state is responsible for gathering outcomes data for all federally-funded/state-administered employment and training programs. Centralization of the data gathering process does not necessarily mean that follow-up for all programs studied by the lead agency is integrated. The central entity may be divided into divisions or "silos" with each conducting follow-up for separate sets of programs - perhaps using different outcome measures and data collection methodologies. Centralized follow-up is not necessarily automated although all three attributes tend to go hand-in-hand.

Follow-Up, Conventional or Traditional

Herein used to refer to post-program surveys which relied on the self-reported behaviors and outcomes of former program participants (as in alumni surveys and JTPA 13th week follow-up) as opposed to efforts which rely primarily on automated record linkages; used erein as interchangeable with "traditional follow-up."

Follow-Up, Integrated

Arrangements for using common operational definitions of outcome measures and a standard data collection methodology across all employment and training programs in a particular state. Note that integrated follow-up is not necessarily automated or centralized.

Follow-Up, Non-traditional

A process of gathering outcomes data that relies predominately on automated record linkages rather than on conventional participant-contact surveys. (See also Follow-Up, Automated.)

Follow-Up, Primary

The use of electronic linkages to public administration databases by a state's lead agency to gather the vast majority of information about the outcomes achieved by subjects in the exit cohorts being studied. In most states, linkages to the Unemployment Insurance wage records and master public postsecondary enrollment files will suffice to locate the vast majority of subjects in any exit cohort. Therefore, primary linkages are used for an entire cohort while auxiliary linkages may be used on an exceptions basis to document results for those not located using primary linkages. (See also Follow-Up, Auxiliary and Follow-Up, Supplemental as well as Exceptions.)

Follow-Up, Supplemental

An effort by parties other than the central follow-up entity to collect outcomes data --usually via traditional methods -- on former students and program participants not located by the central entity through automated record linkages. (See also Exceptions.)

Follow-Up Entity

The agency or subagency unit which facilitates record linkages for the purpose of gathering information about the outcomes achieved by former participants -- preferably distinct from service pro-viders and the agencies or subagency units responsible for either fiscal or operational administration of service delivery. (See also Central Follow-Up Entity and Lead Agency.)

Frame of Reference

An appropriate context for interpreting follow-up data. Putting follow-up data in perspective usually consists of comparing the results achieved by one set of program exiters to external benchmarks or to internal baselines. (See also Benchmarking and Baselining.)

Freedom of Information Act

Federal law under which citizens or the media can request that public entities disclose information in their files; generally at odds with the Data Privacy Act as amended by FERPA. (See also Buckley Amendment, Data Privacy Act, and FERPA.)

FSE&T

Food Stamp Employment & Training: work-first programs to promote the economic self-sufficiency and reduce the welfare dependency of Food Stamp recipients.

Gaming

The behavior or strategies of program administrators or service providers to achieve performance standards through means that do not necessarily provide intended benefits to customers or clients. Includes but is not limited to creaming. (See also Creaming.)

Gantt Chart

In project management, an approach named for its developer, H.L. Gantt. Noted for is capacity to show project activities graphically across a time scale, track them, manage them and print periodic reports on progress and resource consumption.

GAO

General Accounting Office: reports to Congress on the fiscal propriety and cost-effectiveness of federally-funded programs as well as the potential cost-effectiveness of proposed federal programs. (Individual states may have equivalent bodies called Legislative Budget Bureaus or Boards.)

GIS

Geographical Information Systems: combination of software and database structure in which each record contains location information coded in such a way (i.e., geo-coded) that facilitate displaying information on data maps.

Geo-coding

Inclusion of a location field in unit records that can be related to a table of longitude and latitude coordinates to facilitate depicting information on a map.

Goals

Broad statements generally describing a desired outcome for an entity and its programs. (See also Mission, Objectives, and Performance Measures.)

GoTR

Government Technical Representative: a staff person with the DoL national or regional office responsible for managing specific grants and for helping grant recipients: meet the technical specifications of their contractual obligations; complete all necessary paperwork; disseminate project information to interested stakeholders; and close-out projects properly. GoTRs also may assist policy-makers in the DoL in brainstorming service delivery and product development strategies and in evaluating related grant proposals. In conjunction with follow-up, a GoTR would be a key resource person and liaison between a state recipient of demonstration and capacity-building funds and experienced practitioners in other states or in the federal government.

GPRA

Government Performance Reporting Act.

Guidance Letters

More detailed specifications, suggestions and examples issued by the DoL national or regional offices to help state agencies comply with Training and Employment Information Notices. (See also TEIN.)

HEGIS

Higher Education General Information Survey: an old taxonomy for coding programs and courses offered by postsecondary institutions -- now less commonly used than the CIP.

Heuristic Value

In follow-up studies, the degree to which a set of outcomes data has the capacity to lead stakeholders to useful insights about the performance of an employment and training program, service or activity; i.e., the information-conveying capacity or understanding-enhancement capabilities of pieces of data, reports, or presentation formats.

Hit

A successful linkage between an individual's record in one file and that same individual's record in another file.

Hit ratio

The percentage of subjects in an exit cohort for whom outcomes could be documented through electronic record linkages.

HRIC

Human Resource Investment Council: in states moving toward consolidated strategic planning for employment and training programs, an advisory board that may exercise functions formerly mandated under federal law for the State Job Training Coordinating Council, State Council on Literacy and Adult Education, State Council on Vocational Education, etc.

Indicator

In empirical research, labels for properties that are observed directly and which can take on different values. (See also Constructs, Values, and Variables.)

In-Family Use

Any use of data that conforms to the spirit of the law authorizing collection of the data even though the specific use is not identified expressly in the letter of the law or attendant regulations. For example, because one aim of follow-up is to improve the match between the supply of appropriately trained workers and occupational employment demand, linkages to the UI wage records would be an in-family use of those records given the spirit of the Wagner-Pyser Act and the intent of most states' Unemployment Compensation Acts.

Information Delivery Subsystem

In this Guide, the operations which comprise the feedback loop for the employment and training system.

Input

Information about the background and services received by the subject of a follow-up study stored electronically in a format that conforms to specifications that facilitate linkages with external files likely to contain information about the post-program outcomes achieved by that subject. (See also Seed Record.)

Input (Definition #1)

Characteristics of subjects antecedent to or at the time they enter a program offered by a service provider; inputs also may be used to label the resources at the provider's disposal and constraints on the delivery of services -- commonly factors which service providers can measure for themselves without requiring the assistance of an external follow-up entity.

Inputs (Definition #2)

In Systems Theory, the demands (stresses and disturbances), supports (expectations and resources), and processed reactions (feedback) from the external environment which prompt a system to react.

Intervention

A term used generically in empirical research methods to label the purposeful treatment of or ser-vices provided to individuals or members of a group. The mission of follow-up often is to deter-mine if subjects receiving a particular treatment or service achieved intended results.

IPEDS

Integrated Postsecondary Education Data System: a system managed by the National Center for Education Statistics for collecting common data elements from institutions of higher education.

ISD

Independent School District.

ITSC

Information Technology Support Center; designed and operates the WRIS under contract to DoL. (See also WRIS.)

JCL

Job Control Language: a more cumbersome and cryptic precursor of SQLs found in pre-fourth generation programming languages and statistical application software -- but not standardized from one programming language or application software package to another. (See also SQL.)

Job Development

Efforts to get employers to post notices of employment opportunities with the SESA's labor exchange and/or on publicly-supported "real-time" forums such as the America's Job Bank. Differs from economic development in the sense that it presumes that job openings already exist but are not being posted with the SESA or on other publicly-supported forums. (See Economic Development.)

Job-Matching and Referral

Assessing the fit between an applicant's qualifications and the requirements of a job posted with the state's employment security agency through case management and the subsequent efforts to schedule an interview for a suitable applicant with a prospective employer. Now that America's Job Bank and America's Talent Bank are operational, much of this can be done electronically as both the range of job openings and the pool of talent expanded to a national scale.

Job-Search Assistance

Instruction provided to those seeking employment on where to look for job postings, how to network with others to increase access to information about job openings that have not been posted in common forums, how to write a resume, how to fill out a job application, and how to conduct oneself in an interview, etc.

JOBS

Employment and training programs provided to promote the economic self-sufficiency and reduce the welfare dependency of AFDC/TANF recipients under the Job Opportunities and Basic Skills Act.

JSEC

Job Service Employer Committee: local committees which advise the state and substate Job Service operations about employers' needs and concerns regarding labor market conditions.

JTPA

Job Training Partnership Act as amended in 1994 in Public Law 97-300.

KSAs

Knowledge, Skills and Abilities: what it takes to perform successfully the duties and tasks associated with a particular job or occupation. Currently the focus of research under the National Skills Standards Board and its state-level counterparts; also integral to the content model of the O*NET. (See also NSSB and O*NET.)

Labor Exchange Subsystem

Activities by partner entities to get employers to post job openings with the state's employment seurity agency or publicly supported electronic forums (such as America's Job Bank) and to encourage job-seekers to use services such as individual job matching and referral as well as publicly supported electronic forums (such as the Talent Bank).

Lag Time

In project management, a delay between tasks that have a dependency. Usually expressed as a percentage of latitude in a project schedule for delaying the start of a successor task without affecting the on-time completion of other tasks -- particularly those in its critical path. (See also Lead Time and Slack Time.)

LAR

Legislative Appropriations Request: usually contains a budget, an explanation of proposed services and the benefits the public will derive therefrom. For a state's central follow-up entity, the LAR may be a stand-alone request or a rider attached to other pieces of legislation.

LEA

Local Education Agency: an entity such as an independent school district responsible for administering public education (K-12) in the community; usually the direct recipient or subgrant recipient of federal education and training dollars. (See also CEA.)

Lead Agency

Insofar as the administration and delivery of employment and training services may be scattered divided across several state agencies, the one which is assigned principle responsibilities for centralizing follow-up is herein designated the lead agency will the remaining ones are lumped together as partner agencies. (See also Central Follow-Up Entity, Partner Agency and Stakeholder.)

Lead Time

In project management, an overlap between tasks that have a dependency. Usually expressed in a percentage of latitude in a project schedule for starting one task before its predecessor is finished; viz., permissible head start. (See also Lag Time and Slack Time.)

Leavers

In follow-up studies, members of an exit cohort including those who: a) completed an employment and training program; b) terminated voluntarily; c) transferred to another program for services; or d) were involuntarily terminated. (See also Completers, Drop-Outs, Stop-Outs, and Terminees.)

Legacy Systems

The variety of hardware, software, and operating platforms comprising the various information management systems of the partner agencies participating in automated follow-up and which preceded the creation of a state's central follow-up entity. (See also MIS.)

Life-Skills Training

Instruction on matters outside the workplace that may impact a person's ability to achieve economic security, personal satisfaction, and capacity to get a job or hold on to it (e.g., how to prepare and stick to a household budget).

LMI

Labor Market Information: may be used generically to refer to employment demand and supply information or more specifically to a governmental body that analyzes such data and/or to an automated system for delivering those data to varied customers.

Longitudinal Design

Research conducted on the same subjects at two or more points in order to assess changes in their behaviors, attitudes, experiences, or achievements over time. In e mployment and training follow-up, longitudinal designs are used to assess pre-service/post-exit changes and delayed or long-term program outcomes such as learning gains, earnings gains, and employment retention. (See also Snap-Shot.)

LWDB or LWFDB

Local Workforce Development Boards responsible for the strategic planning and evaluation of the JTPA and other employment and training programs at the substate level. Under the Human Resource Investment/One-Stop approach to employment and training, LWFDBs have, by and large, replaced the PICs. (See also HRIC, PIC and SDA.)

Matching

As defined by the Computer Matching and Privacy Protection Act of 1988, the electronic linking of administrative databases for the purpose of determining how to treat or serve a particular individual as opposed to linking done for statistical purposes. (See also Record Linkage.)

MDTA

Manpower Development and Training Act; federal workforce development program that preceded CETA and JTPA.

Micro-Matrix

A product of joint federal state efforts to identify occupational staffing-patterns down to the lowest level of business/industry sector of the SIC taxonomy.

Milestone

In project management, a reference point -- usually a task with zero duration (i.e., a significant event such as a begin date or end date).

MIS

Management Information System: the hardware, software, database structure, the data therein, and the rules used by the responsible entity for operating, maintaining and securing all the above. (See also Legacy Systems.)

Mission

A concise statement of the unique, fundamental current and future purpose of an entity and its pro-rams. (See also Goals, Objectives, and Performance Measures.)

MOS

Military Occupational Specialty.

MTTR

Mean Time To Repair a defect; a concept take from quality control in software development and applied to error detection, capture and repair in follow-up.

Multiple Indicators

Using several measures or observational techniques to record and classify phenomena. (See also Validity, Convergent and Triangulation.)

N

In standard statistical notation, number of observations or subjects (usually written in upper case).

NAFTA

North American Free Trade Agreement. As used in this Guide deals primarily with dislocated workers entitled to employment and training services under TAA as amended by the NAFTA Transition Assistance Act. (See TAA.)

NAICS

North American Industrial Classification System -- a proposed replacement for the Standard Industrial Classification. (See also SIC.)

NALS

National Adult Literacy Survey.

NCES

National Center for Education Statistics: a unit within the Office of Education Research and Improvement of the United States Department of Education. (See also IPEDS, NPEC, and VEDS.)

NCRVE

National Center for Research on Vocational Education located at the University of California - Berkeley.

Nearest Competitor

Closest entity geographically or the entity most likely to attempt to enroll/recruit/provide services to the same potential customers/clients. (See also Benchmarking and Peers.)

NEC

Not Elsewhere Classified: as used herein, a catch-all designation within OEWS occupational clusters often having an OEWS code ending in 99. May also be a catch-all designation in other coding sys-tems as in the DOT and CIP taxonomies.

New Hires

For measurement purposes, an individual whose Social Security number did not appear on the wage records submitted by a particular employer in Q_N but who does appear in the wage records submitted by that employer in Q_N+1.

NGA

National Governors' Association.

NLA

National Literacy Act of 1991 which expanded services and added performance measure requirements to the Adult Education Act of 1966 (AEA).

NOICC

National Occupational Information Coordinating Committee.

Non-Filers

Employers who failed to submit required quarterly reports on the earning of their employees that are covered under their state's Unemployment Compensation Act.

Non-Traditional Students

Individuals pursuing education and training but who have not taken the traditional linear path at a pace commensurate with "normal academic progress" and without interruption from kindergarten to their highest level of educational attainment. Term also is used to describe those who do not fit the typical demographic profile of the client mix served by a particular provider.

Norm-Referenced

Assessment instruments where a passing score is not set at a fixed point but rather is set relative to te performance of others assessed with the same instrument -- as in grading on a curve. (See also Criterion-Referenced.)

NPEC

National Postsecondary Education Collaborative: a representative body of state delegates and national leaders formed under the auspices of the National Center for Education Statistics to assess needs of postsecondary educational institutions to gather and report data elements to address com-mon external demands for accountability.

NSSB

National Skills Standards Board established under Goals 2000 Educate America Act to ratify an industry-by-industry methodology for employer validation of the KSAs they expect their employees to have for each occupational cluster therein. (See also KSAs.)

Objectives

Measurable statements about the results that a service or program is expected to accomplish in a given period of time. (See also Mission, Goals, and Performance Measures.)

OES

Occupational Employment Statistic: an occupational classification system based on annual surveys conducted jointly by the state employment security agencies and the Department of Labor; having the advantage over other occupational employment taxonomies because it includes current and pro-jected employment figures for each title.

Off-Line

Data storage that is either separated physically from a computer or not immediately accessible bythe computer's central processor unit; e.g., a backup file or archived data stored in a locked cabinet.

OIG

Office of Inspector General: a unit within a federal agency responsible for monitoring regulatory compliance and, in some agencies, with increasing responsibilities for monitoring performance outcomes.

OIS

Occupational Information System: a set of automated tools that program administrators and planners can use to digest and understand occupational employment demand and education and training supply information to guide them in planning the delivery of services. While several states have developed their own automated labor market planning tools, OIS most commonly is used in reference to coordinated federal-state efforts through the NOICC-SOICC network to standardize, auto-mate and disseminate a set of tools pursuant to §464(b)(2) of JTPA.

Micro-OIS

A micro-computer based Occupational Information System developed by the NOICC in collaboration with several state OICCs. The micro-OIS consists of standard data elements in a standardized file structure (the OLMID) and an application layer designed for use by program administrators and planners. (See also OLMID.)

OJT

On-the-Job Training: training done in a workplace setting or simulated environment rather than in a classroom -- may connote that the subject's employment is subsidized during the training period, that the person doing the instruction is a practitioner rather than a teacher, and that there may be an expectation of unsubsidized employment after the training period ends.

OLMID

Occupational Labor Market Information Database: the standardized file structure and data elements that serve as the foundation for the micro-OIS distributed by the NOICC. (See also micro-OIS.)

OMB

Office of Management and Budget: a federal executive office.

One-Stop

A federal initiative to house employment and training services at a single location to provide customers easier access, to improve articulation among the programs, and to streamline their adminis-tration; hence a name used to describe such centers, the tools and resources therein, and the subgrants award to entities developing tools and service strategies.

O*NET

Occupational Network: an on-line replacement for the Dictionary of Occupational Titles. The O*NET, however, differs from the manual DOT by consolidating several occupational taxonomies into a single taxonomy while also expanding and enriching the underlying content model through the addition of more data fields to describe occupational employment attributes that are important to employers, job-seekers, educators, and trainers. (See also DOT.)

On-Line

Data stored for immediate access by a computer's central processor unit; commonly the most recent files are stored on-line while older data sets tend to be archived. (See also Archive and Off-Line.)

OPM

Office of Personnel Management: office responsible for administering the federal civil service; a source of labor market outcomes typically not covered in a state's Unemployment Insurance wage records.

Outcomes

What happened to subjects after services were provided -- broadly conceptualized to also include the impacts, payoffs, or returns on the investment made in service delivery. Generally speaking, outcomes are something in which the subject takes an active role and for which the subject has some modicum of responsibility for achieving. (See also Inputs and Outputs.)

Outcomes, Gross

The the level of post-exit achievements obtained by an employment and training program cohort - not adjusted to take into account the pre-service status of individuals in the cohort.

Outcomes, Net

The difference between the values obtained for a variable when measured before and after subjects participated in an employment and training program. (See also Value-Added.)

Outliers

An exceptionally high or exceptionally low value for a variable; an extraordinary observation. (See also Anomaly.)

Output Record

Information about the post-exit outcomes achieved by former program participants obtained through record linkages when appended to the participants' original seed records; also known as an enhanced record.

Outputs (Definition #1)

Attributes or characteristics of subjects at the point when they exit a program or when services are terminated -- commonly factors which service providers can measure for themselves without requiring the assistance of an external follow-up entity.

Outputs (Definition #2)

In Systems Theory, the actions and decisions of a system in response to demands (stresses and disturbances) and supports (expectations and resources) emanating from its environment.

p <

In standard statistical notation, probability that data support the null hypothesis is less than the trailing value. (Written in lower case.)

Parameter

A feature of software or report generating templates that facilitates subsequent reuse by allowing substitution or even prompting end-users to input values or specifications on key variables -- as in specifying the beginning date and end date of a school year when prompted on screen in an application designed to extract seed records for secondary or postsecondary education and training programs' exit cohorts.

Participant

An individual who was provided a service or who received a treatment or intervention; commonly the subject of follow-up studies after the service, treatment, or intervention is terminated as in former participant; may also be known as client, student, inmate, etc. depending on the nature of the program in which the individual participated.

Participant Contact

As used in this Guide, refers to traditional methods of gathering data that necessitate locating persons who have received services, asking them about their experiences and the out-comes they achieved, and accepting their responses as truthful and accurate.

Partner Agency

Any entity that is provided services by the central follow-up entity or which provides either seed records or outcomes data resources to the central follow-up entity. (See also Lead Agency and Stakeholder.)

(PB)²

Performance-Based Program Budgeting: an initiative in Florida to base an ever growing share of program budgeting for all government sectors on outcomes and outputs; intended to counterbalance accountability and budgeting issues. (See also Enrollment-Driven, Performance-Driven and ROI.)

Peers

For the purpose of putting follow-up data into perspective, entities having a comparable mission, size or caseload, admissions/eligibility criteria or customer mix, or expenditure level. (See also Benchmarking and Nearest Competitor.)

Pending

The status assigned to a record for which additional inquiries or statistical manipulations must be done in order to assign a value to one of its variables or fields. (See also Exceptions, Residual Titles, and Resolution.)

Performance

The relative success of a program, service, intervention, or activity in achieving desired results; often conceptualized as exhibiting degrees of effectiveness and/or efficiency.

Performance

The relative success of a program, service, intervention, or activity in achieving desired results; often conceptualized as exhibiting degrees of effectiveness and/or efficiency.

Performance-Based Contract

An arrangement whereby the vendor of a service agrees in advance to a minimum level of achievements by the persons served; often includes a provision for withholding some portion of budgeted dollars after contract period is closed with final payment made upon receipt of documentation that contractually-specified outcomes were achieved. Before entering into such arrangements, a service provider may have to supply proof of adequate performance in some prior base period.

Performance-Driven

Arrangements for allocating funds to service providers in one program year based wholly or in part on the outcomes achieved by the participants served during some prior program year or base period. (See also Enrollment-Driven.)

Performance Measures

On-going, quantitative indicators of the extent to which objectives are being achieved. (See also Mission, Goals, and Objectives.)

Performance Measure, Core

A performance measure that is applied universally and defined consistently across all programs in a state's employment and training system; e.g., post-exit employment.

Performance Measure, Program-Specific

A performance measure that is applied to all like programs but not universally across all programs comprising the employment and training system. Training-relatedness, for example, might be a performance measure applied to all occupationally-specific training programs but which would not be applied to academic-transfer or basic education programs.

Performance Measure, Provider-Specific

A performance measure that is used by a specific service provider for on-going program management requirements and diagnostics but which might not applied to all like programs or universally to all components of the employment and training system.

Performance Measure, Tiers

A way of conceptualizing the distinction and hierarchical relationship among core performance measures (Tier I), program-specific performance measures (Tier II), and performance measures necessary for on-going program management require-ments and diagnostics (Tier III).

Performance Standard

A minimum level of achievement for a program or service provider as established by an authoritative body or by contract.

Perkins Act

Carl D. Perkins Vocational Education Act (Public Law 98-524) as amended by Public Law 100-392 became the Carl D. Perkins Vocational and Applied Technology Education Act; both are referenced rather interchangeably as the Perkins Act.

Persistence Effort

The use of resources by an entity engaged in survey research to obtain the minimally acceptable response rate as opposed to seeking clarification of responses already received. (See also Cohort Coverage, Employer Response Rate, and Follow-Up, Additional.)

Personal Responsibility Work Opportunities Reconciliation Act

At the federal level, welfare reform legislation that replaced AFDC with TANF and introduced the "work first" model. Noted for establishing a time limit on both current receipt of public assistance and receipt of public assistance over a lifetime. Also noteworthy because it sets upper limits on the duration of public assistance as a matter of federal policy but allows the states greater latitude in setting tougher time restrictions. (See also AFDC, TANF and Work-First.)

PERT

Program Evaluation Review Technique for project management developed by Lockheed when it served as prime contractor on Polaris missile project for US Navy. This technique uses statistical probabilities to forecast project duration. It is used widely because of its capacity to display task relationships visually in a network diagram.

PIC

Private Industry Council: responsible for strategic planning and evaluation of JTPA programs at the substate level; by and large, PICs are being replaced by Local Workforce Development Boards (LWFDBs) as states consolidate administration of employment and training programs under Human Resource Investment Councils and the One-Stop initiative. (See also LWFDB.)

Placement Verification System

Terminology used to describe a follow-up entity and its activities under education, training and workforce development program consolidation proposals introduced during the 104^th and 105^th Congresses chiefly by William Goodling in the House of Representatives and, before her retirement, by Nancy Kassenbaum in the Senate.

Planning

Formulating a course of action in orderly fashion to achieve desired goals and objectives at some point in the future.

Planning, Operational

Deciding how to deliver services to eligible customer groups in the next program year or current fiscal cycle (i.e., planning into the future only so far as the timeframe for which a budget is known or for which funds in theory have been authorized even if not yet appropriated).

Planning, Strategic

Anticipating who will need services and what kind of services they will need at some point in the future beyond the current program year or budget cycle and deciding how to meet those needs.

Planning Subsystem

In this Guide, the conversion process through which decision-makers in the employment and training system receive, evaluate, and respond to demands and supports emanating from private citizens and organized stakeholder groups in the external environment.

Processes

The actual services, treatments, or interventions and how they were delivered -- commonly factors which service providers can measure for themselves without requiring the assistance of an external follow-up entity. To some extent, process measures have, in the past, been taken as empirical indi-cators of the quality of services provided.

Proxy Measure

In the absence of data elements best suited as an indicator of some construct, a substitute indicator consisting of available data which comes closest to measuring the construct. (See also Construct, and Indicator.)

Public Education

Elementary and secondary education (K-12) provided at public expense as distinguished from private education at any level and from publicly-funded education and training at the postsecondary level.

Purposive Sample

A non-random sample used in proof-of-concept efforts and exploratory studies to focus attention on a particular problem or to generate testable hypotheses rather than to make inferences about a larger population.

Q_N

Reporting Quarter of the Unemployment Insurance wage records tapped for a particular piece of data where: Q₀ represents the service delivery exit quarter; a negative subscript (Q_-N) represents the Nth full quarter preceding enrollment in an employment and training program; and a positive subscript (Q_+N) represents the Nth full quarter after program exit.

R&A Unit

Research and Analysis Unit: usually a division within the state's employment security agency; may also be known as the state's labor market information - or LMI - unit. (See also LMI and SESA.)

Real-Time

In the context of follow-up, electronically capturing transactions between job-seekers and employers as they occur and instantaneously updating supply and demand data in labor market information systems.

Record

Information about a specific individual, event or activity stored as hardcopy or on electronic media in a standard format.

Record Linkage

Connecting individual records from two data sets electronically by matching them on a unique identifier (usually the Social Security number) common to both sets of data.

Redress

For the purpose of this Guide, the actions, decisions, and mechanisms in program accountability for compelling providers to either discontinue a service or revise its delivery in order to meet or exceed performance standards; analogous to preventing or seeking repayments for illegal procurements detected through a fiscal audit or the tightening of procurement procedures and procedural guidelines to combat wasteful and inefficient practices uncovered by a management audit.

Referral

Sending a customer or client elsewhere for services. In a One-Stop setting, the transfer of an individual's information to one or more appropriate service providers after going through intake, eligibility screening, and assessment. In the labor exchange, an arrangement by a case manager to send a job-seeker to apply for an opening posted with the SESA. (See also Job-Matching and Referral.)

Reliability

As used among persons doing empirical research: when the application of measurement rules results in consistent and stable results. (See also Validity.)

Reliability, Inter-Coder

The degree of consistency between two or more researchers or data entry clerks as they apply the measurement rules of a particular research design to assign a value to comparable observations. As oppose to Intra-Coder Reliability, this usually is a question of how clear and unambiguous the measurement rules are.

Reliability, Intra-Coder

The likelihood that an individual researcher or data entry clerk will assign the same value consistently assigning a value to comparable observations encountered at different times. As opposed to Inter-Coder Reliability, this usually is a question of how well the researcher or data entry clerk has been trained regarding the application of the research design's mea-surement rules.

Repository

Library of automated subroutine, boilerplates and report formats, systematically cataloged and indexed for easy retrieval and reuse. (See also Reusability.)

Residual Titles

Employer-provided occupational titles (known as lay titles or payroll titles) that can't be walked to a standard taxonomy without additional research to determine the duties and tasks performed.

Resolution

Assigning a value to a variable or field when additional inquiries or observations provide sufficient information to apply an operational definition. (See also Exception, Pending File, and Follow-Up, Additional.)

Response

Answer returned for a mail survey or replies to telephone survey that are sufficiently complete.

Response Analysis

Additional research used to determine if there are statistically significant differences in the characteristics of those providing sufficiently complete answers to a mail or telephone survey versus those who did not reply, refused to answer, or provided incomplete answers.

Response-Set Bias

Systematic sources of error in data collection based on the differential probabilities that subgroups within the sample or universe being studied will respond to follow-up surveys. (If the degree and source of response-set bias can be determined, statistical adjustments can be made to affected data sets. However, the dimensions of response-set bias often are either not investigated or -- relative to the required confidence level and need for precision in a particular study -- would be too costly to determine.)

Results

Outcomes experienced rather passively or without purposeful action by a subject after receiving a service or intervention; i.e., outcomes over which the subject has little or no control and for which the subject may not be held personally accountable. (See also Outcomes and Outputs.)

Reusability

The characteristic of modularized software, report templates, and wordprocessing boilerplates that are designed intentionally from the outset to minimize the need for modification if retrieved and applied at a later date to meet anticipated needs likely to arise under circumstances similar to those surrounding the item's initial creation. (See also Parameter Substitution and Repository.)

Reverse Linkage

The process of harvesting additional explanatory variables from a partner agency or service provider's management information system once it has received enhancements of its seed records from a central follow-up entity. (See also Performance Measures, Provider-Specific.)

Reverse Matrix

When constructing a crosswalk between two variables, the data element arrayed on the vertical axis is usually the one which serves as the point-of-entry for lookup purposes (i.e., the "known" or "given" in a lookup routine). For example, an educator may want to enter a matrix used to determine the training-relatedness of post-exit employment situations to look up occupational titles related to the programs he or she teaches or administers. In that case, the matrix would be arrayed with CIP codes on the vertical axis and OEWS codes on the horizontal. A reverse matrix arrays the variables in just the opposite fashion. In the example above, the matrix used to determine training-relatedness could have the OEWS codes on the vertical axis and the CIP codes on the horizontal to facilitate review by employers who are more accustomed to working with occupational titles than they are with the names of program offerings. Herein, reverse matrix may be used more specifically in conjunction with the CIP-to-OES/OES-to-CIP crosswalk for determining training-relatedness and the SIC-to-OES/OES-to-SIC crosswalks used in analyzing industrial staffing-patterns and the distribution of occupational employment.

Right-to-Know

The principle expressed in the Student Right-to-Know and Campus Security Act of 1994 asserting that prospective students are entitled to sufficient relevant data to make informed choices regarding their selection of an education and training provider. In this Guide, the co-authors construe the right-to-know broadly to include the rights of participants in all employment and training programs to access meaningful and comparable data relevant to the services being provided.

ROI

Return On Investment: a phrase borrowed from banking and finance; when applied in an outcomes based accountability context, it refers to the ratio of benefits enjoyed by taxpayers relative to the cost of a program funded with their tax dollars or the benefits achieved by former participants relative to the time, effort, and resources they expended. (See also Const-Consequences Analysis.)

RTW

Return To Work: a desired outcome for Workers' Compensation and Rehabilitation programs.

SACS

Southern Association of Colleges and Schools: the regional body which accredits education and training institutions in several states, including Texas and Florida. Other regional accrediting bodies have comparable rules that require program evaluation based in part on labor market outcomes.

SDA

Service Delivery Area: the entity administering JTPA programs at the substate level or the geographic territory in which JTPA services are provided; under Title III of JTPA may be known as a Sub-state Area. (See also LWFDB and PIC.)

Second Chance Programs

Programs made available to eligible individuals who, because of economic or educational disadvantages, find that they need additional education and training to compete in the labor market to achieve economic self-sufficiency as contrasted to "first chance" programs where learners make progress in a linear or "age-appropriate" fashion through the traditional sequence of elementary, secondary, and postsecondary education and training. (See also First Chance Programs.)

Seed Record

Information about the background and services received by the subject of a follow-up study stored electronically in a format that conforms to specifications that facilitate linkages with external files likely to contain information about the post-program outcomes achieved by that subject; also known as input record.

Selective Perception

The act (deliberate or unconscious) of ignoring evidence that does not support one's foregone conclusion or citing favorable findings out of context or in ways that overstate their importance and relevance.

Service Provider

An entity that renders a treatment or intervention.

Service-Eligible Population

A group whose members, according to authorizing legislation or regulation, are entitled to receive a benefit or service. (See also Equity of Access, Special Pops, and Target Populations.)

SESA

State Employment Security Agency: usually the entity responsible in each state for administering Unemployment Insurance benefits, collecting employer payroll tax contributions, and operating a publicly-supported labor exchange; may also house a research and statistics unit responsible in the state for joint federal-state initiatives to gather, analyze and disseminate labor market information; may be responsible for administering several employment and training programs within the state. Note that the names of the particular agencies serving as their state's SESA will vary (e.g., Texas Workforce Commission, Florida Department of Labor and Employment Security, etc.)

SHEEO

State Higher Education Executive Officers (see also CCSSO and ESC).

SIC

Standard Industrial Classification: a hierarchical coding system to identify firms according to the products and services they provide. (Eventually will give way to the NAICS.)

Simulation

A process of forecasting alternative futures by inserting high, median and low parameters on change variables into empirical models that have been validated post hoc. (See also Validation, Post-Hoc.)

SIS

Oregon's Shared Information System.

SJTCC

State Job Training Coordinating Council: state-level advisory board which makes policy recommendations regarding federally-funded/state-administered employment and training programs. In states moving toward integrated strategic planning, federally-mandated SJTCC functions may now be delegated to a Human Resource Investment Council. (See also HRIC and COVE.)

Slack Time

In project management, the amount of time a task can slip before it affects another task's projected finish date. (See also Lag Time and Lead Time.)

Slippage

In project management, the amount of time a task has been delayed from its original baseline plan. (See also Expectation Slippage.)

Snap-Shot

A type of research design that gathers data about an exit cohort at a single point in time; while adequate for several purposes, snap-shot studies cannot measure change over time. (See also Longi-tudinal Design.)

SOC

Standard Occupational Code: soon to be rolled into the O*NET data delivery system. (See also O*NET.)

SOICC

State Occupational Information Coordinating Committee, now called CDR (Career Development Resources). See CDR.

SOW

Statement Of Work: a document attached to a contract, interagency agreement, or project budget that includes specified deliverables, a timetable for their delivery, and a narrative explaining how re-sources will be used to meet those obligations in a timely and competent fashion.

Special Pops

Special Populations: subpopulations targeted for delivery of a specific service; e.g., individuals with limited English proficiency, physical disabilities, or in need of bilingual education, etc. as defined under authorizing legislation or pertinent regulations. Most commonly used in conjunction with the Perkins Act. (See also Equity of Access, Perkins Act, Service-Eligible Population, and Targeted Population.)

SPIR

Standard Program Information Report: a database containing uniform information submitted by every state on their JTPA participants; it is used for program management purposes and for compiling annual "year-end" performance reports to the Secretary of Labor. The SPIR replaces the separate reports for JTPA Title IIA (called the JTPA Annual Status Report -- or JASR) and for Title III (called the Worker Adjustment Annual Program Report or WAPR).

SQL

Structured Query Language: a standard way used in 4th-generation or newer database management and some statistical application packages for retrieving and manipulating data. (See also JCL.)

SSN

Social Security number: a record identifier used by the Social Security Administration; often used by other entities as the unique personal identifier for information management purposes.

Stakeholder

A person or group having a financial interest in or fiduciary responsibility for some aspect of the employment and training system. While stakeholders are among a follow-up entity's customers, not all customers are stakeholders in the sense that the interest of the later may be less formal and more sporadic. Stakeholders are a subset of customers whose involvement includes more than merely the use of services and final products; e.g. stakeholders are involved more actively in the follow-up entity's on-going processes and the production of deliverables. Partner agencies aare stakeholders but not all stakeholders are partner agencies. An advocacy group, for example, might not interact directly with employment and training programs although they have a sustained interest in them because they represent affected subpopulations of the taxpayers who foot the bills. Ergo, an advocacy group would be a stakeholder but not necessarily a partner. A partner agency contributes seed records, outcomes-enhanced output files, and/or contectual data. (See also Customer and Partner Agency.)

Standard Taxonomy Approach

The practice of providing employers a list of occupational titles common to their industry and asking them to translate the titles they use in their in-house payroll and personnel systems into titles from the lists provided. (See also Employer-Generated Title Approach.)

STI

State Training Inventory: a standardized file structure that lists the fields of study that may be pursued at each education and training site in a state; part of the OLMID. Also a natural place to link a CRS to a CIDS. (See also CIDS, CRS, and OLMID.)

Stop-Out

A participant who voluntarily withdraws from a program but who expresses an intent to complete that program at a later date. (See also Drop-Out and Terminee.)

S-T-W

School-To-Work: activities funded under the School-to-Work Opportunities Act.

SVPT

Specific Vocational Preparation Time: a coding system used in the Dictionary of Occupational Titles to rate the average amount of instruction required to perform the work under any given occu-pational title.

Subpopulation

As used in this Guide, a group whose shared demographic characteristics are antecedent to or independent of the treatment, intervention or services they received. Subpopulation characteristics often are used as control variables in the analysis of a program's impact to help determine if a service is equally effective for all participants; e.g., for males and females alike. (See also Special Pops.)

Subsystem

Units with distinct boundaries within a larger system performing specific functions; usually having sufficient visibility to be identified by actors in a system's external environment.

Support

In Systems Theory, the resources which give a system the capacity to respond to demands from its external environment. Supports may be negative in the sense that they diminish system capacity; e.g., public cynicism and distrust of government would be "negative" supports. (See also Demands, Feedback, and Inputs.)

System

A set of actors perceived to be working together and the processes they use in applying resources to address demands and expectations emanating from their external environment.

System Building

Establishing an entity that will survive beyond its first funding period and/or the tenure of its founding cadre or creating a permanent process that addresses a myriad of similar, recurring but transi-tory issues. Also may be called "institutionalizing."

Systems Theory

An approach to the analysis of purposeful human behavior which examines the way actors process external demands and supports to arrive at decisions and actions and the feedback mechanisms through which reactions by external parties to one round of outputs are transformed into new de-mands and supports requiring new decisions and/or actions.

TAA

Trade Adjustment Act. As used in this Guide, in reference to assistance given to workers whose employment was adversely effected by trade policies that force American employers out of business or lead them to locate off-shore.

TANF

Temporary Assistance to Needy Families: a form of public assistance that replaced AFDC under federal welfare reform. (See also AFDC and Personal Responsibility and Work Opportunities Re-conciliation Act.)

Targeted

Expressly identified for special attention -- usually in an operational plan rather than a strategic plan.

Targeted Industry

A sector of the economy specifically identified as the focus of economic development, job development or workforce development efforts because of its potential for sustained occupational employment growth.

Targeted Occupation

A job classification or cluster of jobs specifically identified as the focus of job development or workforce development efforts because of their potential for sustained employment demand growth and on evidence that occupational earnings potentials are high enough to enable participants meet or exceed the level of financial independence and economic security speficied in a related employment and training program's mission.

Targeted Population

A group specifically identified as the focus of workforce development efforts; in some operational plans, the targeted population may be a subset of the service-eligible populations identified for special recruitment efforts because they are especially difficult to enroll (e.g., the homeless) or because they have been underserved in the past. (See also Equity of Access, Service-Eligible Population, and Special Pops.)

Tayloristic Managment

Organization of work -- particularly in mass production factories -- into highly routinized, closely supervised tasks that can be performed efficiently by unskilled labor. (Named for its chief advocate and father of time-motion studies, Frederick Taylor.)

Tech Prep

Coherent sequences of courses beginning with career exploration at the middle school level and including integrated vocational and academic instruction articulated between the secondary and post-secondary levels that lead to an advanced associate degree; funded with federal dollars under the Carl D. Perkins Vocational and Applied Technology Education Act.

TEIN

Training and Employment Information Notice: a directive from the Secretary of Labor specifying what is required of state recipients of JTPA grants and other federal funds distributed by the DoL. (See also Guidance Letter.)

Terminee

A participant to whom services are no longer provided; usually connotes that the individual did not withdraw voluntarily from a program. (See also Drop-Out and Stop-Out.)

TQM

Total Quality Management: a management theory that emphasizes the need for a feedback loop that uses performance measurement to drive continuous planning and evaluation for improving products or services and customer satisfaction.

Tracking

Studying the activities of participants during the period in which they are provided services; done to monitor progress from one service level or program activity to another and/or to ensure that program delivery complies with procedural guidelines; i.e., tracking is process-oriented rather than outcomes-oriented. (Not to be confused with the pejorative use of the term as in labeling students and locking them prematurely into fixed education and training paths.)

Training- Relatedness

The degree to which instruction provided in a particular field of study corresponds to the requirements of employment for a specific occupation or cluster of occupations; i.e., the degree to which the knowledge, skills and abilities imparted by a particular instructional program meet the needs and expectations of those who are expected to hire that program's completers.

Traditional Labor Exchange

Labor exchange efforts that are state-specific and not fully automated -- relying instead on the intervention of a case manager.

Traditional Students

Individuals generally considered to be of "school age" whose pursuit of education and training is "age-appropriate" or on a pace commensurate with "normal academic progress" without interruption up through the highest level of desired educational attainment.

Transition Support Subsystem

Activities designed to help dislocated workers, displaced homemakers, ex-offenders, first-time labor force entrants, etc. obtain and retain jobs in order to achieve economic security and/or reduce their welfare dependency; a key component of School-to-Work, Welfare-to-Work and One-Stop initiatives.

Triangulation

Observing a phenomenon from several angles or using several tools to measure the phenomenon in order to determine its location and/or dimensions. (See also Convergent Validity and Multiple Instruments.)

UI

Unemployment Insurance: programs established to provide temporary benefits to workers covered by a state's Unemployment Compensation Act; intended to sustain individual members of the workforce and their families economically while they search for work; also the name commonly given to the unit within a state's economic security agency that administer the Unemployment Insurance program.

UI Agency

The entity in a state that administers Unemployment Insurance benefits, collects quarterly employer reports and maintains the UI wage record database -- typically the state employment security agency. (See also SESA.)

UI Benefits

The services and assistance to which an eligible UI claimant is entitled; used almost exclusively, however, to describe income assistance payments.

UI Claimant

Unemployed person who qualifies and registers for benefits under a state's Unemployment Compensation Act.

USPS

United States Postal Service: a source of labor market outcomes typically not covered in a state's Unemployment Insurance wage records.

Validation

Confirming the suitability of a measure and fine-tuning its application to take into account peculiar practices or special circumstances.

Validation, Employer

The process of subjecting assessment instruments and automation tools to the inspection inspection and approval of employers and the fine-tuning thereof based on employer recommendations.

Validation, Post Hoc

Using historic data to validate a model by showing after the fact that the model adequately predicted subsequent historic events; a process used to build confidence enough to rely on the model to predict future events. (See also Validity, Predictive.)

Validity (Definition #1)

A rule of logic where if the premises of an argument or proposition are related to the conclusion in such a way that the conclusion must be true if the premises are true.

Validity (Defnition #2)

Among persons doing empirical research, the extent to which an indicator actually measures what it purports to measure. (See also Indicator and Reliability.)

Validity, Construct

A determination that an indicator relates to other indicators consistent with theoretically derived hypothesis concerning the concepts being measured. Prior to actual data collection, the indicators are presumed to share common implications for the hypothesis being tested.

Validity, Convergent

A condition where confidence in an inference from empirical data or a measure used in the research design increased because the same results are obtained when two or more techniques or instruments were used to measure the constructs. (See also Multiple Instruments and Triangulation.)

Validity, Face

The subjective or intuitive determination that an indicator measures what it purports to measure.

Validity, External

The determination that all the necessary conditions were met in the research design and by strength of the relationships uncovered in order to generalize the results; also known as generalizability.

Validity, Internal

The determination that the research design was sufficiently robust to eliminate spurious interpretations of the results; in empirical research, the degree to which non-experimental research designs approximate the rigor of well-conceived experimental designs.

Validity, Predictive

The determination that an indicator can be used to accurately predict the value or position on some other indicator. (See also Validation, Post-Hoc.)

Value

In empirical research, the number or code assigned to an observation. (See also Variable.)

Value-Added

The contribution made by a program, service or activity in the employment and training system to the knowledge, skills, abilities, employability, or work-habits of participants; net gains achieved as a result of program participation. (See also Outcomes, Gross and Outcomes, Net.)

Variable

In empirical research, a label for properties that are more or less directly observed and that can take on different values. In data processing, fields in records are represent different variables while the specific entry in a field is the value of a recorded observation. (See also Construct and Variable.)

VEDS

Vocational Education Data System: a national database on secondary, postsecondary and adult vocational and technical education; maintained by the National Center for Education Statistics.

Venue-Neutral

A measure or instrument that is not biased in favor of a particular type of service delivery setting or modality. For example, a count of books in a postsecondary institution's library would favor traditional brick and mortar institutions over virtual institutions that deliver distance learning whereas access to information would be a venue neutral measure.

Wage Records

A relational database containing: a) information submitted quarterly by employers to a state's economic security agency on the earnings of workers who are covered by that state's Unemployment Compensation Act; and b) information about the employers who are required to submit quarterly reports. States maintain wage record systems in order to determine the eligibility and level of bene-fits for unemployment insurance claimants. (See also Quarterly Reports and UI.)

Wage Request or Wage Report

A precursor of wage record reporting on a quarterly basis whereby employers submit employment and earnings information on individual workers only upon the request of the entity that administers the state's UI system in order to settle a disputed benefit claim.

Wagner-Pyser

Federal legislation that established the Employment Service and ancillary functions in 1935 and as subsequently amended.

Waiver

A request to be exempted from the detailed specifications but not the intent of a federal regulations or directive; usually predicated on the desire to deploy innovative strategies to achieve some spec-ified objective in a more cost-effective fashion, improve performance, resolve incompatibilities between conflicting directives -- especially regarding paperwork reduction and employer burdens.

Waiver Plan

A document submitted in support of a requested waiver showing that the requesting party thought through the implications of its requests and has devised a workable alterantive that will achieve the desired results expressed in the original directive or regulation.

Walk-Aways

Bargaining positions that an organization or its leaders will not compromise when negotiating with other organizations or individuals.

Whitewash

To suppress negative findings and/or to exaggerate positive findings through selective perception and reporting of data by self-interested parties.

Work First Model

An approach to weaning welfare recipients from public assistance gradually by requiring them to find employment by a date-certain; this model may include additional education and training and transition support services to help individuals move from low-skill/low-wage/high turn-over entry-level jobs to the kind of high-skill/high-wage jobs that are more likely to sustain their long-term economic security. (See also Personal Responsibility and Work Opportunities Reconciliation Act.)

Work-Order

A request -- typically in writing -- that certain duties and tasks be performed in a specific timeframe by a specific individual or entity.

WRIS

Wage Record Information System. (See also ITSC.)

Standard mathematical notation for the "sum of."