|
This report was written by Kelly Maxwell, Kathleen Bernier, Donna Bryant, and Betsy Lowman. We appreciate the collaboration of the Executive Directors from the four participating partnerships and the people who helped collect information for this project. For additional copies of this and other Smart Start evaluation reports,
contact Marie Butts at the Frank Porter Graham Child Development Center,
105 Smith Level Rd., CB # 8180, Chapel Hill, NC 27599-8180 or call (919)
966-4295.
EXECUTIVE SUMMARY SEPTEMBER 1998 Study Purpose and Description Some Smart Start local- and state-level leaders have expressed an interest
in creating a system to count uniquely all children and families served
by Smart Start. This system, referred to in this report as a client information
system, would consist of a database used by all Smart Start funded projects
that contains basic demographic information about each person served, including
a unique identifier assigned to each individual. Using the unique identifier,
data from each project could be combined with every other project to determine
an accurate count of people served. In 1997, the FPG-UNC Smart Start Evaluation
Team worked with four Smart Start partnerships to describe the current
client information systems used by local partnerships and to determine
the feasibility of collecting similar information from participants of
all Smart Start projects, using a unique identifier to link data across
projects.
Findings
Local-Level Recommendations
Feasibility Study The North Carolina Early Childhood Initiative, better known as Smart
Start, is a partnership between state government, local leaders, service
providers, and families to better serve young children. The main goal of
Smart Start is to ensure that all children enter school healthy and prepared
to succeed. Smart Start’s innovative approach requires local community
partnerships to plan how best to meet their own community’s needs, improve
and expand existing programs for children and families, and design and
implement new programs. In 1993, 12 Smart Start partnerships were selected
from geographically and economically diverse areas across the state to
begin planning and implementing programs. New partnerships have been funded
each year since 1993. Currently, 55 of the 100 counties in NC are serving
young children and their families through Smart Start. Each local partnership
funds multiple projects that can be categorized into three general areas
of services: child care quality, availability, accessibility, and affordability;
health; and family support. On average, partnerships fund approximately
20 different projects each year (ranging from 10 to over 50 projects per
partnership).
Background and Purpose Since Smart Start was implemented in 1993, legislators and others have asked the deceptively simple question, "How many children and families are being served through Smart Start?" Although an estimated number has been provided, the only way to know the answer definitively is to document exactly who receives each type of Smart Start service. As noted in the 1994-95 Smart Start Evaluation Annual Report, this exact documentation of people served through Smart Start is difficult for several reasons. First, many service agencies and programs in North Carolina do not count individual families and children served but instead count service contacts (e.g., number of clinic visits) which overestimate the actual number of people served. Second, some Smart Start activities are outreach efforts, such as public education through pamphlets, for which lists of families reached cannot be produced. Third, agencies and programs serving children and families in North Carolina do not share a common database of client information. Thus, people who receive multiple Smart Start services are counted by every different agency or program in which they participate, overestimating the total number of people served by Smart Start. Over the years, some Smart Start local- and state-level leaders have expressed an interest in creating a system to count uniquely all children and families served and to evaluate the relationship between service intensity and outcomes (e.g., Do people who receive more intensive Smart Start services have better outcomes than people who receive less intensive or no Smart Start services?). This system, referred to in this report as a client information system, would be a database used by all Smart Start funded projects that contains basic demographic information (e.g., date of birth) about each person served by Smart Start. The database would contain the same information collected in the same way from all projects to ensure that each project’s information could be combined with every other project’s information. To combine data from different projects, each person served must be identified uniquely. A person’s social security number is an example of a unique identifier that could be used in this system. With a unique identifier, it is possible to determine which people received services from only one project or from multiple projects. Some local partnership leaders have taken the first step in developing a client information system by gathering similar client information from multiple Smart Start sponsored programs. In 1997, the FPG-UNC Smart Start Evaluation Team investigated the feasibility of using a similar database across multiple partnerships to answer more accurately the basic question, "How many children are served through Smart Start?" The two major purposes of the investigation were to:
Client Information Systems This study included four partnerships that had gathered information from clients across multiple Smart Start projects. Three of these partnerships (referred to in this report as Partnerships A, B, and C) began participating in Smart Start in 1993. The fourth (Partnership D) became a Smart Start partnership in 1994. In the winter of 1996, we asked each partnership Executive Director and other relevant staff to describe their client information system. Each participating partnership’s system prior to the 1997 feasibility study is described below. Partnership A’s Client Information System A partnership board committee designed a common client information data sheet in 1995 as one step toward organizing the partnership’s financial and performance accountability. This data sheet included information about the age of the client and his/her immediate family members. Information about both children and adults was collected. A unique identifier for each family member was created using the person's first three letters of the last name, the first letter of first name, and date of birth. The social security number of the oldest preschool child was used as a common family identifier to link people to families (e.g., to identify siblings who received services). From July to December of 1995, about half of the Smart Start projects
gathered these data in paper form. The other half of the projects that
did not collect the requested information were administered by one agency
whose leaders did not want to participate because they did not see a clear
purpose in the effort, believed that they could not obtain the information
from the numerous families they served, and did not want to add to staff’s
workload. The partnership staff did not enter the paper forms from each
project into a database or summarize the information in a meaningful way.
Knowing that half of the projects would not participate, the partnership
did not ask any projects to participate again in a client information system
until the 1997 feasibility study.
Partnership B’s Client Information System In 1994, Partnership B implemented a client information system to help
make program decisions and provide a more accurate count of people served
through Smart Start. The committee wanted to use the information to ensure
that family support programs were serving their intended clients and to
determine whether some populations were not being served. The partnership
designed the system to be used for family support programs only, and did
not collect the information for children receiving Smart Start child care
or health services. The family support client information system included:
the date services began, ethnicity of target child, family income, family
size, single parent household status, primary caregiver’s education level,
number of children under 6 with special needs, number of children under
6 who did not speak English, and the social security number for each child
in the family who was less than 6 years old. This client information was
collected on a form by family support project staff and sent quarterly
to a partnership staff member who entered the information into a computer
database. Each participating project received a quarterly summary report
of their data. Almost all family support projects have continued to participate
to some degree in this system.
Partnership C’s Client Information System A Partnership C staff member developed a client information system in 1994 as a step toward providing a more accurate count of the people served through Smart Start. Information was collected for children served by Smart Start and professionals who received Smart Start sponsored training (e.g., child care providers, nurses). The data sheet for children included the following information: the date services began, target child’s social security number, first three letters of child’s last name, child’s date of birth, child’s ethnicity, family size, family income, single parent household status, primary caregiver’s education level, number of children in the family who were less than 6 years old, number of children under 6 with special needs, number of children under 6 who did not speak English, and the number of children under 6 who attended child care. The data sheet for professionals included the following: social security number, ethnicity, name of employing agency, highest education level, and number of children under 6 served by the professional. During the 1994 fiscal year, almost all Smart Start projects in this
partnership collected the requested client information for children and
professionals. In 1995, far fewer projects (only 12%) collected the requested
information, possibly because the data collected in 1994 were not summarized
in a meaningful way. Until the 1997 feasibility study, the partnership
did not request client information again from projects.
Partnership D’s Client Information System Partnership D began designing a client information system during their
first year of planning to provide an accurate count of the people served
through Smart Start, to help develop coordination among community services,
and to create a procedure for following children over time (which could
help them evaluate the long-term child outcomes of Smart Start projects).
They implemented a client information system during 1995, their first year
of funding service delivery projects. Each Smart Start funded project was
responsible for gathering the name, date of birth, and social security
number for each child under 6 served. Each project could collect additional
information, such as attendance or pre- and post- intervention measures.
Approximately 75% of Smart Start projects collected the client information,
which was reported to the local partnership, summarized by partnership
staff, and reported monthly to the partnership board. Partnership D has
continued using this client information system.
Procedures For this study, the four partnerships agreed to try gathering the same basic client information from funded projects over a 4-month trial period. The FPG-UNC Smart Start Evaluation Team provided technical assistance and some financial support for a person at each site to help collect this information and document the a) process used to collect the information, b) problems and solutions, c) time spent on the task, and d) response rates when possible. The basic client information included:
Each partnership handled data collection slightly differently, although they all hired someone to help gather data for the client information system. Because child care centers served many children and required a special effort to collect the data, three of the four partnerships focused their data collection efforts primarily on child care centers. Partnerships anticipated difficulty in obtaining information from child care centers because, unlike agencies, center staff do not have extended, routine contact with parents during which the information could be collected. Although parents come to child care centers daily, they typically are rushing either to pick up or drop off children on the way to or from work. Because of the expected difficulty in obtaining information from parents of children enrolled in child care centers, partnerships and the FPG-UNC Smart Start Evaluation Team were interested especially in determining the feasibility of gathering information from child care centers. Staff in Partnerships A and B worked primarily with child care centers to collect client information data but also facilitated data collection from other programs (e.g., obtained administrative support for project staff to gather data and designed a data collection form for project staff). Staff in Partnership C worked with child care centers only. Staff in Partnership D worked with all projects to modify their existing data collection form and coordinate data collection. Findings None of the partnerships were able to gather client information from all projects. Partnerships excluded some projects from their client information database because it was not feasible to gather information from participants (e.g., outreach projects, library projects). Other projects were excluded because program staff expressed concerns regarding confidentiality (e.g., adult literacy projects, women’s shelters, and immigrant support service programs). Another group of projects was excluded because project administrators were reluctant to participate (e.g., large agencies that had hundreds of clients receiving Smart Start funded services). Several factors seemed to support project willingness to collect client information. In this database effort, participation among Smart Start projects varied widely from partnership to partnership. Projects were more likely to collect client information if they had a trusting relationship with the local partnership and other Smart Start funded projects collecting client information data. Projects were also more likely to participate if they understood the importance of collecting the information, knew how the information was to be used, and knew about the client information system requirements when they were first awarded a Smart Start contract. Collecting client information was time consuming. For some projects, staff could collect the required client information easily because it was already collected through their routine procedures; other projects required more effort. In one partnership, staff spent approximately 44 hours collecting client information from parents of 372 children attending nine child care centers, averaging 7 minutes per form. (This does not include the time spent by child care directors distributing and gathering the forms or by families completing the forms.) Partnership staff spent those 44 hours explaining the system to child care directors, preparing parental consent forms, developing and posting reminder notices on classroom doors, traveling to the centers to collect completed forms, and talking with parents during pick-up hours at the centers. In another partnership, staff from Smart Start funded projects (not including child care centers) estimated their time spent collecting client information for Smart Start. Although there was a wide range of time required, project staff reported spending an average of 5 minutes per form. Because Smart Start is serving a large number of children and families in each community, the time needed to gather this information on all Smart Start participants would be large (e.g., using the 5 minutes per form guideline, it would take 83 hours to gather information from 1000 individuals). Smart Start projects were not able to collect information from all of their clients. Two partnerships documented client response rates. Using estimates from project staff, response rates in one partnership varied from 20% to 100% across participating projects. That is, even when projects tried to obtain and report standard client data, not all families were willing to give it. In another partnership, client response rates were gathered from nine child care centers. For these centers, response rates ranged from 18% to 70% after one attempt to gather the information and ranged from 18% to 90% after two attempts. These data suggest that a client information system will never include information on all Smart Start participants. Some project staff and Smart Start families expressed concerns about providing social security numbers and income level to their local partnership. As mentioned earlier in this report, the client information system used social security numbers as the unique identifier linking children who participated in multiple Smart Start projects. Without a unique identifier of some type, partnerships cannot match children across projects to determine whether someone received services from more than one Smart Start project and, therefore, cannot provide an accurate count of the total number of people served. Some project staff in all four partnerships raised concerns about including social security numbers in a client information system. Some families also expressed concerns about including social security numbers and income. One parent said, "This form is asking a lot about social security numbers and income. Isn’t it a little much? I think they’re basing everything on income and I don’t agree with that. I don’t want to be a part of it." Another parent said, "I suggest that (partnership name) study the quality of services provided with Smart Start money rather than the parents’ income and social security numbers of the persons served." This hesitancy to include social security numbers is evident in the number of forms that were missing social security numbers. In one partnership that had widespread cooperation across projects, 29% of clients did not provide social security numbers. In another partnership, 25% of the parents returning forms to child care centers did not provide social security numbers. Partnerships did not have the staff needed to continue working on the client information system. Although most partnerships asked project staff to gather client information, someone at the partnership had to enter the information into a computer program, summarize the data, and write a report. During this trial period, partnership staff already had multiple responsibilities and could not add to their current workload. In all cases, someone outside the partnership was contracted to work on the client information system during this trial period. At the end of the period, three of the four partnerships had not identified someone to continue the work. Partnerships already have an enormous administrative responsibility, and adding the oversight of a client information system to their current workload does not seem reasonable without substantial additional resources. Partnerships did not have the technical expertise and ongoing support to continue working on the client information system. Three of the four partnerships did not have staff who knew computer
database programs well enough to continue using the client information
system database and generate summary information from the database.
State-Level Recommendations Three general recommendations about a client information system are directed at state-level leaders. A client information system would be useful for the broader North Carolina service system, not just for Smart Start services to young children and their families. A client information system shared across all services would be helpful
not only to count accurately the number of people served but also to evaluate
the effectiveness of the broad range of services provided to children and
adults in North Carolina. With such a broad implementation of a client
information system, we could better understand the relationships between
intensity of services and outcomes (e.g., length of employment, school
success). A client information system would also inform us about the patterns
of services people use, which could help create a more coordinated system
of services. For example, if most people who use the Health Department
also use services within the Department of Social Services, then administrators
could decide to locate those services in the same building or develop just
one enrollment process for both programs.
Cooperation and support across multiple key organizations would be needed before a client information system could be implemented across all partnerships. Many people would need to agree on the importance of developing a client
information system and working collaboratively to implement such a system.
At a minimum, agreement would be needed from the legislature, NC Department
of Health and Human Services, NC Partnership for Children, each local partnership
board, state- and local-leaders of major agencies like the Department of
Social Services, as well as private organizations such as the United Way
that receive Smart Start funding.
A client information system should not be implemented without the financial resources to support it. Creating, implementing, and supporting a client information system would
be expensive. Each local partnership would need one to two more full-time
administrative staff members, and the NC Partnership for Children (or other
support organization) would need many new staff to provide ongoing technological
support. Specialized computer software and upgraded computer systems also
might be needed.
Local-Level Recommendations Six recommendations are provided for local partnerships who are using or want to use a client information system. Social security numbers should not be used as the unique identifier in a client information system. Section 7 of the Federal Privacy Act of 1974 requires all agencies that ask for social security account numbers to tell the person a) whether the disclosure is mandatory or voluntary (if mandatory, the agency must also provide the statute or authority that requires it) and b) how the information will be used. The individual must be asked permission each time an agency wants to use the social security number for a new reason. For example, if an agency first asked to collect social security numbers for their own record keeping, they would need to ask the individual again for permission to use the social security number for Smart Start. Not asking for permission each time is a federal offense and could happen easily if a staff member is unaware of the law and has access to clients’ social security numbers. Violation of the law is a felony punishable by a fine of up to $5000 and/or up to five years in prison. The severity of the violation poses a substantial risk that many agencies may not want their employees to bear. The Electronic Privacy Information Center recommends alternatives to
using social security numbers as a unique identifier. One simple option
is to use only the last four digits of the social security number. It is
also possible to purchase software to create a unique identifier based
on a person’s full name, date of birth, and address. Information about
these and other unique identifier options is available at The Electronic
Privacy Information Center website at http://www.epic.org/privacy/ssn/alternatives.html.
Projects could collect client information more easily during enrollment periods. Partnerships that are implementing a client information system would
obtain data more easily—and possibly from more clients—if they coordinated
data collection with each project’s enrollment period. This seems to be
especially important for child care centers because it is very time consuming
to gather information from parents once they complete the enrollment forms.
Partnerships may find it useful to delineate in contracts with funded projects their expectations about data collection for the client information system. Partnerships could increase project participation in the client information
system by requiring funded projects to ask clients for information. It
is important to note that a partnership could mandate through a contract
a project’s participation in the client information system, but
it could not mandate the participation of individual clients: the
partnership cannot force individuals to provide the requested information
but it could require project staff to ask individuals for the information.
Although Partnership D used this strategy (along with others) successfully,
it should not be the only strategy used to garner support for and
participation in the system. As mentioned earlier, developing trust among
community programs and the partnership and using the information wisely
also facilitate project involvement.
Partnerships should develop guidelines for gathering client information from unique projects. Collecting client information is not always a simple process. For example,
if a child care center receives computers for each classroom, the computers
will benefit children from that year and future years. Which children have
received this Smart Start benefit and how should they be counted? A partnership
may want to count children from multiple years as clients for that project.
As another example, partnerships should develop guidelines for determining
which children benefit from child care provider training. A more elaborate
data collection system than the one used in this trial would be necessary
to link child care providers to the children they teach. These are just
two examples of the numerous difficult decisions partnerships face if they
implement a client information system.
Local partnerships would need at least one knowledgeable, full-time staff member to coordinate a client information system. It is important for partnerships to have at least one staff member to
coordinate data collection, organize the data in a computer database, and
provide summary reports of the client information data. Larger partnerships
may need additional staff. The partnership staff’s role is critical to
ensure that the data are used appropriately, once collected. Data that
sit in a file drawer in someone’s office are not helping programs or partnerships.
People who work with the client information system must have a strong knowledge
of computer databases, be good logical thinkers and problem-solvers, and
have good interpersonal skills.
Partnerships may find it more feasible right now to gather long-term information from a sample of children, rather than implementing a client information system for all children. It is important to document the effects of Smart Start on children’s outcomes. Although a client information system provides one way of identifying children who received one or many Smart Start services, it is not the only way. A smaller group of children who received intensive Smart Start services could be identified and followed to determine their outcomes. For example, partnerships could select children who participated in child care centers that received intensive Smart Start services and compare these children’s developmental skills to children who participated in child care centers that were not intensively involved with Smart Start. As another example, partnerships could survey participants of particular programs, such as Family Resource Centers, to gather information about program satisfaction and family outcomes. Appendix
|