This paper was presented at CUMREC '99, The College and University Information Services Conference. It is the intellectual property of the author(s). Permission to print out copies of this paper is granted provided that the copies are not made or distributed for commercial advantage and that the title and authors of the paper appear on the copies. To copy or disseminate otherwise, or to republish in any form, print or electronic, requires written permission from the authors.
Adding Value: The Challenge of Platform and Package Interfaces
Vicki Andros
Applications Systems Analyst, Principal
Email: Vicki.Andros@asu.edu
Katherine Ranes
Applications Systems Analyst, Principal
Information Technology
Arizona State University
Arizona State University
Main Campus – Tempe, AZ
Other campuses – Metropolitan Phoenix, AZ
49,000 studentsz
Research I University
Abstract: As you add components to existing applications can you "Plug and Play" or is the situation more like, "If it doesn’t fit, force it?" As a way to give added value to its customers, Arizona State University Information Technology has found ways to extend or enhance its legacy applications. Solutions have come from purchased packages and from in-house development. Regardless of how the new functions are provided, developers are spending significant time in interfacing the existing system with the new enhancement. Developers have become system integrators.
The presenters will review some lessons learned from their recent projects. Among the things you should consider before you start your next system integration project are:
Adding Value: The Challenge of Platform and Package Interfaces
When someone knows you work with computers you may be greeted with, "I heard you were the one who knows about computers, I’m having a problem with ...". The technical neophyte has the expectation that as a computer professional you can solve any problem with a computer. The presence of the Internet and the World Wide Web has given the impression that all computers are linked together and you can get information from one computer to another with the click of a mouse. The expectation and the vision are that this should be true, but as we approach 2000, we are not there yet.
In simpler days, a computer professional could know a great deal about how computers worked. After all there were only a few varieties of computers and only a few computer architectures. Now the picture is more diverse. We are faced with supporting systems dating from the IBM card days all the way to the newest networked applications that span organizations. The application developer is faced with the challenge of integrating old applications with new purchased software, with custom solutions and blending various versions of system software. How can you survive when you are the one who knows about computers?
In Support of Incremental Enhancements
Arizona State University is a growing university. This growth means the institutional resources, both financial and human, need to be invested in new buildings, increased staffing and expanding programs. In 1995, ASU reached a key decision point on how to progress forward with our Student Information System. The marketplace did not offer an easy proven way to replace our current homegrown Student Information System and the institutional priorities did not align with a massive effort to build, buy or co-develop a new one. So, ASU made the decision to upgrade our existing legacy system to be Y2K compliant. Various departments within the university needed additional data processing services, but understood that the Y2K upgrade was mission critical. They were willing to wait for additional services, but the backlog of need was growing.
How could we provide the University Community with some additional services while completing the Y2K upgrade? ASU chose to provide a few incremental enhancements.
Making changes to the existing system in the middle of doing Y2K upgrades was not feasible. Enhancements would have to be done with as little interference as possible to the existing system. Purchased packages and in-house development on alternate platforms would be possible.
We wanted to give added benefits to our customers, while building upon the existing system. At the same time, we needed to keep in mind that we would possibly replace the current system some time in the future.
With an eye to the future, we were looking at the newer technologies. Our current system is mainframe Cobol, and the current staff is proficient in this environment. Providing enhancements for our customers would also give us an opportunity to explore the newer technology as it evolves, looking toward possible replacement of the current system.
Business needs drove the selection of projects. These business drivers included:
Our Experience
As projects were presented for new development, they were evaluated based on the criteria of:
The New Applications Team within ASU Information Technology has implemented several enhancements using these criteria. The enhancements use a variety of platforms and approaches.
As these projects were chosen, one criterion that was given little, if any consideration was how the new functions of the proposed project would interface with existing systems. Since our existing student systems were highly integrated, sharing the same database and environment, it was assumed the new systems would interface too. The focus of this presentation is to discuss how the development teams reacted to and met this assumption.
The main lesson we learned is that system integration is a much bigger part of a project than most people recognize. Development of interfaces has consumed at least 50% of project time and is definitely more complicated than other system development tasks. It also requires a redefinition of the developer’s job. System integration requires looking at your job in a different way and a breadth of skills not needed for traditional development.
Lessons We Have Learned (or Things to Consider)
Supporting Operating Systems, Compilers and Platforms
When beginning a project, survey your environment to identify software versions and available system facilities for integration. Even in the old technology, pieces do not always melt together. We knew this, but with the new technology evolving, we tend to think "that is old stuff, it’s easy". The mainframe world has not stood still while we introduced new client/server platforms as part of the administrative computing environment. Compilers and operating systems are still upgrading. What release is your environment, and what release is your package? Does your mainframe have the environment the package needs? Does your technical staff have the expertise needed for the package? Can you find training? Just because it is old technology does not mean that everyone is proficient in every part. (We are an IDMS shop - VSAM? Hmmm, it’s been years!) One strategy we have used to surface software incompatibilities between purchased solutions and our environment is to require a "Proof of Installation" activity as part of package purchase. We require that the vendor assist us in installing the product at ASU and then demonstrating that it can run successfully on our hardware.
When using systems software that is new to your environment, you should remember the support structures that go along with that environment. We often take these things for granted. Inform the "help desk" personnel that you are using an environment outside the norm. Identify a method for moving system components to production. Determine if existing security systems will work with the new environment. Establish secure locations to retain original versions of the software as well as your own adapted production versions. Set your hours of operation and system backups as close to established guidelines as possible. Our strategy has been to mirror establish procedure and practice as much as possible.
Significant time is necessary to coordinate both the "Proof of Installation" tasks and establishment of support structures. In our organization, other areas of Information Technology support production processing and equipment. This means many of our projects cross unit boundaries and become Inter-Unit Projects (IUP). The amount of actual effort may not be great, but each activity requires identifying the right person to contact, explaining the project’s needs, requesting time for the work among all their other priorities and then testing the process to make sure it works. Coordination effort is often underestimated and we have difficulty understanding the work done by these support and system specialists. We have learned to appreciate that these other IT professionals play a key role in our success and we try to get them involved in our projects early.
Designing Bridge Processes
When a new system or process needs to reference or act upon the data from another system then a bridge is necessary. When a new system provides new processing features that are triggered by an event that occurs in an existing system, then a bridge is necessary. This is the most challenging part of system integration. Each bridge will be different. We have used a variety of techniques to bridge data and processes from one system to another. In each design, we needed to identify what options were logically and technically sound and needed to balance processing efficiency with the ease of development and maintenance. Some examples of successful implementations follow.
What thought processes led the unique design of each interface?
Identify the database/system of record. By identifying the authority, you can improve data integrity and eliminate some of the bridge design options. The ideal environment would have one system that is the system of record for everything. In this environment, other systems must directly access or replicate the data in the system of record.
However, it is not always possible to have one system of record. Our DARS environment has segmented data. Our legacy system is the system of record for the student data. The DARS package is the system of record for degree program requirements, transfer articulation data, and student record information that the legacy can not handle. We rely on the DARS processes to connect these two sets of data as needed.
Some implementations may choose a situation where multiple systems have a database and for system processing each system assumes its data is the system of record. When this occurs, it is necessary to synchronize the databases so that all systems have a complete picture of the business information. An illustration of this model is the data about rooms on the ASU Main Campus. We have three systems that work with room data. The Facilities system, the legacy SIS system and the ECA room scheduling system all keep an inventory of rooms in the University. A manual process that keeps these three systems in sync with occasional help from computer generated reports of the discrepancies. This synchronization method is feasible because room information is relatively stable and processing using room data occurs only at key points during the semester.
Determine if the system needs data for reference only and how "current" and "volatile" is the data. Many of our applications rely on ID, demographics, or codes to support data edits. Since these are "read-only" references and having "old" data (a day old, an hour old or a few minutes old) does not significantly impair system functions, we have replicated data across various databases. The presence of a "read-only" replicate saves access between servers, makes development simpler, and improves performance. In this case, if the amount of data is not large, then the phrase "disk space is cheap" is really true. We have used replication in Web development and in the Student Accounts, Accounts Receivable project to consolidate data and processing on the same platform.
Determine which system will support data entry. This problem is a bit trickier because the system used for data entry is more a question of what customers will use rather than what system will provide the easiest interface. Ideally, only one system will be used for data entry. If this is the case, then you can build a one-way bridge from the point of entry to the database/system of record.
Two projects we have implemented have resulted in different results in this area. When the Affiliate Management System was developed, it was designed as a point for data entry. Because Admissions Offices needed to enter more information about a student than was present in the Affiliate Management System, Admissions Offices continued to update data about "student applicants" using the SIS. Updates for another affiliate group "courtesy students" used the Affiliate Management System to update data. This required a bridge that was two-way, from SIS to Affiliate (batch) and from Affiliate to SIS (real-time).
In designing these bridges, we had to be sure to design the process so that a recursive loop did not occur.
The DARS project started with the transcripts entered and stored in the SIS. The data entry of transcripts has now been shifted to the DARS package. This allows use of the editing features of the package, and the use of an automated evaluation system. And even though some minor data entry is still done in the SIS, (the 80-20 rule) we only had to design a one-way bridge to take the information from DARS and store it in the legacy system.
Recognize error conditions and data conversion rules. The thing that makes designing bridge processes so difficult is that you need to understand two systems not just one. Most bridges will need conversion routines that will convert data from structure of one system to the structure of the other. How closely data structures are aligned will influence how involved the bridge becomes. As well as positive results, one system will need to interpret and/or react to error conditions generated by the other system. Developing error-handling can be very time consuming. One strategy we have used to improve our delivery times is to manage errors generically and then enhance the system to respond to common errors. This way you do not invest time in handling errors that happen rarely or never.
Picking the Interface Point
What type of processes are you interfacing? If it is a monthly or semester process, perhaps a process that moved data from one system to the other will work. Technology will drive the options you have for interface points. Does your package have exit points or I/O routines that you can look at for interface points? If these points do not have the data available that you need, do you have the source code for the package? If so, perhaps you can insert a call to a program you can write that will do your interfacing.
Handling data inconsistency and conversion
When moving data from one system to another or converting data from an "old" system to the "new" one, the picture is not pretty. In the easiest case, you can go through the pain once, when you convert data from an old data structure and populate a new one. When integrating two systems and maintaining data representations in both, the dissonance of data continues for some time.
Some of the inconsistencies are introduced because of data definition within the two systems, for example, field length variations, character vs. numeric representation, or supporting only upper case instead of mixed case character fields. These inconsistencies are more of an aggravation than a problem and can be handled by conversion routines in a straightforward manner.
A real problem comes when data domains or business rules have changed because of changes in the business. How is a non-existent or unknown date handled? What if the field is optional in the existing system and is required in the new system? The new data requirements may have been the reason the project was initiated. The basic strategy we have used to work this change is to
This gradual transition has allowed us to implement projects in phases and each phase has a smaller scope. This means quicker and more flexible delivery schedules.
The DARS bridges were implemented in a series of steps. The first step was to have the student’s transfer and home course work bridge to the DARS audits. This coursework was used for evaluation, and was not stored in the DARS system. This bridge allowed us to use our new system in production. The second step required the student’s transfer coursework to be stored in DARS. This step facilitated a new process that allowed more accurate audits. Which coursework would be needed for an audit could not be anticipated, so all current student transfer work was duplicated, and then nightly updates kept it current. The audit now got its transfer work from the DARS system, and the bridge in step one was modified to not send transfer work to the audit. This second bridge quickly allowed more detailed audits, but required additional disk space, and overhead of maintaining two copies of transfer work. The third step in our bridge process built a bridge that would retrieve a copy of the transfer work as it was needed, and coded the transfer work in the system of record as being ‘locked’. This ‘lock’ permitted the audit bridge (step one) to again retrieve transfer work from the system of record, and ignore records which would be duplicates.
When you start using routines or converting data you will learn that your data is not as clean as you thought it was. What ‘added value’ has the customer of the existing system built into the data by ‘adapting’ the current system? How will each system detect and handle duplicates? How much inaccurate or extinct data is dormant in your files? These are the most difficult problems of all. In most of the projects, we have used people power instead of data processing. Plan for the time the project will need in this manual effort. Data clean up and handling of inconsistencies needs the judgment of a person not the swift rigid action of a computer program.
Promoting user transition
The change process is difficult for everyone. Several factors will either hinder or facilitate the customers’ use of a new system.
"What will this do for me?" If the change will directly help the person being affected, they will move to the new system more readily. The change process is more difficult if the person winds up with more work, and someone else reaps the benefit. Another thing that makes change more difficult is if features are taken away. This can be a perceived or real loss. Even if the customer gets a couple of new features, this does not always make up for the loss.
The transition can be accomplished many ways. It can be done incrementally, with a pilot group, train the trainer, or by word of mouth.
It’s the little things that make the difference. If the customer is going to run dual systems, are the columns on the screens in the same order? Do the function keys work the same in both systems, or is "save" key in one system the "delete record" key in the other? The little quirks can be major irritants and build frustration. Is there a hot key that can flip the customer from one system to the other?
Accepting inconsistency
When all systems were built in the same environment and that environment was mature and fairly stable, standards enhanced productivity. Now as we integrate systems that are not based in that environment, rigid standards and procedures can be a barrier to productivity. A willingness to not enforce a rigid standard and try new ways has moved us ahead. The main caution here is to realize that the inconsistency exists and document your "special case".
The inconsistency that system users perceive in data, presentation and timing has been discussed in previous sections. To increase acceptance of systems that appear inconsistent, communication is the key. If you explain why some names are all upper case and some a mixed case then when this inconsistency is encountered, the user will not react as if this is a problem. For irritating inconsistencies, be clear that you realize they are there. Be honest, solutions for these inconsistencies are not always easy or may not be a wise use of institutional resources.
In a time of change, things will be inconsistent. We can’t move from Kansas to Oz in the blink of an eye, like in the movies. The transition is more like the introduction of color TV. Some people had black and white TVs and could only see the program in black and white. Others had a color TV but as the NBC network made the transition to full color programming it used the color peacock as means to let us know that the program was available in color.
Coping Mechanisms
The challenges of system integration have been more easily overcome because the organization and staff members have developed some coping mechanisms. The theme of these coping strategies is to not know it all yourself but build on the resources available to you.
Data Administration and Data Warehouse Implementation: Arizona State University has staff assigned to data administration. These staff members and community awareness of the need for data definition have helped us avoid some data conversion problems and have served as a way to define the new business rules when inconsistencies are present. The presence of the data warehouse allows some system integration in the area of reporting. When system users can build queries against data populated from disparate data sources, this shifts some the need for system integration at the data source. It also surfaces the data inconsistencies that exist among existing systems.
Vendor Assistance: Vendors who supply application packages, middleware, database systems and systems software have provided assistance. The various package vendors have been available to answer questions, and validate the proposed method of installation. The value of maintenance support agreements has grown and we have become more active in seeking answers outside our own organization.
User Groups and Listserves: Vendor listserves provide a network of other users to compare notes and ask questions. Other institutions may have installed the package on the same type of hardware or connected to the same packages. Questions to the listserve about upgrading operating systems can often point out possible problems and how to avoid them.
Broaden project team and commitment to cross-unit cooperation: When project teams are formed often consultants in other units are identified and matched with the project. Adding the project to the Inter Unit Project List raises organizational awareness of project objectives and architecture. The broader skill set available to the team can result in some creative solutions.
Evolving open system standards and cross-platform products: As the computer landscape has become diversified and few organizations have a "one vendor" or "one platform" solution, the market has responded. New products and standards are being developed so that one system can talk to another one. As developers, request information on the API (Application Program Interface) features and ask about development partners. At ASU we have adjusted these capabilities to blend operating environments and get out of the closed box of closed computer environment.
Strong user support for debugging, problem resolution and testing: By maintaining communication with our users (customers), we are able to work together in debugging and problem resolution. By looking at the problems jointly, the users are able to research and share with the technical staff what actions proceed a problem, what seems to help, what makes things worse. The joint effort enhances the effort of resolving problems.
Increased resources for building skills, training, seminars, web references: There are many times when there is no one to ask for help. This is when you need to help yourself. Our organization has recognized the need for continual skill enhancement and encourages additional training. Training may be formally structured, for example a university or community college course, or specific tool training offered in a vendor-training program. It is also informal, gained through experimentation, reading and researching the web references. This learning takes time. One coping strategy is to identify the training needs and skill risks early in the project so everyone knows of the "research and development" aspects of the project and which staff will need to pass through a "learning curve."
The Good News
As we worked on these various systems, it was disheartening at times. We knew we were good application developers, or at least we knew we used to be. But now, every time we turned around, there were toes sticking out and we were tripping over them. As we continued with our applications, we began to realize, "Yes, we were good application developers but we were also learning a new set of skills." Our jobs were changing from application development to system integration. As we analyzed our situation, we noticed other things were also changing. There was more acceptance of diversity. The hard and fast standards of the past were loosening, giving us more options, and enabling us to be more effective integrators. We were involving more people in our consultations as we entered into areas that we did not previously have expertise. And what about our old skills? They are the basis for our new skills. A system integrator who knows at least one of the systems will be more effective in finding out how two systems can go together. We learned that the basics still held. We were just learning a new verse of the same song with maybe some harmony variations. And maybe, we are the ones who know about computers.
CHECKLIST FOR SYSTEM INTEGRATION PROJECTS
THINGS TO DO
Operating Systems/Compilers/Systems Software
HOW TO COPE
Communicate
Ask for Help
Build on Integrating Technology