System Integration And Data Merging: Techniques, Tools, And Applications Essay.

This assignment will take several weeks to complete and will require a good understanding of application and information integration technologies for successful completion. It is imperative that students take heed of the following points in relation to doing this assignment:

1. Ensure that you clearly understand the requirements for the assignment – what must be done and what are the deliverables.
2. If you do not understand any of the assignment requirements – Please ASK the course coordinator or your tutor.
3. Each time you work on any aspect of the assignment reread the assignment requirements to ensure that what is required is clearly understood.
4. Prior to submitting your code, you should ensure not only that it executes as required, but also looks professional. It is expected that you adhere to HTML and python standards for naming and indenting. All methods should be adequately documented such that another programmer examining your code will readily know what the code is doing.

The process of system integration is used to combine two different processes. System and data integration is the combination of technical and business process. Number of operations is performed using system and data integration. Using system and data integration done validation, mapping, transformation, and console the data.

The data give the meaningful information about disparate. Normally data and integrated systems are clearly identified. We can perform the field mapping operation as well as gap analysis also. Using smooth data in data ware house the final result will be accurate and also effective.

Data merging means collections of information’s are merged into a single information. The information’s are frequently used in government and non government organizations. Merge/purge problem is similar to this problem. It will generate the duplicate information about typical data. They are most complicated and depend upon the domain matching process.

Data cleaning task means the unwanted data or information’s are deleted. The marketing type applications are directly connected by the customers. The statistical details about the data are described in final result.

Final result explains the statistical details about the data. By follow this method the final result will be effective and accurate. Transitive technique is the special type of technique.  Using this technique the individual results are combing. The final result in low cost, and also using combing method the generated result will be accurate and effective.

Each and every program provides the rule programming module. The system needs rule programming model for achieve the successful execution. Suppose there was an duplicate environment is there, using the rule programming to find out and eliminate the duplicate environment. We know that each application have ability to handle large amount of information. Using data merging the real world applications are performed well. The final result is used to analyze the statistical data

PETL package

petl is a python package or python package index. To describe the pip follow the below command

$ pip install petl

Using petl python package we can easily run and extract. The following command is used to describe how install the python setup

python setup.py install

finally if we want to check the installation is done or not,  using the following commands

$ pip install nose

$ nosetests – pet1

We are using the latest version of python language like version 2.7 and 3.4 are mostly used. Python is performed by WINDOWS and UNIX operating system.

Data Merging and Cleaning

ETL pipelines

The lazy evolution and iterations are avoided by using ETL pipelines. But this pipeline needs some specific data. It does not execute accurate until the data is required.

For example

>>> example_data = """foo,bar,baz

... a,1,3.4

... b,2,7.4

... c,6,2.2

... d,9,8.1

>>> with open('example.csv', 'w') as f:

 f.write(example_data)

We are using one calling function for this pipeline. petl.util.vis.look() is a type of calling function. Using this function we can easily read the data, write the data, files, and also database.

Using the following codes to achieve the read and write operation

Data extractions are done by the table containers. Each table containers have two major things one is table containers another one is table iterations. All the transformations are run by pipeline.

RESTful web services is defined as representational state transfer in the way of architecture and it describes about the constraints in the form of interface as unique and scalability and it is the best web service in total web services. This is known as the web standards and it is used to the data communication. REST server in the architecture provides the access to the resources and the resources are analyzed using the uniform resource locator such as URI and the representation in the kind of xml and text. Hypertext transfer protocols methods are used in the web service and the methods will be get, delete, put, post. And the RESTful web services is the collection of protocols and the standards are used to the data communication. And this web service is based on the architecture defined in the web service. And it is used to classify by the uniform resource locator.

The mashups content are used from different sources for displaying and creating a new service. It have advanced in the web technology. The companies have shown the information for using other sources so they have shared their own data without any permission. The map mashups is a major mashup. The programmable web have traced the map to the list of comprehensive. The main example of mashup is creating a map mashup by a google map that shows you an address and location. In data integration they are many types of mashups that are performing in the system are URL mashups, HTML mashups, data mashups custom mashups. A mashup is a good web application that has combination of data and functionality from many other external sources and finally it create a good service. The mashups one can use any easily, integration speed have increase, using of APIs and to produce a results from data sources they as no reasons to produce an original data from source.

ETL Pipelines

The python program that is “dataMerger.py” is used to merge given content and the files are imported by using the keyword “import” and every attributes in the codes are used to form the tree and the web service side the python program that is “clinics_locator.py” used to search and locate the address in the nearest tab.

Merging two file:

Figure 1merge the two file

Restful web services:

Code Explain:

Which was carried out to the execution of python files and save the location on .csv files. The results of the operation “import csv” was used. For opening the information “storeopen()”  was used. To read the file we need to use “StoreFileReader()” was used. For length checking purpose we need to use the “If(len !=row)”. To increase the no of rows we need to use “ScoreList[]=ScoreList[]+row”. For exiting from the file we need to use “ScoreFile.close”.

Figure 2Restful web services

To search the Mashup Demo:

Figure 3Search the Mashup demo

Location Display to submit the Button:

Figure 4location display in Map

The clinics_map.html file used to view the exact geolocation and direction of the position of

The clinic wants to know. The MAP which contains the direction for the clinic. It very useful show the location of the clinics services location easily.

Conclusion 

The techniques are used to compute the responsibilities of the system. The system integration of various data the final required data was recovered. Scalability of the system was ensured by the virtualizing techniques. And finally the integrating the information, demonstrations also performed. The position of the stores in the MAP was identified. The IT structure are mainly used to access the data centers and based on the Functionality of the dependent on the type of the Infrastructure.

Reference 

Joglekar, A. (2016). Prediction of Favourable Rules to Identify Suspected Patients of HIV Using Integration of Expert System and Data Mining. International Journal of Mechanical Engineering and Information Technology.

Kaps, A., Dyshlevoi, K., Heumann, K., Jost, R., Kontodinas, I., Wolff, M. and Hani, J. (2006). The BioRSTM Integration and Retrieval System: An open system for distributed data integration. Journal of Integrative Bioinformatics, 3(2).

Mynarz, J. (2014). Integration of public procurement data using linked data. Journal of Systems Integration, pp.19-31.

Dong, X. and Srivastava, D. (n.d.). Big data integration.

Finkelstein, C. (2006). Enterprise architecture for integration. Boston: Artech House.

Haltiwanger, J., Lynch, L. and Mackie, C. (2007). Understanding business dynamics. Washington, D.C.: National Academies Press.

Huang, X. and Zhu, W. (2013). An Enterprise Data Integration ERP System Conversion System Design and Implementation. Applied Mechanics and Materials, 433-435, pp.1765-1769.

ISHII, H. and TEMPO, R. (2009). Computing the PageRank Variation for Fragile Web Data. SICE Journal of Control, Measurement, and System Integration, 2(1), pp.1-9.

Oró, E. and Salom, J. (2015). Energy Model for Thermal Energy Storage System Management Integration in Data Centres. Energy Procedia, 73, pp.254-262.