3. Theoretical Principles
The underlying principle of data mining is that there are hidden but useful patterns inside data and these patterns can be used to infer rules that allow for the prediction of future results (GAO, 2004, p. 4).mining as a discipline has developed in response to the human need to make sense of the sea of data that engulfs us. Per Dunham (2003), data doubles each year and yet the amount of useful information available to us is decreasing (p. xi). The goal of data mining is to identify and make use of the golden nuggets (Han & Kamber, 2001, p. 4) floating in the sea of data.to 1960 and the dawn of the computer age, a data analyst was an individual with expert knowledge (domain expert) and training in statistics. His job was to cull through the raw data and find patterns, make extrapolations, and locate interesting information which he then conveyed via written reports, graphs and charts. But today, the task is too complicated for a single expert (Fayyad, et al., 1996, p. 37). Information is distributed across multiple platforms and stored in a wide variety of formats, some of which are structured and some unstructured. Data repositories are often incomplete. Sometimes the data is continuous and other times discrete. But always the amount of data to be analyzed is enormous.involves searching large databases, but it distinguishes itself from database querying in that it seeks implicit patterns in the data rather than simply extracting selections from the database. Per Benoît (2002), the database query answers the question what company purchased over $100,000 worth of widgets last year? (p. 270) whereas data mining answers the question what company is likely to purchase over $100,000 worth of widgets next year and why? (p. 270).forms of data mining (KDD included) operate on the principle that we can learn something new from the data by applying certain algorithms to it to find patterns and to create models which we then use to make predictions, or to find new data relationships (Benoît, 2002; Fayyad, et al., 1996; Hearst, 2003).important principle of data mining is the importance of presenting the patterns in an understandable way. Recall that the final step in the KDD process is presentation and interpretation. Once patterns have been identified, they must be conveyed to the end user in a way that allows the user to act on them and to provide feedback to the system. Pie charts, decision trees, data cubes, crosstabs, and concept hierarchies are commonly used presentation tools that effectively convey the discovered patterns to a wide variety of users (Han & Kamber, 2001, pp. 157-158).
4. Technological Elements of Data Mining
Because of the inconsistent use of terminology, data mining can both be called a step in the knowledge discovery process or be generalized to refer to the larger process of knowledge discovery.
5. Steps in Knowledge Discovery
5.1 Step 1: Task Discovery
goals of the data mining operation must be well understood before the process begins: The analyst must know what the problem to be solved is and what the questions that need answers are. Typically, a subject specialist works with the data analyst to refine the problem to be solved as part of the task discovery step (Benoit, 2002).
.2 Step 2: Data Discovery
In this stage, the analyst and the end user determine what data they need to analyze in order to answer their questions, and then they explore the available data to see if what they need is available (Benoit, 2002).
.3 Step 3: Data Selection and Cleaning
Once data has been selected, it will need to be cleaned up: missing values must be handled in a consistent way such as eliminating incomplete records, manually filling them in, entering a constant for each missing value, or estimating a value. Other data records may be complete but wrong (noisy). These noisy elements must be handled in a consistent way (Benoit, 2002; Fayyad, et al., 1996).
.4 Step 4: Data Transformation
Next, the data will be transformed into a form appropriate for mining. Per Weiss, Indurkhya, Zhang & Damerau (2005), Ѓgdata mining methods expect a highly structured format for data, necessitating extensive data preparation. Either we have to transform the original data, or the data are supplied in a highly structured formatЃh (p. 1).process of data transformation might include smoothing (e.g. using bin means to replace data errors), aggregation (e.g. viewing monthly data rather than daily), generalization (e.g. defining people as young, middle-aged, or old instead of by their exact age), normalization (scaling the data inside a fixed range), and attribute construction (adding new attributes to the data set, Han & Kamber, 2001, p. 114).
.5 Step 5: Data Reduction
The data will probably need to be reduced in order to make the analysis process manageable and cost-efficient. Data reduction techniques include data cube aggregation, dimension reduction (irrelevant or redundant attributes are removed), data compression (data is encoded to reduce the size, numerosity reduction (models or samples are used instead of the actual data), and discretization and concept hierarchy generation (attributes are replaced by some kind of higher level construct, Han & Kamber, 2001, pp. 116-117).
5.6 Step 6: Discovering Patterns (aka Data Mining)
this stage, the data is iteratively run through the data mining algorithms (see Data Mining Methods below) in an effort to find interesting and useful patterns or relationships. Often, classification and clustering algorithms are used first so that association rules can be applied (Benoit, 2002, p. 278).rules yield patterns that are more interesting than others. This ЃginterestingnessЃh is one of the measures used to determine the effectiveness of the particular algorithm (Fayyad, et al.,1996; Freitas, 1999; Han & Kamber, 2001)., et al. (1996) states that interestingness is Ѓgusually taken as an overall measure of pattern value, combining validity, novelty, usefulness, and simplicityЃh (p. 41). A pattern can be considered knowledge if it exceeds an interestingness threshold. That threshold is defined by the user, is domain specific, and Ѓgis determined by whatever functions and thresholds the user choosesЃh (p. 41).
.7 Step 7: Result Interpretation and Visualization
It is important that the output from the data mining step can be Ѓgreadily absorbed and accepted by the people who will use the resultsЃh (Benoit, p. 272). Tools from computer graphics and graphics design are used to present and visualize the mined output.
5.8 Step 8: Putting the Knowledge to Use
, the end user must make use of the output. In addition to solving the original problem, the new knowledge can also be incorporated into new models, and the entire knowledge or data mining cycle can begin again.
6. Data Mining Methods
Common data mining methods include classification, regression, clustering, summarization, dependency modeling, and change and deviation detection. (Fayyad, et al., 1996, pp. 44-45)
Classification is composed of two steps: supervised learning of a training set of data to create a model, and then classifying the data according to the model. Some well-known classification algorithms include Bayesian Classification (based on Bayes Theorem), decision trees, neural networks and backpropagation (based on neural networks), k-nearest neighbor classifers (based on learning by analogy), and genetic algorithms. (Benoit, 2002; Dunham, 2003).trees are a popular top-down approach to classification that divides the data into leaf and node divisions until the entire set has been analyzed. Neural networks are nonlinear predictive tools that learn from a prepared data set and are then applied to new, larger sets. Genetic algorithms are like neural networks but incorporate natural selection and mutation. Nearest neighbor utilizes a training set of data to measure the similarity of a group and then use the resultant information to analyze the test data. (Benoit, 2002, pp. 279-280)
Regression analysis is used to make predictions based on existing data by applying formulas. U