Network data mining: methods and techniques for discovering deep linkage between attributes

Network Data Mining identifies emergent networks between myriads of individual data items and utilises special algorithms that aid visualisation of 'emergent' patterns and trends in the linkage. It complements conventional data mining methods, which assume the independence between the attributes and the independence between the values of these attributes. These techniques typically flag, alert or alarm instances or events that could represent anomalous behaviour or irregularities because of a match with pre-defined patterns or rules. They serve as 'exception detection' methods where the rules or definitions of what might constitute an exception are able to be known and specified ahead of time. Many problems are suited to this approach. Many problems however, especially those of a more complex nature, are not well suited. The rules or definitions simply cannot be specified. For example, in the analysis of transaction data there are no known suspicious transactions. This chapter presents a human-centred network data mining methodology that addresses the issues of depicting implicit relationships between data attributes and/or specific values of these attributes. A case study from the area of security illustrates the application of the methodology and corresponding data mining techniques. The chapter argues that for many problems, a 'discovery' phase in the investigative process based on visualisation and human cognition is a logical precedent to, and complement of, more automated 'exception detection' phases.