PyCon AU 2025

Hierarchical Clustering: Finding the awkward reunions
2025-09-12 , Ballroom 1

Networks have in recent years emerged as an invaluable tool for describing and quantifying complex systems in many branches of science. Recent studies suggest that networks often exhibit hierarchical organization, where vertices divide into groups that further subdivide into groups of groups, and so forth over multiple scales. Here we present a general technique for inferring hierarchical structure from network data and demonstrate that the existence of hierarchy can simultaneously explain and quantitatively reproduce many commonly observed topological properties of networks(Eg Terrorist Networks), such as right-skewed degree distributions, high clustering coefficients, and short path lengths. We further show that knowledge of hierarchical structure can be used to predict missing connections in partially known networks with high accuracy, and for more general network structures than competing techniques. Taken together, our results suggest that hierarchy is a central organizing principle of complex networks, capable of offering insight into many network phenomena


A great deal of recent work has been devoted to the study of clustering and community structure in networks. Hierarchical structure goes beyond simple clustering, however, by explicitly including organization at all scales in a network simultaneously. Conventionally, hierarchical structure is represented by a tree or dendrogram in which closely related pairs of vertices have lowest common ancestors that are lower in the tree than those of more distantly related pairs—. We expect the probability of a connection between two vertices to depend on their degree of relatedness. Structure of this type can be modeled mathematically using a probabilistic approach in which we endow each internal node r of the dendrogram with a probability pr and then connect each pair of vertices for whom r is the lowest common ancestor independently with probability pr This model, which we call a hierarchical random graph, is similar in spirit (although different in realization) to the tree-based models used in some studies of network search and navigation . Like most work on community structure, it assumes that communities at each level of organization are disjoint. Overlapping communities have occasionally been studied and could be represented using a more elaborate probabilistic model, but as we discuss below the present model already captures many of the structural features of interest.

Given a dendrogram and a set of probabilities pr, the hierarchical random graph model allows us to generate artificial networks with a specified hierarchical structure, a procedure that might be useful in certain situations. Our goal here, however, is a different one. We would like to detect and analyze the hierarchical structure, if any, of networks in the real world. We accomplish this by fitting the hierarchical model to observed network data using the tools of statistical inference, combining a maximum likelihood approach with a Monte Carlo 2 sampling algorithm on the space of all possible dendrograms. This technique allows us to sample hierarchical random graphs with probability proportional to the likelihood that they generate the observed network. To obtain the results described below we combine information from a large number of such samples, each of which is a reasonably likely model of the data.

I am Neha, a passionate PhD researcher currently pursuing my doctorate in Precision Medicine in Oncology and Complex Disorders at the University of Newcastle. As an international student from India, I am deeply invested in exploring the intersection of machine learning, oncology, and complex disorders to advance personalized healthcare solutions.

With over 8 years of experience in the IT sector, I have worked with esteemed companies like Tata Consultancy Services (TCS) and HCL Technologies, where I honed my skills in IT infrastructure, data analytics, and machine learning. My professional journey has provided me with a solid foundation in leveraging technology to solve real-world challenges.

In addition to my research, I am proud to have received fellowships from the Indian government as a Women Scientist and CMIE Fellow, recognizing my contributions to both technology and scientific research. Outside of academia, I work part-time with organizations like Junior Engineers and Code Camp, where I teach coding to students in schools across Australia, inspiring the next generation of tech leaders.

I am deeply committed to empowering women in technology and STEM education, particularly in the areas of machine learning and data science, and I actively strive to make a difference through mentorship and community outreach. My work in precision medicine aims to bridge the gap between cutting-edge technology and clinical application, and I am excited to contribute to advancements in oncology and complex disorder research.