PhoenixMap: Spatio-Temporal Distribution Analysis with Deep Learning Classifications
VAST Challenge 2018
Mini-Challenge 1
Team Members:
Junhan Zhao, Computer Graphics Technology, Purdue University, zhao835@purdue.edu, PRIMARY
Xiang Liu, Computer and Information Technology, Purdue University, xiang35@purdue.edu
Chen Guo, School of Media Arts & Design, James Madison University, guo4cx@jmu.edu
Ryan Guan, Naperville North High School, Illinois, USA, ryguan@stu.naperville203.org
Josephine Zhang, Adlai E.Stevenson High School, Illinois, USA, jzhang9@students.d125.org
Baijian Yang, Computer and Information Technology, Purdue University, byang@purdue.edu
Zhenyu Qian, Art and Design, Purdue University, qianz@purdue.edu
Yingjie Chen, Computer Graphics Technology, Purdue University, victorchen@purdue.edu
Student Team:
YES
Tools Used:
PhoniexMap, developed by Purdue University, West Lafayette, IN
Approximately how many hours were spent working on this submission in total?
200 hours.
May we post your submission in the Visual Analytics Benchmark Repository after VAST Challenge 2018 is complete? Yes
Video
https://va.tech.purdue.edu/vast2018/answer/phoenixMap.mp4
Live System:
https://va.tech.purdue.edu/vast2018/
Questions
1
Using the bird call collection and the included map of the Wildlife Preserve, characterize the patterns of all of the
bird species in the Preserve over the time of the collection. Please assume we have a reasonable distribution of
sensors and human collectors providing the recordings, so that the patterns are reasonably representative of the
bird locations across the area. Do you detect any trends or anomalies in the patterns? Please limit your answer to
10 images and 1000 words.
Answer:
One pattern that we found intriguing was that the total bird population remains relatively stable during recent years(Figure 1.1). By analyzing our compiled data closely, we found that there has been a steady growth in many of the bird populations(Figure 1.2).
Figure 1.1 This figure shows an overall growth in bird population
Figure 1.2 This details the various maps and charts that we compiled in order to solve the problem
However, accompanying the steady growth are still some bird species (Figure 1.3,Figure 1.4) that have experienced a drastic drop in population in recent years, which requires further investigation. Our original hypothesis was that certain birds were growing because they had a favorable location away from the chemical dumping of the Kasios factory.
Figure 1.3 Some bird populations are declining
|
|
Figure 1.4 These two pictures show the clustered, unchanging nature of the Bombadil and Canadian Cootamum birds
It is clear that two significant contributing signs of a decrease in local bird populations is the yearly locational change of the population and the concentration of the population. If there are enough resources and space, a population would not need to expand. As a result, there would be a relatively stable population of birds in the area. On the other hand, a closely clumped population, known as clumped dispersion, is caused by limited resources and an uneven resource distribution, which could lead to smaller, decreasing bird populations.
Two birds that clearly show these signs are the Bombadil and Canadian Cootamum (Figure 1.5). Together, amongst similar birds, they comprise a group that represents the birds that decrease in population. Their population bar charts speak a similar story.
Figure 1.5 This details the various maps and charts that we compiled in order to solve the problem
Two birds that clearly show these signs are the Bombadil and Canadian Cootamum. Together, amongst similar birds, they comprise a group that represents the birds that decrease in population. Their population bar charts speak a similar story.
The opposite end of the spectrum explains why the bird populations are growing. Three birds, the Orange Pine Plover, the Queenscoat, and the Darkwing Sparrow represent consistent growth throughout recent years (Figure 1.6). We can see that this is correlated to the wideness and spread of their populations. Both populations encompass a larger amount of surface area than the bombadil and canadian cootamum (Figure 1.7), making it more reasonable to assume there are more Queenscoat and orange pine plovers. Furthermore, their bar charts spell the same story.
|
|
|
Figure 1.6 These show wider, moving populations of the Queenscoat, Orange Pine Plover, and Darkwing Sparrow birds.
Figure 1.7 These graphs show the steady population growth of the Queenscoat, Orange Pine Plover, and Darkwing Sparrow birds.
Figure 1.8: This graph shows the consistent spike in recordings in the spring months of the bird populations
Lastly, to address the timing of the recordings, we notice that there are a significant amount of recordings during the spring seasons of the year.
We noticed this through the large spikes in quantity when dividing the allBirds recordings by seasonal times. Although there are anomalies, many
individual birds follow this pattern of having a high number of recordings during the spring season (Figure 1.8), presumably their most active months.
These birds include but are not limited to the Darkwing Sparrow, Vermillion Trillian, and the Bombadil.
Overall, we found that while location was our preconceived factor that dealt with population sizing, we found that the population distribution and annual locational change had a lot higher correlation with the overall population change. In addition to that, we measured the changes in Shanon's and Simpson's indices (Figure 1.9). Both indices were relatively stable. However, when come to individual bird speicies, we can see these birds are under different situation as indicated in other figures.
Figure 1.9: The Shanon's and Simpson's indices measure diversity and evenness in a community/environment.
2
Turn your attention to the set of bird calls supplied by Kasios. Does this set support the claim of Pipits being found
across the Preserve? A machine learning approach using the bird call library may help your investigation. What is the role of visualization
in your analysis of the Kasios bird calls? Please limit your answer to 10 images and 1000 words.
Answer:
(1) What is the role of visualization in your analysis of the Kasios bird calls?
First, we extracted the clips from the original bird audios. Each clip is 6-second long with two adjacent clips share 4-second overlap. All clips are converted to spectrograms (Figure 2.1 left) by utilizing short time Fourier transformation.
These spectrogram is furthur processed to reduce the noise in the background (mostly low frequency noise) through scaling method (Figure 2.1 right image). From 2000 audio files, we extracted 13900 spectrograms.
|
|
Fig 2.1 original spectrogram and noise-reduced one
In preparation of deep learning training, we split the whole spectrogram dataset into training and test set. The ratio of training set to test set is 4:1, approximately 10600 and 3300 spectrograms for training and test respectively.
We used Inception V3, a convolutional neural network (CNN) developed by Google, to classify these spectrograms. Inception is a famous and innovative CNN which comes up with an idea that replacing large convolutional kernel with a sequence of smaller ones to reduce the computational time. It contains 48 layers and 1024 neurons in the prediction layer as i's initially designed for ImageNet competition. We replace the prediction layer with a 19 neurons one which represents 19 birds to accommodate our task objectives. We get 99.77% accuracy on training set, and 97% accuracy on test set. Figure 2.2 shows the architechture of our implementation of Inception v3.
Fig 2.2 Inception v3 architechture
The same preprocessing operation is also applied to the Kasios audios so as to obtain the spectrograms. Similarly, predictions based on the spectrograms from Kasios audios is performed. Afterwards, approval or disapproval of the given claim is made based on the prediction.
After we get the well-trained model, we feed the spectrograms generated from D or less quality audios to get additional bird labels, enriching the original dataset. Also, we feed the train data back again to see if we could get more labels. The related spectrograms and saliency maps are stored and displayed on the sound visualization page (Figure 2.3). This site contains the spectrograms and saliency maps for all the bird audios provided by VAST organization. The top of this screen is the menu for all 19 kinds of birds. The visualization of spectrograms and saliency maps are underneath. The first column is spectrograms and saliency maps for Kasios audios. And the last column shows the spectrograms and saliency maps of the additional bird labels. The color and brightness of saliency maps has been redesigned to make it more recognizable from the spectrograms.
Fig 2.3 Sound visualization. Only three sounds out of the 15 audio files provided by Kasios were classified to be Rose-crest Pipit (marked with orange background in the left column).
In training stage, visualization also plays an important role for accuracy improvement. We plot saliency maps (Figure 2.4 right) for hyper parameters tuning, which highlight the areas of the spectrograms that is crucial when making prediction. In this way, saliency maps could inspire us on how to tune some hyper parameters on some level. Meanwhile, saliency maps could also be used to verify the validity of trained inception v3 by checking the pattern appear in the spectrograms and saliency maps (Figure 2.4).
|
|
|
Fig 2.4 spectrogram and saliency map. Saliency map highlights important part of the original image in the computing.
(2) Does this dataset support the claim?
Fig 2.5 phoenix map and bar char for Rose-crested Pipits
However, the population of Lesser birchbeere maybe affected by this dumpsite as the population is decreasing yearly (see the bar chart of Lesser birchbeere below for decrease trendency) and the habitat is right above to the dumpsite.
Fig 2.6 phoenix map and bar char for Lesser Birchbeere
We also find a abnormal case based on our predictions which refers to Bent-beak Riffraff. Below are two maps for Orange pine Plover (Figure 2.7 left) and Bent beak Riffraff (Figure 2.8 right). Most of the prediction locations (e.g. Orange pine Plover, Bombadil, etc.) are align with the original activity area (e.g. Figure 2.7 left). However, the locations of Bent-beak Riffraff from Kasios audios are far from their typical range. Here, we come up with two hypothese for this abnormal case. If the Kasios audios are collected in year 2018, we assume Bent-beak Riffraffs are moving out to a new place which maybe affected by Kasios. Another assumption is this part of data was labelled with wrong X/Y locations.
|
|
Fig 2.7 maps for Orange pine Plover (left) and Bent-beak Riffraff (right)
3
Formulate a hypotheses concerning the state of the Rose Crested Blue Pipit. What are your primary pieces of evidence to support your assertion? What next steps should be taken in the investigation to either support or refute the Kasios claim that the
Pipits are actually thriving across the
Boonsong Lekagul Wildlife Preserve? Please limit your answer to 500 words.
Answer:
Based on the yearly population plot of the Rose Crested Blue Pipit (Rose Pipit) below, we can tell its population has been decreasing from 2015. The boundaries we created also show that the regions are shrunk and restricted.(Fig 3.1) The Pipit tend to migrate towards the south and slightly west. However, there is not enough evidence to assume that the Rose Pipits' movement is caused by the chemical dumping because the dump site is farther south.
Fig 3.1 Rose-crested Blue Pipit Yearly Population
By analyzing the data from the seasonal bar charts, we can conclude that in the first half of the year during spring and summer, Rose Pipits are very active, much more so than the second half of the year (Fig 3.2).
Fig 3.2 Rose-crested Blue Pipit Seasonal Population
Compared with the other birds living in the same Preserve, though, Rose Pipits' population increased and reached peak in 2015, then declined significantly (Fig 3.2),
especially when juxtaposed with another species of the same genus, the Carries Champagne Pipit. Carries Champagne Pipit's population is rapidly growing (Fig 3.3).
Judging from the active region highlighted by the curves, the Carries Pipits have remained in the same area, which is entirely overlapping with the dump site. The third type of Pipit is Green-tipped Scarlet Pipit.
It experienced a big drop in 2015. Only five out of ten were detected compared with 2014. After that, it seems that the Green Pipit has stable increase. However, the exact number of this type of bird is still quite small.
Green Pipits are behavoring more in the southwest before 2015. After that, they preferred northwest. There is no clue showing their activity regions were impacted by the dumping site, but the number of the bird may be.
It is difficult to say if their population would be rising even more than now without the dumping site implanted.(Fig 3.3,Fig 3.4,Fig 3.5)
Fig 3.3 Carries Cahampagne Pipit (top), Green-tipped Scarlet Pipit (bottom) Yearly Population
Fig 3.4 yearly region change of Rose Pipit from year 2013 to 2017. The right end image shows year 2011 to 2017 together.
Fig 3.5 Pipits Regions Maps
From the audios provided by Kasios, we detected three recordings belonging to the Rose Pipit. Although they did not appear in the concentrated area of the bird's region, they are still largely within the active area. (This part can use the sound page - in the left column, sounds belong to the bird will be highlighted in orange. Also the map with green dots)
We suggest paying more attention to human activities at north east region around the spot (150,150). In 2017 MC1, They surreptitiously dumped process waste in the northeast region of the Preserve. We also found that for birds who lived at the bottom of this area, such as Lesser Birchbeere, purple tootling tout, pinkfinch, their population is in sharp decline. Factories are located in this area. It worth further investigation to study how these birds get affected by factories.
We will also recommend recording the birds more based on statistical requirements, regarding the experiment design. The recording time and amount should be constant. Furthermore, it is worth comparing with other Preserve which has Rose Pipits inhabiting. The comparison with similar conditions will provide more evidence of differentials.