Entry Name: UKON-Cakmak-MC1

VAST Challenge 2018
Mini-Challenge 1

Team Members:

Eren Cakmak, University of Konstanz, cakmak@dbvis.inf.uni-konstanz.de PRIMARY

Udo Schlegel, University of Konstanz, udo.3.schlegel@uni-konstanz.de

Matthias Miller, University of Konstanz, miller@dbvis.inf.uni-konstanz.de

Wolfgang Jentner, University of Konstanz, , jentner@dbvis.inf.uni-konstanz.de

Juri Buchmüller, University of Konstanz, buchmueller@dbvis.inf.uni-konstanz.de

Student Team: YES

Tools Used:

SoX - Sound eXchange library http://sox.sourceforge.net/

Librosa - Python library for audio and music analysis https://github.com/librosa/librosa

Dejavu - Audio fingerprinting and recognition in Python https://github.com/worldveil/dejavu

SpectrogramJS – developed by Jim Vallandingham and extended by Eren Cakmak for this project https://github.com/vlandham/spectrogramJS

International Bird Rescue Tool – developed by Eren Cakmak, Matthias Miller, and Udo Schlegel

Approximately how many hours were spent working on this submission in total?

80h

May we post your submission in the Visual Analytics Benchmark Repository after VAST Challenge 2018 is complete? YES

Video

https://youtu.be/eBdkTpPpUrw

Questions

1. Using the bird call collection and the included map of the Wildlife Preserve, characterize the patterns of all of the bird species in the Preserve over the time of the collection. Please assume we have a reasonable distribution of sensors and human collectors providing the recordings, so that the patterns are reasonably representative of the bird locations across the area. Do you detect any trends or anomalies in the patterns? Please limit your answer to 10 images and 1000 words.

1.1

Figure 11: Distribution of bird population recordings in the Wildlife Preserve (a) and a convex hull and a heatmap for each bird population of the whole collection (b).

Figure 1a shows the spatial distribution of all bird call recordings existing in the collection. For instance, the Rose-crested Blue Pipit (182 red dots) is mostly recorded in the north-east of the preserve divided into two Gaussian clusters. Similarly, the bird types Queenscoat, Bent-beak Riffraff, Scrawny Jay, Lesser Birchbeere, Vermillion Trillian, and Orange Pine Plover are also each subdivided into two clusters having single outliers mainly distributed in between. The bird recordings of Blue-collared Zipper, Bombadil, Broad-winged Jojo, Canadian Cootamum, and Carries Champagne Pipit form single clusters with many outliers that are almost distributed over the whole map (Figure 1b). Ordinary Snape and Qax are bird groups that have a very small deviation in their geographic distribution consisting of a more dense cluster compared to those mentioned above. The remaining bird categories Darkwing Sparrow, Eastern Corn Skeet, Green-tipped Scarlet Pipit, Pinkfinch, and Purple Tooting Tout do not tend to form clusters. Instead, these species are broadly distributed over the map and slightly skewed to the Eastern direction. Figure 1b displays a heatmap by using opacity to indicate dense regions in contrast to areas that contain outliers primarily. The color of the convex hulls is linked to the respective bird species. The hulls reveal that the majority of bird generations exist in the east rather than in the western part of the preserve. From last years VAST Challenge, we know that the dumping took place on Tuesdays and Thursdays in from May 2015 - May 2016. The location of the dump overlaps with the northern cluster of the Rose-crested Blue Pipit.

In the above images, we visualize the number of recordings and encode the age of the recording using the opacity to highlight the latest recordings. We distinguish between songs, calls, unknown and map this to the shape of a cross, circle, and a triangle respectively. The Kasios files are in the shape of a star as they are of particular interest. Even though overlapping exists in many areas, single outliers are quickly identifiable for each bird type.

We describe our insights over the whole time span in the following:

a) Vermillion Trillian: Two clusters - center top, left center - calls + songs uniform distributed

b) Scrawny Jay: Two clusters - left center, left top - more calls in recent years

c) Rose-crested Blue Pipit: Two clusters + many outliers - right center and right top - more calls in recent years

d) Queenscoat: Three Clusters - left center (2x), left top - calls + songs uniform distributed

e) Qax: One cluster - left center - more songs in general

f) Purple Tooting Tout: No clusters - spread over the left bottom to the center - calls + songs uniform distributed

g) Pinkfinch: No clusters - spread over the left bottom to the center - only songs - close to Purple Tooting Tout

h) Ordinary Snape: One cluster - right center - calls + songs uniform distributed

i) Orange Pine Plover: Two clusters + some outliers - left bottom + center bottom - calls + songs uniform distributed

j) Lesser Birchbeere: One cluster + many outliers - left bottom - a lot more calls than songs

k) Green-tipped Scarlet Pipit: Three clusters - left bottom, left top (2x) - more songs than calls

l) Eastern Corn Skeet: One cluster + many outliers - center - a bit more songs than calls

m) Darkwing Sparrow: Two clusters + some outliers - left bottom, center top - calls + songs uniform distributed

n) Carries Champagne Pipit: One cluster + many outliers - right bottom - a lot more calls than songs

o) Canadian Cootamum: One cluster + some outliers - left top - more calls than songs

p) Broad-winged Jojo: One cluster + many outliers - left bottom - calls + songs uniform distributed

q) Bombadil: One cluster + few outliers - center top - calls + songs uniform distributed

r) Blue-collared Zipper: One cluster - left bottom - calls + songs uniform distributed

s) Bent-beak Riffraff: Two clusters sparse - left center, left top - a lot more songs than calls

Further insights:

b) Shares cluster with p), s)

c) Was one cluster and moved from top to the center

d) One of the cluster vanishes

e) Slowly vanished from cluster or moves a bit

f) Close to Pinkfinch

h) Leaves in the winter

k) Was one cluster - moved from bottom to top

m) Was one cluster - moved from top to bottom, shares cluster with q)

p) Leaves in the winter, shares cluster with b), s)

q) Leaves in the winter, shares cluster with m)

s) Shares cluster with b), p)

Some small insights are:

The Scrawny Jay (b), the Broad-winged Jojo (p) and the Bent-beak Riffraff (s) share one cluster.

The Rose-crested Blue Pipit (c) and the Ordinary Snape (h) share a cluster.

The Vermillion Trillian (a) and the Qax (e) share one cluster.

The Broad-winged Jojo (p), the Bent-beak Riffraff (s) and the Ordinary Snape (h) leave in the winter and come back for the summer.

The Rose-crested Blue Pipit (c) moves during the duration of the dataset from north-east further to the south. The north-eastern region gets abandoned by the bird species.

The Queenscoat (d) also abandons a region and moves on a cluster to one of the other two. This process starts in the beginning to mid of 2016.

The Qax (e) slowly vanishes from the preserve. This population of this species is already small. A movement of the cluster is visible as well as its disappearance.

The Purple Tooting Tout (f) often forms groups with the Pinkfinch (g) in the preserve.

The Green-tipped Scarlet Pipit (k) moves a few times. First from left top to left bottom and then to the center top.

The Darkwing Sparrow (m) shares at first a cluster with the Bombadil (q) but migrates to another region from the center top to the bottom left.

The Qax (e), the Ordinary Snape (h), and the Bombadil (q) are always at one location and have very little outliers.

Qax, Vermillion Trillian and Queenscoat vanish from a location in the map.
Especially, Qax vanishes completely from their cluster.
The picture above shows the Qax population of the last few years. The top row shows the last five years. A move and a vanishing can be seen there. From 3.2016-3.2017 the population is not in the circle anymore but below.
The bottom row shows exactly this movement a bit to the south. This could be because the Green-tipped Scarlet Pipit moves away from this location. Or because there is another event in the region above.

Figure: The distribution of the bird populations Broad-winged Jojo, Qax, Queenscoat and Vermillion Trillian over many years in half yearly rhythm (see date ranges). The red square highlights an area, where the number of recordings is continually decreasing with one exception in spring 2017. The Qax population does not occur anymore after autumn 2017.

The distribution of the bird populations shown in the figure above shows the movements over the years from January, 2014 to March 2018 by an almost half yearly rhythm. The red square highlights an area where the Qax population seems to become extinct (after July 2017) whereas many recordings where made regularly before. Similarly, the recordings of the Vermillion Trillian population tends to go away after winter 2016. The dumping site seems to change after May 2016 and this image highlights an area, where the dumping location could be changed to this position. The number of recordings after June, 2017 supports this claim, because neither Qax, Broad-winged Jojo nor Vermillion Trillians are recorded anymore which occurred regularly even if not often in this area.

1.2

Figure 2 shows the development of all birds in contrast to the Rose Crested Blue Pipit.
A change from the top right area to an area a bit more to the south is visible.
This is a special event as it is only observable in the last few years starting from the end of 2014 and the beginning of 2015.
At first, only the songs move and then the calls follow. Eventually in the northeastern area, almost no recordings of any bird type occur any more, except some outliers.

1.3

Figure 32: The temporal change of recording locations using a Hilbert Curve on the map to visualize the monthly movement of the cluster centers of every bird population over the whole time range.

This visualization shows the position of the cluster centroid of all bird types within a time range of a month. The vertical position of the lines indicates the location of the center of the respective cluster using a Hilbert Curve. The size of the circles displays the number of call and song recordings for each bird type for every month. Even though the visualization shows some clutter, the areas of the circles indicate that between 2013 and spring, 2016 the majority of recordings exist. From summer 2016 and later the number of recordings drastically decreases for many of the available bird populations in the preserve. Especially the Rose-Crested Blue Pipit population decreases significantly after spring 2016. At the end of 2015, some outliers were recorded in different locations than the typical cluster center.

Figure 45: Positions of cluster centers using polar coordinates with the center of the map as the origin.

1.4 Figure 4 indicates the direction from the center point for each population. The polar coordinate system starts in the east (0 degrees) and continues in a counter-clock direction (e.g., 90° south). This view provides information about the relative movements between the different bird populations. Large jumps in the image either indicate periods with missing values or reveal that the cluster center moved over the center of the map. Generally, the line movements are quite stable. For example, the Rose-Crested Blue Pipit and the Ordinary Shape mainly stay between 200°-240° and are very close to each other. This visualization also shows that in the east (180°) no clusters have built over all years.

1.5

Figure 5: The stars indicate the alleged Rose-crested Blue Pipit recordings from the Kasios company. The red symbols pinpoint the historical recordings from 2011/10/01 – 2018/03/30.

Figure 5 simultaneously shows the alleged Rose-Crested Blue Pipit recordings from the Kasios company (brown stars). The red dots denote the actual historical recordings. These dots form a cluster in the north-eastern region of the Wildlife Preserve. The main recordings from Kasios are located in the western area of the preserve. However, actual tapes of the Rose-Crested Blue Pipit are generally located in the north and eastern part of the preserve. Typically, the “proof” of Kasios does not seem to be significant. Since many other bird types are clustered and settled in this region (see Figures 1-4), it is more likely that these recordings originate from different populations.

2. Turn your attention to the set of bird calls supplied by Kasios. Does this set support the claim of Pipits being found across the Preserve? A machine learning approach using the bird call library may help your investigation. What is the role of visualization in your analysis of the Kasios bird calls? Please limit your answer to 10 images and 1000 words.

We analyzed the Kasios dataset with two tools: an Interactive Audio Classification tool and a visual tool with Audio Glyphs. We could not identify any Rose Crested Blue Pipit in any of the 15 audio files. The following table shows our classification results.

File	Classification Results
Kasios File: 1	Bent-Beak-Riffraff-106265, Vermillion-Trillian-42461, Pinkfinch-387635
Kasios File: 2	Ordinary-Snape-132146
Kasios File: 3	Darkwing-Sparrow-192305
Kasios File: 4	Ordinary-Snape-132146
Kasios File: 5	Lesser-Birchbeere-160790
Kasios File: 6	Green-Tipped-Scarlet-Pipit-278068
Kasios File: 7	Qax-50739
Kasios File: 8	Lesser-Birchbeere-95720, Lesser-Birchbeere-190805
Kasios File: 9	Ordinary-Snape-183362
Kasios File: 10	Eastern-Corn-Skeet-388696, Orange-Pine-Plover-282286
Kasios File: 11	Lesser-Birchbeere-190805
Kasios File: 12	Pinkfinch-387635, Eastern-Corn-Skeet-176000 (partly in the background)
Kasios File: 13	Queenscoat-126385, Queenscoat-367046
Kasios File: 14	Canadian-Cootamum-96742
Kasios File: 15	Bent-Beak-Riffraff-103177, Pinkfich-387635

To analyze the files supplied by Kasios, we preprocessed the ground truth of 2080 bird calls with the SoX (Sound eXchange) library. We removed the audio noise, silent moments and amplified the remaining bird calls/songs. Afterward, we used the dejavu audio fingerprinting and recognition in Python to create 280 million fingerprints. Our classification is a similarity search in a database of 280 million fingerprints that were precomputed using a rolling window approach.

The interactive classification tool works as follows. The tool opens after clicking on a Kasios file on the Bird Map visualization (following Figure). The view (1) shows an interactive spectrogram view which can be played, paused, and stopped. Additionally, it depicts different frequency ranges. The spectrogram can be zoomed and panned to interesting moments. The visible component of the spectrogram view can then be classified against the fingerprint database. The tool then loads a second spectrogram to show which audio file seems to be similar which allows to interactively checking if the bird sounds the same.

These steps allow the visual comparison of the spectrograms of the Kasios file and the classified bird species. The following Figure shows a sequence of Kasios file 8 which is classified as a part of the Lesser-Birchbeere-95720. The analyst can then validate that sound and the spectrogram both match.

The following example shows a classification for the Kasios file 1.wav. In the spectrogram, you can visually identify three different birds. The playback of the file confirms this audibly. The individual birds were zoomed and classified in the interactive classification tool. The classification results are classified as (1) - Pinkfinch-387635, (2) is classified as the Bent-Beak-Riffraff-106265, and (3) is classified as Vermillion-Trillian-42461.

Each file was analyzed and classified as shown in the table above. We also found other artifacts in the Kasio Audio Files.

· The Kasiso file 2.wav, 3.wav, 8.wav, 9.wav, 13.wav and 14.wav has a silent moment at the beginning. During this silence, there is also no noise, which is unusual. This file seem also to be tampered with.

· The Kasiso file 3.wav and 14.wav seems to have no noise. The noise of this file was probably removed.

· The Kasios file 6.wav appears to contain several copied audio recordings. Between these copied audio snippets there are silent moments and the noise in the file also seems to be different in each copied snippet.

· The Kasios file 7.wav is file of multiple copies of one birdcall of the Qax.

· The Kasios file 10.wav has also measurements bigger than 16kHz. This is also quite unusual, since no other birdcall has such a high frequency.

· The Kasios file 15.wav has a silent gap in between the recording, which lets us assume that this file consists of two copied audio snippets. Further, at the end of the file is also silence without noise.

In addition to all this our classification approach also found humans voices in Rose-Crested-Blue-Pipit-237009, Orange-Pine-Plover-244585 and in Blue-collared-Zipper-251570, which should have been removed to improve the classification results.

We also developed also audio glyphs. The features visualized in these glyphs were extracted using the Python audio library librosa. We extracted three characteristics features for the different audio files:

· Spectral centroid: This feature computes the “average” frequency for each frame. The different frequencies are weighted relatively by their energy.

· Spectral bandwidth: This feature computes the bandwidth range for the different frequencies for each frame. This feature could be seen as variance.

· Spectral rolloff: This feature finds the frequency that has a strong energy. This is the 0.85 quantile of the frequency sorted by the energy.

The following Figure shows the spectral rolloff for the 19 different bird species in the preserve. An audio glyph is designed similar to a clock view visualization. In this case the extracted spectral rolloff is mapped to a global color scale for all birds. Different characteristics and outliers of birds calls are clearly visible in the following Figure.

This visualization of spectral rolloffs can then be used to visually compare individual Kasios files against other audio glyphs of the bird species. For instance, the following Figure shows 25 Rose-Crested Blue Pipits. They are visually nearly all the same except for some outliers. These outliers can be produced by noise which could not be removed through the noise removal or other birds.

The following Figure shows the 15 files provided by the Kasisos Company. It is visible that most of the birds do not match to the characteristic patterns of the Figure above. However, some seem to have the same colors and partly also the same patterns. This visualization can be easily used to dismiss multiple files of the Kasios company. The other files have to be analyzed in detail with our Interactive Audio Classification tool.

3. Formulate a hypotheses concerning the state of the Rose Crested Blue Pipit. What are your primary pieces of evidence to support your assertion? What next steps should be taken in the investigation to either support or refute the Kasios claim that the Pipits are actually thriving across the Boonsong Lekagul Wildlife Preserve? Please limit your answer to 500 words.

The Rose-Crested Blue Pipit population has two clusters during the time span of the recordings. However, in most cases, the cluster at the top right corner is filled before the other one. After the dumping at the end of 2014, no bird is recorded in this area anymore. Even after abandoning the region in the past for a short time, the birds migrated back. After the end of 2014, no bird revisited this region. This indicates that in this area the conditions for reproduction might have worsened. This hypothesis can be confirmed through the fact that no other bird population has tried to settle in this region after 2014. It seems that this area is in general unattractive for any bird population. The polluted area is especially dangerous for the Rose-Crested Blue Pipit because it appears that the species has no other habitat to populate within the Wildlife Preserve. Either other species already take all suitable habitats, or the other areas do not provide ideal environments.

Notably, the bird distribution shows that there is nearly no habitat room anymore for another species, due to the destruction of the area in the top corner. Based on the population movements chart, one can view that the number of the Rose-Crested Blue Pipit population recordings decreased significantly after 2015 with the tendency to become extinct (2015: 45, 2016: 27, 2017: 16, 2018: 3). The flourishing period of the Rose-Crested Blue Pipit population was between 2013 and the end of 2015 (2013: 23, 2014: 16, 2015: 45). The lower value in 2014 is a small fluctuation, which appears a few times in the data. However, from spring 2016 and later the Rose-Crested Blue Pipits are declining rapidly as they are perhaps suffering from existence-threating circumstances. There is no fluctuation anymore, but a steep decline is visible. If no countermeasures are taken, the Rose-Crested Blue Pipit will most likely vanish from the preserve. The classification of the recordings from Kasios did not correspond to the Rose-Crested Blue Pipit recordings that were available in the recording history.