Entry Name: "QAX-Zhang-MC3"

VAST Challenge 2019
Mini-Challenge 3

Team Members:

Mengyang Zhang, Qi An Xin, zhangmengyang@qianxin.com PRIMARY

Kaokao Lv, Qi An Xin, lvkaokao@qianxin.com

Liang Tang, Qi An Xin, tangliang@qianxin.com

Chuanming Huang, Qi An Xin, huangchuanming@qianxin.com

Zhaokang Yuan, Qi An Xin, yuanzhaokang@qianxin.com

Xiaobo Luo, Qi An Xin, luoxiaobo@qianxin.com

Student Team: NO

Tools Used:

Python

R

Javascript

Pandas

Numpy

Sklearn

Echarts.js

d3.js

Turf.js

Approximately how many hours were spent working on this submission in total?

300h

May we post your submission in the Visual Analytics Benchmark Repository after VAST Challenge 2019 is complete? YES

Video

Included in the electronic submission package 

Questions

The City has been using Y*INT to communicate with its citizens, even post-earthquake. However, City officials needs additional information to determine the best way to allocate emergency resources across all neighborhoods of St. Himark. Your task, using your visual analytics on the community Y*INT data, is to determine the types of problems that are occurring across the St. Himark. Then, advise the City on how to prioritize the distribution of resources. Keep in mind that not all sources on Y*INT are reliable, and that priorities may change over time as the state of neighborhoods also changes.

1 – Using visual analytics, characterize conditions across the city and recommend how resources should be allocated at 5 hours and 30 hours after the earthquake. Include evidence from the data to support these recommendations. Consider how to allocate resources such as road crews, sewer repair crews, power, and rescue teams. Limit your response to 1000 words and 12 images.

With machine learning algorithms and manual tagging method, we summarize what people are saying by classifying all the messages into several topics, including shaking, power, food and water, sewer, road, gas, building collapsed, casualties, shelter, rumble, fire disaster. The figure 1 shows heat distribution of these topics over time.

Fig. 1-1: heat distribution of topics

Combining the Y*INT messages and shake maps of the earthquake strength, we find that the earthquake happens at around 08:35 on 04/08/2020, at that time most people feel shaking and the conditions of the city changes a lot. More accurately, a 6.7 earthquake just occur off the NE shore of the town of St. Himark at 08:36:00 as stated in messages. The earthquake has a serious impact on the city. People suffer from many destructive disasters, like building collapsed and power outage.

In order to find the best way to allocate resources wthin 5 to 30 hours after the earthquake, we have figured out the problems the neighborhoods encounter from the topic data we divided. And we define 6 main problems, which contain power outage problem, sewer problem, road problem, fire problem, gas problem, rescue problem. For each problem, we mark the affected neighborhoods with color on the map as is shown in figure 2 to 11, and the shade of color indicates the seriousness of the problem. Meanwhile, some supportive messages are presented in the figures. We also extract detail descriptions of problems to help allocate resources.

According to the city background and 6 defined problems, there are 6 resources that can be allocated to neighborhoods, including road crews, sewer crews, power company, rescue teams, fire stations and gas company. Three priority levels (high, medium, low) are considered by the colors in the figure 2 to 11, which is related to the seriousness of the problems. Furthermore, table 1 and table 2 below play a guiding role to allocate resources to different locations at 5 hours and 30 hours after the earthquake.

Within 5 hours, the conditions are:

1. For the power problems, the power of temporary animal shelter and hospital is out at Palace Hills and Terrapin Springs respectively. At Broadview, the power line tilted over causing power outage.

Fig. 1-2: power problems

2. For the sewer problems, Sewer pipe broken occurs at Downtown and Old Town. At Palace Hills, sewer pipe is dry. At Easton, sewer smells so gross. At Scenic Vista, broken water and sewer pipes create risk of contaminating drinking water.

Fig. 1-3: sewer problems

3. For the road problems, At Easton, rubbles in the streets are slowing traffic everywhere. And traffic is gridlocked at the rest of these neighborhoods.

Fig. 1-4: road problems

4. For the rescue problems. Buildings collapse and someone lost contact at Broadview, Weston and Palace Hills. At Palace Hills and Old Town, someone has broken foot and need doctor. At Weston, 2 people injure in school bus crash. At Scenic Vista, 3 blocks of waterfront property at Entitled Acres housing development have slid into the water.

Fig. 1-5: rescue problems

5. For the fire problems, Red Hot Fire Extinguisher Company and shoe store are on fire at Oak Willow and Broadview respectively.

Fig. 1-6: fire problems

Table 1: resources allocating at 5 hours after earthquake

	road crews	sewer crews	power company	rescue teams	fire stations	gas company
Downtown		Low
Safe Town
Pepper Mill			Low
East Parton
Cheddarford
Southton				Low
Oak Willow					Low
West Parton	Medium
Chapparal	Low
Wilson Forest
Terrapin Springs	Low		Low
Broadview			Low	Low	High
Southwest					High
Scenic Vista		High		High
Northwest
Palace Hills		Low	Low	Low
Easton	High	Low			Low
Old Town	Medium	Low		Medium
Weston				Low

Within 5 to 30 hours, the conditions are:

1. For the power problems, most area have no power, especially the hospital by the neighborhood’s power is out at Broadview. At Scenic Vista, the power of the temporary animal shelter by the restaurant and the pharmacy by the Elementary School is out. At Weston, the power of the pharmacy by the neighborhood and the grocery store by the fire station is out. At Southwest, the power of the gas station and the pharmacy by the fire station is out.

Fig. 1-7: power problems

2. For sewer problems, sewer pipes are dry on this end at Southwest and Broadview. At Old Town and East Parton sewer pipes break and water flood.

Fig. 1-8: sewer problems

3. For road problems, road condition is a mess and traffic is jammed owing to bricks and rubbles at these affected locations.

Fig. 1-9: road problems

4. For gas problems, the events of gas leaking, and gas shortage for boiling water are mentioned in these affected locations.

Fig. 1-10: gas problems

5. For rescuing problems, the buildings of house, hospital and the University collapse, and someone can’t be contacted at Weston. At Old Town, church collapses and someone’s building collapse which traps him inside. At West Parton, the Elementary School collapses and someone’s grandma who gets trapped inside has lost contact. At Northwest, Pizza Hut collapses and someone’s mother who gets trapped inside has lost contact. At Broadview, fire station collapsed and someone’s dad who gets trapped inside has lost contact. At Palace Hills, the Elementary School collapses and someone’s dad who gets trapped inside has lost contact. At Easton, Police Station collapses and someone’s friend who gets trapped inside has lost contact.

Fig. 1-11: rescue problems

Table 2: resources allocating at 30 hours after earthquake

	road crews	sewer crews	power company	rescue teams	fire stations	gas company
Downtown	High		Low	High		Low
Safe Town
Pepper Mill
East Parton		Low	Low
Cheddarford	Low		Low			Low
Southton	Low			Medium		High
Oak Willow			Low	Low		Low
West Parton	High			High
Chapparal	High		Medium	High
Wilson Forest
Terrapin Springs	Low		Low	Low		High
Broadview	Low	Low	High	High
Southwest	Low	High	High	Low
Scenic Vista	Low		High	High		Low
Northwest	Low		Low	High		High
Palace Hills				Low
Easton		Low	Medium	High		Low
Old Town	High	High	High	High		Low
Weston	Medium		Low	High		High

2 – Identify at least 3 times when conditions change in a way that warrants a re-allocation of city resources. What were the conditions before and after the inflection point? What locations were affected? Which resources are involved? Limit your response to 1000 words and 10 images.

As time goes on, conditions among neighborhoods are changing. We use the 6 main categories of problems defined in the answer of question 1 to help us to pick up the inflection point. And 5 inflection points are selected with the analysis of problems distribution changing over time. Based on these 5 inflection points, we analyze the alternation of conditions of neighborhoods before and after the inflection points by the figures of problems distribution and the tables that show comparison of between conditions. At last, a resources re-allocation table for affected neighborhoods is given at each inflection point.

The first inflection point is at 2020-04-08 09:00:00. As we can see from table 1 and figure 1, no neighborhood suffers from disasters and people still live a safe life before this point. After this point, many serious problems appear, which include power outage, sewer broken, traffic jam and gas leaking. According to the statistics of table 1, table 2 is established for re-allocating resources to the affected neighborhoods. And it is obvious to get the involved resources from table 2.

Table 1: comparison of conditions between neighborhoods before and after the inflection point

	Before	After
Power problems		Terrapin Springs, Broadview, Palace Hills, Pepper Mill
Sewer problems		Downtown, Scenic Vista, Palace Hills, Easton, Old Town, Terrapin Springs
Road problems		West Parton, Chapparal, Easton, Old Town
Fire problems		Oak Willow, Broadview, Southwest, Easton
Rescue problems		Scenic Vista, Palace Hills, Old Town, Weston, Southton, Broadview
Gas problems

Fig. 2-1: problems distribution before and after the inflection point

Table 2: resources re-allocation for affected neighborhoods

	road crews	sewer crews	power company	rescue teams	fire stations	gas company
Downtown		Need
Pepper Mill			Need
Southton				Need
Oak Willow					Need
West Parton	Need
Chapparal	Need
Terrapin Springs		Need	Need
Broadview			Need	Need	Need
Southwest					Need
Scenic Vista		Need		Need
Palace Hills		Need	Need	Need
Easton	Need	Need			Need
Old Town	Need	Need		Need
Weston				Need

As for the rest 4 inflection points, 3 questions listed in this section could be easily answered from figures and tables below. The names of the neighborhoods written in green show that these neighborhoods are free from the corresponding problems after the inflection points. On the other hand, names written in red mean the neighborhoods which do not have the corresponding problems before the inflection points but have them after the points. Through the figure 2 to 5, conditions before and after the inflection points are displayed in a visualized way. Table 4, 6, 8, and 10 show the affected locations and involved resources.

The second inflection point is at 2020-04-08 16:00:00.

Table 3: comparison of conditions between neighborhoods before and after the inflection point

	Before	After
Power problems	Terrapin Springs, Palace Hills, Broadview, Easton, Pepper Mill, Weston	Weston, Chapparal, Scenic Vista, Oak Willow, Southwest, Cheddarford, Downtown, East Parton
Sewer problems	Scenic Vista, Palace Hills, Easton, East Parton, Old Town, Southwest	Old Town, Southwest, Broadview
Road problems	Old Town, Downtown, Scenic Vista, Terrapin Springs, Broadview, Northwest, Southton, Chapparal, Cheddarford, Southwest, West Parton, Weston	Southwest, Chapparal, Weston, West Parton
Fire problems	Broadview
Rescue problems	Downtown, Easton, Oak Willow, Safe Town, Chapparal, Old Town, West Parton, Scenic Vista, Weston, Southton, Palace Hills	Old Town, Scenic Vista, Weston, Southton, West Parton, Palace Hills, Chapparal, Southwest, Northwest, Broadview, Cheddarford
Gas problems

Fig. 2-2: problems distribution before and after the inflection point

Table 4: resources re-allocation for affected neighborhoods

	road crews	sewer crews	power company	rescue teams	fire stations	gas company
Downtown			Need
Oak Willow			Need
Chapparal			Need
Broadview		Need		Need
Southwest			Need	Need
Scenic Vista			Need
East Parton			Need
Cheddarford			Need	Need
Northwest				Need

The third inflection point is at 2020-04-09 01:00:00.

Table 5: comparison of conditions between neighborhoods before and after the inflection point

	Before	After
Power problems	Old Town, Cheddarford, Southwest	Southwest, Northwest
Sewer problems	Old Town	Old Town
Road problems	Weston	Weston, Old Town, Southwest, Broadview, West Parton, Downtown
Fire problems
Rescue problems	Weston, Scenic Vista, Southton, Easton, Broadview, Terrapin Springs, Northwest, Palace Hills, Safe Town	Old Town
Gas problems

Fig. 2-3: problems distribution before and after the inflection point

Table 6: resources re-allocation for affected neighborhoods

	road crews	sewer crews	power company	rescue teams	fire stations	gas company
Downtown	Need
West Parton	Need
Broadview	Need
Southwest	Need
Old Town	Need			Need
Northwest			Need

The forth inflection point is at 2020-04-09 15:00:00.

Table 7: comparison of conditions between neighborhoods before and after the inflection point

	Before	After
Power problems	Broadview, Cheddarford, Southwest, Oak Willow, Northwest, Downtown, Terrapin Springs	Southwest, Downtown, Terrapin Springs, Northwest, Oak Willow, Chapparal, , Southton, Weston
Sewer problems		Easton, Southwest
Road problems	Chapparal, Southwest
Fire problems
Rescue problems	Southton, Northwest, Terrapin Springs, Cheddarford, Chapparal, Downtown, Broadview, Scenic Vista, Easton, Weston, Southwest, Oak Willow, Old Town	Weston, Southwest, Broadview, Easton, Old Town, Oak Willow, Scenic Vista, West Parton
Gas problem	Northwest, Southton, Terrapin Springs, Scenic Vista, Downtown, Oak Willow, Old Town, Cheddarford, Easton, Weston	Weston

Fig. 2-4: problems distribution before and after the inflection point

Table 8: resources re-allocation for affected neighborhoods

	road crews	sewer crews	power company	rescue teams	fire stations	gas company
Southton			Need
West Parton				Need
Chapparal			Need
Southwest		Need
Easton		Need
Weston			Need

The fifth inflection point is at 2020-04-10 02:00:00.

Table 9: comparison of conditions between neighborhoods before and after the inflection point

	Before	After
Power problems	Palace Hills, West Parton	West Parton, Easton, Downtown, Cheddarford, Old Town, Southwest
Sewer problems
Road problems		Palace Hills
Fire problems
Rescue problems	West Parton, Palace Hills	West Parton, Palace Hills, Scenic Vista, Weston, Oak Willow, Downtown, Northwest, Cheddarford, Easton, Chapparal, Southwest
Gas problems

Fig. 2-5: problems distribution before and after the inflection point

Table 10: resources re-allocation for affected neighborhoods

	road crews	sewer crews	power company	rescue teams	fire stations	gas company
Downtown			Need	Need
Oak Willow				Need
Chapparal				Need
Southwest			Need	Need
Scenic Vista				Need
Palace Hills	Need
Easton			Need	Need
Old Town			Need
Weston				Need
Cheddarford,			Need	Need
Northwest				Need

3 – Take the pulse of the community. How has the earthquake affected life in St. Himark? What is the community experiencing outside the realm of the first two questions? Show decision makers summary information and relevant/characteristic examples. Limit your response to 800 words and 8 images.

The earthquake affects people in many aspects, it interferes normal life in the city. Citizens are experiencing not only physical but also mental harms. Buildings collapse everywhere, people cannot go back home and can hardly find place to rest. Their basic needs can no longer be guaranteed, including accommodation and food. Municipal infrastructure is badly damaged, power is off in many district and phone service is unstable. People are separated with family and friends. Safety is not guaranteed, scheduled activities are interrupted. Fear, panic, sadness and anger hangs over the city. The entire city is in crisis.

Except repair and rescue related topics mentioned in the first two questions, the community also is talking about topics including scheduled activities, finding missing people, resettling citizens, history protection and scam events. Characteristic examples are shown as follows.

Scheduled activities are interrupted

Due to earthquake, activities like concerts and student rodeo are cancelled, schools are closed and students go back home. For instance, it should have been a wonderful night in April 8 for Lacki and her fans if the earthquake didn't happen, she would perform her new hit in the Old Town Theater as schedule. Yet another news that Lacki is missing and possibly buried in the rubble of hotel comes out. People talk about her and the concert is not staged at that night. On the next day Lacki makes appearance at a shelter and fans are relieved that she is safe. Lacki has called for a global moment of silence and asks people send donations. Finally Lakci and several other celebrities will perform in April 10 at the Union Hall for free. Following figure shows the message and trending of Lacki.

Search the missing, watch out life threatening accident

After the earthquake, lots of people couldn’t contact with their family and friends. What is worse, phone service is unstable. Survivors post information online asking for help. There is still many people missing. The City Health Department establishes a family assistance center where people can go to report missing. Meanwhile we should pay attention that landslide and flood could happen at any time, life is in danger. EOC ask citizens to report immediately when landslide and flood happens. Following figure shows the trending of landslide and flood.

Essential living elements cannot be guaranteed

The government worries about the collapse of building, and puts the red tag on those building, which makes people feel frustrated. People have nowhere to go, they have to camp in the park or wander in the streets. At the same time, city ordinance prohibits camping in the parks. If not taking down tents and leave, people will be subject to arrest. According to messages, police also arrested people on the street who have nowhere to go. As for shelters, many of them are already full. People coming from a long way here have to move again to find another place to stay. That’s why Mayor was confronted by an angry mob over the red tag policy when he toured Old Town in April 9.

People rush to grocery stores but return empty-handed. News report says stores shelves are bare all around the city. Supply vehicles cannot enter to replenish supplies because of damaged roads. There is an urgent need to restore traffic for supplies. Following figure shows food and accommodation issues.

Crime happens all around the city

Criminals still commit crimes. Some victims get scammed by those who declare they can get supplies, fix roof and so on. Victims believed in what they said and gave them money as a down payment, however scammers never return. It’s happening all over town, TV host reminded everyone to be vigilant against those scammers.

Historical architecture and artifacts protection

Artifacts in museum are in danger, more volunteers are needed to move precious artifacts out of the basement. The falling down buildings are including historical architectures. Many netizens feel sorry for the loss of history and culture, they appeal that the government should take actions to protect and reconstruct architectural heritage. At the same time, others with different opinions think that lives come first. We should save lives before saving history. It’s a problem the government need to solve. Lives and history are all important. Following figure shows citizens are worried about the history.

After the earthquake, normal life is affected. There are still people missing out there, the first priority is rescue. Decision makers should focus on saving lives and resettling affected people properly. Police should fight against crime and take actions to protect citizens from harms. Besides, plans should be made for future city reconstruction and cultural relic protection. It’s better to bring the city back to normal as soon as possible.

4 –– The data for this challenge can be analyzed either as a static collection or as a dynamic stream of data, as it would occur in a real emergency. Describe how you analyzed the data - as a static collection or a stream. How do you think this choice affected your analysis? Limit your response to 200 words and 3 images.

We analyzed the data as a static collection. Data pre-processing, word cloud and basic statistic are used to show the data distribution and trending of community involvement. Unsupervised and supervised algorithm is used to classify messages. Classification result along with manual analysis leads to the classified topics, which is the foundation for analysis that follows. We can learn what’s happening in town. Then we process the data with time interval of an hour, and analyze with dimensions of topic, location, time and messages. Using visual analysis tools, we can find the topic heat is changing throughout districts as scrolling timeline. It’s helpful to find out what city resources should be allocated.

Static analysis is chosen because it is better than dynamic analysis. Dynamic classification is uneasy to stabilize due to the rapidly change of data which leads to less stable and reliable result. And it’s difficult to find town’s unique topics when using stream data and public corpus.

Five topics’ data are selected to show visual effect of cluster result. Fig. 4-1 shows static-data-based cluster result. Fig. 4-2 and 4-3 show comparison classification result of static-data-based and stream-data-based with 12-hour messages in two days, which indicates static-data-based result is better.