dauvi - UFO sightnings report

UFO Sightnings
geo & text

Data mining analysis
and most relevant results

G. Marelli @ dauvi

Outline

  1. data preparation
  2. descriptive analysis
  3. data viz
  4. geo
  5. predictions

Data preparation

Cleaning

#format date and time
fs$date = as.Date(fs$datetime,format="%m/%d/%Y %H:%M")
fs$datet = strptime(fs$datetime,format='%m/%d/%Y %H:%M')
fs$posted = as.Date(fs$date.posted,format="%m/%d/%Y")
fs$time <- as.POSIXct(fs$datetime,format='%m/%d/%Y %H:%M')
#parse descriptive time
fs[set,"hour"] = fs[set,"duration..hours.min."] %>% gsub("[[:alpha:]]","",.) %>% gsub("-.*$","",.) %>% gsub("[[:punct:]]","",.) %>% as.numeric(.)
#...
fs$duration = fs$sec + fs$min*60 + fs$hour*3600
#stopwords, stemming, cleaning
x <- tryTolower(x)
x <- removeWords(x,c(stopwords("english"), 'yeah', 'yep','&','"','quot','that','from','like'))
x <- gsub("[[:punct:]]","",x)
x <- gsub("[[:digit:]]","",x)
x <- gsub('(http.*\\s*)[^[:blank:]]+','',x)
x <- gsub('(\\.|!|\\?)\\s+|(\\++)',' ',x)
x <- stripWhitespace(x)
x <- stemDocument(x,language="english")
		    
R functions for data cleaning and stemming

Transformation

Rearth = 6371;
normF = pi/180.
sigDist = data.frame(shape=fs$shape)
sigDist$theta = (fs$longitude-refPoint[2])*normF ##Euler's rotation
sigDist$ang = sin(refPoint[1]*normF)*sin(fs$latitude*normF) + cos(refPoint[1]*normF)*cos(fs$latitude*normF)*cos(sigDist$theta)
sigDist = sigDist[grepl("us",fs$country),]
sigDist$ang = acos(sigDist$ang)
sigDist$dist = sigDist$ang*Rearth
sigDist = sigDist[!is.na(sigDist$dist),]
sigDist$area = sin(sigDist$ang)*2.*pi*Rearth
sigDist$count = sigDist$dist/sigDist$area
sigB = seq(1,100)*max(sigDist$dist)/100
sigDist$br = cut(sigDist$dist,breaks=c(0,sigB),labels=sigB)
sigD = ddply(sigDist,.(shape,br),summarise,rad=sum(count),count=length(count))
sigD$br = as.numeric(as.character(sigD$br))
		    
Euler's angles transformation for distance calculation

import dataset in elastic search

notA = pd.read_csv("~/lav/dauvi/webapp/motion/" + "raw/clean.csv")
notA.fillna("",inplace=True)
sucS = ""
for i in range(0,len(notA)):
    idS = {"index":{"_index":notA.loc[i,'date'],"_type":"log"}}
    timest = list(notA.loc[i]['datet'])
    timest[10] = 'T'
    timest.append('Z')
    tS = "".join(timest)
    sData = {"timestamp":tS,"shape":notA.loc[i]['shape'],"posted":notA.loc[i]['posted'],"dur":notA.loc[i]['duration..seconds.'],"dur_text":notA.loc[i]['duration..hours.min.'],"geo":[notA.loc[i]['longitude'],notA.loc[i]['latitude']],'city':notA.loc[i]['city'],'state':notA.loc[i]['state'],"country":notA.loc[i]['country'],"comment":notA.loc[i]['text']}
    resq = requests.post(baseUrl+indexN+'/external/'+str(i)+"?pretty",headers=headers,data=json.dumps(sData))
    sucS += " " + str(resq.status_code) #+ resq.text
    if i%20 == 0:
        print sucS
        sucS = ""
		    
python script for elastic search population

configuration in kibana

Descriptive

Relevant quantities

  1. time of the day
  2. shape
  3. location vs shape/time
  4. time evolution
  5. text and emotions

time of the day

Most sightnings in the early night

shape sightning evolution

sightnings reports have significanlty increased

shape and location

most common is a light saw in California

Most common word used

Most sightings refer to a fas moving light

text by shape

hexagon, round and delta the most peculiar shapes

text by emotion

negative feelings win over positive 7:1

Data Viz

Kibana

each panel works as a filter

Sightnings in US

most of the sightnings where seen in US

Light as most common shape

Light is the most common phenomenon

California has most observations

Zooming into California

Area 51

No particular sightnings in area 51

Geo

Overview

Triangle in area 51, most sightnings by the coast, red circles the top ten hot spots

By location

Us is the far the most active location

Normalized by population

Most souspicious is Washington compared to population size

Sightnings from area51

Area51 below the average, distances are normalized by radial areas

Predictions

Time series

  1. we use three different prediction models: arma, ets, tbats
  2. all models agree on # of sightnings growth after a slight deviation
  3. arma works well on larger timescale
  4. ets is most suited for short scale (exp devay)
  5. tbats exhagerates seasonality

ARMA

Auto regressive model - conservative

ETS

Exponentially decaying processing - agressive

Tbats

All models predict growth - exhagerated seasonality

Month and weekday check

Most sightnings are lights in summer in california by saturday night???

Summary

Overview

  1. Sightnings are growing
  2. The most common sightnings are lights in west/east coasts on weekends nights in summer
  3. people are worried
  4. Area 51 does not play a particular role

Resources

dauvi

intertino

print css