dauvi - UFO sightnings report

UFO Sightnings
geo & text

Data mining analysis
and most relevant results

G. Marelli @ dauvi


  1. data preparation
  2. descriptive analysis
  3. data viz
  4. geo
  5. predictions

Data preparation


#format date and time
fs$date = as.Date(fs$datetime,format="%m/%d/%Y %H:%M")
fs$datet = strptime(fs$datetime,format='%m/%d/%Y %H:%M')
fs$posted = as.Date(fs$date.posted,format="%m/%d/%Y")
fs$time <- as.POSIXct(fs$datetime,format='%m/%d/%Y %H:%M')
#parse descriptive time
fs[set,"hour"] = fs[set,"duration..hours.min."] %>% gsub("[[:alpha:]]","",.) %>% gsub("-.*$","",.) %>% gsub("[[:punct:]]","",.) %>% as.numeric(.)
fs$duration = fs$sec + fs$min*60 + fs$hour*3600
#stopwords, stemming, cleaning
x <- tryTolower(x)
x <- removeWords(x,c(stopwords("english"), 'yeah', 'yep','&','"','quot','that','from','like'))
x <- gsub("[[:punct:]]","",x)
x <- gsub("[[:digit:]]","",x)
x <- gsub('(http.*\\s*)[^[:blank:]]+','',x)
x <- gsub('(\\.|!|\\?)\\s+|(\\++)',' ',x)
x <- stripWhitespace(x)
x <- stemDocument(x,language="english")
R functions for data cleaning and stemming


Rearth = 6371;
normF = pi/180.
sigDist = data.frame(shape=fs$shape)
sigDist$theta = (fs$longitude-refPoint[2])*normF ##Euler's rotation
sigDist$ang = sin(refPoint[1]*normF)*sin(fs$latitude*normF) + cos(refPoint[1]*normF)*cos(fs$latitude*normF)*cos(sigDist$theta)
sigDist = sigDist[grepl("us",fs$country),]
sigDist$ang = acos(sigDist$ang)
sigDist$dist = sigDist$ang*Rearth
sigDist = sigDist[!is.na(sigDist$dist),]
sigDist$area = sin(sigDist$ang)*2.*pi*Rearth
sigDist$count = sigDist$dist/sigDist$area
sigB = seq(1,100)*max(sigDist$dist)/100
sigDist$br = cut(sigDist$dist,breaks=c(0,sigB),labels=sigB)
sigD = ddply(sigDist,.(shape,br),summarise,rad=sum(count),count=length(count))
sigD$br = as.numeric(as.character(sigD$br))
Euler's angles transformation for distance calculation

import dataset in elastic search

notA = pd.read_csv("~/lav/dauvi/webapp/motion/" + "raw/clean.csv")
sucS = ""
for i in range(0,len(notA)):
    idS = {"index":{"_index":notA.loc[i,'date'],"_type":"log"}}
    timest = list(notA.loc[i]['datet'])
    timest[10] = 'T'
    tS = "".join(timest)
    sData = {"timestamp":tS,"shape":notA.loc[i]['shape'],"posted":notA.loc[i]['posted'],"dur":notA.loc[i]['duration..seconds.'],"dur_text":notA.loc[i]['duration..hours.min.'],"geo":[notA.loc[i]['longitude'],notA.loc[i]['latitude']],'city':notA.loc[i]['city'],'state':notA.loc[i]['state'],"country":notA.loc[i]['country'],"comment":notA.loc[i]['text']}
    resq = requests.post(baseUrl+indexN+'/external/'+str(i)+"?pretty",headers=headers,data=json.dumps(sData))
    sucS += " " + str(resq.status_code) #+ resq.text
    if i%20 == 0:
        print sucS
        sucS = ""
python script for elastic search population

configuration in kibana


Relevant quantities

  1. time of the day
  2. shape
  3. location vs shape/time
  4. time evolution
  5. text and emotions

time of the day

Most sightnings in the early night

shape sightning evolution

sightnings reports have significanlty increased

shape and location

most common is a light saw in California

Most common word used

Most sightings refer to a fas moving light

text by shape

hexagon, round and delta the most peculiar shapes

text by emotion

negative feelings win over positive 7:1

Data Viz


each panel works as a filter

Sightnings in US

most of the sightnings where seen in US

Light as most common shape

Light is the most common phenomenon

California has most observations

Zooming into California

Area 51

No particular sightnings in area 51



Triangle in area 51, most sightnings by the coast, red circles the top ten hot spots

By location

Us is the far the most active location

Normalized by population

Most souspicious is Washington compared to population size

Sightnings from area51

Area51 below the average, distances are normalized by radial areas


Time series

  1. we use three different prediction models: arma, ets, tbats
  2. all models agree on # of sightnings growth after a slight deviation
  3. arma works well on larger timescale
  4. ets is most suited for short scale (exp devay)
  5. tbats exhagerates seasonality


Auto regressive model - conservative


Exponentially decaying processing - agressive


All models predict growth - exhagerated seasonality

Month and weekday check

Most sightnings are lights in summer in california by saturday night???



  1. Sightnings are growing
  2. The most common sightnings are lights in west/east coasts on weekends nights in summer
  3. people are worried
  4. Area 51 does not play a particular role




print css