Step 1 - First Impressions

Before diving into data exploration, let's take a peek into one of the files... It's always good to have an idea of the data we'll be working with!

with open( "data/user1.txt" ) as f:
    for _ in range( 10 ):
        print( f.readline(), end="" )
print( "... and so on." )

timestamp,date_id,dow,utc_time_sec,offset_time_sec,accuracy,longitude,latitude
2015-03-01 07:16:39,20150301,Sun,1425190599,-3600,10,-64.97920478,-15.68742782
2015-03-01 07:17:12,20150301,Sun,1425190632,-3600,29,-64.97919636,-15.68745956
2015-03-01 07:17:25,20150301,Sun,1425190645,-3600,33,-64.97922924,-15.68733408
2015-03-01 07:18:19,20150301,Sun,1425190699,-3600,25,-64.9791967,-15.68745926
2015-03-01 07:18:25,20150301,Sun,1425190705,-3600,17.6770000457764,-64.9791967,-15.68745926
2015-03-01 07:18:58,20150301,Sun,1425190738,-3600,36.6500015258789,-64.9792025,-15.68750054
2015-03-01 07:19:28,20150301,Sun,1425190768,-3600,69.0550003051758,-64.97919298,-15.68740562
2015-03-01 07:19:59,20150301,Sun,1425190799,-3600,25,-64.97919666,-15.68745918
2015-03-01 07:20:28,20150301,Sun,1425190828,-3600,66.8030014038086,-64.9791805,-15.68737422
... and so on.

Notes

Nice. The data files have "txt" extensions, but contain structured, tabular data. In other words - CSV! So we can use the Pandas library, which will make our lives easier.

---

Step 2 - Loading the Data

Now we load the contents of the other files into the same structure, and differentiate the data source (file) with the user number.

import pandas as pd

data = pd.DataFrame()
for user_num in range(1, 5):
    dataframe = pd.read_csv('data/user%s.txt' % user_num, parse_dates=["timestamp"])
    dataframe.insert(0, "user", str(user_num))        
    data = data.append(dataframe)       
data

	user	timestamp	date_id	dow	utc_time_sec	offset_time_sec	accuracy	longitude	latitude
0	1	2015-03-01 07:16:39	20150301	Sun	1425190599	-3600	10.000000	-64.979205	-15.687428
1	1	2015-03-01 07:17:12	20150301	Sun	1425190632	-3600	29.000000	-64.979196	-15.687460
2	1	2015-03-01 07:17:25	20150301	Sun	1425190645	-3600	33.000000	-64.979229	-15.687334
3	1	2015-03-01 07:18:19	20150301	Sun	1425190699	-3600	25.000000	-64.979197	-15.687459
4	1	2015-03-01 07:18:25	20150301	Sun	1425190705	-3600	17.677000	-64.979197	-15.687459
...	...	...	...	...	...	...	...	...	...
30603	4	2015-04-01 00:08:25	20150401	Wed	1427839705	-3600	776.625977	-64.609553	-15.031567
30604	4	2015-04-01 00:09:29	20150401	Wed	1427839769	-3600	872.018982	-64.609553	-15.031567
30605	4	2015-04-01 00:10:30	20150401	Wed	1427839830	-3600	963.453003	-64.609553	-15.031567
30606	4	2015-04-01 00:11:31	20150401	Wed	1427839891	-3600	1326.000000	-64.609553	-15.031567
30607	4	2015-04-01 00:12:38	20150401	Wed	1427839958	-3600	1326.000000	-64.609553	-15.031567

137824 rows × 9 columns

Notes

We have a reasonably large dataset: 137824 readings for our 4 users.

---

Step 3 - Plotting Movement

Since this data describes user movements let's see what it looks like to project one into a surface - it's good to have a feeling about our users'movement patterns. We could have used a map, but these coordinates were transformed for privacy reasons so the map would just be noise.

import plotly.express as px
from IPython.core.display import HTML

def caption( number, explanation ):
    display( HTML('<div style="text-align:center"><b>Figure %d.</b> %s</div>' % ( number, explanation ) ) )

fig = px.scatter(data, x="longitude", y="latitude", color="user")
fig.show()
caption( 1, "Scatter plot of users 1-4 location readings." )