Coding

What Donald Trump is Tweeting (Analyzing Tweets with NLTK and Pandas)

How does @realDonaldTrump (Donald J. Trump) tweet? Before the President-elect became the president-elect, I didn’t pay much attention to his tweets, but I did know that he seemed to have a unique style of writing them. To me, it felt like he tweeted like he spoke in public. But what was that tone or Trump brand?

From a quick glance of his account, the usage of exclamation points seemed prominent. I wondered if that was consistent, and I wondered what else I could find. So I ran an analysis Trump’s tweets as well as @HillaryClinton, @CNN and @FoxNews’ for some comparison.

My strategy for selecting the accounts:

  1. Find another individual user’s account with opposing views, but who had some similar goals over the last year + (to be come President of the United States)
  2. Compare the individual users’ accounts with “objective” news sources’ Twitter accounts. Since source objectivity is always a topic of hot debate (or a hot topic of debate), I took @CNN and @FoxNews.

The Method

  • Pull as many of Trump’s Tweets as I could via Twitter API. The best service to perform this operation is /user_timeline. You can only get the last 3,200 tweets from any particular user handle, which is unfortunate because plenty of accounts have authored far more than 3,200 tweets (e.g. Trump has written over 30,000 tweets).
  • Take the tweets and do some Natural Language Processing with Python’s NLTK.
    • Get the parts of speech tags for every word in every tweet
    • Do word frequency counts
    • Classify the sentiment of the tweets
  • Classify the reading level or difficulty of each word. Determine the reading level of the account. 
    • I didn’t implement this. I started, but then I got distracted.
  • Pull out an aggregate view of some other interesting tweet data, like #HASHTAG usage.
  • Compare data.

And Now, The Data

Punctuation! Punctuation! Punctuation! Trumpunctuation?

If you’ve seen Trump speak, you probably may have noticed his strong intonations and general emphatic demeanor. His tweets seem to capture this partially through punctuation alone. His words are sprinkled with the strongest punctuation mark in the English language, the exclamation mark.

“!” occurs 2336 times over 3200 Trump tweets. 

And “!” occurs at least once 1954 times over 3200 Trump tweets.

Which means that 61% of Trump’s tweets contain an exclamation mark (based on my 3200 tweet dataset). This seemed astonishingly high; when I first pulled this I thought my code was incorrect. So I opened my file of tweets and eyeballed it to confirm. I figured if 60+ % of tweets contained an exclamation mark, then it would be easy to confirm my sanity (or lack thereof) from the text file.

trump's usage of exclamation mark

Some of Trump’s Tweets

The highest frequency of “!” in a single tweet was five. And that occurred in the following tweet:

“#WheresHillary? Sleeping!!!!!”
Which was Retweeted 27,158 times and Favorited 61,084 times. Created on August 20, 2016.

biggest_excla_tweet

Here’s the !!!! comparison among the entire group:

@realDonaldTrump Absolute Count (!) @HillaryClinton Absolute Count (!) @CNN Absolute Count (!) @FoxNews Absolute Count (!)
2336 171 43 134

Adjectives Are Very Great

Trump’s top twenty favorite adjectives are listed below, along with the top 20 adjectives from the comparison group. It’s not really a surprise that “great” was number one for Trump because, among other things, his campaign’s slogan was “Make America Great Again.” Notably, I removed stopwords and only grabbed lowercase words to help filter out some noise from the data. Relative frequency is the percentage relative to the other adjectives in the dataset.

trump_word abs_frequency rel_frequency
great 211 4.953052
bad 81 1.901408
many 81 1.901408
big 77 1.807512
last 66 1.549296
new 56 1.314554
good 50 1.173709
much 34 0.798122
amazing 32 0.751174
total 30 0.704225
wonderful 30 0.704225
nice 27 0.633803
massive 25 0.586854
first 25 0.586854
presidential 23 0.539906

 

Below is the list of top 20 adjectives from @realDonaldTrump, @HillaryClinton, @CNN, @FoxNews from their last 3200 tweets.

adj_comparison

 

General Sentiment of Tweets

To calculate the sentiment of the tweets, I used a function in the nltk library called demo_liu_hu_lexicon() which classifies each word of the sentence as Positive, Negative, or Neutral, and then does a basic count of each word-classification category. Whichever group has the highest count is how the text will get assigned. There are definitely better ways to do this. I considered an integration with IBM’s Watson, but time was doing its thing, being time, and being of the essence and such.

@realDonaldTrump @HillaryClinton @CNN @FoxNews
Positive 0.463750 0.468750 0.304688 0.235625
Neutral 0.285313 0.401875 0.425938 0.455937
Negative 0.250937 0.129375 0.269375 0.308437

Top Hashtags

This section contains a top 5 hashtag summary table for the entire analysis group, and also has the top 20 hashtags for each account listed afterward. You can infer what you will from this data.

I did find it interesting that the top-used hashtag by Trump was one of self-promotion, and Hillary Clinton used a lot of hashtags relating to the debates. Just looking at the hashtag data makes me think that Trump’s social media strategy was much stronger throughout the campaign.

In addition, all four Twitter accounts had at least one hashtag with “Trump” in it. From a marketing perspective, that’s good brand awareness.

Top 5 Hashtag Summaries

@realDonaldTrump Top 5 Hashtags @HillaryClinton Top 5 Hashtags @CNN Top 5 Hashtags @FoxNews Top 5 Hashtags
#Trump2016, #MakeAmericaGreatAgain, #MAGA, #AmericaFirst, #DrainTheSwamp #DebateNight, #DemsInPhilly, #VPDebate, #RNCinCLE, #debate #CNNHeroes, #CNNSOTU, #Aleppo, #CNNNYE, #JadonAndAnias #KellyFile, #Trump, #FoxNews2017, #Hannity, #Christmas

@realDonaldTrump Hashtags

abs_frequency hashtag rel_frequency_pct
238 Trump2016 15.130324
202 MakeAmericaGreatAgain 12.841704
111 MAGA 7.056580
79 AmericaFirst 5.022250
78 DrainTheSwamp 4.958678
57 ImWithYou 3.623649
57 BigLeagueTruth 3.623649
53 VoteTrump 3.369358
38 CrookedHillary 2.415766
36 Debate 2.288620
35 TrumpTrain 2.225048
34 TrumpPence16 2.161475
24 Debates2016 1.525747
22 ICYMI 1.398601
20 SuperTuesday 1.271456
18 VPDebate 1.144310
15 RNCinCLE 0.953592
14 Debates 0.890019
13 WIPrimary 0.826446
12 ThankYouTour2016 0.762873

Below are the top hashtags for the entire group.
all_hashtags

It’s also worth sharing how many total hashtags were used in the last 3200 tweets of each Twitter account in the dataset group:

@realDonaldTrump @HillaryClinton @CNN @FoxNews
total_hashtags_used 1573 482 143 810

 

Top Favorited Tweets

Top Favorited of @realDonaldTrump

text favorite_count created_at
Such a beautiful and important evening! The forgotten man and woman will never be forgotten again. We will all come together as never before 639428 Wed Nov 09 11:36:58 +0000 2016
TODAY WE MAKE AMERICA GREAT AGAIN! 577008 Tue Nov 08 11:43:14 +0000 2016
How long did it take your staff of 823 people to think that up–and where are your 33,000 emails that you deleted? https://t.co/gECLNtQizQ 296236 Thu Jun 09 20:40:32 +0000 2016
The media is spending more time doing a forensic analysis of Melania\’s speech than the FBI spent on Hillary\’s emails. 248370 Wed Jul 20 15:36:06 +0000 2016
Just had a very open and successful presidential election. Now professional protesters, incited by the media, are protesting. Very unfair! 234619 Fri Nov 11 02:19:44 +0000 2016
Love the fact that the small groups of protesters last night have passion for our great country. We will all come together and be proud! 224497 Fri Nov 11 11:14:20 +0000 2016
Nobody should be allowed to burn the American flag – if they do, there must be consequences – perhaps loss of citizenship or year in jail! 216064 Tue Nov 29 11:55:13 +0000 2016
Fidel Castro is dead! 212487 Sat Nov 26 13:08:11 +0000 2016
This will prove to be a great time in the lives of ALL Americans. We will unite and we will win, win, win! 204239 Sat Nov 12 15:05:33 +0000 2016
A fantastic day in D.C. Met with President Obama for first time. Really good meeting, great chemistry. Melania liked Mrs. O a lot! 194927 Fri Nov 11 02:10:46 +0000 2016

 

Top Favorited of @HillaryClinton

text favorite_count created_at
“I never said that.” \u2014Donald Trump, who said that. #debatenight https://t.co/6T8qV2HCbL 160098 Tue Sep 27 01:19:47 +0000 2016
Where was this kind of comedy last night? https://t.co/71JhLG55G5 134956 Fri Oct 21 17:51:34 +0000 2016
“Trump just criticized me for preparing for this debate. You know what else I prepared for? Being president.” #DebateNight 112647 Tue Sep 27 02:01:59 +0000 2016
Women have the power to stop Trump.\\n\\nhttps://t.co/tTgeqy51PU\\nhttps://t.co/VH3woeAf9Q https://t.co/NjvbkPsjPR 111546 Fri Oct 07 23:54:35 +0000 2016
“Nobody respects women more than me.” \u2014Donald Trump earlier tonight\\n\\n”Such a nasty woman.” \u2014Donald Trump just now #DebateNight 108114 Thu Oct 20 02:36:56 +0000 2016
Don\’t stand still. Vote today: https://t.co/jfd3CXLD1s #ElectionDay #MannequinChallenge https://t.co/4KAv2zu0rd 95882 Tue Nov 08 11:47:18 +0000 2016
Happy birthday to this future president. https://t.co/JT3HiBjYdj 94281 Wed Oct 26 13:03:18 +0000 2016
RT if you\u2019re proud of Hillary tonight. #DebateNight #SheWon https://t.co/H7CJep7APX 93121 Thu Oct 20 02:37:43 +0000 2016
RT this if you\’re proud to be standing with Hillary tonight. #debatenight https://t.co/91tBmKxVMs 92721 Tue Sep 27 02:45:44 +0000 2016
This is horrific. We cannot allow this man to become president. https://t.co/RwhW7yeFI2 91471 Fri Oct 07 20:55:13 +0000 2016

 

Top Favorited of @CNN

text favorite_count created_at
At least 60 people have been hurt in an explosion at a fireworks market near Mexico City, local media report.\u2026 https://t.co/2UWK6ECpsf 22570 Tue Dec 20 22:49:24 +0000 2016
This little boy is the newest face of OshKosh B\’gosh\’s holiday ads after initially being turned down by a talent ag\u2026 https://t.co/bdc1Zig7lE 22506 Sat Dec 10 07:01:29 +0000 2016
“We\’ve been friends for a long time”: Kanye West and President-elect Trump appear together at Trump Tower\u2026 https://t.co/KIdFwkVTGr 19082 Tue Dec 13 14:59:55 +0000 2016
This little boy is the newest face of OshKosh B\’gosh\’s holiday ads after initially being turned down by a talent ag\u2026 https://t.co/8ppZdDs3Te 12099 Sat Dec 10 23:45:25 +0000 2016
Clinton jokes at Reid portrait unveiling: “After a few weeks of taking selfies in the woods, I thought it would be\u2026 https://t.co/zpVwUAnjId 11868 Thu Dec 08 22:23:56 +0000 2016
BREAKING: President Obama vows retaliatory action against Russia for its meddling in the US presidential election\u2026 https://t.co/SJPGxWQB8d 9767 Fri Dec 16 01:36:31 +0000 2016
More Americans voted for Hillary Clinton than any other losing presidential candidate in US history https://t.co/6TBbQi3Nsn 9273 Wed Dec 21 23:28:00 +0000 2016
This little boy is the newest face of OshKosh B\’gosh\’s holiday ads after initially being turned down by a talent ag\u2026 https://t.co/LG0rdFFVkc 9117 Mon Dec 12 00:42:57 +0000 2016
Mariah Carey, Adele, Elton John and Lady Gaga bring the holiday spirit in special Christmas-themed \’Carpool Karaoke\u2026 https://t.co/8aQ5BDIbKt 7475 Mon Dec 19 05:30:24 +0000 2016
\’Dear world, why are you silent?\’: Desperate pleas from inside Aleppo https://t.co/1RvPlSDB7a https://t.co/XYwbJibFTD 7355 Wed Dec 14 19:00:04 +0000 2016

 

Top Favorited of @FoxNews

text favorite_count created_at
JUST IN: President-elect #Trump announces @Sprint will bring 5,000 jobs back to the U.S., and OneWeb will hire 3,00\u2026 https://t.co/CVWrTyuFNR 18974 Wed Dec 28 22:17:35 +0000 2016
.@JudgeJeanine: “Michelle, you may not realize it, but Americans rejected you and everything you stand for.”\u2026 https://t.co/tGrzIlZjxN 15054 Sun Dec 18 17:17:51 +0000 2016
#Breaking News: President-elect @realDonaldTrump has garnered the 270 #ElectoralCollege votes needed to become pres\u2026 https://t.co/5CDoiZCs4d 11164 Mon Dec 19 22:35:53 +0000 2016
Peters: “Without the least exaggeration, we can say that President Obama has been the worst foreign policy presiden\u2026 https://t.co/JbIGCpwKUk 10263 Sat Dec 31 03:32:46 +0000 2016
.@realDonaldTrump: “Michelle Obama said yesterday that there\’s no hope, but I assume she was talking about the past\u2026 https://t.co/V1BuKztapK 10021 Sat Dec 17 22:43:31 +0000 2016
.@realDonaldTrump: “We have to protect Israel. Israel, to me, is very, very important. We have to protect Israel.” https://t.co/R8CZWsGfvX 10021 Sun Jan 01 03:26:17 +0000 2017
Giuliani: \u201cThe U.S. Constitution doesn\u2019t give anyone in this world the right to come to the U.S. That\u2019s a privilege\u2026 https://t.co/TyKS7REAPR 8885 Wed Dec 21 03:41:24 +0000 2016
.@KatrinaPierson: This president is the divider-in-chief. His entire political career revolved around racism, sexis\u2026 https://t.co/vQP27Mxc1B 8700 Fri Dec 30 01:43:04 +0000 2016
.@GovMikeHuckabee: Can you name me one Muslim country that welcomes Christians to build & protect churches? No, you\u2026 https://t.co/baLyxGAkL6 8419 Thu Dec 29 01:39:42 +0000 2016
DJT: “I think the Democrats are putting it out because they suffered 1 of the greatest defeats in the history of po\u2026 https://t.co/2bBtHDwqu3 8175 Sun Dec 11 19:08:47 +0000 2016

About the Data

If you would like to access some of the code/data, it is publicly available on my GitHub repo. I’ve also included all four files that contain all of the tweets on which I ran the analysis in the data/ directory.

I had plans to include Date/Time data analysis in this post and many other things (if you’d like to see more data, let me know), but you have to stop somewhere great!!!!!! #MakeCodingProjectsSmallAgain

Standard
Travel

Japan, Part I: Traipsing Through Tokyo

I’ve never let a potential trip wait so long to be blogged as I have this trip to Japan.

Relevant for that leading sentence:

I remind myself that one reason I write about my trips is so I can look back in later years and remind myself of some good times. So here goes nothing.

First off: Jared and flew from NYC-> Hong Kong -> Sapporo (New Chitose Airport). The route from NYC-> Hong Kong only took 16 hours, took 16 hours, 16 hours, 16. That’s long enough for two sleeps. I took about 1, on account of the Advil PM; I also finished a book (without pictures).

My ass went numb at one point, too.

Hey, stop thinking about my butt.

flying to japan

I’m ready! (No I’m not.)

By the time we got to New Chitose Airport (close to Sapporo), we realized we were in Japan, which meant our brains were still working! That’s when we first encountered Onigiri.

Salmon wrapped in rice, wrapped in seaweed. $1.50 Onigiri!!

Salmon wrapped in rice, wrapped in seaweed. $1.50 Onigiri!!

And also, we discovered Japanese attention to detail/organization:

At that moment, I wouldn’t have been able to tell you what time it was, even if I had a watch. We stopped at New Chitose because it is in Hokkaido, and Hokkaido was our final destination, since this was a ski trip after all. But before skiing, we had other business to take care of. Neither of us were “going to fly 32+ hours and not visit Tokyo.” So we went to bed, because the next morning, we were on our way to Tokyo.

I think we did a lot of the stuff in Tokyo that folks often do. In the interest of clarity, I’m going to break down our trip to Tokyo in categories.

Shibuya

We stayed in an AirBnb in Shibuya on Dogenzaka street. It was clearly an investment property — the owner never lived there and only rented it out. It was spacious.

The perfect fit.

The perfect fit.

There were sweet-ass arcades, apparently owned by SEGA?

Tekken UNLIMITED (whatever that means). I still play Paul Phoenix.

Tekken UNLIMITED (whatever that means). I still play Paul Phoenix.

We also ate at Freshness Burger.

It's so... fresh.

It’s so… fresh.

Checked out some views

Squad goals.

Squad goals.

Chilled with a doge

Hachikō the dog, frozen in time.

Hachikō the dog, frozen in time.

Yoyogi Park and Meiji Jingu Shrine

We sauntered.

I guess it's just a park. I don't know.

I guess it’s just a park. I don’t know.

And we found the bathrooms.

If the portos are unisex, then these signs must be an indication for...

If the portos are unisex, then these signs must be an indication for…

Ah, the signs were both actually just saying: "CAUTION."

Ah, the signs were both actually just indicating: “CAUTION.”

Then we discovered the entrance to the Meiji Shrine, and some decorative sake barrels.

Raise the roof.

These Sake Barrels are actually empty.

These Sake Barrels are actually empty.

And made it to the main shrine

Zen times Ten.

Zen times Ten.

Shinjuku Golden Gai

Drinking in the Golden Gai area was one of the most memorable and cool things we did in Tokyo. Quite a few of the bars won’t even allow tourists in, which is judged by one’s ability to speak Japanese. No Japanese? No entry.

The bars in the area are tiny. What you see here is pretty much what you get.

The bars in the area are tiny. What you see here is pretty much what you get.

full_bar_size

Tsukiji Fish Market

It’s the fish market you’ve all heard of. It’s massive, and they apparently auction off massive tunas in the early morning (5 a.m. or something). We couldn’t be bothered to wake up for that. But we took photos later anyway!

Off with his head.

Off with his head.

It's a real operation.

It’s a real operation.

Served on a leaf.

Served on a leaf.

Tsukiji SUV. 217 mpg.

Tsukiji SUV. It gets 217 mpg.

Tokyo Skytree and the Hyatt Hotel (“Lost in Translation” bar)

Easy tower, EASY.

Easy tower, EASY.

Mt. Fiji in the background. Civilization in the foreground.

Mt. Fuji in the background. Civilization in the foreground.

A bad nighttime shot of the skyline from the Hyatt Hotel.

A bad nighttime shot of the skyline from the Hyatt Hotel.

Harajuku & Takeshita Street

I imagine this is something like New York City’s St. Marks street.

Pals!

Pals!

Onesies on SALE. Get them while they're hot, or not.

Onesies on SALE. Get them while they’re hot, or not.

A good look for a pilot.

A good look for a pilot.

Food Things

The food, the food, the food. Best Ramen I had was at this place that has 3.5 stars on Google Reviews: Samurai Noodle. Oh well. A 3.5 ramen to the Japanese is 35.5 to me.

Heads down for ramen at Samurai Noodle, JP.

Heads down for ramen at “Samurai Noodle, JP”.

Ramen meal 1 of 74.

Ramen meal 1 of 74.

Plastic food displays! Very common and tasty.

Plastic food displays! Very common and tasty.

The stove-tops for DIY Okonomiyaki, "Japanese Pancakes."

The stove-tops for DIY Okonomiyaki, “Japanese Pancakes.”

Okonomiyaki in the making.

Okonomiyaki in the making.

Okonomiyaki, preparing to be eaten.

Okonomiyaki, preparing to be eaten.

I'll have THAT (pointing) one, please.

I’ll have THAT (pointing) one, please.

Vending machines on the exterior or entrance of restaurants will take your order and make you pay on the spot.

Vending machines on the exterior or entrance of restaurants will take your order and make you pay on the spot.

Ramen meal 2 of 74.

Ramen meal 2 of 74.

Japanese version of "street meat."

Japanese version of “street meat.”

Get ready to get eaten, little fishy.

Get ready to get eaten, little fishy.

Art and Decorations

Self-explanatory?

Always.

Always.

Collage?

Collage?

I think this was outside of a love hotel. Not sure if there is some deeper meaning here.

I think this was outside of a love hotel. Not sure if there is some deeper meaning here.

Sorry, we're open.

Sorry, we’re open.

This was written in backwards so that it could be read from a mirror opposite of it.

This was written backwards so that it could be read from a mirror opposite of it.

street_art

Random Things!

Also self-explanatory?

The karaoke bar we went to was 5+ stories; the bathrooms were also filled with puke. :(

The karaoke bar we went to was 5+ stories; the bathrooms were also filled with puke. 😦

Smokers must stand in a sectioned off area.

Smokers must stand in a sectioned off area.

The decibel level for some construction is tracked.

The decibel level for some construction is tracked.

Some clubs have self-serve lockers!! Groundbreaking, folks.

Some clubs have self-serve lockers!! Groundbreaking, folks.

The public transportation manners were shocking. Look at that perfect escalator behavior.

The public transportation manners were shocking. Look at that perfect escalator behavior.

Those are the highlights. Tokyo was one of the most exciting cities I’ve been to in my life. I want to go back and stay for a few months, although I’m not sure when I’d carve the time out for that wishful endeavor. But, when we departed Tokyo, we weren’t really sad, and that’s because we were on our way back to Sapporo — our starting point for our ski exploration trip on the northern main island of Hokkaido. Read about the skiing in Japan Part II: Heaven is Actually Called Hokkaido.

Arigatou gozaimasu, Tokyo. Konnichiwa, Sapporo.

Bye!

Bye!

Standard
Travel

Japan, Part II: Heaven is Hokkaido

After arriving in Sapporo and collecting the rest of our squad, we picked up the rental cars and started driving to Niseko. Worth mentioning: the steering wheels were on the Right side of the cars, and we were driving on the Left side of the road. But that’s normal for plenty of folks, I’ve heard, even when they’re sober.
"I think this is the GPS."

“I think this is the GPS.”

No clue, but they were great!

No clue, but they were great! (Beef jerky substitute.)

Once we learned how to drive again, we finally arrived at our condo in Niseko. We had a healthy weather forecast.
IMG_2200
We went out to the town to grab food around 7 p.m. and that’s when we realized that, during peak season (or Australian/Chinese holidays), you need a reservation in this town. So without a reservation, we starved… until we finally found a place that would seat us as a walk-in.
Japanese Pancakes!

Japanese Pancakes!

But, let’s talk about the skiing. From here, I’m just going to break this post down into sections for each ski area.

Niseko Annupuri

Niseko has four resorts on the one mountain, and plenty of interesting skiing. Some results from day one:

niseko_pano
niseko_night_ski

We even got to do some night skiing, which was exciting with dark-lens goggles on.

The next day, we were up skiing again (surprise surprise), and Scott discovered his Snowlipop.
snowlipop

I don’t really know how these things amass, but we saw them amongst the plentiful powder while we were there, and I have never seen them skiing anywhere else.

We also saw some alpenglow:
niseko_pano_alpenglow
And we got above the clouds:
above_niseko_clouds

Rusutsu

Once we arrived at Rusutsu, we were a bit surprised with the layout the resort. We parked at the “base,” which was home to a massive empty parking lot. We walked into a hotel, purchased our tickets, took an escalator and were walking on marble/tile floor. After walking past gift shops, we finally made it outside, where we loaded onto a gondola that took us across the highway, to another base lodge. The ski experience was starting to become familiar again once we took a high-speed quad for our first run. However, we were reminded that we were in a foreign country at the bottom of the run when we skied through a snow-covered amusement park. WTFs were whispered, spoken and even yelled.
Just tall enough to ride.

“Is that a ferris wheel?” … “Yup, that’s a ferris wheel.”

The amusement park sighting was a positive omen, indicating how the rest of our day was going to go. Because it went like this:
knee_deep

POWDER ALERT

At lunch, we discovered one of the most efficient ramen-making machines ever:

Kiroro

Two thumbs up for Kiroro. I think we hit it while we were in transit, on the way do Asahidake. But, despite being on a bit of a tight timetable, we still had plenty of powder to ski, and there were great views. This was a cool resort.

DCIM100GOPRO

Furano

Furano was kinda weird, but maybe it felt that way because we visited at the end of our trip. I always feel weird towards the end of a vacation. Here’s a photo of the trail map, because the skiing was average on that day, and that’s all I’ve got:
furano_map

Asahidake

At 7,517 feet, Asahidake is the tallest mountain (actually a volcano) in Hokkaido, duh. At the base of the mountain is a large hotel/resort called the Asahidake Manseikaku Hotel Bear Monte (quite the mouthful). This place comes equipped with a traditional Japanese onsen, which, was an unusual and through-provoking experience. It also had bus-loads of people dropping in to visit while we were there. The dining area was kind of a trip, since it was a buffet-cafeteria, which was unexpected at such a massive “up-scale” resort tucked away in the mountains. I will say that one of the most memorable experiences of the trip was the day spent trekking in Asahidake, where you could see plumes of smoke coming up from the surface of the volcano. The weather was pretty spectacular.
asahidake_hiking_up
asahidake_plumes

“Put that cigarette out!”

asahidake_on_the_moon
asahidake_melting

Apparently volcanoes can get hot.

asahidake_moon_crater

Traversing across the moon.

asahidake_in_trees
Despite our last ski day of the trip occurring in Furano, I decided to end this post with Asahidake because it was so spectacular. Nobody wants to end on average. Ski you later.
Standard
New York City, Random

Flipping Coins to Decide on Surgery — My Little Torn Meniscus

The coin flipped through the air — heads meant I would go in for the surgery, tails meant I would not. The coin landed on heads.

“Damn,” I thought. Maybe, I’ll do best two out of three. I flipped it again. Heads. “Damnit.” I thought maybe I should flip it one more time, just to be sure. The quarter made that high-pitched zipping sound after it left my hand, cutting its way through the air. It hit its apex and then descended into my palm. Heads. “DAMNIT!”

I brushed my teeth, put my shorts on, and headed to the hospital. I had just flipped a coin to help me decide if I would have surgery on my meniscus. It’s hard to say if that was a good choice, since I’m still recovering, but I’ll know in a few weeks. I mean, I’m not dead, so there’s at least that.

My knees aren’t in the best shape. This is my good one:

Scar+Knee = Scarknee. A circus scarknee!

Scar+Knee = Scarknee. A circus scarknee!

And the good news about my good knee is that it belongs to my good (dominant: right) leg. The bad news about my bad knee is that it belongs to my bad left leg. Right now, this is my bad knee on my bad leg:

IMG_3629

For comparison:

Knee model. Yeah, that's me.

Knee model. Yeah, that’s me.

Over the last 15 years, my left leg has been through some shit. It all started in 2000 when I dropped into a half-pipe and traveled up the wall for the first hit: 3-2-1 lift-off. I dropped my shoulder and head to start the rotation — the trick to be performed was a backflip-180. As I made my way around, I was expecting my skis to meet with the hard snow of the half-pipe’s wall within the second, but the sound and feeling didn’t come. What?

And finally, I crashed hard on the tails of my skis in the bottom of the pipe. The impact from the crash was so powerful that I broke my left leg’s tibia and fibula clean through.

BAM! POW! WHAM!

The surgery for that was, well, gnarly. The results left me with a titanium rod extending knee-to-ankle, and three screw (two in ankle, one in knee area) implants. This is the nicest video I could find of the procedure, which still gives me the chills, even though it’s just drawings/cartoons:

And if you’re really looking for some bloody stuff, you can watch this one:

A year after that surgery, I was playing soccer for my high school team. The season had just started, and we were in our second or third game. I was blazing down the right wing with the ball and a defender came in cleats up from a frontal/side angle. He smashed right into my shin guard and missed the ball, from what I recall. But for what he missed in ball, he made up for by fracturing my shin.

One might say that that kind of damage/trauma to a leg leaves a lasting impression. Physically, I had a shin that protruded near the site of the break, and it would change colors (get black and blue-ish) when I ran around for a long time. Naturally, I kept skiing and playing soccer, because it takes more than broken legs to stop this guy!

Fast forward about 10 years to February of 2015. I skied about 15 days, and everything was just dandy for my knees — at least I thought so. Then, in March, I was playing an indoor soccer game (super important playoff game, obviously) on a hard court.

It was late in the second half and I was backpedaling to get in a good offensive position, when, all of a sudden, my left leg just kinda gave out.

It felt weak and uncontrollable, like someone kicked all the mojo out of it. There were only two minutes left in the game, so I kept playing with my newly acquired hobble-run.

We lost the game, and afterward, I remember thinking that my leg felt weird, maybe a little shocked; I was also limping around like a feeble old man. When I got home, I got a good look at my knee; it was completely swollen. “Shit,” I thought. “Did I tear something? Shit! Shit! Shit!” I started icing it, took Alleve, grabbed a shower and went to bed.

That knee on that fateful night. Curse you, I say. A pox on you!

Left knee on that fateful night of injury. Curse you, I say. A pox on you!

The next day, I tried to get out of bed and couldn’t put weight on it; I wanted to cry.

I probably did. I had been in this position twice in my life already, with the same leg — “always this goddamned left leg,” I said, likely in a more impressive combination of four-letter words. But I tried to stay hopeful. I had my brother get me a pair of crutches from Craigslist and I set up a doctor appointment.

After getting a MRI, it was clear: my meniscus was torn.

The physician recommended that I do physical therapy for 4-6 weeks before thinking about surgery, because for a meniscus, it’s usually not something that is “repaired.” More often, he told me, the torn/frayed part of the meniscus is scooped out, which is analogous to removing a hangnail that bothers you. (One little difference between a hangnail and a frayed meniscus is that fixing one requires some clippers, and fixing the other one requires anesthesia, and, uh, knives and shit.)

I stopped playing soccer for a couple months and switched to running. Eventually, the pain I felt while traveling up and down stairs subsided and I could run a few miles pain free. A couple more months and I was playing soccer again, mostly without pain. Then one day, I kicked the ball in a weird way and it hurt so much I had to stop playing. Again, I stopped soccer and turned only to running. In June, I decided to schedule my surgery for September so I could enjoy some sweat-like-you’re-gonna-water-California’s-plants summer runs and the great outdoors.

Well, as it turns out, time turns (with the hands on the clock, duh), and it was September, and my surgery was in five days. To be certain about the decision for surgery, I went out to play soccer and gave it a good romping. To my dismay, I could elicit no pain. This is when I started to freak out and think, “Maybe I don’t need the surgery? Maybe I can just keep using it like this? This is fine. I like using it like this. Nothing to see here, folks.” This couldn’t be true — where had the pain gone?

To add to my predicament, I knew I would be skiing this winter, and had no idea how my knee would react to skiing and more specifically, skiing moguls (my favorite). To guarantee that I would be skiing pain-free this season, I had to decide if I would:

  • A) use the lack of pain I felt while playing soccer as a predictor for what I would feel while skiing and not get surgery (but risk making the tear worse), OR,
  • B) just get the surgery and settle with the fact that I’d have less meniscus to help protect my knee from arthritis in the future, but probably prevent future tearing, and probably be pain-free.

By the way, these options suck.

I slept on it, and the next day (Wednesday), I played soccer again. I hammered on it with lateral movement, twists and turns, but I couldn’t get much pain out of it. And that made my decision even more difficult. I was hoping that two days of soccer would bring some real pain to it. But instead, I felt like the 14-year-old boy I once was.

Surgery was scheduled for Friday, and I had all of Thursday to decide.

I called my doctor and my old physical therapist — one said to do it, and the other said not to. Perfect.

So, I slept on it again. And when I woke up on Friday, I was really hoping to have a definitive answer, but I didn’t have one. I was standing in the kitchen when I decided I’d flip a coin. I walked into my room to grab a quarter. And then…

…The coin flipped through the air and landed on heads.

I woke up in the post-operation room and I looked down at my leg; it was all bandaged up. Time had passed, and events had occurred while my consciousness was away. Now that it had returned, I took stock of the situation and was relieved that it was over. A nurse came by and I stated: “thirsty,” for which I was awarded a small styrofoam cup of ice chips.

It was time to go home.

 

Some Notes

  • I don’t recommend flipping coins to determine whether you are going to do a surgery or not. You should be certain about your decision. In the end, if I really did’t want to get the surgery, I wouldn’t have shown up at the hospital. I think my brain calculated the odds and determined I was more likely to be pain-free this ski season if I went with the surgery. But, it really felt like a bet. And I hate betting.
  • A meniscectomy (the procedure) is a quick recovery compared to other things, like tearing/repairing your labrum (https://sweet-as-tandy.com/2014/07/30/what-its-like-to-have-surgery-on-your-labrum/). If the procedure involves scooping, and not stitching, you shouldn’t need crutches, and you’ll walk out of the hospital the same day of your surgery. Your knee will be swollen though (GET SWOLL, YO).
  • Your throat can hurt post-operation. Oh, you didn’t know? That’s usually because they stick a endotracheal tube (https://en.wikipedia.org/wiki/General_anaesthesia) down that guy.
Standard
Coding

How to Retrieve and Analyze Your iOS Messages with Python, Pandas and NLTK

I’m one of those people that keeps every text message I send or receive — I never delete them. Meet a girl at a bar, text her the next day and never hear back from her? I keep that. Weird wrong-number texts? I keep those too. Ex-girlfriend texts? Definitely keepers.

I had 65,378 messages on my phone at the time of writing this post.

I’m not a digital hoarder or anything, but I primarily do this because I like the idea of being able to search through the past. But, digital hoarder or not, collecting anything takes up some sort of space, and when I found that my text messages were taking up 4GBs of space on my phone, I decided it was time to back them up. It was at that point that I realized I could also probably analyze them.

As it turns out, you can do this, and I’ll tell you how. For this project, I used Python/Pandas/NLTK for the analysis and an iPython Notebook to render the datasets. I’ve also uploaded the code to GitHub, which you can view here.

An overview of the steps to make this happen:

  1. Sync/back up your iPhone because the messages need to be stored on your computer.
  2. Load the SQLite file and retrieve all messages
    • You can follow the directions for retrieving the right file here.
  3. Analyze those mensajes (I used Pandas)!!

Let’s get into some details.

You need to sync and back up your phone’s contents to your computer. There’s a great post on how to do this here. In case you want to skip that read, you’re ultimately getting a file with the text messages in it; copying it and moving it into your working directory.

You can find the file with this bash command:

$ find / -name 3d0d7e5fb2ce288813306e4d4636395e047a3d28

Now, loading the SQLite file — you can actually see what’s in this file via the command line:

 $ sqlite3 3d0d7e5fb2ce288813306e4d4636395e047a3d28 

Then you can check out the available tables:

sqlite> .tables
_SqliteDatabaseProperties chat_message_join
attachment handle
chat message
chat_handle_join message_attachment_join

From here, the main tables I found useful were “message” and “handle.” The former contains all of your text messages, and the latter contains all of the senders/recipients. I only wrote code around the messages table, primarily because I could never figure out how to make a join between message and handle, but that was probably something trivial that I overlooked. Please tell me how you did it, if you did!

Continuing on, the message table has lots of columns in it, and I chose to select from the following:

['guid', 'service', 'text', 'date', 'date_delivered', 
'handle_id', 'type', 'is_read','is_sent', 'is_delivered',
'item_type', 'group_title']

The key field is “text,” which is where the content of the message is stored, which includes emojis! (A cool thing is that your emojis will show up if you try to plot them in something like an iPython notebook. You could run an entire analysis on emoji usage…)

My analysis, however, ultimately breaks down into two pieces:

  1. Analyzing the content of the “text” field (excluding emojis).
  2. Analyzing the messages themselves (for example, total text messages, or, what I sent vs. what I received, for instance).

For #1, I wrote code that:

  • Classifies all words and assigns a part of speech to them, then check the counts of each part of speech.
    • You should get a table looking like this.

      You should get a table looking like this.

  • Counts the number of times each word appears in the dataset, and gives an overview of the dataset:
    • total_words_filtered
  • Excludes boring words, like prepositions, and words that are < 2 characters.
  • Classifies all words as is_bad=1 or 0. I did this by using a .txt file full of bad words, found here:
  • Plots usage of bad words
    • I’d love to show you my plot, but let’s just assume I never swear…

For #2, the code allows you to:

  • Plot the number of text messages received each day (check out the spike on your birthday or during holidays). You can see my data below has a huge gap (that’s when my phone was replaced and not backed up for many months. My timestamp conversions are also apparently incorrect, but I haven’t looked into it.
    • The timestamp conversion is off, so someone can fix that... we're not in 2016, yet... Are we??

      The timestamp conversion is off, so someone can fix that… we’re not in 2016, yet…Or am I??

  • Count the number of sent versus received messages.

Anyway, I hope you can get some use out of this, and instead of blabbing on about the code here, I’ll just let you read it and use it on your own. Please check out my git repo, and please reach out to me with questions, comments, etc.

Standard
Travel

This Is Why You Should Care About Horse Racing

When the gates open for Race 11, the crowd cheers, but when there’s a Triple Crown on the line, the crowd erupts. With every galloping hoof imprinted on the sand, the crowd crescendos. Hundreds of thousands of dollars are on the line. People are no longer sitting, they’re standing, or they’re jumping up and down.

There’s jostling in the crowds, and those who aren’t in a great position to see the race are trying their hardest to catch just a little glimpse of it from anywhere they can. The screaming and cheering continues, along with the horses, around the turns and over the straightaways. After the horses pass the final turn, the majority of the crowd realizes that the Triple Crown-hopeful is in the lead.

The crowd realizes that their hopes, which are sabotaged too often by life, may actually be fulfilled.

They’ve bet on this horse, not just to win money, but to see a dream come true. The galloping hooves continue and the crowd gets louder, and louder. And when American Pharaoh crosses the finish line, the sound is deafening. The cheering becomes one solid pitch of noise that happens to carry a multitude of human emotions.

You realize you just witnessed a being gain a piece of immortality and you hope that one day, you will too.

Here are some pictures from the day:

belmont_track_pano
American Pharaoh and Co. warming up.

American Pharaoh and Co. warming up.

Not a bad bet.

Not a bad bet.

I'm happy because it was my bet.

I’m happy because it was my bet.

And the day is over.

Standard
Coding, Hackathon, New York City

Coming Through In the Clutch at Sports Illustrated’s Hackathon

“Hackathons are, like, my new favorite thing,” said not a lot of people. But I’m saying it, because I like building proof of concepts in a short period of time.

Hackathons are great reminders that great things can be built in no time at all.

Earlier this month, I participated in Sports Illustrated’s first hackathon, and this is a brief recounting of that sugar-filled, sleep-deprived, product-building experience. (No sports were actually played, unfortunately.)

We showed up around 9:30 a.m. because we were told that more people RSVP’d than the event space could handle. When I arrived, there were already about 10 over-achievers who had gotten there before me. I wanted to say, “damn them,” but then I realized, we were all probably sewn from similar fabric — the kind that welcomes waiting in lines to do work on a Saturday morning for the prospect of a unknown prizes (prizes had not been announced yet).

Around 10 a.m., the event started and we were allowed into the space (Time Inc. Photo Studio), which had a nice view.

The proof is in the pudding.

The proof is in the poster pudding.

A good view for building things that matter.

A good view for building things.

I can't say I read those two issues.

I can’t say I read those two issues.

There was coffee, donuts, bananas and plenty of other snacks to get fired up. My teammate, Brett, and I picked out a table and got settled in. Within about 15 minutes, there were speakers giving presentations to the group of hackers. This was a bit odd, and unfortunately, fell on the note of contrived. But alas, no donut is free. And the final speaker, Alex Bresler, said something that inspired what we would actually be building.

“It’s really difficult to determine who is more clutch than someone else.”

When I heard this, I thought, “Why? We can do that. We’ll do that.” I messaged Brett (sitting next to me), and asked what the thought about building something that rates the “clutchness” of players in sports, and allows users to compare different players. He responded that he liked that idea.

Left: Brett paying attention to speakers. Right: Me not paying attention.

Left: Brett paying attention to the presentations. Right: Me not paying attention.

Shortly thereafter, the speech wrapped up, and one of the guys sitting at our table asked us what we were working on. I stated that we had an idea, but we weren’t totally sure. We asked him what he did, and he said he worked on “statistical modeling,” or something like that. I was skeptical, as I’m sure Brett was as well — I could see it on his face. I decided to give a little more info about what we were working on, and this statistician dude asked if he could join our team. I thought, “I don’t know anything about this guy, but, why the hell not. He could be great.” I said, “yeah sure, but let me ask Brett.” Brett was not yet convinced, but finally, he decided to say yes too. And so we accepted this stranger onto our team.

I asked him what his e-mail was so I could invite him to our git repo, and he said, [actual-email-address]@alum.mit.edu.

I laughed a good laugh inside. I knew regardless of what would happen, I was probably going to learn something. Our new teammate’s name was Dan (and still is, actually).

After officially becoming team-complete, I gave Dan the idea straight up, he seemed to like it. We discussed how we would build it, and then we were off and running. The unfortunate time was roughly 1 p.m.; this was not a true 24-hour hackathon.

We coded through the “early” night and Dan worked up some awesome model for calculating clutchness. By around 1 a.m., we had something pretty serviceable. Subsequently, this was about the same time my bed started to call me. The problem was, my bed was miles away, and it started to rain, and I did not want to walk anywhere in the rain. We also had some nasty bugs in our code that needed to be squashed, and squash them we did. It was around this time, maybe a little bit earlier, that we started to pair program, because this is when 1 brain is worth about 1/2 a brain. Dan took off for his bed sometime around 2 a.m., I think? Not sure.

Around 4 a.m., delirium struck, and I told Brett I was going to need some nappy time. Brett, as though he were cast of iron, turned to me and said that he was probably going to keep working. I admired that, but I also admired my potential time with the couch. I briefly napped and I returned to our table 30 minutes later, hardly refreshed, but it was better than nothing.

Brett was working on some insane bash command that was downloading videos, compressing them, converting them to another format, and probably sending them to the International Space Station. I kinda thought it was magical, but that’s because everything was magical at that hour, including water. A toothbrush would have been magical, too.

The sun came up; that was good. Dan came back early that morning, and pushed all of his work to our repository. Suddenly, we not only had data, we had a real mathematical model computing the clutchness, or, Clutch Rating of a particular player. We were ecstatic.

As we got closer to stopping-time, I built out a brief outline of our pitch, and we made some last-minute tweaks. The judges arrived (including the CMO of Sports Illustrated, Damian Slattery; the Executive Editor of Sports Illustrated, Ryan Hunt; and the CTO of Time Inc, Colin Bodell) and said some short words, and with that, the pitches started.

Check out that high-tech digital display.

Check out that high-tech digital display of the pitching order.

We got up on stage, and it felt like we nailed it. Our product was quite complete, by hackathon standards. Users could:

  • Search through players, see a snapshot of each player and their overall Clutch Rating.
  • Click on a player and see more stats about the player, including a profile photo.
  • View all of a particular player’s plays that went into computing their overall clutch rating.
  • Click on a play to watch game footage of that play (the product of Brett’s insane shell command).

We were the last team to pitch, and after finishing, the judges went on with their judging.

About 5 minutes later, the winners were announced, and we took first place.

This is what winning looks like.

This is what first place looks like. From left to right: Alex (me), Dan, Colin Bodell, Brett

Plenty of post-hackathon networking ensued, and we went to get a few whiskeys — not that we needed them. We were already mostly drunk with delirium. After the second or third drink, I turned into what some would call, a zombie. I promptly made my way to the train station without eating anyone, and found my bed.

The next day, we were all off to work like nothing ever happened.

Check out our project at clutchratings.com.  

Standard