NDC Sydney talk

In August this year, I gave a talk at NDC Sydney on Real-time Twitter Analysis with Reactive Extensions. NDC is the Norway Developers Conference, so it’s a natural progression for them to come to Sydney. This was their first time down under, but they’ve already announced they’ll be back in August 2017.

It was a three day conference with over 100 speakers, some international and some local. That’s a lot of speakers, and it translates into 7 parallel tracks, or 7 concurrent talks.

Full talk video and code online

The video of my talk is on NDC’s vimeo channel, and the code and data driving the visualizations in the talk is on github. The repository is fairly large because the data files total a couple of hundred megabytes.

I’ve written a few articles covering parts of the material in the talk and discussing the code approach:

Continue reading

Twitter Analysis: Hillary and Obama’s Convention Speeches

Hillary Clinton and Barack Obama at the Democratic National Convention

While preparing for my recent NDC Sydney talk on real-time Twitter analysis, I was looking for interesting events to watch play out on Twitter. One great candidate was the 2016 Democratic National Convention, where Hillary Clinton accepted the nomination for president – the first woman in history to be nominated. There were a lot of major speeches from people such as Michelle Obama, former president Bill Clinton, Barack Obama and of course Hillary herself.

In a previous post I discussed the approach I used to identify the topics of Twitter discussion over time. Mapping them out into a histogram in real-time made for an interesting view of the event from the audience’s perspective.

Another view that can be taken using the same analysis approach is to plot key topics over time, to get an idea of the strength and duration of each topic’s impact. It’s interesting to see the topics that got the biggest response on Twitter during Hillary Clinton and Barack Obama’s convention speeches.

Continue reading

Tracking Twitter Discussion Topics in Real Time with Reactive Extensions

During my recent NDC Sydney talk on real-time Twitter analysis with Reactive Extensions, I talked about the approach I used to track current discussion topics as they changed over time. This is similar to Twitter’s trending topics, but changing more dynamically.

The source data came from Twitter traffic during two episodes of the ABC’s Q&A show in the lead up to Australia’s 2016 federal election. Each of the candidates for Prime Minister – incumbent Malcolm Turnbull and opposition leader Bill Shorten appeared as a solo guest to face questions from the audience.

I wanted a live view of the current topics of discussion as the show progressed, to get a feel for which topics the Twitter audience was responding to.

Continue reading

Detecting spikes in time series with Reactive Extensions

I recently spoke at NDC Sydney, which was a great experience. My talk was on Real-Time Twitter Analysis with Reactive Extensions. I wanted to have a deeper look into the data and approaches I’d started with the Women Who Code workshop.

I wanted some compelling Twitter data, and given the year we’ve had so far in 2016, politics seemed a good choice. Between Australia’s federal election, the EU referendum in the UK and the US presidential primaries, there was a lot going on in this space. Twitter engagement was huge across all of these events.

One thing I wanted to be able to do was to plot the rate of Twitter traffic in real-time. This was relatively easy with a couple of lines of Rx, and it gave me a good grasp of the tweets per minute rate through my data.

Continue reading

James Bond Data Analysis

This Bloomberg article on James Bond is pretty much perfect. These are some of my favourite things: data analysis, data visualization, and cheese.

I like that the authors sat through all Bond films to classify such things as:

  • Time spent flirting
  • Number of double entendres
  • Aggregate time spent in tuxedos – with (or the morning after adventurous nights) without jacket

I also like that the method of data gathering and delineation of time in/out of tuxedos (or shirts) is precisely specified. Accuracy is important in maintaining significance and respectability.

Also important is the attention to detail in Bond’s romantic escapades, and kudos must go to the authors for including “making eyes” in this category – an easily missed, but crucial and subtle part of any Bond film worth its salt.

Scoldings from M, Q and anyone else alphabetically named are all scientifically categorised. Gadgets are carefully distinguished from plain, unsophisticated weapons.

Poker, Baccarat, and a kaleidoscope of cocktails do not escape the authors’ scrutiny, and of most importance is recognition of Bond’s surname first self introductions.

What I like most of all of is the deep, rewarding insights the authors have uncovered:

  • Sean Connery spent the most time in top drawer dress, followed by Daniel Craig
  • Connery was also the most shirtless of all bonds (possibly a reason to stick to the newer ones)
  • Daniel Craig managed to find something other than romance to do for over 95% of his movies. It must be hard resisting the temptations of being so tempting.
  • The safest Bond to hook up with is Timothy Dalton, the only Bond to have no love interests perish during their movies. Of course, Timothy Dalton may not be the most appealing Bond to choose from, and so perhaps some of the others are worth the risk.
  • If you want an Aston Martin to stay in good shape, give it to me, not James Bond. Please.
  • Brosnan’s Bond was the most prolific with puns and double entendres. Perhaps if he was more focused on delivering value, his movies would have been more appealing.

Particularly worth attention, I think, is the in-depth analysis of correlation between flirting, love interest and mortality for Bond’s female companions. While most Bonds manage to endanger their love interests to the point of fatality in at least some of their movies, Craig, Connery and Moore all manage to lose two love interests in the same movie at least once.

I love the timelines of each movie, which reveal the typically short timeframe Bond takes to recover from the loss of a loved one before moving onto the next opportunity:

bondkey

bondromance.png

rogermooreromance

If only all scientific method was so exhaustive. Most recommended, A+++, would buy again.

Read the original article here.

Service health monitoring with Reactive Extensions

When integrating independent services to build larger systems, it’s often important for services to keep track of the status of the other services that they depend on. Especially for a microservices approach, where services should expect their dependencies can be absent at any point in time. Services that cope with dependencies being unavailable make a less flaky and more resilient, hands off system.

Services can dynamically change their behaviour as the states of their dependencies change. If a dependency is offline, a service can decide to re-route work, buffer it until processing can resume, or explicitly reject requests. Doing this proactively by reacting to changes in dependencies’ statuses makes this much more fluid.

This post will look at a simple way of using Reactive Extensions to keep track of dependency status.

Continue reading