In August this year, I gave a talk at NDC Sydney on Real-time Twitter Analysis with Reactive Extensions. NDC is the Norway Developers Conference, so it’s a natural progression for them to come to Sydney. This was their first time down under, but they’ve already announced they’ll be back in August 2017.
It was a three day conference with over 100 speakers, some international and some local. That’s a lot of speakers, and it translates into 7 parallel tracks, or 7 concurrent talks.
Full talk video and code online
The video of my talk is on NDC’s vimeo channel, and the code and data driving the visualizations in the talk is on github. The repository is fairly large because the data files total a couple of hundred megabytes.
I’ve written a few articles covering parts of the material in the talk and discussing the code approach:
Hillary Clinton and Barack Obama at the Democratic National Convention
While preparing for my recent NDC Sydney talk on real-time Twitter analysis, I was looking for interesting events to watch play out on Twitter. One great candidate was the 2016 Democratic National Convention, where Hillary Clinton accepted the nomination for president – the first woman in history to be nominated. There were a lot of major speeches from people such as Michelle Obama, former president Bill Clinton, Barack Obama and of course Hillary herself.
In a previous post I discussed the approach I used to identify the topics of Twitter discussion over time. Mapping them out into a histogram in real-time made for an interesting view of the event from the audience’s perspective.
Another view that can be taken using the same analysis approach is to plot key topics over time, to get an idea of the strength and duration of each topic’s impact. It’s interesting to see the topics that got the biggest response on Twitter during Hillary Clinton and Barack Obama’s convention speeches.
During my recent NDC Sydney talk on real-time Twitter analysis with Reactive Extensions, I talked about the approach I used to track current discussion topics as they changed over time. This is similar to Twitter’s trending topics, but changing more dynamically.
The source data came from Twitter traffic during two episodes of the ABC’s Q&A show in the lead up to Australia’s 2016 federal election. Each of the candidates for Prime Minister – incumbent Malcolm Turnbull and opposition leader Bill Shorten appeared as a solo guest to face questions from the audience.
I wanted a live view of the current topics of discussion as the show progressed, to get a feel for which topics the Twitter audience was responding to.
I recently spoke at NDC Sydney, which was a great experience. My talk was on Real-Time Twitter Analysis with Reactive Extensions. I wanted to have a deeper look into the data and approaches I’d started with the Women Who Code workshop.
I wanted some compelling Twitter data, and given the year we’ve had so far in 2016, politics seemed a good choice. Between Australia’s federal election, the EU referendum in the UK and the US presidential primaries, there was a lot going on in this space. Twitter engagement was huge across all of these events.
One thing I wanted to be able to do was to plot the rate of Twitter traffic in real-time. This was relatively easy with a couple of lines of Rx, and it gave me a good grasp of the tweets per minute rate through my data.
In a previous post, I looked at how Rx can be used to neatly track the status of service dependencies. In this post, I’ll look at how the same technique can be used to manage streams that update with a varying frequency, and a specific application – tracking market data staleness in financial applications.
When integrating independent services to build larger systems, it’s often important for services to keep track of the status of the other services that they depend on. Especially for a microservices approach, where services should expect their dependencies can be absent at any point in time. Services that cope with dependencies being unavailable make a less flaky and more resilient, hands off system.
Services can dynamically change their behaviour as the states of their dependencies change. If a dependency is offline, a service can decide to re-route work, buffer it until processing can resume, or explicitly reject requests. Doing this proactively by reacting to changes in dependencies’ statuses makes this much more fluid.
This post will look at a simple way of using Reactive Extensions to keep track of dependency status.
I recently ran a workshop on Rx, and as part of preparing I had to hunt down some useful resources for learning. I found that things had come a long way since the early days, when people mostly learned from videos of Wes Dyer, Jeff van Gogh and Bart de Smet drawing on whiteboards at Microsoft, and Lee Campbell’s blog.
With all this new exposure, there are now a lot more resources for learning Rx. Here are some that I think are particularly useful.
The Reactive Extensions (Rx) library is one of those tools that opens up your mind to new ways of thinking about your how your data and application work together. In this post I’m going to talk about an easy technique you can use to replay time series data as if it were live.
Replaying time with HistoricalScheduler
Schedulers in Rx are a deep subject, but one of the things schedulers are responsible for is managing how time based operations run. All time based operations in Rx, like Timeout, Delay, etc allow you to provide your own scheduler.
HistoricalScheduler inherits from VirtualTimeScheduler, and that name gives us an idea what we can do with it. VirtualTimeScheduler gives us control of time – at least as far as our code goes. You can’t go back in time and buy stocks to become rich, but you can make your code think it’s doing that.
I had the opportunity to run a workshop on the Reactive Extensions library with the Sydney Women Who Code group earlier this year. Rx is something I’m passionate about – it really has the power to change the way you approach your data and application design to open up a lot of possibilities.
This post gives some details about the workshop, but is mostly about the process of creating it and some insights into how it went. The workshop and code are on github if you want to check it out.