391 lines
24 KiB
HTML
391 lines
24 KiB
HTML
|
<!DOCTYPE html>
|
||
|
|
<html lang='en'><head><meta charset='utf-8' /><meta name='pinterest' content='nopin' /><link href='../../../../static/css/style.css' rel='stylesheet' type='text/css' /><link href='../../../../static/css/print.css' rel='stylesheet' type='text/css' media='print' /><title>A Faster Feed Apart / Steve Losh</title></head><body><header><a id='logo' href='https://stevelosh.com/'>Steve Losh</a><nav><a href='../../../index.html'>Blog</a> - <a href='https://stevelosh.com/projects/'>Projects</a> - <a href='https://stevelosh.com/photography/'>Photography</a> - <a href='https://stevelosh.com/links/'>Links</a> - <a href='https://stevelosh.com/rss.xml'>Feed</a></nav></header><hr class='main-separator' /><main id='page-blog-entry'><article><h1><a href='index.html'>A Faster Feed Apart</a></h1><p class='date'>Posted on April 30th, 2010.</p><p><a href="http://aneventapart.com/">An Event Apart</a> is a conference for web developers and designers that
|
||
|
|
happens a few times a year in various cities. <a href="http://afeedapart.com/">A Feed Apart</a> is a site
|
||
|
|
that aggregates tweets during each conference and displays them in a live
|
||
|
|
stream so attendees can follow them during the conference, and people not
|
||
|
|
attending can see what the attendees are talking about.</p>
|
||
|
|
|
||
|
|
<p>A Feed Apart was originally written by <a href="http://nicksergeant.com/">Nick Sergeant</a> and <a href="http://pkarl.com/">Pete
|
||
|
|
Karl</a> of <a href="http://lionburger.com/">Lion Burger</a> during one of the conferences. Since
|
||
|
|
then a lot of people have used it and love it.</p>
|
||
|
|
|
||
|
|
<p>The current A Feed Apart site is not without its problems. It was written in
|
||
|
|
a single night so it's not perfect. During the last conference it went down
|
||
|
|
several times and lost some tweets along the way.</p>
|
||
|
|
|
||
|
|
<p>I work at <a href="http://dwaiter.com/">Dumbwaiter Design</a> with Nick and one day he mentioned that
|
||
|
|
it would be cool if we rewrote A Feed Apart from the ground up. He's learned
|
||
|
|
a lot about how people use the site and what the big problems are, so we'd have
|
||
|
|
a better idea of what we need to accomplish.</p>
|
||
|
|
|
||
|
|
<p>Nick, myself and <a href="http://alialithinks.com/">Ali Ali</a> have risen to the challenge and started
|
||
|
|
rewriting A Feed Apart. Ali is a designer and is taking care of the design of
|
||
|
|
the new site. Nick is handling all the frontend HTML, CSS and Javascript. My
|
||
|
|
job is the backend.</p>
|
||
|
|
|
||
|
|
<p>Ali posted a <a href="http://www.dcamm.com/blog/archives/765">blog entry</a> about the design of the new version so
|
||
|
|
I figured I'd write about the backend to show how I'm trying to improve it. If
|
||
|
|
you have any advice I'd love to hear it — the next An Event Apart conference
|
||
|
|
is about three weeks away so there's still time to add improvements.</p>
|
||
|
|
|
||
|
|
<ol class="table-of-contents"><li><a href="index.html#s1-how-afa-is-used">How AFA is Used</a><ol><li><a href="index.html#s2-people-attending-the-conference">People Attending the Conference</a></li><li><a href="index.html#s3-people-that-wish-they-were-attending-the-conference">People That Wish They Were Attending the Conference</a></li><li><a href="index.html#s4-speakers-and-organizers">Speakers and Organizers</a></li></ol></li><li><a href="index.html#s5-goals">Goals</a></li><li><a href="index.html#s6-staying-sane">Staying Sane</a></li><li><a href="index.html#s7-being-real-time">Being Real Time</a><ol><li><a href="index.html#s8-retrieving-updates">Retrieving Updates</a></li><li><a href="index.html#s9-storing-updates">Storing Updates</a></li><li><a href="index.html#s10-sending-updates-to-users">Sending Updates to Users</a></li></ol></li><li><a href="index.html#s11-organizing-the-conversation">Organizing the Conversation</a></li><li><a href="index.html#s12-staying-consistent">Staying Consistent</a></li><li><a href="index.html#s13-getting-bigger">Getting Bigger</a></li><li><a href="index.html#s14-a-work-in-progress">A Work in Progress</a></li></ol>
|
||
|
|
|
||
|
|
<h2 id="s1-how-afa-is-used"><a href="index.html#s1-how-afa-is-used">How AFA is Used</a></h2>
|
||
|
|
|
||
|
|
<p>You can't hope to improve on a site unless you know how people are going to use
|
||
|
|
it. AFA has been around for a while and Nick has learned a lot about what
|
||
|
|
people want to get out of it.</p>
|
||
|
|
|
||
|
|
<p>There are three main kinds of users of AFA:</p>
|
||
|
|
|
||
|
|
<ul>
|
||
|
|
<li>People attending the conference</li>
|
||
|
|
<li>People that wish they were attending the conference</li>
|
||
|
|
<li>The speakers/organizers of the conference</li>
|
||
|
|
</ul>
|
||
|
|
|
||
|
|
<h3 id="s2-people-attending-the-conference"><a href="index.html#s2-people-attending-the-conference">People Attending the Conference</a></h3>
|
||
|
|
|
||
|
|
<p>People that attend AEA are web developers and designers. They're up-to-date on
|
||
|
|
the latest technology and almost all of them use <a href="http://twitter.com/">Twitter</a>.</p>
|
||
|
|
|
||
|
|
<p>During the conference you'll see a sea of laptops in the audience. Attendees
|
||
|
|
will tweet about whatever is happening <em>right now</em>. There's a huge amount of
|
||
|
|
conversation that happens between attendees that AFA is trying to collect and
|
||
|
|
present.</p>
|
||
|
|
|
||
|
|
<p>A simple example is one that Nick mentioned to me: if a presenter mentions
|
||
|
|
a website during her presentation someone will tweet the URL so other people
|
||
|
|
can have a link to easily click on. Attendees will also tweet their agreement
|
||
|
|
or disagreement with what the presenter is saying <em>as they're speaking</em>.</p>
|
||
|
|
|
||
|
|
<p>The most important aspect of AFA for this group of people is the <em>real time
|
||
|
|
conversation</em>. If AFA doesn't show tweets until a minute after they've been
|
||
|
|
posted it's useless to these people.</p>
|
||
|
|
|
||
|
|
<p>Another important part of AFA for attendees is that it's a one-stop-shop for
|
||
|
|
the conversation behind AEA. Switching between windows/tabs/applications to
|
||
|
|
read and contribute is annoying.</p>
|
||
|
|
|
||
|
|
<h3 id="s3-people-that-wish-they-were-attending-the-conference"><a href="index.html#s3-people-that-wish-they-were-attending-the-conference">People That Wish They Were Attending the Conference</a></h3>
|
||
|
|
|
||
|
|
<p>The next group of users is those that can't attend the conference for a variety
|
||
|
|
of reasons but are still interested in what's going on.</p>
|
||
|
|
|
||
|
|
<p>For this group the real time nature of AFA is unimportant. They're probably
|
||
|
|
not going to be following the conversation 24/7 so if a tweet takes a few
|
||
|
|
minutes to display it's not a big deal.</p>
|
||
|
|
|
||
|
|
<p>What these people <em>do</em> care about is the integrity of the stream. They don't
|
||
|
|
want to miss any part of the conversation. If AFA loses tweets they're better
|
||
|
|
off doing a simple search on Twitter because they'll get the whole story.</p>
|
||
|
|
|
||
|
|
<h3 id="s4-speakers-and-organizers"><a href="index.html#s4-speakers-and-organizers">Speakers and Organizers</a></h3>
|
||
|
|
|
||
|
|
<p>The last main group of users is the speakers and organizers of AEA.</p>
|
||
|
|
|
||
|
|
<p>Any speaker worth their salt will want to know what people are said about their
|
||
|
|
presentation. Obviously they can't be reading AFA while they're presenting,
|
||
|
|
but afterword they'll certainly want to go back and see what people were saying
|
||
|
|
during their specific presentation.</p>
|
||
|
|
|
||
|
|
<p>Likewise, organizers want to know which presentations people liked and which
|
||
|
|
ones people didn't enjoy. This could easily influence who they choose to
|
||
|
|
invite to future conferences.</p>
|
||
|
|
|
||
|
|
<p>For this group the most important aspect of AFA is the organization of the
|
||
|
|
conversation into chunks, each of which applies to a single presentation. By
|
||
|
|
reading through each chunk of conversation they can get an idea of the general
|
||
|
|
response to each presentation.</p>
|
||
|
|
|
||
|
|
<h2 id="s5-goals"><a href="index.html#s5-goals">Goals</a></h2>
|
||
|
|
|
||
|
|
<p>Once we identified the main users of AFA we were able to come up with the goals
|
||
|
|
for the new version of the site:</p>
|
||
|
|
|
||
|
|
<ul>
|
||
|
|
<li><p><strong>Stay sane while developing.</strong> Ali, Nick and myself are rewriting the site
|
||
|
|
as a side project, so we don't want it to take <em>too</em> much of our time or cause
|
||
|
|
<em>too</em> much stress.</p></li>
|
||
|
|
<li><p><strong>Provide the real time conversation.</strong> The site needs to be fast and
|
||
|
|
responsive, so people at the conferences can use it to converse.</p></li>
|
||
|
|
<li><p><strong>Don't miss anything.</strong> People following along from home shouldn't feel
|
||
|
|
like they're missing anything. We want to provide a <em>complete</em> version of the
|
||
|
|
conversation happening behind the event.</p></li>
|
||
|
|
<li><p><strong>Organize the conversation.</strong> Presenters and speakers need a way to see
|
||
|
|
what people are saying about them. They want to know what people think about
|
||
|
|
each "chunk" of the event.</p></li>
|
||
|
|
<li><p><strong>Grease the wheels of (physical) social interaction.</strong> This is something
|
||
|
|
that AFA hasn't tried to address before, but that we'd like to work on with
|
||
|
|
this version. There are a lot of people at each conference and we'd like to
|
||
|
|
help them get together. Whether it's going out for dinner or meeting up at the
|
||
|
|
<a href="http://mediatemple.net/">Media Temple</a> party we want to get people talking to each other. I won't
|
||
|
|
talk about this goal in this post because we're still figuring out the best way
|
||
|
|
to do it.</p></li>
|
||
|
|
</ul>
|
||
|
|
|
||
|
|
<h2 id="s6-staying-sane"><a href="index.html#s6-staying-sane">Staying Sane</a></h2>
|
||
|
|
|
||
|
|
<p>If the three of us tried to create the site with nothing more than a couple of
|
||
|
|
laptops and a few chats in person we'd go crazy. We use a few tools to help us
|
||
|
|
manage the development of the site.</p>
|
||
|
|
|
||
|
|
<p>To create <strong>wireframes of the design</strong> we're using a free account at <a href="http://hotgloo.com/">Hot
|
||
|
|
Gloo</a>. Hot Gloo is a great tool that lets us quickly sketch out ideas
|
||
|
|
and comment on them.</p>
|
||
|
|
|
||
|
|
<p>To share <strong>design comps</strong> we're using <a href="http://dropbox.com/">Dropbox</a>. It's simple to set up
|
||
|
|
a shared Dropbox folder and Nick and I can get real time updates when Ali makes
|
||
|
|
changes to the design.</p>
|
||
|
|
|
||
|
|
<p>To <strong>work together on the code</strong> Nick and I use <a href="http://hg-scm.org/">Mercurial</a> repositories.
|
||
|
|
Mercurial lets us work on the same code bases simultaneously and we almost
|
||
|
|
never have to worry about merging. We use <a href="http://codebasehq.com/">Codebase</a> for hosting and issue
|
||
|
|
tracking.</p>
|
||
|
|
|
||
|
|
<h2 id="s7-being-real-time"><a href="index.html#s7-being-real-time">Being Real Time</a></h2>
|
||
|
|
|
||
|
|
<p>The previous version of AFA wasn't <em>truly</em> real time. When you went to the
|
||
|
|
site your browser would ask AFA for the newest updates every 10 seconds.</p>
|
||
|
|
|
||
|
|
<p>There are two main problems with this approach. The first is that it's not
|
||
|
|
really <em>real time</em>. I've noticed this being an issue in my own experience.</p>
|
||
|
|
|
||
|
|
<p>When I watch any of the various "live streams" of <a href="http://apple.com/">Apple</a> press conferences
|
||
|
|
I'm usually at work where other people are also watching. We very rarely load
|
||
|
|
the pages of the streaming sites like Gizmodo at the exact same second, so our
|
||
|
|
browsers will be out of sync with each other. Nick might get an update that
|
||
|
|
I would have to wait 8 more seconds to see.</p>
|
||
|
|
|
||
|
|
<p>When I glance over at his screen and see an update that I don't have
|
||
|
|
I instinctively refresh the page to get it. This defeats the purpose of the
|
||
|
|
"live updating" code that the developers of these sites worked on. They may as
|
||
|
|
well have just made a static page and told me to refresh.</p>
|
||
|
|
|
||
|
|
<p>The second problem is that querying every 10 seconds can be taxing on the
|
||
|
|
site's database. We're doing this as a side project so we don't have unlimited
|
||
|
|
funding for a hefty database server. If 1,000 people are querying for updates
|
||
|
|
every 10 seconds that's 100 requests per second to the database. This means we
|
||
|
|
need to have some kind of in-memory cacheing if we want the site to feel snappy
|
||
|
|
on modest hardware.</p>
|
||
|
|
|
||
|
|
<p>We want the new AFA to be truly real time. To do this we need to use <a href="http://en.wikipedia.org/wiki/Push_technology#Long_polling">long
|
||
|
|
polling</a> by users' browsers to wait for updates and return them as
|
||
|
|
soon as they come in. We also need to retrieve the updates as fast as we
|
||
|
|
possibly can, and they need to be stored in memory to avoid hitting the
|
||
|
|
database constantly.</p>
|
||
|
|
|
||
|
|
<h3 id="s8-retrieving-updates"><a href="index.html#s8-retrieving-updates">Retrieving Updates</a></h3>
|
||
|
|
|
||
|
|
<p>The bulk of the conversation about AEA conferences comes from Twitter. Tweets
|
||
|
|
are the most important items that we need to display on the site, so we're
|
||
|
|
using Twitter's streaming API to pull them in.</p>
|
||
|
|
|
||
|
|
<p>Since we don't want to tie up an entire server to pull in tweets I've decided
|
||
|
|
to create a <a href="http://dieselweb.org/">Diesel</a>-based application called <strong>The Nozzletron</strong> to parse the
|
||
|
|
streaming API. Diesel is a <a href="http://python.org/">Python</a> framework that takes an elegant approach
|
||
|
|
to asynchronous communication with clients and servers.</p>
|
||
|
|
|
||
|
|
<p>Twitter's streaming API accepts HTTP requests and returns "chunked" responses,
|
||
|
|
each of which is a tweet. Unfortunately the Diesel's built-in HTTP client
|
||
|
|
doesn't handle chunked HTTP responses so I had to write some code to handle
|
||
|
|
them myself.</p>
|
||
|
|
|
||
|
|
<p>The Nozzletron will connect to Twitter's streaming API and wait for data to
|
||
|
|
come in. If there's no data to process it will relinquish the server's
|
||
|
|
processor so it can do other things. Processing a single tweet that comes in
|
||
|
|
doesn't take much time, so the server is free to do other things most of the
|
||
|
|
time.</p>
|
||
|
|
|
||
|
|
<p>I've also created another application called <strong>The Flickrtron</strong> to pull in
|
||
|
|
<a href="http://flickr.com/">Flickr</a> photos. Unfortunately Flickr doesn't have a streaming API like
|
||
|
|
Twitter so I have to resort to polling Flickr's API every few minutes for new
|
||
|
|
photos. Flickr is much less of a real time medium than Twitter though, so
|
||
|
|
I don't think this is a very big problem.</p>
|
||
|
|
|
||
|
|
<p>I'm using a Python library called <a href="http://stuvel.eu/projects/flickrapi">Beej's Flickr API</a> to talk to
|
||
|
|
Flickr. It is horrible. It calls itself an "API" but is really just a thin
|
||
|
|
wrapper around calls to the Flickr API. The objects it returns for API calls
|
||
|
|
are Elementtree objects representing the XML of the response.</p>
|
||
|
|
|
||
|
|
<p>I wish I could do something like:</p>
|
||
|
|
|
||
|
|
<pre><code><span class="code">thumb_url = photo.thumbnail_url</span></code></pre>
|
||
|
|
|
||
|
|
<p>Instead I have to use this monstrosity:</p>
|
||
|
|
|
||
|
|
<pre><code><span class="code">thumb_url = <span class="string">"http://farm%s.static.flickr.com/%s/%s_%s_s.jpg"</span> % <span class="paren1">(<span class="code">
|
||
|
|
photo.get<span class="paren2">(<span class="code"><span class="string">'farm'</span></span>)</span>, photo.get<span class="paren2">(<span class="code"><span class="string">'server'</span></span>)</span>, photo.get<span class="paren2">(<span class="code"><span class="string">'id'</span></span>)</span>,
|
||
|
|
photo.get<span class="paren2">(<span class="code"><span class="string">'secret'</span></span>)</span>,
|
||
|
|
</span>)</span></span></code></pre>
|
||
|
|
|
||
|
|
<p>I wish there were a better Python Flickr API out there but there doesn't seem
|
||
|
|
to be one. If I've missed it please let me know!</p>
|
||
|
|
|
||
|
|
<h3 id="s9-storing-updates"><a href="index.html#s9-storing-updates">Storing Updates</a></h3>
|
||
|
|
|
||
|
|
<p>We need a fast way to store and retrieve updates so AFA can provide a real time
|
||
|
|
view of the conversation happening at AEA. With this in mind I've chosen
|
||
|
|
<a href="http://code.google.com/p/redis/">Redis</a> to store the updates of the currently-happening event.</p>
|
||
|
|
|
||
|
|
<p>Redis is an easy-to-set-up, disgustingly-fast, in-memory data store. It's
|
||
|
|
similar to <a href="http://memcached.org/">memcached</a> but has more intelligent data structures that make my
|
||
|
|
life easier for this project.</p>
|
||
|
|
|
||
|
|
<p>As updates (tweets and flickr photos) are scraped by The Nozzletron and The
|
||
|
|
Flickrtron they're placed at the tail end of a Redis list. That means I can
|
||
|
|
quickly and easily get all the items since item <code>N</code> when a user's browser
|
||
|
|
requests them with a single <code>LRANGE {list-key} N -1</code> call.</p>
|
||
|
|
|
||
|
|
<p>I'm also keeping a few other pieces of information in Redis. For example,
|
||
|
|
there's a set of photo IDs held in the <code>{item-key}:flickrtron:grabbed_photos</code>
|
||
|
|
key that keeps track of all of the photos we've already seen.</p>
|
||
|
|
|
||
|
|
<p>This makes it easy to tell if we've already seen a photo (and therefore don't
|
||
|
|
need to query Flickr for more information) — it's a simple <code>SISMEMBER
|
||
|
|
{item-key}:flickrtron:grabbed_photos {photo-id}</code> call.</p>
|
||
|
|
|
||
|
|
<p>I'm also using Redis to store statistics about the site like:</p>
|
||
|
|
|
||
|
|
<ul>
|
||
|
|
<li>How many tweets we've scraped.</li>
|
||
|
|
<li>How many photos we've scraped.</li>
|
||
|
|
<li>How many people are currently waiting for updates.</li>
|
||
|
|
</ul>
|
||
|
|
|
||
|
|
<p>This kind of information will be extremely valuable in the future when we're
|
||
|
|
planning improvements to the site. Redis makes it fast and safe to update this
|
||
|
|
information using the <code>INCRBY</code> and <code>DECR</code> commands.</p>
|
||
|
|
|
||
|
|
<p>There's one more component to The Nozzletron and The Flickrton that I haven't
|
||
|
|
mentioned. Both use Redis' <code>PUBLISH</code> command to push new updates out to users'
|
||
|
|
browsers as they arrive. I'll talk more about that in the next section.</p>
|
||
|
|
|
||
|
|
<h3 id="s10-sending-updates-to-users"><a href="index.html#s10-sending-updates-to-users">Sending Updates to Users</a></h3>
|
||
|
|
|
||
|
|
<p>As I mentioned before we want to send updates to users <em>as soon as they're
|
||
|
|
received</em>. To do this I've created another Diesel-based application called
|
||
|
|
<strong>Halley</strong> to handle this <a href="http://en.wikipedia.org/wiki/Comet_(programming)">Comet</a>-style communication.</p>
|
||
|
|
|
||
|
|
<p>Halley has a few components. The first uses Diesel's <a href="http://bitbucket.org/boomplex/diesel/src/tip/diesel/protocols/redis.py">Redis API</a>
|
||
|
|
to subscribe to a Redis channel like <code>live:{event-id}:items</code> and
|
||
|
|
<a href="http://bitbucket.org/boomplex/diesel/src/tip/diesel/core.py#cl-90">fire</a> off messages whenever something new comes in. As soon as
|
||
|
|
a new update comes in from The Nozzletron or The Flickrtron all of Halley's
|
||
|
|
clients will get it.</p>
|
||
|
|
|
||
|
|
<p>When I started working on AFA Diesel's Redis client didn't support the very new
|
||
|
|
<code>PUBLISH</code>/<code>SUBSCRIBE</code>/<code>UNSUBSCRIBE</code> commands. I implemented them, put my
|
||
|
|
changes on <a href="http://bitbucket.org/">BitBucket</a>, sent a pull request, and started talking to the
|
||
|
|
Diesel crew on IRC.</p>
|
||
|
|
|
||
|
|
<p>One of the maintainers pulled my patches and made them even better. Now
|
||
|
|
Diesel's Redis client has full <code>PUBLISH</code>/<code>SUBSCRIBE</code>/<code>UNSUBSCRIBE</code> support.
|
||
|
|
It's a great example of how open source projects can produce awesome results.</p>
|
||
|
|
|
||
|
|
<p>The other main component of Halley is an HTTP server that listens for
|
||
|
|
connections from browsers. The Javascript on the site will call Halley and say
|
||
|
|
"I need updates since number <code>X</code>", where <code>X</code> is the length of the Redis list of
|
||
|
|
updates at the last time it spoke to the server.</p>
|
||
|
|
|
||
|
|
<p>Halley uses Diesel's <a href="http://bitbucket.org/boomplex/diesel/src/tip/diesel/protocols/http.py#cl-156">HTTP Server</a> to manage these requests.</p>
|
||
|
|
|
||
|
|
<p>If a client is asking for everything since <code>X</code> and <code>X</code> is a smaller number than
|
||
|
|
the current number of items it will return the updates that have happened since
|
||
|
|
then. This might happen if we return an update and then two more updates
|
||
|
|
happen before the browser gets around to sending another request.</p>
|
||
|
|
|
||
|
|
<p>If a client is asking for everything since <code>X</code> and <code>X</code> is equal to the current
|
||
|
|
number of items Halley will wait for a new message to be fired from the
|
||
|
|
<a href="http://bitbucket.org/boomplex/diesel/src/tip/diesel/core.py#cl-179">Loop</a> that's watching the Redis channel. While Halley waits she
|
||
|
|
relinquishes the processor to the server so other requests can be handled.</p>
|
||
|
|
|
||
|
|
<p>There's a bit of code to prevent DoS attacks that request every item in the
|
||
|
|
queue over and over again, of course.</p>
|
||
|
|
|
||
|
|
<h2 id="s11-organizing-the-conversation"><a href="index.html#s11-organizing-the-conversation">Organizing the Conversation</a></h2>
|
||
|
|
|
||
|
|
<p>The live stream is an important component of AFA, but it's not the only one.
|
||
|
|
We also need to organize updates into logical chunks by event and presentation,
|
||
|
|
and provide archives of old events so people can see what happened.</p>
|
||
|
|
|
||
|
|
<p>The main AFA site is built with <a href="http://djangoproject.com/">Django</a> and served with <a href="http://gunicorn.org/">Gunicorn</a> and
|
||
|
|
<a href="http://nginx.org/">Nginx</a>. It uses a <a href="http://www.postgresql.org/">Postgresql</a> database to store data that's not "live".
|
||
|
|
Because queries for live data are handled with Diesel and Redis we don't need
|
||
|
|
to send those request through the full Django/Postgresql stack. Django and
|
||
|
|
Postgresql are only involved when you load a fresh page, and they're <em>more</em>
|
||
|
|
than capable of handling the amount of traffic that AFA gets for those kind of
|
||
|
|
requests.</p>
|
||
|
|
|
||
|
|
<p>I've created an application called <strong>The Strainer</strong> to copy data from the live
|
||
|
|
stream to the Postgresql database. The Strainer looks at the list of live
|
||
|
|
items in Redis, parses those items into Django models and saves them to the
|
||
|
|
Postgresql database.</p>
|
||
|
|
|
||
|
|
<p>Using two different types of stores (Redis and Postgresql) means we can get the
|
||
|
|
best of both worlds for AFA:</p>
|
||
|
|
|
||
|
|
<ul>
|
||
|
|
<li><p>We can keep the live data that's accessed <em>constantly</em> in an in-memory Redis
|
||
|
|
datastore which makes it blazingly fast.</p></li>
|
||
|
|
<li><p>We can keep the less-frequently-used data in an on-disk Postgresql database
|
||
|
|
which lets us to keep our memory usage low and hosting costs down.</p></li>
|
||
|
|
</ul>
|
||
|
|
|
||
|
|
<p>Django's models and managers make it very easy to separate updates out into the
|
||
|
|
various sessions and presentations that happen at the conferences.</p>
|
||
|
|
|
||
|
|
<p>Sessions and presentations are much less in-demand than the live stream so we
|
||
|
|
can take advantage of Django's abstractions without worrying about the extra
|
||
|
|
memory/CPU usage we incur by doing so.</p>
|
||
|
|
|
||
|
|
<h2 id="s12-staying-consistent"><a href="index.html#s12-staying-consistent">Staying Consistent</a></h2>
|
||
|
|
|
||
|
|
<p>Another problem AFA has faced in the past is losing tweets. No code is perfect
|
||
|
|
(and mine <em>certainly</em> is not) so we need to anticipate that some of the
|
||
|
|
applications we're using will crash at some point.</p>
|
||
|
|
|
||
|
|
<p>I'm using <a href="http://supervisord.org/">Supervisord</a> to monitor the various processes of AFA on the
|
||
|
|
server. If a process crashes for some reason it will be restarted
|
||
|
|
automatically.</p>
|
||
|
|
|
||
|
|
<p>Supervisord also has a wonderful Python API, so I've created a simple Dashboard
|
||
|
|
view in the Django site that lets us stop/start/restart each individual process
|
||
|
|
with a simple web interface. The dashboard also shows us the current memory
|
||
|
|
usage of the server and some other statistics so we can monitor how things are
|
||
|
|
working through a web browser (instead of SSH'ing into the server, which is
|
||
|
|
a pain on a phone).</p>
|
||
|
|
|
||
|
|
<h2 id="s13-getting-bigger"><a href="index.html#s13-getting-bigger">Getting Bigger</a></h2>
|
||
|
|
|
||
|
|
<p>An Event Apart is a large event, but it's not a <em>huge</em> event. Despite this
|
||
|
|
I've been trying to build the backend in a way that can be easily scaled.</p>
|
||
|
|
|
||
|
|
<p>Right now it's hosted on a single server, but each of the individual components
|
||
|
|
could be moved to a separate server with less than an hour of work each:</p>
|
||
|
|
|
||
|
|
<ul>
|
||
|
|
<li>Redis</li>
|
||
|
|
<li>Postgresql</li>
|
||
|
|
<li>Django, Nginx and Gunicorn</li>
|
||
|
|
<li>The Nozzletron</li>
|
||
|
|
<li>The Flickrtron</li>
|
||
|
|
<li>The Strainer</li>
|
||
|
|
<li>Halley</li>
|
||
|
|
</ul>
|
||
|
|
|
||
|
|
<p>Moving Django/Gunicorn, Nginx, Redis, Halley, and Postgresql to dedicated
|
||
|
|
servers would increase the performance of the site <em>immensely</em>. I can't
|
||
|
|
imagine an event that would provide enough traffic to require more than that.</p>
|
||
|
|
|
||
|
|
<p>Even if there were an event that needed that kind of throughput, we could
|
||
|
|
easily split the single Halley and Redis servers into multiple load-balanced
|
||
|
|
servers.</p>
|
||
|
|
|
||
|
|
<h2 id="s14-a-work-in-progress"><a href="index.html#s14-a-work-in-progress">A Work in Progress</a></h2>
|
||
|
|
|
||
|
|
<p>This new version of A Feed Apart is still being built. I'm learning new things
|
||
|
|
every time I work on it, and I'm sure there's still room for improvement.</p>
|
||
|
|
|
||
|
|
<p>If you have any questions, advice, or want me to go more in-depth about
|
||
|
|
a specific aspect of the site's backend please let me know!</p>
|
||
|
|
</article></main><hr class='main-separator' /><footer><nav><a href='https://github.com/sjl/'>GitHub</a> ・ <a href='https://twitter.com/stevelosh/'>Twitter</a> ・ <a href='https://instagram.com/thirtytwobirds/'>Instagram</a> ・ <a href='https://hg.stevelosh.com/.plan/'>.plan</a></nav></footer></body></html>
|