About two months ago, I wanted to go on a vacation. I had the hotel more or less picked out, but the transportation was still up in the air.
I began by scouring the web for cheap plane tickets, like normal travel folks seem to do. I scoured all of the fancy airlines, but all of their fares were too high for my liking.
I decided to look into Southwest, the de facto airline for low fares (not really). I had heard from a few internet-goers that ticket prices would drop on Tuesday nights, so I set out to test that claim. I started checking fare prices periodically throughout the day.
Days passed.
Reload.
Reload.
“Would you like to resubmit this form?”
Yes.
Reload.
Nothing.
I started to get annoyed with the repetition (surprise, surprise!), so I set out to automate things as best I could. What if I wrote a script to do these annoying tasks for me? Yeah. I liked that idea.
Crawling the Southwest website
A quick look at Southwest’s site revealed no API. (No surprise there.)
I figured that straight up scraping their website would be an effective approach. I could write a script which would, you know, scrape their website for the data I wanted.
I chose Node because I figured they would have great tooling for web scraping, and I was more or less correct on that one.
As I began experimenting with the algorithm… err, script, I realized that it was a lot harder than I thought it would be to fake a form submission on Southwest’s website so that I could get to the results page.
I tried cURL.
I tried “copy as cURL.”
I tried faking the user agent.
I tried all the things.
The solution was to use osmosis to literally fill out and submit the search form, and then scrape the results page.
So, I built a quick and dirty command line script that scraped their website every so often. I then let it do its thing every night while I slept. It wasn’t fancy, but it did what I told it to do.
Analyzing the Fares
I wrote a Node script … wait I already said that. Okay. So, I have a script running. It’s printing out fare prices, but I’m not really sure what to do with all of this data. It’s not like I want to sit there and watch it all night.
So what did I do? I hooked up Twilio to send me a text message whenever a fare hit a certain threshold. Twilio offers a free account as long as you verify the phone number that you’re texting to, so that’s pretty rad. So I verified my number and let the texts come flowing in.
And since I didn’t think it through, I also chose to send the text message at a low interval so that it would keep pinging me until I stopped the script, even if I was out on a date with my lovely wife.
It wasn’t annoying at all. /s
But it worked quite well. I could see that fares did indeed drop, especially late at night. The internet-goers didn’t lie to me this time.
Plotting the Data
After a few days, I got kind of annoyed (what’s new?) with looking at a stream of text and thought of complex ways to make it more appealing to look at for the few seconds that I looked at it every day.
So what did I do? I hooked up blessed to plot me some graphs and make things look pretty. I had seen blessed be put to use in the lovely webpack-dashboard, so I thought “why ever not?”.
It was awesome. Thanks, Ken Wheeler.
Booking the Flight
Days later, on a normal Tuesday night, it happened: the prices dropped dramatically, something that did not happen every Tuesday, but rather quite randomly. I snagged our tickets for under $100. (I, of course, booked them manually because my script is not that fancy.)
Sweet, Sweet Victory
A month later, I went on vacation. And that’s about it. You can check out the project on GitHub if you’d also like to go on a nice vacation.
Note: as of July, 2017, the project is no longer available on GitHub due to a takedown request by Southwest.