I help run a Twitch channel with a fellow group of nerds called 2Dorks. We produce a whole bunch of content on our Twitch Channel and last week we decided that we should put that all to good use. Jacob, a fellow dork and quite the philanthropist, decided that a 12 hour charity stream would be an awesome way to “do some good in the neighborhood.” We all agreed and quickly threw together plans to make this happen.

The meat of this post is on the alert system. We had a few problems:

  * Toys for Tots didn't have any kind of alerting at all, and no connection to Twitch.
  * They don't send an email to the donation recipient when someone donates, so no parsing emails for a notification.
  * The only place to get your donations is from either table in your admin panel, or a downloadable csv file.

This doesn’t sound like anything too bad, but it got a little hairy. We didn’t know a few things;

  * Does the list populate by most recent or are donations appended?
  * Does the list paginate, and at what point?
  * How can we grab data from behind the login wall with a script?
  * Is the data unique enough that we won't trigger duplicates?

Like all good problems, you should always start with the questions. We had plenty of them.

I set to work on planning this thing out. There was no way that we were going to do a donation heavy stream without some way of showing an alert on the screen, or a tracking meter for when someone donated. I can imagine a lot of Twitch streamers would love to donate to something random and maybe this info can help someone. I am NOT a seasoned python dev, but I dabble. So take what you see here and improve on it if you think there’s a better way.

I knew we needed to be able to send alerts to Streamlabs, a common service used by Twitch streamers to display alerts on top of their stream. The good news on that front is that I had worked on a python based alert system utilizing the Streamlabs API, so I had that code already. I just needed to find out how to send the alerts from data I had no idea how to get.

GOOGLE

There’s you answer. I googled a bunch of stuff and finally got pointed to selenium. It’s a python module (oversimplification) that contains a function I need called webdriver. It can not only open webpages, but also click buttons and fill in boxes. I used that to login to the donation administration page, and grab the table. Here’s the code:

[code language=“python”]

initiate

driver = webdriver.PhantomJS() # initiate a driver, in this case PhatomJS. No Window PopUp driver.get(“https://p2p.charityengine.net/ToysforTotsFoundation/Dashboard/Donations/") # go to the url

log in

username_field = driver.find_element_by_name(‘UserName’) # get the username field password_field = driver.find_element_by_name(‘Password’) # get the password field username_field.send_keys(“username”) # enter in your username password_field.send_keys(“password”) # enter in your password password_field.submit() # submit it

gotta wait while the new site loads

time.sleep(3)

html = driver.page_source

#turn the html into ascii so we can iterate over the lines html = html.encode(‘ascii’,‘ignore’)

#close PhantomJS Instance driver.quit() [/code]

That chunk of awesome logs into the given URL with my admin creds, then pulls down the source code of the page. I had to put a sleep command in there to stop it from pulling down the login page code. Three seconds seemed to do the trick. Just long enough for the page and the actual data I needed to load. I grabbed the HTML because I needed the table where the names and donations lived. Luckily, it was the only table on the page, so I could parse each line one by one and just search for instances of name. The issue there is that I needed to separate actual names from other stuff. ENTER REGEX LAND! More googling plus a regex tester and I ended up with this:

[code language=“python”] for line in str.splitlines(html): if re.search(”(

That little piece of code loops through each line in the HTML and searches for a tag with something like Bob Whatever and considers it a match. Later on, I realized that this was a huge mistake, because it would ONLY match instances of names of either 1 or 2 words. Not 3 or 4. It also wouldn’t work on people like Nikolaj Coster-Waldeau should Jamie Lannister actually desire to donate. It also wouldn’t work on Edith and Archie Bunker or Conan O’Brien. My revised code after the entire thing blew up looks like this:

[code language=“python”] for line in str.splitlines(html): # Find the username line in the table if re.search("(

That part with the Opted out of communications line also had to be added so that the looper would skip instances of that. There is a hilarious moment where Mr. Opted out of communcations donated a bunch of money. Had to kill the script in a bad way. Lesson learned!

Onward to Redis!

Not only did I need to find the names and the amount they donated, I also needed to keep that data somewhere. What could I use to define two simple values that I could reference easily? REDIS!

I have never worked with Redis, but I knew enough to know it was a simple key/value store that I could use to store stuff like donator_name: value. It was perfect! I also knew how to access it with python since I had goofed around with another idea from before.

So I stored each line discovered with the loop in a dictionary that looked like this: {'Hulk Hogan': '$7000.00'}. That was in a list so it was a bunch of those over time. After I had the list of key/values, I looped over that list and ONLY stored values in redis that weren’t already there. I would check to see if name = donation, and if it didn’t, I’d add it. I figured people might donate twice. NOTE: The big issue with this is if someone donated twice the same amount, I wouldn’t alert nor include it. I took a risk and I knew it.

[code language=“python”]

iterate over the result list, load each dict into the redis db as a keypair

for i in result: if r.get(i.get(’name’)) != i.get(‘donation’): f.write(i.get(’name’) + ‘: ’ + i.get(‘donation’) + ' ’ + ‘\n’) r.set(i.get(’name’), i.get(‘donation’)) [/code]

That’s the code that does the storing of the value. You can kind of see what I’m doing there. I also stored the donation into a donations.txt file. That file would be used for a bunch of stuff. I needed that one to load into a webpage so we could feed it to Open Broadcaster Software which is what we used to make all the ovelays.

Alert. Alert.

So now we havea all the data in a file and we are currently holding it in result list. That result list is handy because I could use it inside the loop to call my alert function. By the way, the alert function looks like this:

[code language=“python”] #Twitchalerts API url = ‘http://www.twitchalerts.com/api/v1.0/alerts'

Payload. Still need to add a gif and a sound

payload = { “access_token”: “no_you_can’t_see_this”, “duration”: “5”, “type”: “subscription”, “special_text_color”: “#ff0000”, }

Define the alert function that will be used to actually fire to streamlabs

def alert(name, donation): payload[‘message’] = ‘’ + str(name) + ‘ has donated ’ + str(donation) + ‘ to Toys For Tots!’ session = requests.Session() response = session.post(url, data=payload) [/code]

So I would do something like alert(i.get('name'), i.get('donation')) at the end of the result loop. I threw a time.sleep(15) in there because I don’t know how streamlabs handles a ton of alerts coming in at once. I wanted to avoid any missed alerts.

Meter Time

Okay, so we have our data alerting. GOOD! Job done, right?

WRONG!

Our good buddy, Drest, from our WoW streams surprised us with a meter for keeping track of donations. “All you have to do is plugin your data,” he says. Sounds great!

It was so much trickier than I thought and required more of that good ol’ Google. I figured out after a lot of trial and error that I could take the text from files and load it between <div> tags using jQuery. That way, I could define the data on demand rather than having to statically put it in the webpage. This was important because data was going to change and I wanted to make sure the website, AKA meter, got all the data.

So, I decided to just write out all the info into files that the jQuery could load from. Here’s what that looks like. IT’S UGLY

[code language=“python”] with open(’/foo/bar/donations.txt’, ‘r’) as content_file: content = content_file.read()

f = open(’/foo/bar/total.txt’, ‘w+’) total = 0 for line in content.splitlines(): dollar = re.search("[0-9]+.[0-9]{2}", line) dollar = dollar.group() total = total + float(dollar) f.write(’${:,.2f}’.format(total) + ‘\n’) f.close()

f = open(’/foo/bar/percent.txt’, ‘w+’) percentage = float(total) / 500 f.write("{:.0%}".format(percentage) + ’ \n’) f.close()

num_lines = sum(1 for line in open(’/foo/bar/donations.txt’)) f = open(’/foo/bar/lines.txt’, ‘w+’) f.write(str(num_lines) + ‘\n’) f.close [/code]

I would have done this smarter, but I didn’t have enough time and I don’t know HTML or PHP. I knew how to load the text from a file, but not how to separate it. The jQuery that actually pulls in the data looks like this:

[code language=“javascript”] $(document).ready(function () { function callAjax() { $("#amount").load(“total.txt”); } setInterval(callAjax, 10000); }); [/code]

That setInterval deal came a lot later. I learned that I needed to not only dynamically update stuff on page load, I also needed to do it in increments! That became apparent when we realized that the OBS plugin we used to display web content didn’t actually refresh or even have a refresh interval.

The good part about all this is that Drest was awesome and programmed div id’s around all the spots where I needed to put the text. So all I had to do is load the text file into whatever div I needed it to go to and everything was fine.

Lessons Learned

Double check your REGEX and consider alternates. This one burned pretty bad. When a four name person came in, it totally broke the entire engine. Since a name wasn’t found, the next value of a dollar amount didn’t have anywhere to go and the script broke. It broke again later when a person with an apostrophe in their name donated, but by then I had gone to bed and nothing could be done at the time.

Stuff is gonna break. Roll with it. We were troubleshooting bugs for an hour while LIVE. Though not ideal, it happens. Stuff is going to break. The key to this is knowing enough about the code you wrote to know where the issue is. This is where copy/paste can really burn you. If you just pull from websites and don’t understand your code, then you’re going to get hurt.

PHP/HTML/jQuery I need to know more of this. That’s all.

If you can draw it, you can build it A network teacher told me that a long time ago. I envisioned how I would hold the data early on in the project. Once I had that vision, I was able to put the pieces together. The blank page was terrifying. THe first thing I did was paste in my alert code I wrote a long time ago, because I knew how it worked. After that, some of the intimidation went away and I just started plowing away at it.

Charity streams are awesome We were able to raise over $500 for Toys for Tots just by sitting down and playing video games all night. Granted a lot of effort went in to promoting it and getting people on board with us, but at the end of the day we just hung out and played. It was amazing. We definitely want to do it again, but we’ll pick a charity early next time and hopefully have the bugs worked out.

Code

Yep. I did the thing. I wrote an app that alerts a service by pulling data from another service. I wrote my first python script in July of this year, and now I wrote something on my own because I could. I would have never imagined this even a year ago, and now here I am. Once you know how to code, the possibilities are endless. I’m no expert, but I made a thing that did a job for me that otherwise would have been impossible. I think that counts for something.

Hopefully this post provides some info for other folks who want to send alerts for charities that don’t necessarily have a great system for weird off-shoot stuff like a Twitch stream. Extra-Life and Child’s Play cater to this kind of audience but there are plenty who don’t.

Questions

Got anything you want to know about this stuff? Just leave a comment on the post. I’ll try to answer them. I hope to put my code for this up on Github once I sanitize it a bit and add some comments. I also need to get Drest’s permission since the meter site was his minus the jQuery statements.