Contract Work

Tuesday, April 29, 2014

Building Analytics

A key part of any build is figuring out what colleagues want to track and how to effectively manage that data. In this case, there is a lot of stuff to track. I recently finished building a hefty analytics components to the app and wanted to share my approach and some things that drove me crazy.

First, I had to really think through which analytics were important. The initial ask was for about 30 different types of analytics, but then, when adding in timeframes, and additional category components, we were talking about close to 400 queries… not super useful for an early launch. Additionally, through my experience with Neighborsations, I could look at these requests and know which ones were most important for raising funds, for identifying user paths, and which ones wouldn’t really be useful until we had a larger critical mass using the application. I also made sure to ask my team what the most important success factors were to them (ie- if this number isn’t what we want it to be, then the business is not successful and we need to make some hard choices, quickly… that’s actually a vital part to determining core metrics and something I learned from Steve Wendel who’s awesome at pushing you on those hard questions.)

The easiest way to do the queries was through writing DB queries, putting them into a model with methods and then creating a view. The app is an ember-appkit-rails app which kinda mushes the rails api and ember together but I decided to keep this simple and just run everything outside of the ember piece and just keep it as typical rails.

There were also a couple of options for running the number… we could have done a rake task or set up a chronjob. Right now, the approach is just to have the business folks hit that page whenever they want new numbers (with the understanding that they shouldn’t hit it too often because it kicks off a bunch of queries that hit the database).

To seed the data, I was originally leaning towards factory girl but decided to start with just creating the data I needed in the tests. What I didn’t realize is that the app automatically pulls in fixture data, so everytime I had a to eq it would fail because the number would be incorrect. That taught me… after spending probably too much time trying to figure out how to not have the test pull in the fixtures, I realized I should just embrace them, learn how to set up the fixtures for my tests correctly and use them to generate the data I needed.

Then I created a model with the class AdminAnalytics. Each analytic was a separate method. I think if I wanted to go back and continue to refactor further, then I could easily break the model into separate classes. For example, instead of having one overarching AdminAnalytics class, it would probably make more sense to have a Users class, a Posts class, and Ingredients class, etc.

Then it was down to the queries. I didn’t have much SQL experience and some of these queries were pretty complicated so it helped me to first write out all of the steps that I was looking for before translating it into an actual query. Once I did that, I could take those queries and use ActiveRecord to give me the rails magic that made the queries a little easier. For example, when joining tables in queries, you don’t have to note the join table… you can just note the two tables and activerecord will figure out the relationship between the two tables on its own. Then, because a lot of the queries involved profile type names or time parameters, I refactored by making those things arguments on the method that could be put in place in the view.

Then I created a controller which was literally one line and was able to create a view that just called the instance variable @analytics and the method with whatever arguments I needed. Bam! Analytics.

Now, once this was all done, we actually decided to pull the analytics out of the app. Instead of bundling these things together, we made it a separate app entirely that talks to the primary app in order to get the numbers needed. When we pulled it out, we used sequel which made it easier to pull those queries in (although still more difficult than just doing it right in the app). The nice part about the app was being able to use ActiveRecord but it also made the analytic piece dependent on a bunch of different models. For example, to get the total number of users, you just do User.count or to get the total number of posts, Post.count. You’re already depending on two different models here, User and Post. So, part of pulling this piece into a separate app was so that we were no longer relying on multiple models in order to get these numbers.

The other thing I really wanted to do was set up a simple dashboard using Dashing. I’d seen people whip these dashboards up in no time flat, so I figured I’d take a stab at it… boy was I mistaken. I think my journey to a dark place started with the decision to use dashing-rails instead of dashing. See, I figured, if I used dashing then I would have to create the connections to the database in order to get the information for the queries I was running. If I used dashing-rails, then the dashboard would be a part of the app and this part would be easier. (If you’re thinking about the paragraph above and thinking, wait a second, you pulled the whole thing into it’s own app anyway in the end, yes. I realize this and boy is hindsight 20/20). Dashing-Rails involves a bit more setup. You’ve gotta set up concurrency, you have to use a database that has lots of threads like puma, etc. The issues started there and just didn’t stop. First, I had an issue with puma and so I upp’ed the number of threads possible and that seemed to fix it. Then, there was a database connection issue. The DB connection would time out almost immediately. We got this working as well, but it still times out. After a certain number of rounds, the thing just kicks the bucket. Then, I had an issue where all my dashboard widget boxes would show up but nothing would show up in them. Sometimes, if I commented things out and then reimplemented them one at a time, they would work again. And there was no rhyme or reason about when the dashboard would start up and kick off, versus when it wouldn’t.

So, here’s the really annoying part. I FINALLY got the issues fixed (with the help of lots of pairing with a few different, very patient people) and pushed the dashboard to production, where everything promptly broke and nothing rendered correctly. Then we got it rendering correctly, but the queries still aren’t running correctly. All the while, I’m kicking myself because I was the one who said, “oh, and I’ll build this awesome dashboard to go along with the simple analytics view. It’ll be fun and shouldn’t take too long.” I think the dashboard was mostly a lesson in when to give up. I should have scrapped it after the first week and a half, but I was a little too stubborn and a little too determined to get it done.

So, that’s how I built the analytics. For my final thoughts, I think the moral of this story is to always keep it simple. Start small, get that shipped and continue to add and improve.

Friday, April 25, 2014

Interesting reads from the week 4/19 - 4/25

Just three short interesting reads for this week... I imagine/hope there will be more next week with some good blog posts coming out of Railsconf so enjoy these and look forward to some more next week.

Design for good and interesting thoughts

Interesting read on TDD

    In response to this initial post

Thursday, April 24, 2014

Errors in Ember

Most recently, I had to add a back button to a 404 page. I know, sounds simple, and it should be simple, but it was actually a little more complex than I thought which led me to learning about errors in Ember.

When originally looking in the codebase for where this change would need to happen, I found the ember template which simply said {{status}} {{statusText}} and a ember routes files that didn’t have anything in it except the standard code. I couldn’t figure out where the info being passed into the 404 was coming from and what made most sense for changing it.

When it comes down to it, the answer is actually pretty simple. Ember data has an automatic error route. In this case, we were looking in the user route. In this route, there is a model hook sets up a promise. always returns a promise.

If the promise is returned from the model hook or in other words filled with user data, then the path completes and you arrives at the correct node in the user path. However, if the promise isn’t returned or is rejected, then you get thrown into the error route. Ember always waits for a promise to be fulfilled and if it isn’t, you are led to an entire default error state.

The ember guide gives a pretty good explanation and walk through of what happens to get to these error substates here:

Saturday, April 19, 2014

Interesting Reads 4/12 - 4/18

This week, we've got some ember, some soft skills, some git practice, and more.


One of my goals is to understand and get better at git. This looks like a good resource... now I've just gotta make time for it!

To do, three things:

A great article on thinking through giving feedback.

Interesting read, not sure I agree with all the parts… what are your thoughts?

Since Ember is my first javascript framework, i'm always looking for good articles that explain the similarities and differences between what's out there. This was a good comparison.

This is the sort of beginner's guide I've been waiting for in Ember! If you're curious about getting started, check out this link to walk through building a simple app.

Video on code architecture and making things better

Friday, April 11, 2014

Interesting Reads 4/5 - 4/11

Some really good reads this week on a variety of different topics.


Read on effective flows

Conference proposals and a good step-through process to create your proposal

Really in depth thoughts on crafting a talk

Improv and coding

Scientists and Code reviews

Write Code Everyday. I talk about this a lot in regards to learning to code. This post breaks things down really well.

Saturday, April 5, 2014

Interesting Reads 3/22-4/4

You'd think with two weeks of reading I'd have more interesting things, but not so much that past two weeks. The reads below are really good but I've been a little more heads down recently and haven't had the chance to read as much. Anyway, there are some good, interesting posts below, so check them out and enjoy!

Doing Your Best Work:

The Last Developer was Terrible:

Old but still interesting:

A good first day: