Searching through logs

Recently, I needed to check out which endpoints in a specific version of our api were being hit/ if they were being hit at all. This involves searching through the paper trail logs, which was an interesting process so I thought I’d write about it.

First, I want to say, there is a better path than the one I took. It wasn’t until after I did all this log downloading and searching that I discovered the paper trail CLI. I haven’t had a chance to take a deep dive into it yet, but I assume using it is better than downloading 30 days of log files and than combining them into 1 log file.

Which is what I did first. I had to look at 30 days worth of logs, so I went to the paper trail heroku add-on and downloaded all 30 days worth of logs. Then I used cat *.tsv >merged.tsv to put all of the 30 files into 1 file. After that, I checked which endpoints I needed to look at by just looking at the routes file. Luckily, there are a pretty finite number of endpoints for the specific version I was looking at. I think, if I was dealing with significantly more endpoints, i’d need to figure out a better systems for ensuring that I’ve checked all the endpoints.

Then, for the simple endpoints (ie- /livestreams or /livestreams.json for the index endpoint or /livestreams/ for the show endpoints) I just ran a simple grep -c command which greps for that path and then counts it. For example cat FILE_PATH | grep -c /v1/livestreams.json.

When I got to a slightly more complex endpoint, for example, one that had a path like /v1/livestreams/something_unique_here/watched I couldn’t just grep for the endpoint, I needed to use a grep compatible regex to find the number of results that had “watched” at the end. I did this by searching cat FiLE_PATH | grep -c ‘/watched\b’ which looks only for the last bit of the url.

And that’s it. Pretty simple but I hadn’t handled searching through large log files or any sort of complex grepping before.

Arlington Ruby's Ruby Retrosession 2016

Arlington Ruby RetroSession… Retro ruby for short was this past weekend. I’ve been to retro ruby the past few years, with the exception of last year when I was in labor. It was so fantastic to be back at a community event. I love the unconf idea and feel like I always get a lot out of the conversations so it was great to be able to participate.

The sessions were as follows:

• Rspec
• Advanced Newbie (including mentorship, cs without a cs degree, importer syndrome, and ways to step up as an advanced newbie)
• Freelancing
• Everything APIs
• Functional web stacks
• productivity tools hack
• infrastructure as code

and

• making a safe space.

Since this was a 2-track conference, I only went to half of these. There was also ice cream, a retro at the end, and lightning talks.

In addition there was a separate newbie track. Notes and resources gathered from that track are here: https://gist.github.com/kalimar/1a23222b59fd0208f88d

So first I went to the advanced newbie session. We started by talking about mentorship and setting up both formal and informal mentorship opportunities. We never really got into ways to step up, but one CS subject that was mentioned as important for non-cs folks to know was Cryptography and a specific place to do this is through CryptoPals. Another suggestion related to learning cs fundamentals was by watching university courses. A list of these are available through the Github repo awesome courses.

Next I went to everything APIs. The conversation kicked off by asking what do you want the JSON return to look like and how that is a major decision what crafting an API… looking at the response types. I took some notes about things related to APIs that I didn’t know like how api versioning relates to 2 things, the schema and endpoints, where, I think, only the endpoint is usually considered.

There was a great discussion about good practices for external APIs which includes different strategies of putting docs into the actual api. And another discussion about how people like to create APIs… both grape and active model serializers were mentioned.

Some random tools mentioned and things to look up were: Swagger yard was also mentioned as a way to auto-generate documentation. Pact which writes tests in web mock style and then generates a JSON fixture file which gets shared with the micro service where the JSON is generating the response. It then uses a rake task, I think, to execute the tests. This goes along with pact broker which is an intermediate layer. Graphql is really strict about endpoints but generates great docs and is just 1 endpoint… you tell it what object or data structure you want and then it just gives you back only that stuff. Statsd helps you figure out what is still being used which helps with what endpoints are still needed or can be deprecated. I/O docs generates query explorers.

Finally, for API design, someone mentioned tasty pie which is big in the python community and would be great if something similar existed for the ruby community and HATEOAS which is a response call that tells you everything else you can do with your API.

After I bounced between productivity hacks and functional web stacks but spent most of my time in productivity hacks. There’s always great stuff out there. A full-ish list of what was discussed can be found in the hackpad.

The last session I attended was about infrastructure as code. This was really interesting to me because, I feel like one of the next steps for me is to learn more about devops and how deployments, etc. actually work, without the magic of Heroku. The session started out talking about what we think about when we think about infrastructure as code… things like Docker, Ansible, version control, Puppet, and Chef. It is humans changing code and code changing computers. The reason we do this is so that the infrastructure code is repeatable and can be replicated in the future. There are lots of challenges to infrastructure as code but the main thing mentioned was that if you don’t stay up to date on it, then there can be lots of security holes. Additionally, if you want to start learning about thing like Docker, etc. the barrier is pretty high. The tutorials are fairly difficult and since it’s all changing so quickly, lots of information, tutorials, and documentation is out of date.

Some tools mentioned: flynn.io which is a platform as a service. There’s an interface for Docker, push heroku-like commands, and buildpacks. Open shift by Red Hat. Teraform which is a command line utility that can go across services and configure them (i’m not sure this is the best explanation but this is at least a piece of what it does). Otto by Hashicorp which is an opinionated system which seems like it might be a good option for an off-the-shelf solution.

Some good resources to get started include AWS in plain english and twelve factor app which talks a lot about the decisions that need to be made and the things you should be thinking about when you’re looking at infrastructure stuff. Finally, it was suggested that to get started learning about this, to just create a VM on your machine and play around using something like vagrant. And remember, devops is stressful so find a mentor, find a support group, and practice #hugops often.

I didn’t take as many notes on the lightning talks. Someone gave a great talk about note-taking (which I obviously love doing) and suggested to take notes in markdown and to create a notes folder in every repo (or just in your projects folder) and add it to your .gitignore. I actually love this suggestion because recently, I find i’ve got over 10 notebooks filled with notes and without having the time to add color coded tabs to these notebooks, it’s getting more difficult to find things I KNOW I learned so I’ve been debating starting to take these notes in a more searchable fashion (but don’t really want to just have a gazillion notes in evernote) so this sounds like a great suggestion. Also, taking the notes in markdown would force me to become better at it which I think would be super beneficial in general. The other thing I wrote down was a github repo called Dug which provides a more transparent view into open source software and allows you to organize github notifications better than you can with just gmail filters. Also, there were a few talks that related to sleep but I didn’t take notes on any of those because I have an 11 month old. One day I’ll sleep again.

That’s all! Retro Ruby was awesome. It was so much fun to see people again, learn stuff, and have great conversations. Here's the Hackpad with notes from the whole conference.
Plus the t-shirts glow in the dark!!!!