We’re Bad at Interviewing Developers (and How to Fix It) – Interview with Kerri Miller

In this interview with Kerri Miller, Lead Software Engineer at LivingSocial, we discuss how to hire and interview developers. We typically don’t get trained on interviewing and we’ve all experienced the haphazard approaches of those new to it – poor organization, repeated questions, fizz-buzz… Kerri tells us how to run interview days, the types of questions to ask, how else we can evaluate candidates and what to do after the interview. For more tips, Kerri writes about software development and hiring on her blog.

Introduction

Derrick:
Kerri Miller is a lead software engineer at LivingSocial. She is also a RailsBridge instructor and frequent conference speaker. She talks about software development and hiring, including the talk, ‘We’re Bad at Interviewing and How to Fix It’. Kerri, thank you so much for taking the time to join us today. Do you have a bit more to share about yourself?

Kerri:
I am actually, in fact, a lead software engineer at LivingSocial. Part of that is working with junior developers, or more junior developers, leading software teams and projects, and I also do a fair bit of work in our engineering culture team, so doing things like how do we propagate a good culture for code reviews, post-mortems, and hiring.

“You want them leaving the interview process regretful that they didn’t get hired, not resentful that they didn’t get hired”

What’s Broken with Developer Hiring?
Derrick:
What do you think is broken with the current way a lot of companies hire and interview?

Kerri:
We don’t do a really good job of hiring with intent. We decide that we need more people, but we don’t do a really good job of figuring out what we need those people to actually do, and who we actually need to hire. I like to think of my software teams as little ecosystems, little, tiny arcologies that exist in a bottle. They’re not entirely a closed environment, and, like any ecosystem, anytime you introduce anything new to that realm, there will be changes. There will be impacts.

Any time you hire somebody, you’re changing that ecosystem. You’re introducing a new species or a new variable to things and it’s going to change. Thinking about what you want to change means that you have to have laid that groundwork to understand where you are at the moment. A lot of teams and companies don’t do a really great job of understanding that. They’re just simply, “We need more bodies. Let’s hire bodies.” They don’t go into these things with a conscious sense of where they are and what they need, and how the future’s going to change by adding more people.

How to Structure and Run Interview Day
Derrick:
Let’s talk about the interview day. How should we structure it, and what are some key aspects you need to get right?

Kerri:
You need to go into it having a plan, and that plan starts with knowing what questions you’re going to ask and why. Understanding that every question you ask that a candidate can’t answer, or every step of that process is an opportunity for a candidate to filter themselves out of that process, it’s a point for you to get information to make that final decision. I think it’s really important that you take a look at what that plan is going to be. If you have, say, three people, and you’re hiring for a front-end developer, you should have one person ask about JavaScript. You should have one person ask about, perhaps, browser interaction, or working with designers, or what have you. Just splitting up that interview so that you’re not asking the same questions over and over again, you’re really able to get really solid signal on a person’s skill sets, what they’re comfortable with, and what their concerns are. What kinds of decisions are they making?

Good Types of Questions
Derrick:
What are good kinds of questions that we should be asking?

Kerri:
Well, I’m not a big fan of whiteboarding, because I think that’s something that we just automatically do, and we don’t think about, “Well, what questions are we trying to answer by asking a candidate to solve a problem?” Are we dinging people for trivia questions, for not remembering, “Oh, I need this third option flag or an obscure method from a core library.” Instead, I really want to focus on questions that are asking about decisions that they’ve made, what choices have they made, and what choices would they make again in the future? Are they reflective about mistakes that they’ve made? Are candidates looking for opportunities to improve, and how do they actually go about it? Do they make plans for themselves, like how they would improve a certain skill set, whether that be a technical skill set or a more soft skill set, for example, management, or project shepherding for example. Those are the kinds of questions that I think really get you at the heart of not necessarily what somebody knows, but what they’re capable of.

Beyond the Interview – How Else to Evaluate Developers
Derrick:
You’re a proponent of evaluating candidates in other ways than just an interview. How else should we be finding out more about potential employees?

Kerri:
I’m a really big fan of pairing on projects, like actually working with somebody. It doesn’t have to be a formal or traditional pair programming situation with one computer and two people, talking through the technical choices that they would be making as they programmed on something. At LivingSocial, we do a code challenge like a lot of companies do, using that as, then, a launching pad to have a discussion with a candidate to say, “You solved the problem using this technique. Why didn’t you choose this other technique? Why did you choose this one? How would you do it better? What if we sat down and refactored?” That’s one really good way to really get the heart of why are they making the decisions they’ve made? Not just did they make this choice because they didn’t know, or are ignorant, or did they make this choice because they had a certain belief about what the requirements of a given project were? That’s one way to do it.

Other ways you can be finding out more about potential employees … I’m a really big fan of asking the employee to explain something to me or teach something to me. In the past, we’ve done this with simply just saying, “You can teach me anything, something that I don’t know, and preferably is non-technical.” How well do they communicate about something that they’re a local expert in but they’re intended audience is not? Could they then go off and go and learn a new framework, or go have a meeting with, perhaps, a stakeholder, or a client, and come back and explain what the actual requirements are to me, to distil down what I need to know and communicate that well? Communication is such a big part of what we do in this job, and so testing for that essential skill in a really clear and explicit way can be really useful and get you a really good signal about who that candidate is and how they’re going to fit into your organization.

“We don’t do a really good job of hiring with intent”

After the Interview – Making the Hiring Decision
Derrick:
After the interview, what are key things that employers should be doing?

Kerri:
I think it’s really important that we don’t just say, “We’re going to get back to you,” but to say, “We will get back to you by Thursday, end of the day.” Then, if you can’t make your decision within those three or four days, communicating that to the candidate so they have expectations that you can meet, because it’s not just good for the candidate, it’s good for you as a company to have that discipline, because you want people to, whether you hire someone or not, you want them leaving the interview process regretful that they didn’t get hired, not resentful that they didn’t get hired. Being professional and upfront and just friendly and encouraging about the entire process is great.

I try always to make sure that, if we can’t hire somebody for whatever reason, we make sure that we give them constructive advice or feedback afterwards, or at least make that available. If you did like somebody, if it came down to either Joe or Mary, and you hire one or the other, keep that person on file, and follow up with them in a few months to see how are they doing, what’s going on? “Hey, we have an open position, would you like to re-apply, or would you like us to consider you for that?” That gets into the part of how you keep metrics on things as well because if you didn’t hire somebody, figure out why you didn’t hire them and then follow up and see, are they actually doing that work, and did we hire the … Not necessarily the wrong person, but did our process let us down? If you assume that somebody didn’t know anything about, say, SQL, and now they’ve gone on to work on a SQL-heavy project, for example, what in our process missed that step?

“It’s really hard to look at who you hire and decide that you have a good or bad process. But you can look at who you don’t hire.”

Derrick:
Great, so we talked about having a plan as part of the hiring process, what’s a good process to follow to make a hiring decision?

Kerri:
When you split up the interview topics, the questions you’re going to ask, and you’re going to consistently ask all of your candidates, it feels a little bit like reading a script, but it really lets you compare apples to apples as much as possible. Once you’re done with your little section of the interview, you should immediately go back to your desk and not get back to work but write down what your impressions were. What were the pros and cons, the bullet points, and find something good about the candidate and something not-so-good about the candidate, something that you wish they did have? Doing that at that moment and passing that back to a central person so as not to … Don’t pass it back to a group, pass it back to a central person, whether that be an HR or the hiring manager, to collect that, so you’re not coloring the impressions of other people.

When you get back into that room with everybody else, whether it’s virtual or real, to really discuss your opinions, you’ve got your opinions of the moment and you can’t be swayed by the impressions of somebody else. For example, if you were supposed to interview them about JavaScript, and the senior JavaScript person, who’s got twenty years of experience in JavaScript, just really did not like that person, how would that color your opinion if you had to give your opinion in that moment? If you wrote it down previously, no, this person really is good at JavaScript, then you’ve captured that honestly and you can really give honest feedback about what that person’s qualities are and what their strengths are without being colored by other people in that discussion.

Measuring and Improving Your Hiring
Derrick:
You hinted at this earlier, but a key part of your approach to hiring is measuring the process to improve it. How could we go about measuring the effectiveness of our hiring?

Kerri:
It’s very seldom that we ever hire anybody bad. When you hear horror stories about hiring, it’s always somebody else’s team that hired that one jerk, or that one idiot, so it’s really hard to quantify because now we know that person, and we’ve worked with them, and we understand their strengths and their weaknesses. It’s really hard to look at who you hire and decide that you have a good or bad process. You can look at who you don’t hire. You can look at that in terms of what were the false negatives? Did we bounce this person out of the process for a specific reason and then it turns out that that reason wasn’t good based on where they ended up going to work?

It’s really easy to LinkedIn stalk people, and peak into their GitHub profiles if they’re doing that sort of work, to see what they’re doing a few months later. It can be really useful to, four, or five, six months down the road, go back and look at the candidates that you passed over and see what they’re doing to understand, if you keep records of the questions that you ask, and the reasons why you maybe didn’t hire somebody, to see if those reasons are still valid.

Other metrics that I think are really, really important to an organization are understanding what your pipeline for candidates consists of. At each step, you have a certain amount of leakage, because people just simply don’t make it through the process or they abandon the process, they disappear. How many people are you losing at each step, and is there one step that you’re losing a lot of people at? Maybe you need to refine that step, remove it, or move it earlier or later in the process based on what your organizational needs are. I think it’s also important to look at who you’re losing as well. Are you losing junior developers at a step that you really don’t want to be losing them at? Are you losing more diverse candidates? Are more women abandoning your process at a certain step than men are, and understanding, or questioning at least, your process to see, is that a problem? Can we fix it? How do we fix it?

“You should immediately go back to your desk, and not get back to work, but write down what your impressions were”

Common Developer Hiring Mistakes
Derrick:
What are some common mistakes you see companies making when hiring developers?

Kerri:
Some of the more common mistakes are hiring from our friend networks. I think that the friend network is such an important part of how we get jobs, but it also tends to reinforce our monocultures a little bit. We tend to be friends with people who are mostly like us, and so those are the people that we’re going to be recommending, and so those are the ones that get hired more often. When I was mentioning earlier how the team is an ecosystem, it’s important to have some diversity there, and not just the diversity we talk about in terms of gender or ethnicity or race, but age, class, looking at people’s technical backgrounds, do they come out of CS programs versus being a self-taught or a boot camp?

Industry backgrounds, did they work in, perhaps, consumer electronics testing before they became an SDET at Microsoft? Were they at startups versus large enterprise companies, or somewhere in between? All those pieces of diversity are going to be influential and improve the health of the ecosystem of your team, and so those friend networks are important for getting candidates in the door, but understanding that that sometimes is going to lead to a certain amount of self-selection for candidates.

You have to, like in soccer, they say, “Run to where the ball will be, rather than where the ball is.” If you have those early conversations about who you need to hire, and what you want to look for, what sort of energy and person do you want to add to your team, to influence it into a good direction? And then go to those people, find them, whether it be through meetups or user groups, or extending your extended network, not just your immediate friend network.

Derrick:
Are there any other resources you can recommend for those looking to improve how they hire developers?

Kerri:
Looking at the different boot camps you’re doing, and how they’re talking to their students, as well as to their sponsoring companies, or the companies that are hiring. I’m a really big proponent of hiring more junior developers, because no one is ever going to know our exact technology stack and our exact way of working, we always have to teach people, so looking at what those boot camps are doing, and how they’re talking about the industry, because they’re trying to set people up for success over the next five years. There’s a lot of wisdom. They’re spending a lot of time to gather wisdom that they can relate to us about who we should be hiring over the next five years, and what skills we should think are important.

Finally, I tell everybody this, go take a relationship skills class. Although they’re sold as being aimed at couples, a lot of that is really about listening to other people and understanding what their concerns are. Once you can start to build those sorts of skills for understanding the perspectives of other people, just generally improves everything about your hiring process, and your team, and how you work with each other.

Derrick:
Kerri, thank you so much for joining us today.

Kerri:
I’m really excited about this topic. I’m glad to see more and more people talking about it. There’s no one size fits all solution. We all face some really unique problems, but there are some commonalities.

9 Effective Code Review Tips

9 Code Review Tips

For everyone:

  • Review the right things, let tools do the rest

You don’t need to argue over code style and formatting issues. There are plenty of tools which can consistently highlight those matters. Ensuring that the code is correct, understandable and maintainable is what’s important. Sure, style and formatting form part of that but you should let the tool be the one to point out those things.

  • Everyone should code review

Some people are better at it than others. The more experienced may well spot more bugs, and that’s important. But what’s more crucial is maintaining a positive attitude to code review in general and that means avoiding any ‘Us vs. Them’ attitude or making code review burdensome for someone.

  • Review all code

No code is too short or too simple. If you review everything, then, nothing gets missed. What’s more, that makes it a part of the process, a habit and not an afterthought.

  • Adopt a positive attitude

This is just as important for reviewers as well as submitters. Code reviews are not the time to get all alpha and exert your coding prowess. Nor do you need to get defensive. Go into it with a positive attitude of constructive criticism and you can build trust around the process.

For reviewers:

  • Code review often and for short sessions

The effectiveness of your reviews decreases after about an hour. So putting off reviews and doing them in one almighty session doesn’t help anybody. Set aside time throughout the day including breaks not to disrupt your own flow and help form a habit. Your colleagues will thank you for it. Waiting can be frustrating and they can resolve issues quicker whilst the code is still fresh in their heads.

  • It’s OK to say “It’s all good”

Don’t get picky, you don’t have to find an issue in every review.

  • Use a checklist

Code review checklists ensure consistency – they make sure everyone is covering what’s important and avoid common mistakes.

For submitters:

  • Keep the code short

Beyond 200 lines, the effectiveness of a review drops significantly. By the time you’re at more than 400, they become almost pointless.

  • Provide context

Link to any related tickets or the spec. There are code review tools that can help with that. Provide short but useful commit messages and plenty of comments throughout your code. It’ll help the reviewer and you’ll get fewer issues coming back.

9 Integration Testing Do’s and Don’ts

Integration tests check whether your application works and presents properly to a customer. They seek to verify your performance, reliability and of course, functional requirements. Integration tests should be run against any of your developer, staging and production environments at any time.

Writing good tests proving your solution works can be challenging. Ensuring these tests to perform the intended actions and to exhibit the expected outcomes requires careful thinking. You should consider what you are testing and how to prove it works – both now and in the future. To help you create tests that work and are maintainable, here are 9 Do’s and 9 Don’ts to contemplate:

When Creating Integration Tests Do…

1. Consider the cost vs. benefit of each test

Should this be a unit test? How much time will it save to write this test over a manual test? Is it run often? If a test takes 30 seconds to run manually every few weeks, taking 12 hours to automate it may not be the best use of resources.

2. Use intention revealing test names

You should be able to figure out or at least get an idea of what a test is doing from the name.

3. Use your public API as much as possible

Otherwise, it’s just more endpoints and calls to maintain when application changes are made.

4. Create a new API when one isn’t available

Rather than relying on one of the Don’ts

5. Use the same UI as your customers

Or you might miss visual issues that your customers wouldn’t.

6. Use command line parameters for values that will change when tests are re-run

Some examples include items like site name, username, password etc.

7. Test using all the same steps your customers will perform

The closer your tests are to the real thing, the more valuable they’ll become.

8. Switch your system under test back to the original state

Or at least as close to it as you can. If you create a lot of things, try to delete them all.

9. Listen to your customers and support team

They will find ways to use your systems that you will never expect. Use this to your advantage in creating real-world beta tests.

When Creating Integration Tests Don’t…

1. Write an integration test when a unit test suffices

It’ll be extra effort for no benefit.

2. Use anything that a customer cannot use

Databases, web servers, system configurations are all off limits. If your customer can’t touch it, your tests have no business touching it either.

3. Access any part of the system directly

Shortcuts just reduce the quality of your tests.

4. Use constants in the body of your tests

If you must use constants, put them in a block at the top of your test file or a configuration file. There is nothing worse than having to search through all your source files because you changed a price from $199.95 to $199.99.

5. Create an internal-only API

Unless necessary for security or administration.

6. Create an internal only UI

You’re supposed to test what the customer will see after all.

7. Make your test too complex

No matter how brilliant your test is, keep it simple. Complexity just breaks later. If you are finding it hard to write, it will be hard to maintain too.

8. Test more than one thing

Stick to what you need to test. If you try to do too much in one test, it will just get more complex and more fragile.

9. Leave the test system in a bad/unknown state

This means a broken or unusable site, database or UI.

 

A Guide to Open-Sourcing Your Project at Work

Congratulations! You’ve written something at work that is amazing and you want to share it with the world! This guide covers three key areas that you should consider before making the leap: Why, when and how to do it.

Why Should I Open-Source My Work Project?

Open-sourcing your project at work can be a great idea. It can:

Help you build a developer-friendly brand

  • From those with a developer-focused product, like Stripe and Twilio, to those with APIs, like Facebook, Google and Square. Open-sourcing your code can be a good way to build your company’s relationship with developers.

Allow you to give back to the community

  • Just think of all the libraries and software you use on a daily basis that make use of open-source code. Adding your own is a good way of paying it forward so that others can benefit from your contribution. We’ve open sourced a number of libraries and even whole products.

Help you to recruit

  • Take Yahoo and LinkedIn for example. They’ve found that through their commitment to Open-Source projects (like Hadoop and Kafka), that they’ve been able to encourage developers to join them who otherwise might not have.

Gain more contributors than your project ever would have in-house

  • Like for example Square’s Dagger, a dependency injector for Android and Java. Having released it, many developers are contributing to it, including those at Google. In fact, Google developers have been contributing more than Square’s developers do themselves.

When Should I Open-Source My Work Project?

There are two conditions that you would want to meet before open-sourcing your project. You want to make sure that:

It won’t hurt your business

  • It may be an impressive, complicated bit of code that would be useful for other products beyond your own. Yet if that development is your secret sauce, then giving it away would be bad for the business. Likewise, if your library is an integral part of what makes your product unique or even what makes it possible, then you might want to keep it in-house.

Your code is helpful to others

  • Consider whether anyone else would actually want what you’ve created. Is it so uniquely tied to your workflow or infrastructure that it wouldn’t be useful for others? As a rule of thumb: if making it suitable for general consumption would make it less useful for yourself, then it’s probably not worth the effort.

Ok, so you’ve met those two requirements. Then let’s move to the mechanics of open-sourcing some code.

How Do I Open-Source My Work Project?

Step 1: Audit your code for security leaks

  • Chances are higher than you might like to admit that you or a colleague have left some passwords, usernames, IP addresses, machine names, personal contact information or other security hazards somewhere in your code. Keep in mind that this applies not only to your final master code but also to all the changesets you’ve had in the past.

For that reason, we recommend you do two things:

1. Make a brand-new repository
    • Chop off all the history of the code up to that point. There will be a new history and it saves you having to audit all the historical versions of your code. Plus, no one needs to know that it took you two weeks to wrap your head around C++11 lambda syntax.
2. Audit the code for security problems
    • This will take a lot less time than you think. Look especially at test suites and any places that are near connection points to other systems.

Step 2: Strip your code of profanity and immature pot-shots

  • While you’re in there, also rip out anything inappropriate that makes you sound more like a teenager than a professional. This doesn’t mean you can’t have any humor in your source code. But it does mean that jokes made at the expense of your competitor, a customer or the decrepit browser you’re forced to support might not be appropriate.
  • If in doubt, think about whether you’d feel comfortable reading your code loud to those beyond your team.

Step 3: Make sure your code adheres to best-practice naming and formatting

  • You’ll want your open-source code to be examples of your best work. Make sure you are using good, standardized naming conventions and formatting. Use tools like pyflakes/pep8, jslint, gofmt, ReSharper and others to help.
  • Also, keep in mind that if you’ve been wanting to do the One True Naming Standardization for your project, now’s a good time. Once you open-source your code, there will be a lot of inertia to avoid breaking changes. Get those done before you release. It’ll also make it easier for other contributors to get started with your code.

Step 4: Document it

  • You don’t have to write ninety pages of info docs but you should at least have a nice Markdown-formatted README.md in your root directory that explains what your software is, how to use it, and (if applicable) how to build it.
  • If you’re releasing a library, you should also make sure your code has docstrings/JavaDoc/whatever so that you can generate API documentation.

Step 5: License your code

  • You may want to get some proper legal advice on this. But before releasing your code, you should pick a license. Unless you have a compelling reason to do otherwise, the MIT license will probably suffice. It’s short, sweet, well-understood, liberal and makes integrating third-party changes back into your own products headache-free. But if you’re contributing to the code that you want to include in a project that already has its own license, you might want to use that license instead. Here’s a useful overview of license types for more info.
  • You’ll want to put a LICENSE file in your repository and have a copyright notice somewhere prominent — either in that file or in the README. Such as ‘(C) 20XX Your Name. All rights reserved.’

Step 6: Name your library or tool

  • Pick a name. Make sure it’s not offensive and avoids having the same name with other existing libraries and trademarked products.

Step 7: Push your code

  • Put it on GitHub, create your own organization, repository and push your code.
  • Keep in mind that some communities have secondary systems that you should consider utilizing as well. If you’re writing .NET, then another one might be Codeplex. If it’s Ubuntu-specific then a Bazaar mirror on Launchpad etc.

Step 8: Publish your package in the appropriate package archive

  • If you’re publishing a library, submit it to the appropriate package manager. For .NET, that would be NuGet; for Python, it’s PyPI; for Perl, it’s CPAN; for Ruby, it’s RubyGems; for Node, it’s NPM; and so on. Also, make sure that someone else at your company, such as a sysadmin, has the ability to continue maintaining the library under the unfortunate circumstance that you get hit by a bus.

Step 9: Announce your code

  • You’re all good, time to announce it! You’ll want to blog and tweet it out. You should also consider publishing on /programming on Reddit and Hacker News etc.

And that’s it! You’re all done!

…well, nearly.

Step 10: Don’t forget about your code

  • Just because you’ve published it doesn’t mean you’re done. You’ve unleashed a new-born into the world; you need to take care of it. Monitor pull requests and bug reports on your new project. If you realize that keeping your project going is overwhelming, then a hearty congratulations! You should remember that it is your responsibility to at least find an extra or substitute maintainer. It’s okay if your project ultimately forks but it’s best not to do so just because you dropped the ball incorporating freely and submitted improvements to your code.

That’s it. For real this time. So go out, contribute, and have fun!

 

How Low Should You Go? Level of Detail in Test Cases

It can be difficult to know just how much detail you should include in your test documentation and particularly in test cases.

Each case has a different set of needs and requirements in terms of purpose, usage, frequency and admin needs.

If it’s written at a too high level, then you’re leaving it open to too much interpretation and risking the accuracy of the testing. If it’s at a too low level, you’re just wasting your own time. It makes the maintenance more difficult and there’s an opportunity cost to other projects with demands on your time.

In this post, we break down some of the factors you should consider helping you find the right level.

Understand the Wider Context

Each of your project’s stakeholders will have concerns that will impact the amount of detail you need to provide. From your organization’s internal politics and appetite for risk to the extent to which the product is relied upon etc. This will provide a wider context for your test cases and start to improve your thinking. The documentation expectations at a lean startup may even differ greatly to that at some of the financial institutions.

Test Requirements and Resources

You need to provide enough information to describe the intent of the test case. This should clear all the elements that need to be tested. A special consideration should be given to any specific input values or a particular sequence of actions.

The amount of time you have to invest in the test case and the human or IT resources you have to enact the tests is obviously another key factor.

Know Your Audience

Also, consider the audience for each case. How technical are they? How much product knowledge do they have and how experienced at testing are they? More experienced testers who are familiar with the product will need fewer details but is the team likely to change in a foreseeable future? If so, then you might want to head off re-writes later by providing extra details now for those with less experience.

Some organizations have specific requirements to provide evidence of test coverage. Usually, it’s to show adherence for compliance to a standard or certification or for other legal issues.

Test and Product Considerations

Each test is different, from the importance of the test, to how long it will be in use for. If it’s likely to convert to an automated test script in the future, then including more details at that time might make it easier to do. There are similar considerations about the product you’re testing. Will the application be used in long-term? And are whereabouts in its lifecycle? The amount of change that you can expect for a recently built, an agile application is far greater than for some old system you’re maintaining. Unless it’s a wild, testless code beast that is.

There’s a Balance to be Found

These factors don’t necessarily mean you should include more detail but crucial and long-lasting tests justify the time if needed. However, there’s a balance to be sought. If you create highly specific tests, then even minor design changes or functionality alterations may mean you have to re-write the cases. They also lead testers to raise bugs for what end up creating problems with the test documentation, rather than impacting customers. They can have a knock-on effect too. They encourage the tester to only consider the specific paths through the application detailed in the case. Meaning they might not consider the functionality from a broader perspective.

There’s no silver bullet for coming to a conclusion, each organization’s requirements differ. And these requirements change depending on the project, product and individual tests. However, considering the factors above, you can find a level that works for you and your team.

Taming a Wild, Testless Code Beast — 4 Steps To Improve Test Coverage

Whether you’re working on an existing or new application, you’ll often find yourself playing catch up when it comes to tests. Soon deploying code changes feels like poking at some ugly, sleeping code monster — you aren’t sure what’s going to happen, but you know it won’t be good.

Here are the 4 things you should do first to tame the beast and improve test coverage:

1. Add the Right Tests

Start by adding tests in the areas where it is easiest. It’s important to consider the order in which you do this to make sure you get the most out of your scarce resources. You want to start adding tests in the following order:

  • i. Create tests as you fix bugs

Add tests to prove that your specific fix is working and keep them running to show that this does not break again. This prevents from being somewhat targeted — you are creating tests in your weakest areas first. The weaker it is (i.e. more bugs) the faster you will build up tests.

  • ii. Create tests with all new features

All new features will need to include tests created to prove that the feature works as expected. If you’re covering the new aspects of your application, then at least things aren’t getting worse.

  • iii. Create tests as you add to old features

When updating old features, add tests as you go to show that the older functionality is not breaking in unexpected ways.

  • iv. Create tests in identified risk areas

Talk to the developers and testers on your team and ask them to point out any weak spots or issues they have experienced. Also, you talk to your support team — they are an excellent resource with a direct line to the customer. They’ll know the features of your product that frequently causes issues.

2. Turn on Code Coverage

Code coverage is a tool included in most continuous integration systems (or one that can be added with a plugin). This tool will instrument and monitor your code as you run the tests to determine how much of your code used by the tests. For this to be useful, follow these steps:

  • Start running code coverage against all your code
  • Get a baseline

Find out what the tool can see, where you are currently at etc.

  • Determine areas that you want to exclude.

There are likely areas of your code that you don’t want to cover — third-party tools, ancient code untouched for years etc.

  • Determine coverage goals

Sit down with your team and discuss what your current coverage is and what your ideal can realistically be (usually 90% or above).

  • Work-out steps to improve your coverage

You aren’t going to fix this problem overnight. Put in place some specific tasks which are going to help you achieve your goals over time.

  • Determine your pass/fail criteria

Is staying the same OK, or should it always go up? Do you define any drop as a fail?

  • Run Code Coverage constantly

Use automation to run your coverage test and use the criteria you determined as a team to report a pass/fail and do this constantly. It is a lot easier to add tests when the code is still front and center in your mind than later on.

3. Run your Tests on a Scheduled Basis

You should run your tests regularly, on several schedules:

  • Run them on every check-in

Use CI tools like Jenkins to run (at least) your unit tests on every check-in. Run them simultaneously if they are taking too long to run at this frequency.

  • Run them on every build (package)

Depending on how your systems work, your CI infrastructure can help you with this. This could be on every check-in if you are on Continuous Deployment, or every day/week/month that you use. This should be a clean install on a test environment and a full run of all your automated tests.

  • Run them on every deploy

You should run all your automated tests against your environments immediately after a deploy.

  • Run them every X days/hours/minutes

Run your automation suite against your constant environments as often as you can (Production, Staging etc). This should be at least a “once a day task” and take place during ‘off-peak’ times when it does not interrupt others too much. You can increase the frequency further if your tests are short, just be mindful not to overload the system.

4. Provide a Button to Run the Tests

Again, use a tool like Jenkins to turn test runs into a self-service operation. A developer should not be delayed by asking a QA person to run a test for them. Get a system in place where your tests will run and just give them a button to press. Remove as many barriers for everyone to run the tests as possible.

If you follow these steps, over time, you’ll see that you are able to turn an unwieldy application into something more manageable. First, by adding tests to the key areas, then making things as easier as possible, you can build confidence around your code changes and deploys.

Stop More Bugs With This Code Review Checklist!

Checklists are a great tool in code reviews — they ensure that reviews are consistently performed throughout your team. They’re also a handy way to ensure that common issues are identified and resolved.

Research by the Software Engineering Institute suggests that programmers make 15–20 common mistakes. So by adding such mistakes to a checklist, you can make sure that you spot them whenever they occur and help drive them out over time.

To get you started with a checklist, here’s a list of typical items:

Code Review Checklist

General

  • Does the code work? Does it perform its intended function, the logic is correct etc.
  • Is all the code easily understood?
  • Does it conform to your agreed coding conventions? These will usually cover the location of braces, variable and function names, line length, indentations, formatting, and comments.
  • Is there any redundant or duplicate code?
  • Is the code as modular as possible?
  • Can any global variables be replaced?
  • Is there any commented out code?
  • Do loops have a set length and correct termination conditions?
  • Can any of the code be replaced with library functions?
  • Can any logging or debugging code be removed?

Security

  • Are all data inputs checked (for the correct type, length, format, and range) and encoded?
  • Where third-party utilities are used, are returning errors being caught?
  • Are output values checked and encoded?
  • Are invalid parameter values handled?

Documentation

  • Do comments exist and describe the intent of the code?
  • Are all functions commented?
  • Is any unusual behavior or edge-case handling described?
  • Is the use and function of third-party libraries documented?
  • Are data structures and units of measurement explained?
  • Is there any incomplete code? If so, should it be removed or flagged with a suitable marker like ‘TODO’?

Testing

  • Is the code testable? i.e. don’t add too many or hide dependencies, unable to initialize objects, test frameworks can use methods etc.
  • Do tests exist and are they comprehensive? i.e. has at least your agreed on code coverage.
  • Do unit tests actually test that the code is performing the intended functionality?
  • Are arrays checked for ‘out-of-bound’ errors?
  • Could any test code be replaced with the use of an existing API?

You’ll also want to add to this checklist any language-specific issues that can cause problems.

The checklist is deliberately not exhaustive of all issues that can arise. You don’t want a checklist which is so long no-one ever uses it. It’s better to just cover the common issues.

Optimize Your Checklist

Using the checklist as a starting point, you should optimize it for your specific use-case. A great way to do this is to get your team to note the issues that arise during code reviews for a short period of time. With this data, you’ll be able to identify your team’s common mistakes, which you can then build into a custom checklist. Make sure to remove any items that don’t come up (you may wish to keep rarely occurring, yet critical items such as security-related issues).

Get Buy-in and Keep It Up To Date

As a general rule, any items on the checklist should be specific and if possible, something you can make a binary decision about. This helps to avoid inconsistency in judgments. It is also a good idea to share the list with your team and get their approval on the content. Make sure to review the checklist periodically too, to check that each item is still relevant.

Armed with a great checklist, you can raise the number of defects you detect during code reviews. This will help you to drive up coding standards and avoid inconsistent code review quality.

What A Great Software Does?

Great software helps you out when you misunderstand it. If you try to drag a file to a button in the taskbar, Windows pops up a message that says, essentially, “You can’t do that!” but then it goes on to tell you how you can accomplish what you’re obviously trying to do (try it!)

Great software pops up messages that show that the designers have thought about the problem you’re working on, probably more than you have. In FogBugz, for example, if you try to reply to an email message but someone else tries to reply to that same email at the same time, you get a warning and your response is not sent until you can check out what’s going on.

Great software works the way everybody expects it to. What great software has in common is being deeply debugged and the only way to get software that’s deeply debugged is to keep track of your bugs.

A bug tracking database is not just a memory aid or a scheduling tool. It doesn’t make it easier to produce great software, it makes it possible to create great software.

With bug tracking, every idea gets into the system. Every flaw gets into the system. Every tester’s possible misinterpretation of the user interface gets into the system. Every possible improvement that anybody thinks about gets into the system.

Bug tracking software captures the cosmic rays that cause the genetic mutations that make your software evolve into something superior.

And as you constantly evaluate, reprioritize, triage, punt, and assign these flaws, the software evolves, it gets better and better. It learns to deal with more weird situations, more misunderstanding users and more scenarios.

That’s when something magical happens and your software becomes better than just the sum of its features. Suddenly it becomes reliable. Reliable, meaning, it never screws up. It never makes its users angry. It never makes its customers wish they had purchased something else.

Getting Started with Iteration Planner for Agile and Sprint Planning

Iteration Planner unlocks the power of the project management software as an Agile planning tool for software development. With it, you can combine sprints and milestones to graphically group cases into the scope of work that you’ll complete in each sprint. You can also balance the allocation of resources by dragging and dropping cases from one assignee to another. Iteration Planner is a lightweight way to plan work and manage teams using FogBugz.

Here are a few key suggestions to help you apply priorities in your projects:

  • 1–3 Minimum Viable Product
  • These are all of the cases that must be completed in order for your sprint to provide new value to the customer in the form of a usable feature.
  • Priorities 4: Hygiene
  • This is non-essential work that improves the product but is not required in order for the product or feature to be useful to a customer. Small amounts of this work are often spread throughout sprints to continue making small improvements to the product.
  • Priority 5: Incidental work
  • Don’t allow your scope to creep from a manageable size to something so huge that it couldn’t be completed in less than a decade by Steve Jobs and a host of efficiency experts. If you add something as incidental work that needs to be done during a sprint you need to subtract or deprioritize some other work that has an equivalent number of estimated hours. Keeping all your incidental cases under one priority allows you to group cases by priority and see how much time you are spending on cases that are not part of your MVP. At the end of your sprint add up the hours and determine whether or not the interrupts were a worthwhile use of your time. This discussion can be part of your retrospectives or done separately. Either way, the numbers should be useful for evaluating whether or not the work that was done during the sprint was part of the planned work.
  • Priority 6: Long-range work
  • Frequently teams are asked to take on a category of work that is not part of the MVP but does contribute to the overall plans of the organization. These tasks may be expected to take many sprints to complete and their progress needs to be tracked across multiple milestones.
  • Priority 7: Stuff we won’t do
  • It’s useful to declare this as a way of focusing the team and managing expectations about the work that will be completed in the sprint.

For Agile software development teams, Iteration Planner is a useful, graphical tool that allows you to visually manipulate the information you need for planning Sprints. With it, you can set the direction for your team’s work and monitor progress so you can deliver on your organization’s goals.

How to keep track of cases in multiple projects at once?

Life is complicated and we all accept it. While we tend to categorize the different aspects of our lives, in reality, things rarely fit neatly inside the little boxes we carve out for them.

Our work life is another major compartment and when it comes to planning, it’d be nice if a single project captured all the details we might need to organize work for a project or product and across teams, but often that’s not possible.

This meant at times, working with the Iteration Planner and Kanban boards in FogBugz could be kind of awkward. What if you maintain multiple products and each one has its own project? You’d want the “Planner” to show both at the same time, but it couldn’t. Or, what if several teams had multiple projects relating to a single product? You couldn’t plan without affecting the other teams.

Well, such things are not a problem anymore! With the cross-project planning capabilities in Iteration Planner and Kanban, you can keep track of cases in multiple projects at once.

That means you can view all cases relating to a product on one board, even if they’re in separate projects. And multiple teams can plan things without disrupting or being interrupted by others.

Here’s how it works

Site admins can create a new planner and associate it with one or more projects. Each planner may contain any global milestones and any per-project milestones for its projects. You can also optionally filter by project or area, and filtering down to multiple projects is allowed too.

For existing customers, each of their single-project planners migrated to the new versions. If you would like to add additional projects to a planner, view that planner and click “Edit Planner Settings” in the top-right, then “Add Another Project”, select additional projects and click “Save”. With that done, you can add milestones from those projects and any global milestones, and the filter columns shown will include cases from your selected list of projects.

If you use our Kanban board, the project selections for your planner also apply to each milestone when you click through to the Kanban view.

Take a look at our help site for more details. If you have any questions or feedback, please get in touch!