Tester Interview Questions

There’s a lot more to hiring than the interview questions but recently I came across someone looking for a list so I thought I would share my own.

The context of these questions is an interview for a mobile-focused, agile-ish, exploratory test role in an agency environment.

Please use as you see fit.

Style/Opinion/Purview

  • Do you have a smartphone, what’s your favorite app? How would you test it?
  • Tell me about your favorite bug that you found and reported.
  • What are the benefits of an exploratory approach to software testing? Downsides?
  • Where do test cases fit into your typical project lifecycle in an agile project in the client services world?
  • Where does design fit into your test coverage strategy? How do you work with designers?
  • Do you have any view on QA versus QC versus Test nomenclature?
  • Oh, you found a bug! What is the ideal lifecycle of a bug you just observed?
  • What makes a good tester? Are you a good tester? What’s the biggest challenge in being a tester?
  • When you hire testers, what do you look for? Why?
  • Why do we test? Why doesn’t Dev just do that? Or the client? What is the value?
  • How should the development team handle test coverage? How much Unit/etc testing coverage do you expect from the dev team?
  • How do you know when you are Done testing a feature? A pull request? A project?
  • Do you prefer more bugs filed or better bugs filed? why?
  • What is testability? How do you advocate for testability on a project?
  • If the PM expects you to “test the build” before it goes out to the client every day, what do you do with it?
  • What kinds of technical debt does a software tester incur? How would you refactor your testing efforts or output?

Communication/Collaboration

  • How do you communicate your testing progress over the course of a project?
  • Tell me about collaboration with other testers. When/why/how.
  • You’re filing bugs that aren’t making it into the sprint, at the next sprint planning meeting, they don’t get brought in from the backlog, what do you do?
  • Does your team/PM/etc know what your role is/what you do? How? Problems?
  • What kinds of documentation would you produce as a tester?
  • Your Project manager wants you to create a test plan but it’s not a deliverable, what do you do?
  • Bugs/enhancements you encounter but know are out of scope, do file them anyway?
  • How do you assist in managing scope creep? What if your cases get triaged into a sprint but you know they are out of scope?
  • A dev is asking you to write unit or UI tests, how do you respond?

Tenacious/Problem Solving

  • When there is nothing to test at the beginning of a project, what do you spend your time on?
  • How do you test your own testing efforts?
  • Project is overtime and budget, you’ve got 40 resolved cases to validate, your own backlog of things you want to look at, the client is filing about 25 new bugs a day, they want to ship the app Friday, it is Wednesday, you are asked to test an RC candidate, what do you do? What’s your first move? And then?
  • Can you tell me about a particularly toxic/buggy/troubled/deathmarch project that you helped turn around in some way?
  • A majority number of bugs you file come back to you after a fix and you have found the changes to be be mostly insufficient, what do you do? How? Why?
  • In reviewing mockups/designs they do not include UX/motion/interaction or indications about behavior, how do you figure out the intended behavior?
  • How do you know when you are in over your head and need assistance? Who would you ask (test dept or manager or dev team or ?)
  • How would you handle lack of sufficient test devices?
  • How do you test a mobile app that’s API is in active development also?
  • How empowered do you feel/are you to change and address problems as they arise?
  • Your PM says VoiceOver support is out of scope but VO is part of the company’s internal Quality Standards, what do you do?

Devious/Inquisitive/Curious/Self Motivated

  • When you researched our company, what did you find?
  • Why are you a tester? Why aren’t you a designer or developer?
  • What kinds of goals do you set for yourself as a tester?
  • You’re told to focus on crashers only, no feature/functional/etc testing, what do you do/test?
  • How do you handle usability issues that fit requirements but you find troublesome?
  • How do you know what to test and when?
  • Where do your test ideas come from?
  • Have you heard the phrase “break software” how do you feel about that? Is that what you do?
  • What are good test interview questions that I have not asked you but should?
  • What’s your favorite thing to try when you first encounter an app/site/sytem under test?

Passionate/Learning/Leading/Growing

  • What testing courses, trainings, seminars have you been to?
  • How do you stay current with development and platform technologies?
  • How do you share your knowledge with the test community at large?
  • Are you on any well known beta groups or open source projects contributing in any way?
  • Do you think you would be more of a leader coaching other team members or more in a learning position seeking perhaps a mentor?
  • How would you contribute to the test org at large? Is team building important to you?
  • Many people do testing as a stepping stone elsewhere (PM, design, development, management, etc) is testing that for you? If so, where are you headed. How can you meet your goals while also functioning well in your current role?

Technical/Experienced/Platform

  • How can you ensure your testing meets what a customer would expect? How do you keep the end user in mind?
  • What do you really think of agile/scrum/kanban? Is it important to you?
  • Can you speak to your expectations as a tester from the IT/DevOps/Operations dept/people?
  • How do you manage state modeling in your testing?
  • Do you have any experience in the build/deploy/release cycles? Interest?
  • Your PM tells you to go through the backlog and add testing hours to each feature case on file, how do you approach this?
  • Are you comfortable hopping around Pull Request and feature/fix branches in Git?
  • How do you contrive test data? How do you manage it/etc. Are you comfortable fiddling with JSON and serving it to the app yourself?
  • Where does combinatorial testing fit into testing of a mobile web app?
  • Tell me about security bugs you look for or have found.
  • In what context would you use Git bisect in your testing efforts?
  • Does Swift/Kotlin/Node interest you at all? Is there anything in the new Android Studio/Xcode/etc that excites you?
  • How do you test localization, internationalization?
  • Do you prefer Native, Web or Hybrid? Have you done desktop testing? Any windows experience?
  • What is your experience with accessibility testing?
1,197 Words

Get To Know Your Friendly Local Developer

Befriend and bemuse them with your uncanny ability to keep them on their toes.

It should go without saying, but let’s do it anyway in case you’re new here. Your team’s software tester should be generally studied on UX principles, IT basics, platform specifics, project management, development/test tools, and general software engineering. These are the main ancillary areas of “QA” knowledge for any software tester. A solid tester will keep these domain areas in mind while aiming for Expert level in general QA practices. This is pretty typical scope for high functioning testers who will hopefully have an additional specialization of some sort to offer as well. This might sound like a lot but it isn’t. I have worked with many testers functioning at this level and each of them have their area of expertise that compliments their testing skills which are honed by way of design and development knowledge.

There is another aspect that is incredibly valuable but not talked about much in regards to working with an embedded tester on your development team. It’s a combination of observance and collaboration: fostering and honing the interaction between your developers and your testers. Sometimes this looks like pairing and sometimes it’s simply empathetic perception. This stratagem is one of the things you lose when you disconnect your QA from your development team. Don’t do that. If QA is an add-on or optional “evil” to you or your team, you’re simply not functioning on the level I’m talking about. You have someone like me on your team, don’t you? Get it together, man.

What happens in both instances is the most virtuous of circles where the tester has to get better at finding bugs because the developer keeps learning their own blind spots by way of QA and covering the corner cases more thoroughly.

This can happen directly which is the crème de la crème of testing in my humble opinion but we’re not always that lucky. We either pair or run a close facsimile thereof.

Tester, Pair with your Developer

If your branching and merging policy requires feature work to be tested before merging to a main branch, this state is perfect for a tester and developer to sit down and review the work before it’s unleashed on the team.

Review what the feature does, what’s expected, what issues the developer is concerned about. Test cases that must pass and other basic functionality. Go through the Happy Path together. Your Tester should write out an exploratory test charter with their basic plan of attack and things like Negative Personas, Worst Case Scenarios and other quicktests, heuristics and tools of the trade. This should be reviewed with the developer and serve as the other volley to their initial introduction of their work. Now you’re both on the same page of each other’s books.

Time for testing. Before you dig in, figure out how the dev wants to get the info. This could be anything from a Google hangout, a text-dump of the exploratory charter, to a play-by-play in a chat conversation. I’ve found a play by play to be pretty effective as many things could be addressed in the time it took me to find a new thing. This meant we could find and fix a large number of issues on the fly. After your session, let the dust settle and file some bugs. You should have enough ammo to inform further tests or pass along test-cases and update your regression checklist.

Pairing in this way is probably my favorite thing to do and honestly when I’ve felt like I’ve functioned at my highest level. It’s very raw and terrifying.

Why is it scary? Well, the role of QA is in part to hold developers (and designers!) to their own muster. We’re usually the scrutinizers and when you’re doing this one-on-one you’re showing exactly what you are made of. The tables are turned (play nice!). Don’t get too scared though, this is an excellent learning opportunity as your developer will probably mention things they hadn’t yet thought of that were inspired by something you brought up. This is the virtuous circle shimmering.

If you need to convince your developer or QA to afford time for this, simply do a first session and neither of you will be questioning its value in the future. I promise. Well, unless one or both of you suck then I can’t help you but perhaps one of you will help the other until you figure it out.

Now, Without Bothering Your Developer

Testers establish oracles. A lot of times these are things like competing products, PRDs, Apple’s HIG. There is another fleshy oracle and that’s your friendly local developer. Yeah, I know that seems a little creepy. It’s not. When you’re not pairing with them they and their work are still there and can be a passive resource to inform your testing.

Read each commit and the code along with it. If you’re allergic to code, no you’re not, grow up. You don’t have to fully understand each line. You don’t have to know what those APIs do. No one’s going to ask you about subclassing or pointers. Just get a picture of the basic idea and what classes and parts and features the new work touched. Between this and the comments you should have something to work with. Compare again with the bug tracker (or kanban board or user story or whatever) to further confirm your approach for testing this particular thing.

But before that, there was probably some discussion, in the chatroom, in a status meeting, at a whiteboarding session. You, dear tester, should be monitoring all those channels as well. Not only monitoring but actively participating as much as you can.

Even deeper than reviewing someone’s commits is just being empathetic to where they are as a human and worker. Is the PM asking them to do too much fiddling and it’s taking away from their available time to code? Maybe their partner is sick. Maybe they are just coming on or off another project. Getting a temperature read of the pressures on everyone might not spell out explicit test cases but it can show you where and whom to focus on or back off of. Try to be of assistance and sensitive to your team’s needs.

You’ll be finding and filing bugs all the while. The developer will start to see what you find. Over time you will both see patterns. You will know your team members quirks and idiosyncrasies. And they’ll know yours. You’ll be attuned to their thought processes and should be able to find their ever-shrinking weak spots. A software tester gorges on information, their synthesis skills must be top notch. Put that into play outside of the actual application with our empathetic selves and everything and everyone gets better.

1,184 Words

The Tester on Your Team Does Not Assure Quality

Having a tester on your team is no assurance of a Quality end product. That carrot dangles in front of all of our faces. It is a shared burden and responsibility by the whole team. Your Tester is simply highly trained in hunting bugs and keeping watch for potential issues that may lead to a weaker product.

The tester doesn’t believe in “bug free” software and will never tell you the system under test has no bugs. An experienced Tester functions under the belief that the number of bugs in a product is infinite and that their job is to get the big bad ones exposed as rapidly as possible. They will report defects and show you things they tried in order to expose anomalies and will make every effort to find the bugs that matter expeditiously.

They will, however, assure you that they will put their best effort into finding issues and voicing concerns with the goal of mitigating risk and aiming for the highest quality reasonably achievable.

They are not magicians, software testing is a trade. Putting one on your project won’t magically make everything better. They show up and promise to do the best work they can. They don’t outright eschew or blindly embrace any particular form of testing. They take them all and decide which ones are appropriate for the project/feature/situation or not. Sometimes that is, in fact, verbose test cases (but those are sparing to tell you the truth) and sometimes it’s a well crafted Exploratory Charter. They might end up writing some lean test cases (those can be written in one line) or creating some MindMaps to help flesh out their approach. Their test plans will lay out the intentions and ideas and strategies but writing down every single thing they might do would be wasting everyone’s time and therefore money. They find that idea repulsive. Wasted time is not an option when you are on a deadline, a budget, and are facing down an incalculable number of defects. A Tester will document what makes sense to keep themselves organized and their team informed, but anything more rubs against their professional integrity.

In order for them to do a good job, they need to spend a lot of time learning about the product, its purpose, its technologies. This is one area where the ol’ Agile “working software over comprehensive documentation” is crystalline. A Tester needs to pool together every oracle, data point, and resource available. Once they have a foothold on understanding the large picture, they can dive in and start finding issues and getting a sense of the risks. The more a Tester learns, the deeper they can test. The more things they figure out that there are, the better a Tester can ensure they’ve made a reasonable effort to cover them.

They will not wait for permission or guidance, a good tester will dive in head first and start exploring. The developers probably started implementing features that weren’t quite fleshed out to begin with (look at literally any PRD on the planet). So many bugs come from this lack of clarity. The Tester will turn that information firehose on and monitor it continuously; probably by watching commit logs in the repository or User Stories in the bug-tracker, likely both. They appreciate and will absorb any information you can give them but are comfortable in ambiguity and will self direct their time and attention diligently. The tester will compose the best bug report they can in the shortest amount of time that they can. They know that sometimes they will not be able to reproduce the anomaly but will report it nonetheless, even if lack of details is to everyone’s shared chagrin.

They are more investigative journalists and archaeologists than button mashers. They study software testing specifically. They can’t assure quality. They get good at what they do and they report their findings and provide feedback but it’s up to others to take that information and act on it. All the boundary analysis and combinatorial testing, all the memory profiling and fuzzing, all the bug reporting in the world, any of it, all of it, it’s all noise if it’s not acted upon.

If you want your product to be of the quality that we’re all aiming toward it, get someone on your team who is trained at it. Process and act on what they find. Vow as a team to only put well-scrutinized bits and bytes out in the world.

763 Words

Making Accessibility a Team Standard

As a UX-focused software tester who is passionate about the usability of software, accessibility has been of particular interest to me. People accept technology into their life because of an unspoken promise. They want a better life and hope that accepting some bit of software into their routine will give that to them. Software is for people and that shouldn’t just be people exactly like the ones who built it.

When I talk about accessibility, I refer to a broad spectrum of the human condition where we deviate from what is commonly accepted as standard issue. Not everyone has eyeballs or has arms or can read or can speak or can hear. So many people have some sort of trait that would be considered a disability, including ones not governmentally classified as such including common learning and reading issues. From paraplegics to dyslexics. From the deaf to the hard of hearing. We are different.

Attending the Accessibility focused talk is something I have unwaveringly done at every conference I’ve been to, from WWDC to MobileUXCamp, Xcoders, and well beyond. I’ve also delved deep into resources and articles out there focusing on various aspects of the challenge of accessibility support in software.

A recurring issue I hear from people on the ground floor who care, and want to move toward more accessible software, is concern about how they can convince others in their department or team to allow for it. They need help getting Buy In. They want approval and permission.

That’s the wrong approach. You’re not going to ask for permission or later, forgiveness. You’re just going to do it because it’s the right thing to do. You never need permission to do your job.

The Design Team

Have your UX team make reusable Personas out of variously abled folks. This is a good way to keep designers in the Universal Design mindset.

Find ways of including people with varying abilities in your Usability Testing design phase. Many colleges have centers that you can reach out to, to help connect you with people willing to participate and bring their unique perspective. If your Design department starts solving problems that include everyone, you are baking in support from the get-go that will flow through the entire project.

The Engineering Team

Your programmers want to do that whole “surprise and delight” thing even if they groan at the phrase. They want their work to shine and be usable by all. They want to participate in design when and where they can, and they want to implement the intent of the Design just as beautifully and deeply as the UX team does. They also love puzzles and challenges, and sometimes implementing support for accessible features is just that.

They don’t see themselves simply as implementors (even if sometimes their managers do) and that means they stand by what they produce, to put out something not fully usable into the world makes them itchy at best. They want their work to last.

Encourage the team to seek out experts and deputize someone to be your team’s expert. Enable that person to delve deeply into implementation and be a resource for the team.

When conducting code reviews and merging in those feature branches, check that everyone is on the same page, headed toward the same goal. Likewise, if you notice a tester hasn’t covered something or a designer hasn’t accounted for a use case, raise that flag high.

The Test Team

Ensure your testers have heuristics and test cases that address your applications functionality and usability when using available assistive settings. They should be using state models for each applicable setting. Yes, each one. Your test coverage should be multi-pronged, and you should be able to ask external testers to focus on areas of coverage during Beta testing. Have your QA folk gather the ideation cruft from Design to help inform test coverage. If you’re on iOS having accessible UI elements means your project can delve into UIAutomation too.

Your Stakeholders

The attention and care you give toward accessibility must be so embedded into your design, development and test structures that to have it as a SOW line item would be laughable. Try telling your programmers that to cut down on time & budget they can no longer put comments on their code and see what happens. If you survive the tidal wave of laughter, sobbing and eye rolling, perhaps a compassionate soul will explain to you that it’s an ingrained and compulsory part of their trade and practice.

Creating software for humans with their various variants in mind is simply part of the art of making great software. If you are called to defend your accessibility support to stakeholders, gently explain to them that their software is meant for customers and 100% of them are human beings, unless your stakeholders have commissioned a shell script to live on a server or something. To press on further point them to ADA and 508 and various lawsuits and class action settlements out there. More attention paid from the design, development and test teams is far cheaper than a lawsuit, let alone lawyers fees.

Consider Your Users

A typical user of your application might not think of themselves as needing an assistance. Consider your user however. How deeply does a user pay attention to your application? Have you noticed people with their chin to their chest and a device in hand wandering around your city or town? I know, it seems like they aren’t paying attention to their surroundings but they are. They are apportioning their attention to an application, their immediate surroundings and whatever is floating through their beautiful mind.

Sometimes this means they will use VoiceOver to help them read an ebook, sometimes it’s using Zoom so they can better see a website. Maybe they have a headache and are using inverted colors to ease the pain. Maybe they just like a large or bold text better aesthetically. Maybe they are trying to get better at another language and using VoiceOver better enforces their usage. Consider what your eyesight was like when you were 15, then 35? People decline. All of us.

I promise you that these features are in use by people you wouldn’t normally think of when implementing support specifically for “accessibility”.

Consider your Users who require assistive features

These users don’t need convincing now do they? This is their daily life and they want to accept the promise too. Putting in the time and attention for assistive features is non-negotiable to those who require it.

There are communities out there like AppleVis who are largely dedicated to weeding out the muck and shining a light on accessibility-friendly applications. How many simply bounce your app directly into the bin if they can’t achieve the basic functionality? How many simply do not tolerate anything less than a stellar experience? We don’t know. Why not just stop putting them in that position?

Can you imagine buying a product to solve a problem and when you tried to use it you figured out it could only be used if you didn’t have sight, hearing, vision, etc? Ridiculous. Why wouldn’t you take the opportunity to include communities who have been largely ignored?

The gains here are incalculable and unknowable.

The rest of us

May 15th was Global Accessibility Awareness Day and you missed it. That’s OK. There are things you can do outside of your project or company to help raise awareness and help in general.

When you see an Accessibility or Universal Design talk at a conference, attend and ask your colleagues to as well. Most are full of attendees who need help and you’re an expert they can confer with and the presenter usually will have some keen insights you may have overlooked.

Another way to show you care and demonstrate that these issues are important is to give direct feedback to developers of apps you use. File a bug report or enhancement request, show a use case and the business value. Let them know this is important to you as a user of the app, regardless if you yourself are “disabled” or not.

If you are in a position of leadership in your group, allow your team members to use you as a resource for guidance and implementation. Show them how by a job well done.

If empathy is what drives you to create useful bits of technology, craft your software for all, and champion others to do so as well. Be a promise keeper.

1,440 Words

Reporting Bugs My Way

As a fervent exploratory tester, a single test session may produce upwards of 40 issues I’d like to investigate further or report in some way. Turning a 90 minute QA flâneurship into 40 stellar bug reports doesn’t scale so well. So how does one move from a bug dump to flawless compelling bug reports? Well that is a tough nut to crack. Mostly we just want to give our paired developer a rough list, and let them cherry pick the ones they don’t need further info on, as they tend to be living and breathing the project. Let them skim the crème of the crap, as it were. Then let QA file the rest, as god and project managers intended. The pairing bit, well, that doesn’t always get to happen because every day can’t be your birthday. Bug reports are QA’s greatest deliverable so, time to file whatever is left of what we shook out.

My aim is to write such an interesting and compelling bug report that no follow-up with the reporter is necessary, the developer is persuaded to want to fix it and perhaps a product owner is impelled to triage as we, the lowly QA bug report submitters, suggest.

The majority of bugs a tester will find can thankfully be observed directly in the UI of an application and that ends up being the predominant way in which to find them when black box testing. Lucky for us on OS X we have a bunch of cool tools to capture them where they lay. A bit of extra effort on our part can ease the managing of the case for the bug triager and provide a greater depth of clarity for the bug fixer.

Let me walk you through what that looks like on the ground. Typically I’m running an iOS app from the simulator or on device, looking for observable issues (bugs found via logging or other methods is a subject for another time). And then all of the sudden, I see you, Bug!

Having observed—and hopefully reproduced—the anomaly, it’s time to capture it. For static UI issues, it’s really easy to just snap a screenshot from the iOS Simulator with ⌘S or by using the Xcode Organizer. I have set up some Hazel rules to watch the folders where those files will land and move new images into Skitch automatically. Skitch then opens with my screenshot and I can notate and highlight the image to better point out the defect or its effects. If the defect is something that a picture can’t capture, I will record a video with Reflector and turn that into a gif with Gif Brewery. Adding videos to a bug, with our current tracker, means the developer has to download and open the whole file. If the bug shows itself for only a second or two after some actions, how long will the video be and will things be clear? Maybe I’ll have to edit and notate it to highlight the issue – well that’s just absurd. A few quick strokes in Skitch is one thing, video editing is quite another and ain’t no one got time for that. Just drag the file into GifBrewery, constrain the video to the relevant section and export to .gif for embedding in the case. The app has some controls to assist here but 9 out of 10 times no further editing is needed and a few seconds in and out of the app will get me most of what I need.

From there, it’s a quick hop over to my favorite Markdown editor, Mou. Create a new document and in it goes my per project TextExpander snippet that I’ve carefully preconfigured. Setting up a per project snippet at the beginning of the project saves an enormous amount of time. It also has the advantage of uniformity which will help the team hone in on the content they are most concerned with. Crafting these with your standard bug report fare in the Markdown format is trivial. I like to add a few secret sauce ingredients to save me time. One is adding a short shell script that will go to my HEAD and run a git log --pretty=format:"%H%x20%an%x20%ad%x20" -n 1 or perhaps a git log -n 1 --decorate to notate which commit I built with. Another handy trick is to populate a dropdown menu with a pre-input device list of my specific test devices. I also employ TextExpander fill-ins to format my bug summary in the “If, Then, When” acceptance story format like so: If %filltext:name=SetupCriteria% when %filltext:name=ActionCriteria% then %filltext:name=Result%.

When I’ve got the report all gussied up and ready to go, I create a new case in the bugtracker. Then it’s a paste of my nicely formatted report into the new case and drag my Skitch or GifBrewery file into the drop target. Fiddle with the title, set some fields, and voila, I’m done. The composition of the report in Mou also has the advantage of not risking the browser feeding your hard work to the ether if you accidentally misbehave.

Once you’ve got your set up dialed in, you can really crank the reports out. This happens to be so streamlined that often, it can get rather dull. I like to reward my reader—usually the developer whose favor I want to curry—for having read it. Sneaking funny things in whenever I can reasonably do so one way I do that. I do recall an autolayout anomaly where some text in an article was stretched beyond all recognition and to me looked rather full of Eldritch horror. In lieu of a bug summary, I put in Ph’nglui mglw’nafh Cthulhu R’lyeh wgah’nagl fhtagn which is the sort of thing you can get away with if you have a lovely moving .gif of the bug in question. On another report, I listed “if your eyes are not open, open them now” as the first item in the steps to reproduce; you’re damn right I want to know if the reader is paying attention. Putting in some Easter eggs will show you when they do. While that is beneficial to me, I think bringing some levity to the process helps builds a camaraderie that you just can’t get by showing up and doing a good enough job.

1,068 Words