Asking developers to own QA is broken because developers are naturally biased towards the happy path. If you want to build a bar, you need someone to order -1 beers[1].
Handing off QA to an external team is broken because those people don't have the necessary experience with the product, nor can they quickly and easily engage with development to get to the heart of a problem (and a fix.)
Having QA rely exclusively on automation brings quality down as the application's complexity goes up. Writing tests to cover every possible edge case before the product ships isn't feasible.
The best solution I've seen in the decades I've been in software development has been to have QA and Dev as part of a team. Automation covers as much of the product as possible, and continually grows as issues are identified that can be covered by CI from here on out. QA become essential at the boundary of new features, and require a deep understanding of the product in order to know what knobs to turn that might break the application in ways the developer never thought about. Once those sharp edges are also automated, QA pushes the boundary forward.
I've also arrived at this approach and don't think it's that uncommon - IMO the article is presenting a false dichotomy.
There are still gotchas to look out for in the team-embedded QA approach. In typical team sizes, you often end up with only one QA per team - you need to make sure they have cover (everyone needs a break), and they need support in their discipline (do something to share QA knowledge across teams).
Absolutely. Disappointing to see this sort of shallow sales pitch blogspam making it to the frontpage. I'm surprised more HN readers don't see through this.
The entire purpose of this article is to self-servingly attempt to convince the reader that their product is the only solution to QA problems.
It correctly identifies some challenges with QA, but this solution is certainly not the only way to have effective QA. That Rainforest is resorting to such a disingenuous presentation of solutions to QA issues makes me think they probably don't actually solve QA problems very well.
"We have researched what makes QA successful and X, Y, and Z are what we found. Here's how we believe we're solving Z for our customers" would be a much more honest pitch.
QA also works best for features that the user can meaningfully interact with. The more esoteric, interconnected, or complex the failure is, the more it gets muddled with other reports and weird conditions (it breaks on 3pm when the moon is in equinox and I have my mouse pointed north). That’s an over exaggeration but the sentiment is accurate. QA will frequently lack a deeper understanding of how the software works and is connected. That ignorance can be valuable (finds blind spots) but has trade offs (assumptions are made based on imperfect observation of cause and effect where human biases lead you down dark hallways). Speaking from experience with lots of manual QA which is actually a rarity these days.
The other thing to consider is, if you’re careful, user feedback can be obtained more quickly. If you can keep things stable, then you won’t upset your users too badly and you’ll avoid the need for QA to act as a stand in.
Agreed it's presenting a false dichotomy. The article ends with a pitch for Rainforest's "no-code" QA platform so it makes sense they want to present their product as the clear solution. I could take the article more seriously if it at least mentioned a more integrated approach.
Most projects I've worked on have been games where the studio was split into teams of about 6-10 people with one dedicated in-house QA member for some but not all teams, a few in-house QA workers not assigned to a specific team, plus a much larger external QA team (from the parent company/ publisher). That has worked great.
I've also been on projects with only external QA. It works, but the lack of internal QA has been a frequent annoyance.
>Asking developers to own QA is broken because developers are naturally biased towards the happy path.
I've been arguing the same thing for a while now. There's a clear mismatch in incentives. If you hand off the QA to developers you might initially get less bug reports but that doesn't mean your software isn't garbage.
Some of the best QA engineers I've worked with were developers in a previous life but they've since spent years cultivating a very different way of thinking about software. Believing you can get your engineers to adopt that mindset overnight is pure fantasy.
[author here] You are absolutely right: the optimal setup is a cross-functional team of domain experts collaborating on solving the customer's problem. The article is meant to point out that siloing QA to either just dev or just QA is an anti-pattern that is unstable and will not work, long term, without significant pain.
I think the challenge is that, while we agree on the optimal setup, it's almost never done like this. There's lots of reasons why, but the main one (imo) is organizational scaling. The logic of scaling teams tends towards specialization, especially because developer time is extremely expensive.
There are many ways to address the problem of QA. From our perspective, if they don't broaden the ownership of QA from just developers or just QA engineers (which is where all the current products are targeted), they will exacerbate this specialization problem.
I think that to say developers should not be interest or responsible for quality is most certainly a cop out, there can and should be a level of pride of workmanship that is focused on the customer.
The main challenge with software is the possibility of unexpected regression in areas that are non-obvious when implementing a given feature. One of the more scalable ways of catching these regressions is via some sort of automated testing. I think that a QA support team can provide the tools and support that allow for teams to easily add end to end testing, which is vitally important when build large distributed software.
If the QA team is responsible for testing every feature from many teams there is definite risk of bus factor and finger pointing.
In the super ideal world the testing is specified via something like Cucumber and the orginal author could be the product owner with feedback from the engineer as he works to that specification. But I have never seen this though have heard it can happen, rarely.
Hi author, I found parts of this downright wrong, especially on the Cypress side, which is where I have the most familiarity:
>Compare this to code-based automation solutions like Selenium or Cypress, where a coder would need to manually update the code across all the relevant tests any time the product changes.
... Well, no. Those tools can have reusable chunks and references that work just like your steps and are organized in the same manner, so that things used multiple places are stored once and reused. That's a basic programming good practice. When a change is needed to, say, a login flow, you don't change 100 UI tests, you change the login function that those tests are all calling.
> Tests created in code-based automation solutions like Selenium and Cypress evaluate what’s going on in the code behind the scenes of your product’s UI. That is, these solutions test what the computer sees, not what your users see.
and
> For example, if the HTML ID attribute of the signup button in the example above changes, a Selenium or Cypress coded test that identifies that button by its ID would likely break.
This is all very disingenuous in my book. In these tools the tests interact with the rendered web page and _can_ be made to use IDs, classes, and other brittle stuff as selectors, but that's a naive approach that a professional wouldn't typically use, because the brittleness surfaces early and there are better patterns that avoid it. One of those is to use selectors based on something that itself should be present - eg, click a button based on its label being "Log In". If that test fails because the button label was changed or removed, that's a reasonable failure, because labels are important in the UI. In the case where we are selecting/making assertions about DOM elements that don't have meaningful user-facing labels, dedicated test-id attributes solve the brittleness problem. But by and large, if something is interactive to the user, it should have a label, so the fallback to test attrs for selectors is rare, and used for the case where we want to measure some specific effect took place that is not itself an interactive element.
It's unfair to suggest that these tools inherently create brittle tests.
> Rainforest QA tests don’t break when there are minor, behind-the-scenes code changes that don’t change the UI.
but
> It works by pixel matching and using AI to find and interact with (click, fill, select, etc.) the correct elements.
Look, maybe your tool is GREAT, I haven't tried it. But a functional UI test based on label contents may be more robust than something that breaks when the visual structure of a page changes, which happens itself pretty frequently. Maybe the AI is good enough to know that a bunch of stuff moved around but the actual flow is the same and can find its way through. With tests based on labels and dedicated selectors, devs can make massive structural changes to the DOM and not break any tests if all the workflows are the same.
I would sure love if this post contained more info on how the visual AI system of selecting elements is better than using labels, test attributes etc as is conventional in automated testing done by professionals. Likewise it would be good to compare with an actual competitor like Ghost Inspector, where non-programmers have been able to generate step-based tests like this for years. The main gripe I had with Ghost Inspector was that it creates brittle selectors and so tests need to be changed a lot in response to DOM changes, unless a developer gets in there and manually reviews/picks better selectors.
If what you have a is a tool that _makes more robust tests than Ghost Inspector_ but is _as easy for a PM to use_, then that is interesting.
I actually support this underlying idea completely - I love systems where non-developers can create and maintain tests that reflect what they care about in the UI. I even love it when tools like this create low-quality tests that are still expressive enough that a dev can quickly come in and fix up selectors to make them more suitable. Cypress Studio is a currently-experimental feature that looks promising for this too, allowing folks to edit and create tests by clicking in the UI, not editing code files. It's a good direction for automated test frameworks to explore.
I'm just really uncomfortable after reading this post. It strays beyond typical marketing hyperbole to be actually deceptive about the nature of these other tools and the practices of teams using them. Instead of highlighting the actual uniqueness of your product, you exaggerate the benefits of this approach vs other solution. Come on, you can do tests in parallel with many tools, and you can avoid captcha for automated testing in various ways. Some of the process points you make are fair but also, again, kind of weak, because sure "QA at the end with no consultation during planning and development" is bad, but that's inherently bad for well known reasons.
What you've said is basically "our tool is better than every other idea about QA, if those ideas are deliberately implemented in the worst possible way, and our tool is implemented in the best way". Well sure. But also, of course it is.
Sorry if this seems harsh. To give you the benefit of the doubt: Marketing copy walks a fine line when trying to be persuasive and you might not have expected to create this impression. It's also possible that you did some research into weaknesses of automated test frameworks and just don't understand that those things are compensated for quite easily and routinely, because maybe you don't have the background. I don't know, but I hope future materials are a little more grounded in reality.
i don’t agree with this at all. i’ve seen qa automation team members work solely as a resourcing issue: that if you have X developers you always have X feature developers and never enough people to automate. it’s not that devs aren’t capabale. i fundamentally disagree with rainforests mission that you need “other people” to achieve success in testing. your top engineers have no problem succeeding here
For the first ~15 years of my career, multiple teams, the model was always that engineering was responsible for all the test coverage of all areas. It was wonderful. A separate dedicated QA team will never, almost no matter how good they are, have the same deep insight into the code both big picture and small picture. So they won't be able to exercise it via tests as well as the engineers who wrote it. I always thought, this is the only way that makes sense.
To my surprise in subsequent roles, I've seen many development teams, even good ones, who don't have that deep instinct to break their code via tests. Which was very alien to me. I've seen enough cases now to realize it's somewhat common. For such teams, separate QA engineers are a must.
I still feel having quality responsibility in the engineering team ultimately can offer the best quality outcome, but only if you can build a team with the right attitude about testing. If not, having a separate QA team is vital.
> Asking developers to own QA is broken because developers are naturally biased towards the happy path
This overlooks that "no customer calls with product issues that dev needs to get fixed asap on a Saturday evening" is part of the happy path to optimize for. If those things happen, the dev has deviated from the happy path, including the embarrassment in front of fellow devs "but that's really a case we should have tested". This is another of those details the article is lacking.
I would agree with this. In a past job doing UI work, I had a tester sitting across from me. As soon as I got things working reasonably, I would give her a build to test. It was great, because we found a lot of bugs super early, very shortly after the code was written...
Problem comes from BigCo approach to people. Such corporations would like to have a QA team that should be utilized to a 100%.
So that if they don't have much work on project X you just drop them into project Y.
In reality we all know that is just not possible, because people have to have domain knowledge and keep up with changes. It is not like if you come back to project after 2 months you will be proficient with it as the world changed.
This way if project X has slow times, dedicated QA will have less to do and BigCo has to eat that "loss".
In small startup it is easier to have QA incorporated into dev/product team because as software is constantly changed there is work to be done all the time.
Many years ago I asked a developer which was a bigger deal:
- Having an outage
- Missing a deadline
They answered: "oh having an outage is WAY worse". I then asked: "if that's the case, why do you push so hard to hit your deadlines with code you know and I both know is probably not ready?"
They didn't really answer at the time but it eventually dawned on me what's happening:
- odds of being yelled at if you miss a deadline: 100%
- odds of being yelled at due to an outage: unclear as it depends on the odds of an outage in general so let's say "less than 100%"
Therefore, they are "gambling" rationally by pushing for deadlines vs pushing out good code.
The point I'm getting to is that if the goal is "hit the deadline" vs "deploy code that runs in production for two weeks with no errors", QA is going to be less of a priority.
Couple this with the fact that many firms think of QA as "they have to be cheaper than devs!" and then compensate them accordingly, means that QA people are fighting down both the comp front and the incentives front.
I've seen this happen so many times that I'm not really sure why people are surprised by it any more.
(NOTE: you could say "well then testing should be automated" but you get into a similar argument on who is building and maintaining the testing frameworks).
> - odds of being yelled at if you miss a deadline: 100%
> - odds of being yelled at due to an outage: unclear as it depends on the odds of an outage in general so let's say "less than 100%"
To reframe this, the odds of a dev having to crunch to hit a deadline they're behind on is 100%, but the odds of any developer catching a support escalation or on-call page from an outage are usually way less, especially on larger teams, because it's rare for every dev to always be on the hook for escalations. That's why things like goalies and pager rotations exist; the perception is that saddling one person with the responsibility occasionally is better than splitting it to everyone all the time. One weekend of abject hell a month is better than four weekends of annoyance.
But when any developer can shirk ownership of an outage, they all effectively do. Even from a support perspective, that doesn't even make me mad — who wouldn't want to sleep in, ignore Slack on weekends, and not feel dread every single time your phone pings with a notification?
On the other hand, teams _never_ let developers off the hook when there's a deadline that might slip. If you don't have something to do, you're pairing off to help someone else who does, or if you can't then you're more likely to be working on the next thing down the pipe so there's not as much deadline pressure, than supporting on stuff (like tests! and docs!) that won't be considered tech debt until someone (probably support!) hits something related to it and calls it out later.
Dedicated QA doesn't lift the outage ownership problem, it helps mitigate it before it happens. But QA teams that deflect outages struggle to provide data-driven reasons for their existence, because they can't track a negative, and credit for _n_ 9s of uptime is always split multiple ways such that nobody gets a big piece. QA winds up forever underappreciated because their wins are so much harder to count, but the times QA causes a deadline to slip are _always always always_ flagged.
Nevermind that outage response pulls engineering resources off hitting deadlines... so that becomes a self-perpetuating cycle...
The best route is to never have deadlines. Just convince sales and marketing of this and you're golden. /s
That (developers only being responsible for a fraction of bad rollouts they personally cause) reminds me of the water dynamic at a lot of apartments I've rented:
For "reasons" the water meters are per building rather than per unit, but the landlords are adamant that residents have to pay their fair share to ensure water isn't wasted. The scheme envisioned to meet those goals is that the total cost of water for a building is averaged out across all units (perhaps normalized by unit size). Looking at the net effect however, using $X of water only costs $X/N because my personal excess is split between the rest of the residents. Consequently, the entire building uses and pays for substantially more water than they would if the meters were more finely distributed.
This is a great description of the dysfunction in development work management. I've always said that deadlines are just made up numbers and the work will get done when it's done. However after leading a team I see how that attitude can lead to a ton of bike shedding instead of prioritizing and shipping features.
This is the classic case of development velocity pitted against operational stability. Entirely different incentives, and when vested into the same role that role is bound to prioritize one incentive over the other. For this reason I think they must be separated at least by body if not by team. They certainly need to separated by different managers.
I tend to think QA is perhaps well situated alongside Ops("DevOps"), and very close to Product + Design.
We were a small and good team developping hardware/software combos.
We ruined a demo to a client by promising some hard to do feature, and during the demo, the said feature had not been well tested for a particular environment.
The debriefing of that failure was memorable. The big boss was yelling at us, saying we should do better, work harder, longer, whatever was required to succeed.
When this calmed down, I only asked one question: when going back to my desk, should I work on this new feature promised to some other customer, or test this old one for any combination of inputs/environments?
The response was: "You do both"
I insisted that I will do both, but which one first?
He responded with some blabla I do not remember, but no response to my question.
To any manager which cannot decide between feature and stability:
If you cannot prioritize, the dev will do it, with whatever information/incentive they have. You may not be happy of the result.
But that's just bad management. It's most of the time better to drop/delay some features, than compromising stability. If the manager doesn't get that, he/she is the problem. And not the software developer.
OTOH, a manager doesn't see outage - they give you an assignment and expect you, the developer, do to a good job. Outages are not because you missed a deadline, but because you didn't do the job that is specified on your job description.
I'm sure you can make analogies. If you get a new kitchen installed, you expect it to be done properly, if it's finished within the day or so they quoted for you but the doors fall off, you won't be happy. You never SPECIFIED that the doors should be firmly attached - you assume they will be, because you trust in the competency of the people installing it.
Why can’t the QA people just live on the dev team? Why do they need to be siloed away or not exist?
I had this in my past job. Have 1 QA per two developers, the QA sits with those two developers, and you are constantly telling them when a build is done and on staging. They write tests, do some manual checks, and then tell you how it is going relatively immediately. They also handle the problem of reproducing bugs and narrowing down exactly where they occur, which is not trivial.
For all the faults in that organization (including letting our test runner be broken for months), we didn’t put out a lot of bugs and we found all manner of strange edge cases very quickly.
The key part in that is communication. That’s makes all the difference whether it be with BA’s, QA’s, or the end user. Speeds up the development cycle and greatly reduces the number of bugs.
The best experience I had was on a team that had essentially 2 BA’s and 7 devs. There was constant communication to clarify actual requirements, devs would build automated tests off them, BA’s would test against the requirements and then a business user would do a final look over. All in all features were able to be released usually within a day and there would be days we’d get out 3 or 4 releases. Only in one case did a major bug get released to production and the root cause of that was poorly worded regulations which had a clarifying errata come out at the 11th hour.
For as many faults as that company had that caused me to move on I’ve yet to run across a team that functioned as well as that one did.
Communication is great until someone becomes unreasonable and doesn't want to do something. Trust but the chain-of-command must verify. Shouldn't need to, but it should be there as insurance.
I’ve experienced both, were the dev team also had to do QA, and a dev team with QA people.
I personally hated writing automated tests, but it did give me a much deeper understanding of the product. Especially when writing tests for features I wasn’t working on.
That said, having a dedicated QA person within the team is far more effective. The dev team can build features, while our QA person works with the business to come up with test cases and test data.
We had that in one of my assignments. It kinda works, but at the same time, they were a gatekeeper, a barrier, and usually a release was delayed for a day while they tested out their new testing approach and went through their excel sheets of things to test for.
I think the ideal team setup is one you allude to: a cross-functional team with experts from each domain collaborating together. My intent was not to say that can't happen - but that it typically doesn't, because of specialization and organizational politics. The vast majority of software teams have a siloed QA team. Why? That's for another thread :)
Makes a lot of sense to me. The best and highest quality software I've ever worked on (people could die if it failed) had the devs doing testing and writing automated tests, with an intermingled QA team doing further testing, and often jumping in on dev.
QA here, disagree. In this situation QA is focused on the quality of the new features, so they're already excellent when they go out into the current release.
That's false. QA can also be concerned about growing new features in the next release. We have embedded QA's on our teams. They QA new features, as well as help to triage current bugs.
I work at a startup so we have a small team. But for us it goes:
1. Develop some feature (me and the other engineer are full stack)
2. Send to other engineer for code review
3. Deploy to a staging site, with test data. Send to product girl to QA
If she finds a bug (or realizes her design doesn’t look as right in practice as she envisioned) she sends back with notes, we implement or fix them, and then start back at 1-2.
I think QA is different for every org but I believe there should be steps. At bigger orgs there should be QA for every feature developed (so each team should have a QA person) and then an integration QA of the current beta.
I agree in theory and there are certainly teams where this could happen, but what happened on this team was that the bugs were found so quickly than the fixes were also quick (I would always get a bug report the same day and often within the hour), making shipping stuff with few bugs relatively painless.
Was there pressure to keep releasing? Yes. But with the rapid feedback, it never became too onerous to get them done anyway.
> Why can’t the QA people just live on the dev team?
Because there is little incentive for devs to police themselves and there could be multiple dev teams spanning client/server that needs to be integrated and tested.
A slightly better org to own QA would be product team.
Sure there’s incentives for devs to police themselves. I’d rather catch problems early than have a to deal with a shitstorm from pissed off product owners and VPs when bugs are found in production. But our QA person is on another team. I’d much rather have them on my team.
Each team should have some QA resources. Server can be tested independent of client, etc. Then the QA staff work together to ensure appropriate systems integration tests.
Siloed QA team runs the risk of becoming a bottleneck as work from disparate teams comes through.
I've been a QA Automation Engineer for 15 years. This is the best thread ever on HN! I've read every single comment.
I've found the best approach is for QAs to be embedded right into the product team - where the product team manager is the same person for both Dev and QA. QA and Dev all review each others code. QA reviews the Dev unit tests, and Devs review the QA integration tests (API and UI). QA may not be able to provide in-depth reviews on dev code, but at least they can double check that unit test coverage is adequate.
As for UI automation, one point of friction that often happens is that QAs want unique IDs on each UI element. If the devs fail to provide those unique IDs, then QAs are often forced to do some convoluted workaround. However, there is a better option where QAs simply add the unique IDs themselves and request a code review from the dev. This works well because the change to developer code is benign and QA is not slowed down.
Thank you for your comment (useful and interesting information).
So it looks like your suggestion is to have 2 types of engineers in the team, where at least one of them is focused on QA + encourage bi-directional code reviews.
Could you tell a bit about the involvement of PMs in the QA process from your experience? What's working and what isn't?
The PM is busy doing their own job (writing new stories, managing old stories, talking to clients, etc). They typically only do a superficial amount of testing on new features (and usually only on their pet-features). The vast majority of testing comes from QA and Dev.
Having QAs that technical would be amazing. Most of the time I find that anything more complex than basic JSON editing is the limit of their technical abilities.
Another QA/Automation engineer here - keep looking, we definitely exist! I was a dev but prefer QA, and still enjoy doing code reviews for the devs as well. Also helps to identify missing unit tests and consider what tests I'll be building if I've missed some too.
I've yet to see the automated test suite that replaces a skilled, sapient, human, functional tester. The automation takes away the drudgery of repeating tests, but it takes a skilled human to figure out what risks are in the code and figure out coverage to determine whether those risks are realized. If you have developers write good unit and integration tests, and build their work to their local environment to make sure it basically works, you avoid most of the "throw it over the wall" mentality. You also deliver way fewer needles in the haystack, if you will. Testers are free to look for more impactful bugs manually, and then to automate coverage as appropriate.
Writing code is generally easy. Figuring out what code to write is hard. I can spend 80+% of my day thinking about an overall system, what things are important, and how to design things so that they both work and don't prevent change later. Then sitting down and writing the code to do that is (usually) the easy part.
Writing automated tests is generally easy. Figuring out what tests to write is hard. Does the code break on normal cases? Does it break on edge cases? Does it handle all the known use cases? Does the implementation actually achieve what the client is asking for? Does what the client asked for actually make sense?
You can't just automate away testing. Automated testing (unit, functional, system, etc) is a power multiplier for both dev and QA... not a total solution.
Business thinks that you can do test automation in a vacuum without any domain knowledge and without knowing the system.
You always need people to understand the system, to understand the automation and to keep domain knowledge in house.
If your whole QA team leaves and you have automation (or even documentation) - you are still in deep shit because hiring new people and training them on the system will still be needed and only then they can learn to run the automation. They still need to spend time reading documentation and understanding it, because that something is written down or documented does not mean other person has the same understanding of it and the same reference framework to use that documentation.
Well you can hire some new people and let them run automation built by previous team but ... good luck with that.
I think you're pretty on point here; today human nuance is needed to decide WHAT to test as well as HOW MUCH to test. It's also useful for executing some types of tests too (which is why we support both automation, and human testing). IMHO Unit and integration tests should be a given. Human testers should be used with the highest leverage where possible. Being a functional-tester though, today is partly limited by what tooling is accessible to you - which we want to fix.
I've seen them used to do a decent smoke test to allow the QA people to make forward progress on other testing. I worked at a place that had a large number of computers running some testing software (robo something?) that ran through all the mainline scenarios. They constantly added new scenarios so the QA folks were working on much deeper flows until those too became automated.
Totally agree that automation will never fully replace the value of human-powered testing. (Though it is great for the rote regression-testing stuff. The "drudgery", as you put it.)
Isn't the problem with relying too much on unit and integration tests that they don't consider the end-to-end app experience? (Which, in my mind, is what ultimately matters when customers think of "quality".)
IMHO, yep - it's a balance, but the great thing is they can be quick to run, and run locally-easily; which is great for developers to get fast-feedback. Unit testing is unlikely to catch all end-user level issues though; traditional automation too, which is why human-testing is still valuable today.
Company from the article is not claiming that but all the business people are talking about it.
They would like that "test-automation" would mean instead of 2 QA engineers now you need 1 doing 2x more.
Sad reality is that with "test-automation" you actually still need those 2 QA engineers and maybe part time dev to help them out. The upside would be that those QA people would spend less time repeating clicks and improve quality of checks.
In a model where developers adhere to a devops philosophy and are product owners, there's no need for a split. Developers should not be measured only quantity of releases, but also on metrics related to availability, customer impact, latency, operational costs, etc.
I'm not opposed to a model where non-technical roles are empowered to define test cases and contribute to approval workflows of a release pipeline, but that doesn't absolve developers of the primary responsibility of being end-to-end owners.
I know "devops" is an overloaded term that means different things to different people. To me, it's when developers are involved in product research, requirements gathering, user studies, design, architecture, implementation, testing, release management, and ongoing operations. They're supported by a dedicated product manager who leads the research/requirements gathering/user studies initiatives.
This is how I've worked for most of the last decade. Empowered devs that are accountable for their releases and supported by metrics related to customer experience want good quality releases. Ownership works.
Agree but a lot of companies, particularly larger ones, don't have this sort of culture. They are filled with people whos sense of worth(at the company at least) is defined by a narrow job title and set of skills. Their worldview requires them to categorize everyone else on a narrow set of attributes as well:
"Oh, you're a programmer so you must not be good at soft-skills and communicating with the business."
"Ah, you have soft-skills and you also program? You must not actually be very good at programming so you should stop that and just become a PM."
"Ah, they are PMs/Sales/Whatever so SQL is too hard for them and they can't be expected to learn git or markdown; that's for nerds."
This leads to cringy, insulting sentiments like:
> The most valuable thing you can be doing is writing code
There's a little bit of an insane assumption in this pitch that PMs should be writing tests because they are the ones that "care a about quality."
With my engineering hat on, I hate this idea because the message that devs don't care about quality (but PMs do) is not how I want to operate.
With my PM hat on, I hate this because having the PM do test automation has got to be the lowest value activity you can ask the PM to do. That is, whatever the PM doesn't get to do because he's writing tests is almost certainly more valuable.
So yeah this pitch is nuts. If your system makes it easy to define tests with no code, awesome and that should be useful by the teams that do testing today. Weird that it isn't their angle.
A problem with having engineers own the tests is that if the engineer misunderstands how the feature is supposed to work, then both the code and tests will be wrong, but because they are consistently wrong the tests will be green. So the product owner still needs to do ad hoc testing to verify that things work as expected.
I like the “three amigos” approach from BDD as a solution here.
The product owner writes the initial acceptance criteria, ie specifies the product. Then sits down with an engineer and tester to edit the ACs into sensible Gherkin so they can be executed as tests.
The tester goes off and builds the tests (implementing the Gherkin steps) and the developer builds the feature. At the end you bring them together and if the tests pass, you are quite confident that you met the requirements. (If the error is not obvious you all huddle and figure out where the misunderstanding was).
The important thing here is that everyone gets together and makes sure they share an understanding of the feature. You could skip the Gherkin and write the tests some other way. But having the product owner think through the process of specifying the acceptance criteria cleanly is a good exercise, I think. Now I don’t think that means they need to own the tests… but there is a case for it not being the engineer that’s implementing the feature, for sure.
> The important thing here is that everyone gets together and makes sure they share an understanding of the feature
So true. Communication is everything, but because it's hard it gets short circuited everywhere. Where I work, devs write their own tests. It's as terrible as you can imagine while still having a successful product. I once suggested we split to a designer/verifier model...and got the weirdest blank stares. I honestly think it's because no one I work with wants to talk to each other.
Devs doing QA worked fine for us. You need to have a team that actually cares about the product, and you need figurehead devs - not necessarily seniors or team leads, but charismatic people others will fall into line with - who model the behaviours you want.
The problem with almost anything else is that it increases the number of hand-offs between groups of differently-aligned people on the route to production. If you're aiming at multiple deployments per day, with sub-hour commit-to-prod times, the latency in those hand-offs adds up way too fast.
My main problem with devs doing QA is that QA is huge, knowing the entire product is hard.
In my case, devs were just testing what they changed and never noticed something horribly broken somewhere else.
I think all devs should test that what they built works in production (you don't need QA for that), you need QA to keep everything else tested and working.
> devs were just testing what they changed and never noticed something horribly broken somewhere else.
There's your problem. If they broke it but didn't test what they broke, then they changed it but didn't realise they had. "Horribly broken elsewhere" sounds like the sort of thing that should show up in an automated test, no?
If the problem is "we don't have a good enough automated test suite and our architecture doesn't limit the blast radius of changes" then I can see how throwing QA headcount at it might look tempting.
A person doing manual testing, which is a gate to deployment would certainly be hard to make work for deploys as fast as you're targeting. It may still be valuable to have someone separate to the devs testing your product (in production, not as a gate to release) looking for things your automated testing, customer feedback or metrics may have missed. Whether this would be valuable is really context-dependent.
Readit News
Handing off QA to an external team is broken because those people don't have the necessary experience with the product, nor can they quickly and easily engage with development to get to the heart of a problem (and a fix.)
Having QA rely exclusively on automation brings quality down as the application's complexity goes up. Writing tests to cover every possible edge case before the product ships isn't feasible.
The best solution I've seen in the decades I've been in software development has been to have QA and Dev as part of a team. Automation covers as much of the product as possible, and continually grows as issues are identified that can be covered by CI from here on out. QA become essential at the boundary of new features, and require a deep understanding of the product in order to know what knobs to turn that might break the application in ways the developer never thought about. Once those sharp edges are also automated, QA pushes the boundary forward.
[1]: https://twitter.com/brenankeller/status/1068615953989087232
There are still gotchas to look out for in the team-embedded QA approach. In typical team sizes, you often end up with only one QA per team - you need to make sure they have cover (everyone needs a break), and they need support in their discipline (do something to share QA knowledge across teams).
The entire purpose of this article is to self-servingly attempt to convince the reader that their product is the only solution to QA problems.
It correctly identifies some challenges with QA, but this solution is certainly not the only way to have effective QA. That Rainforest is resorting to such a disingenuous presentation of solutions to QA issues makes me think they probably don't actually solve QA problems very well.
"We have researched what makes QA successful and X, Y, and Z are what we found. Here's how we believe we're solving Z for our customers" would be a much more honest pitch.
The other thing to consider is, if you’re careful, user feedback can be obtained more quickly. If you can keep things stable, then you won’t upset your users too badly and you’ll avoid the need for QA to act as a stand in.
Most projects I've worked on have been games where the studio was split into teams of about 6-10 people with one dedicated in-house QA member for some but not all teams, a few in-house QA workers not assigned to a specific team, plus a much larger external QA team (from the parent company/ publisher). That has worked great.
I've also been on projects with only external QA. It works, but the lack of internal QA has been a frequent annoyance.
I've been arguing the same thing for a while now. There's a clear mismatch in incentives. If you hand off the QA to developers you might initially get less bug reports but that doesn't mean your software isn't garbage.
Some of the best QA engineers I've worked with were developers in a previous life but they've since spent years cultivating a very different way of thinking about software. Believing you can get your engineers to adopt that mindset overnight is pure fantasy.
I think the challenge is that, while we agree on the optimal setup, it's almost never done like this. There's lots of reasons why, but the main one (imo) is organizational scaling. The logic of scaling teams tends towards specialization, especially because developer time is extremely expensive.
There are many ways to address the problem of QA. From our perspective, if they don't broaden the ownership of QA from just developers or just QA engineers (which is where all the current products are targeted), they will exacerbate this specialization problem.
The main challenge with software is the possibility of unexpected regression in areas that are non-obvious when implementing a given feature. One of the more scalable ways of catching these regressions is via some sort of automated testing. I think that a QA support team can provide the tools and support that allow for teams to easily add end to end testing, which is vitally important when build large distributed software.
If the QA team is responsible for testing every feature from many teams there is definite risk of bus factor and finger pointing.
In the super ideal world the testing is specified via something like Cucumber and the orginal author could be the product owner with feedback from the engineer as he works to that specification. But I have never seen this though have heard it can happen, rarely.
>Compare this to code-based automation solutions like Selenium or Cypress, where a coder would need to manually update the code across all the relevant tests any time the product changes.
... Well, no. Those tools can have reusable chunks and references that work just like your steps and are organized in the same manner, so that things used multiple places are stored once and reused. That's a basic programming good practice. When a change is needed to, say, a login flow, you don't change 100 UI tests, you change the login function that those tests are all calling.
> Tests created in code-based automation solutions like Selenium and Cypress evaluate what’s going on in the code behind the scenes of your product’s UI. That is, these solutions test what the computer sees, not what your users see.
and
> For example, if the HTML ID attribute of the signup button in the example above changes, a Selenium or Cypress coded test that identifies that button by its ID would likely break.
This is all very disingenuous in my book. In these tools the tests interact with the rendered web page and _can_ be made to use IDs, classes, and other brittle stuff as selectors, but that's a naive approach that a professional wouldn't typically use, because the brittleness surfaces early and there are better patterns that avoid it. One of those is to use selectors based on something that itself should be present - eg, click a button based on its label being "Log In". If that test fails because the button label was changed or removed, that's a reasonable failure, because labels are important in the UI. In the case where we are selecting/making assertions about DOM elements that don't have meaningful user-facing labels, dedicated test-id attributes solve the brittleness problem. But by and large, if something is interactive to the user, it should have a label, so the fallback to test attrs for selectors is rare, and used for the case where we want to measure some specific effect took place that is not itself an interactive element.
It's unfair to suggest that these tools inherently create brittle tests.
> Rainforest QA tests don’t break when there are minor, behind-the-scenes code changes that don’t change the UI.
but
> It works by pixel matching and using AI to find and interact with (click, fill, select, etc.) the correct elements.
Look, maybe your tool is GREAT, I haven't tried it. But a functional UI test based on label contents may be more robust than something that breaks when the visual structure of a page changes, which happens itself pretty frequently. Maybe the AI is good enough to know that a bunch of stuff moved around but the actual flow is the same and can find its way through. With tests based on labels and dedicated selectors, devs can make massive structural changes to the DOM and not break any tests if all the workflows are the same.
I would sure love if this post contained more info on how the visual AI system of selecting elements is better than using labels, test attributes etc as is conventional in automated testing done by professionals. Likewise it would be good to compare with an actual competitor like Ghost Inspector, where non-programmers have been able to generate step-based tests like this for years. The main gripe I had with Ghost Inspector was that it creates brittle selectors and so tests need to be changed a lot in response to DOM changes, unless a developer gets in there and manually reviews/picks better selectors.
If what you have a is a tool that _makes more robust tests than Ghost Inspector_ but is _as easy for a PM to use_, then that is interesting.
I actually support this underlying idea completely - I love systems where non-developers can create and maintain tests that reflect what they care about in the UI. I even love it when tools like this create low-quality tests that are still expressive enough that a dev can quickly come in and fix up selectors to make them more suitable. Cypress Studio is a currently-experimental feature that looks promising for this too, allowing folks to edit and create tests by clicking in the UI, not editing code files. It's a good direction for automated test frameworks to explore.
I'm just really uncomfortable after reading this post. It strays beyond typical marketing hyperbole to be actually deceptive about the nature of these other tools and the practices of teams using them. Instead of highlighting the actual uniqueness of your product, you exaggerate the benefits of this approach vs other solution. Come on, you can do tests in parallel with many tools, and you can avoid captcha for automated testing in various ways. Some of the process points you make are fair but also, again, kind of weak, because sure "QA at the end with no consultation during planning and development" is bad, but that's inherently bad for well known reasons.
What you've said is basically "our tool is better than every other idea about QA, if those ideas are deliberately implemented in the worst possible way, and our tool is implemented in the best way". Well sure. But also, of course it is.
Sorry if this seems harsh. To give you the benefit of the doubt: Marketing copy walks a fine line when trying to be persuasive and you might not have expected to create this impression. It's also possible that you did some research into weaknesses of automated test frameworks and just don't understand that those things are compensated for quite easily and routinely, because maybe you don't have the background. I don't know, but I hope future materials are a little more grounded in reality.
To my surprise in subsequent roles, I've seen many development teams, even good ones, who don't have that deep instinct to break their code via tests. Which was very alien to me. I've seen enough cases now to realize it's somewhat common. For such teams, separate QA engineers are a must.
I still feel having quality responsibility in the engineering team ultimately can offer the best quality outcome, but only if you can build a team with the right attitude about testing. If not, having a separate QA team is vital.
This overlooks that "no customer calls with product issues that dev needs to get fixed asap on a Saturday evening" is part of the happy path to optimize for. If those things happen, the dev has deviated from the happy path, including the embarrassment in front of fellow devs "but that's really a case we should have tested". This is another of those details the article is lacking.
So that if they don't have much work on project X you just drop them into project Y.
In reality we all know that is just not possible, because people have to have domain knowledge and keep up with changes. It is not like if you come back to project after 2 months you will be proficient with it as the world changed.
This way if project X has slow times, dedicated QA will have less to do and BigCo has to eat that "loss".
In small startup it is easier to have QA incorporated into dev/product team because as software is constantly changed there is work to be done all the time.
- Having an outage
- Missing a deadline
They answered: "oh having an outage is WAY worse". I then asked: "if that's the case, why do you push so hard to hit your deadlines with code you know and I both know is probably not ready?"
They didn't really answer at the time but it eventually dawned on me what's happening:
- odds of being yelled at if you miss a deadline: 100%
- odds of being yelled at due to an outage: unclear as it depends on the odds of an outage in general so let's say "less than 100%"
Therefore, they are "gambling" rationally by pushing for deadlines vs pushing out good code.
The point I'm getting to is that if the goal is "hit the deadline" vs "deploy code that runs in production for two weeks with no errors", QA is going to be less of a priority.
Couple this with the fact that many firms think of QA as "they have to be cheaper than devs!" and then compensate them accordingly, means that QA people are fighting down both the comp front and the incentives front.
I've seen this happen so many times that I'm not really sure why people are surprised by it any more.
(NOTE: you could say "well then testing should be automated" but you get into a similar argument on who is building and maintaining the testing frameworks).
> - odds of being yelled at due to an outage: unclear as it depends on the odds of an outage in general so let's say "less than 100%"
To reframe this, the odds of a dev having to crunch to hit a deadline they're behind on is 100%, but the odds of any developer catching a support escalation or on-call page from an outage are usually way less, especially on larger teams, because it's rare for every dev to always be on the hook for escalations. That's why things like goalies and pager rotations exist; the perception is that saddling one person with the responsibility occasionally is better than splitting it to everyone all the time. One weekend of abject hell a month is better than four weekends of annoyance.
But when any developer can shirk ownership of an outage, they all effectively do. Even from a support perspective, that doesn't even make me mad — who wouldn't want to sleep in, ignore Slack on weekends, and not feel dread every single time your phone pings with a notification?
On the other hand, teams _never_ let developers off the hook when there's a deadline that might slip. If you don't have something to do, you're pairing off to help someone else who does, or if you can't then you're more likely to be working on the next thing down the pipe so there's not as much deadline pressure, than supporting on stuff (like tests! and docs!) that won't be considered tech debt until someone (probably support!) hits something related to it and calls it out later.
Dedicated QA doesn't lift the outage ownership problem, it helps mitigate it before it happens. But QA teams that deflect outages struggle to provide data-driven reasons for their existence, because they can't track a negative, and credit for _n_ 9s of uptime is always split multiple ways such that nobody gets a big piece. QA winds up forever underappreciated because their wins are so much harder to count, but the times QA causes a deadline to slip are _always always always_ flagged.
Nevermind that outage response pulls engineering resources off hitting deadlines... so that becomes a self-perpetuating cycle...
The best route is to never have deadlines. Just convince sales and marketing of this and you're golden. /s
For "reasons" the water meters are per building rather than per unit, but the landlords are adamant that residents have to pay their fair share to ensure water isn't wasted. The scheme envisioned to meet those goals is that the total cost of water for a building is averaged out across all units (perhaps normalized by unit size). Looking at the net effect however, using $X of water only costs $X/N because my personal excess is split between the rest of the residents. Consequently, the entire building uses and pays for substantially more water than they would if the meters were more finely distributed.
I tend to think QA is perhaps well situated alongside Ops("DevOps"), and very close to Product + Design.
We were a small and good team developping hardware/software combos. We ruined a demo to a client by promising some hard to do feature, and during the demo, the said feature had not been well tested for a particular environment.
The debriefing of that failure was memorable. The big boss was yelling at us, saying we should do better, work harder, longer, whatever was required to succeed.
When this calmed down, I only asked one question: when going back to my desk, should I work on this new feature promised to some other customer, or test this old one for any combination of inputs/environments? The response was: "You do both" I insisted that I will do both, but which one first? He responded with some blabla I do not remember, but no response to my question.
To any manager which cannot decide between feature and stability: If you cannot prioritize, the dev will do it, with whatever information/incentive they have. You may not be happy of the result.
I'm sure you can make analogies. If you get a new kitchen installed, you expect it to be done properly, if it's finished within the day or so they quoted for you but the doors fall off, you won't be happy. You never SPECIFIED that the doors should be firmly attached - you assume they will be, because you trust in the competency of the people installing it.
I had this in my past job. Have 1 QA per two developers, the QA sits with those two developers, and you are constantly telling them when a build is done and on staging. They write tests, do some manual checks, and then tell you how it is going relatively immediately. They also handle the problem of reproducing bugs and narrowing down exactly where they occur, which is not trivial.
For all the faults in that organization (including letting our test runner be broken for months), we didn’t put out a lot of bugs and we found all manner of strange edge cases very quickly.
The best experience I had was on a team that had essentially 2 BA’s and 7 devs. There was constant communication to clarify actual requirements, devs would build automated tests off them, BA’s would test against the requirements and then a business user would do a final look over. All in all features were able to be released usually within a day and there would be days we’d get out 3 or 4 releases. Only in one case did a major bug get released to production and the root cause of that was poorly worded regulations which had a clarifying errata come out at the 11th hour.
For as many faults as that company had that caused me to move on I’ve yet to run across a team that functioned as well as that one did.
I personally hated writing automated tests, but it did give me a much deeper understanding of the product. Especially when writing tests for features I wasn’t working on.
That said, having a dedicated QA person within the team is far more effective. The dev team can build features, while our QA person works with the business to come up with test cases and test data.
The dev team manager is focused on growing new features in the next release.
QA is focused on the excellent of the current release.
Subordinating quality to new releases is the result.
In a similar situation, devops supposedly addressed similar tensions of dev and ops.
1. Develop some feature (me and the other engineer are full stack)
2. Send to other engineer for code review
3. Deploy to a staging site, with test data. Send to product girl to QA
If she finds a bug (or realizes her design doesn’t look as right in practice as she envisioned) she sends back with notes, we implement or fix them, and then start back at 1-2.
I think QA is different for every org but I believe there should be steps. At bigger orgs there should be QA for every feature developed (so each team should have a QA person) and then an integration QA of the current beta.
Was there pressure to keep releasing? Yes. But with the rapid feedback, it never became too onerous to get them done anyway.
Because there is little incentive for devs to police themselves and there could be multiple dev teams spanning client/server that needs to be integrated and tested.
A slightly better org to own QA would be product team.
Siloed QA team runs the risk of becoming a bottleneck as work from disparate teams comes through.
I've found the best approach is for QAs to be embedded right into the product team - where the product team manager is the same person for both Dev and QA. QA and Dev all review each others code. QA reviews the Dev unit tests, and Devs review the QA integration tests (API and UI). QA may not be able to provide in-depth reviews on dev code, but at least they can double check that unit test coverage is adequate.
As for UI automation, one point of friction that often happens is that QAs want unique IDs on each UI element. If the devs fail to provide those unique IDs, then QAs are often forced to do some convoluted workaround. However, there is a better option where QAs simply add the unique IDs themselves and request a code review from the dev. This works well because the change to developer code is benign and QA is not slowed down.
Could you tell a bit about the involvement of PMs in the QA process from your experience? What's working and what isn't?
Writing code is generally easy. Figuring out what code to write is hard. I can spend 80+% of my day thinking about an overall system, what things are important, and how to design things so that they both work and don't prevent change later. Then sitting down and writing the code to do that is (usually) the easy part.
Writing automated tests is generally easy. Figuring out what tests to write is hard. Does the code break on normal cases? Does it break on edge cases? Does it handle all the known use cases? Does the implementation actually achieve what the client is asking for? Does what the client asked for actually make sense?
You can't just automate away testing. Automated testing (unit, functional, system, etc) is a power multiplier for both dev and QA... not a total solution.
Business thinks that you can do test automation in a vacuum without any domain knowledge and without knowing the system.
You always need people to understand the system, to understand the automation and to keep domain knowledge in house.
If your whole QA team leaves and you have automation (or even documentation) - you are still in deep shit because hiring new people and training them on the system will still be needed and only then they can learn to run the automation. They still need to spend time reading documentation and understanding it, because that something is written down or documented does not mean other person has the same understanding of it and the same reference framework to use that documentation.
Well you can hire some new people and let them run automation built by previous team but ... good luck with that.
Isn't the problem with relying too much on unit and integration tests that they don't consider the end-to-end app experience? (Which, in my mind, is what ultimately matters when customers think of "quality".)
They say that instead of being developers writing codee, they should be non technical product managers using no-code.
They would like that "test-automation" would mean instead of 2 QA engineers now you need 1 doing 2x more.
Sad reality is that with "test-automation" you actually still need those 2 QA engineers and maybe part time dev to help them out. The upside would be that those QA people would spend less time repeating clicks and improve quality of checks.
I'm not opposed to a model where non-technical roles are empowered to define test cases and contribute to approval workflows of a release pipeline, but that doesn't absolve developers of the primary responsibility of being end-to-end owners.
I know "devops" is an overloaded term that means different things to different people. To me, it's when developers are involved in product research, requirements gathering, user studies, design, architecture, implementation, testing, release management, and ongoing operations. They're supported by a dedicated product manager who leads the research/requirements gathering/user studies initiatives.
"Oh, you're a programmer so you must not be good at soft-skills and communicating with the business."
"Ah, you have soft-skills and you also program? You must not actually be very good at programming so you should stop that and just become a PM."
"Ah, they are PMs/Sales/Whatever so SQL is too hard for them and they can't be expected to learn git or markdown; that's for nerds."
This leads to cringy, insulting sentiments like:
> The most valuable thing you can be doing is writing code
Ughh.
With my engineering hat on, I hate this idea because the message that devs don't care about quality (but PMs do) is not how I want to operate.
With my PM hat on, I hate this because having the PM do test automation has got to be the lowest value activity you can ask the PM to do. That is, whatever the PM doesn't get to do because he's writing tests is almost certainly more valuable.
So yeah this pitch is nuts. If your system makes it easy to define tests with no code, awesome and that should be useful by the teams that do testing today. Weird that it isn't their angle.
I like the “three amigos” approach from BDD as a solution here.
The product owner writes the initial acceptance criteria, ie specifies the product. Then sits down with an engineer and tester to edit the ACs into sensible Gherkin so they can be executed as tests.
The tester goes off and builds the tests (implementing the Gherkin steps) and the developer builds the feature. At the end you bring them together and if the tests pass, you are quite confident that you met the requirements. (If the error is not obvious you all huddle and figure out where the misunderstanding was).
The important thing here is that everyone gets together and makes sure they share an understanding of the feature. You could skip the Gherkin and write the tests some other way. But having the product owner think through the process of specifying the acceptance criteria cleanly is a good exercise, I think. Now I don’t think that means they need to own the tests… but there is a case for it not being the engineer that’s implementing the feature, for sure.
So true. Communication is everything, but because it's hard it gets short circuited everywhere. Where I work, devs write their own tests. It's as terrible as you can imagine while still having a successful product. I once suggested we split to a designer/verifier model...and got the weirdest blank stares. I honestly think it's because no one I work with wants to talk to each other.
The problem with almost anything else is that it increases the number of hand-offs between groups of differently-aligned people on the route to production. If you're aiming at multiple deployments per day, with sub-hour commit-to-prod times, the latency in those hand-offs adds up way too fast.
In my case, devs were just testing what they changed and never noticed something horribly broken somewhere else.
I think all devs should test that what they built works in production (you don't need QA for that), you need QA to keep everything else tested and working.
There's your problem. If they broke it but didn't test what they broke, then they changed it but didn't realise they had. "Horribly broken elsewhere" sounds like the sort of thing that should show up in an automated test, no?
If the problem is "we don't have a good enough automated test suite and our architecture doesn't limit the blast radius of changes" then I can see how throwing QA headcount at it might look tempting.