icon cloudapp icon email
Cypress: End-to-end Testing For Developers & Quality Engineers
pre-game:
STICKERS!
dad's root beer
javascript
go-time:
thanks to organizers
thanks to centare
thanks to you
Tech Lead at Artsy
NYC, MKE
our mission is to expand the art market,
so that everyone can be moved by art daily.
and we're doing that with a platform for collecting and discovering art.
Testing Strategies
Testing strategies
Mike Cohn, Succeeding with Agile, 2009
explain types of tests
opinions over shape, names of tests, validity, etc
but it's still a useful metaphor to talk about tradeoffs in testing
Testing Strategies
it's a pyramid because there are tradeoffs
1) cost: to write & maintain
2) speed
3) reliability: how often does this test fail or break?
4) confidence: how confident about your app?
3 on the left have better outcomes lower in the pyramid
1 on the right has better outcomes higher in the pyramid
we'll talk about the top of they pyramid today, e2e
and I'm especially interested in addressing one question:
Testing Strategies
Why are end-to-end tests hard?
• Many moving parts
• State
• Data
0) those 3 left columns all have poorer outcomes for e2e. why?
1) parts: you're not just testing individual units anymore
(parts): more could go wrong
2) state: to complete a test, you need to put the system into a specific state; state is hard to deal with
3) data: you need reliable data so that your tests aren't flaky. If they're flaky, they'll get turned off or deleted.
4) tooling - may vary depending on tech stack (rails vs js)
focus on tooling
but we'll also circle back to this list at the end
Cypress
Fast, easy and reliable testing for anything that runs in a browser.
cypress.io
Cypress
All-in-one testing framework, assertion library, with mocking and stubbing, all without Selenium.
cypress.io
Cypress
Cypress focuses on doing end-to-end testing REALLY well.
cypress.io
not unit, not integration
what it looks like
We'll look at an actual running test later
left panel: every single event in browser
right panel: your app
Cypress
Architecture
It's not Selenium
kind of a joke but also not really
many e2e tools are , under the covers
Cypress
Architecture
Selenium
Cypress
Architecture
Cypress
executed in the same run loop as your app
so it can interact with your app & all events in your app,
including all network activity
also makes it faster because the communication is more direct
Cypress
Architecture
It's JavaScript
and so are the tests you write
Cypress
Architecture
It's Chrome only
for now...
let's get that out of the way
this might be a dealbreaker for you
but it might also mean you can use Cypress for most tests, and another tool for cross-browser testing
Cypress
Features
Tests automatically wait
so they're more reliable
up to 4.5 seconds
if you look for an element and it's not there because React is still rendering,
you have up to 4.5 seconds before the test fails
...
flake!
anyone experience e2e tests that fail because of timing issues?
Cypress
Features
Tests run when files update
tightening your feedback loop
When you're running the tests locally,
Cypress
Features
You can time travel through tests
for easier debugging
successful or failing
Cypress logs every event along the way
...
debug using the chrome dev tools at any point in your test
Cypress
Features
Cypress runs in your browser
making your tests fast
relatively. it's still e2e, so db, network latency, etc.
fewer layers between your code & the browser
Cypress
Features
You have access to all network activity
so you can inspect it or mock it
or do whatever you want with it
maybe you don't want to do this;
we'll talk more about the tradeoffs of mocking.
Cypress
Setup
npm install cypressthat's really it.
If you're not familiar with npm, it's the "node package manager", which is how modern JS projects manage their dependencies.
This command is telling a js project to pull down the latest version of cypress, and add it to the project manifest so that it works on everyone's machine.
The installation of cypress itself is really this simple - one line in a terminal.
Depending on your org, using npm might be significantly more difficult than getting cypress running.
artsy homepage
searchbox - search for artists you love
let's build a cypress test for it
Cypress
Example
Syntax of a test
describe("searching" , () => {
it("searches for an artist" , () => {
})
})
describe = "namespace"
fat arrow functions
describe can be nested
it = test
Cypress
Example
cy.visit
describe("searching" , () => {
it("searches for an artist" , () => {
cy.visit("/" )
})
})
Cypress
Example
cy.contains
describe("searching" , () => {
it("searches for an artist" , () => {
cy.visit("/" )
cy.contains("Learn about and collect art" )
})
})
you'll probably want to look for more specific data than this
automatic waiting! (up to 5 seconds by default)
Cypress
Example
cy.get
describe("searching" , () => {
it("searches for an artist" , () => {
cy.visit("/" )
cy.contains("Learn about and collect art" )
cy.get('[placeholder="Search"]' )
})
})
Cypress
Example
click
describe("searching" , () => {
it("searches for an artist" , () => {
cy.visit("/" )
cy.contains("Learn about and collect art" )
cy.get('[placeholder="Search"]' )
.click()
})
})
Cypress
Example
type
describe("searching" , () => {
it("searches for an artist" , () => {
cy.visit("/" )
cy.contains("Learn about and collect art" )
cy.get('[placeholder="Search"]' )
.click()
.type("goldsworthy" )
})
})
Cypress
Example
cy.contains, click
describe("searching" , () => {
it("searches for an artist" , () => {
cy.visit("/" )
cy.contains("Learn about and collect art" )
cy.get('[placeholder="Search"]' )
.click()
.type("goldsworthy" )
cy.contains("Andy Goldsworthy" ).click()
})
})
Cypress
Example
cy.url
describe("searching" , () => {
it("searches for an artist" , () => {
cy.visit("/" )
cy.contains("Learn about and collect art" )
cy.get('[placeholder="Search"]' )
.click()
.type("goldsworthy" )
cy.contains("Andy Goldsworthy" ).click()
cy.url().should("contain" , "andy-goldsworthy" )
})
})
Cypress
Example
cy.contains
describe("searching" , () => {
it("searches for an artist" , () => {
cy.visit("/" )
cy.contains("Learn about and collect art" )
cy.get('[placeholder="Search"]' )
.click()
.type("goldsworthy" )
cy.contains("Andy Goldsworthy" ).click()
cy.url().should("contain" , "andy-goldsworthy" )
cy.contains("Andy Goldsworthy creates outdoor sculpture" )
})
})
Cypress at Artsy
Smoke Tests
We started with some real simple POC tests
to prove we could get value from cypress
and that meant writing some smoke tests
to make sure the site was up after each deployment.
Cypress at Artsy
Smoke Tests
describe("Artwork" , () => {
it("/artwork/:id" , () => {
cy.visit("artwork/andy-goldsworthy-ammonite" )
cy.get("h1" ).should("contain" , "Andy Goldsworthy" )
cy.get("h2" ).should("contain" , "Ammonite" )
})
})
this is what our test looked like.
Not a lot to it
But it proves to us that we didn't totally break the artwork page
Cypress at Artsy
Critical Flows
Cypress at Artsy
Critical Flows
Logging In
Cypress at Artsy
Critical Flows
Logging In
it("logs in admin user" , () => {
})
Cypress at Artsy
Critical Flows
Logging In
it("logs in admin user" , () => {
cy.visit("/" )
cy.get("header" )
.find("button" )
.contains("Log in" )
.click()
})
Cypress at Artsy
Critical Flows
Logging In
it("logs in admin user" , () => {
cy.get("input[type=email]" )
.type(Cypress.env("ADMIN_USER" ))
cy.get("input[type=password]" )
.type(Cypress.env("ADMIN_PASSWORD" ), {
log : false ,
})
cy.get("button" )
.contains("Log in" )
.click()
cy.get("header" ).find("a[href='/works-for-you']" )
})
Cypress at Artsy
Critical Flows
Finding Art
Cypress at Artsy
Critical Flows
Finding Art
it("filters by medium" , () => {
cy.visit("/collect" )
cy.contains("Ways to buy" )
cy.get("input[type=radio]" ).contains("Painting" ).click()
cy.url().should("contain" , "/collect/painting" )
})
...
and if you have consistent reliable results, you could specify which artworks appeared
we don't - our inventory is changing frequently,
and our artwork sort has a touch of "trending" to it
Cypress at Artsy
Critical Flows
Buying Art
when they find something they love, we want to make sure they can
buy art
negotiate with the gallery
make an offer
Cypress at Artsy
Critical Flows
Buying Art
it("can buy an individual work" , () => {
})
no code - too much!
you start to smell the aroma of one of our biggest challenges so far
how to deal with data & state?
We're chaining together a series of actions for this test
And that has tradeoffs
Cypress at Artsy
Challenges
Cypress at Artsy
Challenges
Why are end-to-end tests hard?
• Many moving parts
• State
• Data remember?
Before we even started, we wanted to figure out how we would approach the data issue.
Cypress at Artsy
Challenges
Data
1. Use a live database
👍🏼 Tests the entire stack
👎🏼 Depends on live data not changing
👎🏼 Requires fake users
live db - from staging or production site
1) one of our goals was to prevent "breaking the site" & without testing entire stack we couldn't be sure
(confidence)
2) (not much control)
3) fake users:
3a) have to store them in the live db
3b) have to store creds somewhere the tests can access
Cypress at Artsy
Challenges
Data
2. Use a test database
👍🏼 Tests the entire stack
👍🏼 We can seed it however we want
👎🏼 Hard to set up
1) confidence
2) control
3) We don't have one source of data; we have five to ten. Mongodb, postgres, elastic
Cypress at Artsy
Challenges
Data
3. Use no database (mock data calls)
👍🏼 Easier to set up
👍🏼 We can mock it however we want
👎🏼 Doesn't test the entire stack
Cypress at Artsy
Challenges
Data
1. Use a live database
2. Use a test database
3. Use no database (mock data calls)
Cypress at Artsy
Challenges
Data
1. Use a live database 👈🏼2. Use a test database
3. Use no database (mock data calls)
We settled on #1
it was really important to us to test the whole stack
2 seems like the "right way" for testing the entire stack,
but it also has the biggest upfront cost - setting up all that data, in many different systems.
I wouldn't be surprised to see us end up here eventually
also: 3 might be great for you (cory)
Cypress at Artsy
Challenges
Finding Elements
Cypress at Artsy
Challenges
Finding Elements
Rule #1: Find elements the way your users would• Text
• Images: alt
• SVG: title
0) key to meaningful tests that last
1) Text: that they can read on the screen
1a) target buttons and links with the text they see
...
In general, the closer you are to testing your app the way a real user uses it, the more reliable & resilient your tests are going to be.
But sometimes you need to find an element and there's no way to target it that a user would experience
We use these rails all over our site to display different contexts of artwork
And in each rail is a series of artwork cards
If we imagine a scenario where we want to find an artwork
but it appears in two separate rails
How do we target the one under "Your favorite works?"
Cypress at Artsy
Challenges
Finding Elements
Semantic CSS
<div class ="recently-viewed" >
<h2 > Recently viewed</h2 >
<ul >
<li > <...Andy Goldsworthy artwork card... /> </li >
</ul >
</div >
<div class ="your-favorite-works" >
<h2 > Your favorite works</h2 >
<ul >
<li > <...Andy Goldsworthy artwork card... /> </li >
</ul >
</div >
semantic css: css selectors describe an element's usage in the app
so you might put a class on the element surrounding the card you want to grab
Cypress at Artsy
Challenges
Finding Elements
Semantic CSS
cy.get(".your-favorite-works" ).contains("Andy Goldsworthy" )
and then to find it, your query looks like this
it's definitely better than trying to use indexes of elements on the page
and it works.
until it doesn't, because a dev removes the semantic class or renames it.
...
At Artsy, we don't even write semantic markup
Cypress at Artsy
Challenges
Finding Elements
React + Styled Components
<div class ="sc-bdVaJa jVrsza" >
<h2 > Recently viewed</h2 >
<ul >
<li > <...Andy Goldsworthy artwork card... /> </li >
</ul >
</div >
<div class ="sc-bdVaJa bOgaFD" >
<h2 > Your favorite works</h2 >
<ul >
<li > <...Andy Goldsworthy artwork card... /> </li >
</ul >
</div >
Styled components: more stable styling
We have no semantics in our markup
Compounding this: our test project is in a separate codebase than UI
Cypress at Artsy
Challenges
Finding Elements
data- Attributes<div class ="sc-bdVaJa jVrsza" >
<h2 > Recently viewed</h2 >
<ul >
<li > <...Andy Goldsworthy artwork card... /> </li >
</ul >
</div >
<div class ="sc-bdVaJa bOgaFD" data-test ="your-favorite-works" >
<h2 > Your favorite works</h2 >
<ul >
<li > <...Andy Goldsworthy artwork card... /> </li >
</ul >
</div >
Cypress at Artsy
Challenges
Finding Elements
cy.get("[data-test=your-favorite-works]" ).contains("Andy Goldsworthy" )
Cypress at Artsy
Challenges
Finding Elements
cypress-testing-library
github.com/testing-library/cypress-testing-library
cy.contains("Learn about and collect art" )
cy.findByText("Learn about and collect art" )
cy.get('[placeholder="Search"]' )
cy.findByPlaceholderText("Search" )
a tool that enforces finding elements by what the user sees
this is a rewrite of our example test
findBy....
Cypress at Artsy
Challenges
Finding Elements
cypress-testing-library
github.com/testing-library/cypress-testing-library
cy.findByText("..." )
cy.findByAltText("..." )
cy.findByLabelText("..." )
cy.findByPlaceholderText("..." )
cy.findByRole("..." )
cy.findByTitle("..." )
cy.findByDataId("..." )
different ways to search for elements how the user experiences your app
dataId - escape hatch
different methods to fail or not fail when something's not found
Cypress at Artsy
Challenges
Limitations
There will never be support for multiple browser tabs .
cypress.io
tests for a CMS tool that pops the page you're editing in another tab
and you want to inspect the value that's rendered there
it's not as simple as cy.contains()
workaround: request page programmatically & inspect contents
Cypress at Artsy
Challenges
Limitations
You cannot use Cypress to drive two browsers at the same time .
cypress.io
We've talked about doing some complex auction bidding scenarios between multiple bidders
workarounds described at url
Cypress at Artsy
Challenges
Limitations
Each test is bound to a single origin .
cypress.io
SSO
if you're using a 3rd party auth provider,
you'll need to do some trickery to visit that second origin
(request page programmatically & inspect response)
(me: set cookies/local storage with spoofed auth)
Verdict
Does Cypress replace Selenium?
No the cypress team thinks it will eventually, they just have to resolve:
1) javascript only
2) cross-browser testing
but they wield strong opinions (limitations)
Verdict
Cypress is young.
It has a lot of growing to do
But it shows a ton of promise
Verdict
Developers like to write Cypress tests.
developers enjoying writing end-to-end tests can end one of two ways
which you end up following says a lot about the relationships of your organization.
1) no need for qa
2) ...
Verdict
Developers and QA can share ownership .
This is the holy grail for me
historically there's been a wall that devs chuck things over for qa to look at
but if we are co-owning the codebase
it's a chance for us to smash silos
and get totally cross-functional
and help each other out more
Thank you!
Thank you for your time!
Questions afterward
Enjoy the rest of ___
Resources
Further Reading
icon cloudapp
icon email
icon twitter
Cypress: End-to-end Testing For Developers & Quality Engineers
Steven Hicks
@pepopowitz
steven.j.hicks@gmail.com
stevenhicks.me/cypress-for-devs-and-qa