Kevin: This is another question that is hard to answer in broad terms, but here are just a handful of thoughts: With all this said, are there best practices to show how we moved the needle? Or what types of metrics do we capture to show that? (E.g. QA Metrics I am looking to implement TDD and CI and increase testing in the development environment. Depending on your needs, you might also look into more beta-testing related tools like HockeyApp, TestFlight, Fabric, or TestFairy, though I don’t think those offer any production monitoring. Kevin: I’m not personally aware of mobile monitoring tools other than New Relic and AppDynamics. Mobile Testing What do you recommend for mobile monitoring? We use New Relic but our mobile developers hate it. Movement from one branch to another requires a pull request. Feature branch -> Develop (backed by staging) -> Master (backed by production). CD Branching Strategy Do you think there is a particular Git repository/branching strategy that is best suited for CD?įred: Yes absolutely. In practice these are usually unit tests (run with a tool like junit, nosetests, rspec, etc) or browser tests (using something like selenium webdriver, cucumber, capybara, etc).
There are a lot of added niceties–for example a lot of common tools are pre-installed and there are special directories to store dependency caches, build artifacts, and test output for later access–but it is ultimately just a place to run your own tests. Kevin: CircleCI provides an environment for you to run whatever tests you want. For Kevin: how does CircleCI perform its functional integration tests to fit in with its CI/CD process? Manually? Automated? Ad-hoc? We mock the external APIs for our unit tests, and use sandboxed versions where possible. We don’t use microservices, but we do use external APIs (payment processing etc), which I assume are equivalent for the purpose of your question.
CRITICAL OPS TESTFLIGHT CODE FULL
Do you have a full environment with all microservices for your integration tests?įred: Yes. How long should an integration test suite run? (Obviously as quickly as possible, but how fast are yours?)įred: Our entire integration suite (Rainforest + cucumber) takes about 25 minutes. If this is unrealistic then at least on every pull request. Integration Testing How often do you recommend running integration test suites?įred: As often as possible. I would also definitely recommend production performance monitoring, as unexpected things will happen in production. I would emphasize controlling as many variables as possible and keeping the benchmark realistic.įor example, at CircleCI we run new build container images against a bunch of open source repos before deploying them. In some cases, time myprogram > /dev/null might be enough, but in others you may need a HTTP API client/test harness to bang on some web service.
I’m afraid this will vary greatly from case to case.
Kevin: I’m assuming this means performance testing. Performance Testing What tools do you recommend for benchmark testing? This is a good example of an area where doing CD does involve some overhead of engineering effort, but it is worth the overall productivity gain of constant deployment. Kevin: Yes! Rainforest QA actually did a great blog post about exactly this. Database Migrations Not sure if you are going to talk about it, but the most challenging thing about CD is database schema changes. Read on to find out more of what Kevin and Fred have to say about how organizations can get to continuous deployment, and what advice they have on everything from integration testing to QA metrics. If you tuned in for our “Getting to Continuous Deployment” Webinar, you already know that we had more questions than we had time to answer! Fortunately, both Kevin Bell of CircleCI and Fred Stevens-Smith of our own Rainforest QA had time to sit down and answer a few more questions that our webinar viewers sent in.
0 kommentar(er)
