I did that a while ago, I’m just late writing about it.

After some experimentation, I’ve succeeded to setup syntax check, linting, idempotency check and installation success within docker.

Create an inventory (tests/inventory), containing:

localhost

Also the test playbook (tests/test.yml):

---
- hosts: localhost
  remote_user: root
  roles:
    - ansible-syncthing-debian

Then the Dockerfile, I used Debian « stretch » current testing to have a recent version of Ansible and ansible-lint from pip.

FROM debian:stretch

ENV ROLE_NAME ansible-syncthing-debian
ENV WORKDIR /build/${ROLE_NAME}
ENV PYTHONUNBUFFERED 1

WORKDIR ${WORKDIR}
RUN apt-get update -qq \
    && apt-get -y --no-install-recommends install ansible python-setuptools python-pip \
    && apt-get purge -y \
    && apt-get autoremove -y \
    && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
RUN pip install ansible-lint
ADD . ${WORKDIR}
ADD . /etc/ansible/roles/${ROLE_NAME}
ADD ./tests/inventory /etc/ansible/hosts

# Syntax check
RUN ansible-playbook -i $WORKDIR/tests/inventory $WORKDIR/tests/test.yml --syntax-check
# Lint check
RUN ansible-lint $WORKDIR/tests/test.yml
# Run the playbook
RUN ansible-playbook -i $WORKDIR/tests/inventory $WORKDIR/tests/test.yml --connection=local
# Idempotency check
RUN ansible-playbook -i $WORKDIR/tests/inventory $WORKDIR/tests/test.yml --connection=local \
    | grep -q 'changed=0.*failed=0' \
    && (echo 'Idempotence test: pass' && exit 0) \
    || (echo 'Idempotence test: fail' && exit 1)
RUN syncthing -version

Finally the .travis.yml:

---
sudo: required
services:
  - docker
script:
  - docker build .

Refs