Big Data, Little Tests

This is the content for my Agile2012 presentation. (Download BigDataLittleTests-Heintz)

Slides are attached to this post, and these are the GitHub repos:

  • https://github.com/jheintz/sample-hadoop-testing
  • https://github.com/jheintz/sample-riak-testing
  • https://github.com/jheintz/riak-download

Here are the scripts that demonstrate everything:

Hadoop Sample:

#/bin/sh

git clone https://github.com/jheintz/sample-hadoop-testing.git
cd sample-hadoop-testing.git

git checkout step1 # initial MRUnit tests
mvn test
# eclipse: show mapper
# eclipse: show reducer
# eclipse: show unit test
# eclipse: try open hadoop source…

git checkout step2 # Use Cloudera repo for sources
mvn test
# eclipse show pom
# eclipse: open hadoop source

git checkout step3 # parameterized many tests
# eclipse test ManyWordCountTest.java

git checkout step4 # parameterized non-mr tests for speed
# eclipse test ManyWordTest

git checkout step5 # add cluster test
# eclipse show HadoopClusterBase
# eclipse test ClusterTest

git checkout step6 # add parameterized cluster test
# eclipse test MultipleClusterTest

git checkout step7 # split unit and integration tests
# eclipse show test names
# eclipse show pom failsafe
mvn test
mvn verify

Riak Sample:

#/bin/sh

git clone https://github.com/jheintz/sample-riak-testing.git
cd sample-riak-testing.git

git checkout step1 # initial test, fails
mvn test
# eclipse show SmokeTest failure

git checkout step2 # add riak downloader
mvn test
# I don’t know why eclipse has maven failure…
# eclipse show riak-build.xml
# eclipse show pom.xml

git checkout step3 # add parameterized many tests
# eclipse show ManySmokeTest

ant -f riak-build.xml stop

Cassandra Sample:

#/bin/sh

git clone http://github.com/jsevellec/cassandra-unit.git
cd cassandra-unit

mvn test
mvn eclipse:eclipse

 

4 thoughts on “Big Data, Little Tests

    • Thanks for attending my presentation Vinayak, I’ve fixed the download now and will email you a copy!

      What did you think of the session? What did you want more of? Less of?

  1. Hi John,
    Great Work. Really appreciate this a very useful contribution.

    Do you have any recorded material of you presentation? If yes can you please give me some download location of it.

    As I have not attended your session, so can you please help me out if possible.

    Thanks,
    Amar

    • Hello Amar

      I do not have a recording of the session, but the git repo I listed in the blog post is actually a step-by-step walk through.

      Let me know if you have specific questions or need more support with the materials.

      Cheers
      John

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>