Distributed jMeter through VPN and SSL

This week I created a jMeter test setup for distributed testing. I thought it would be straight forward but I ran into some interesting things you might want to know if you are considering distributed testing using jMeter.

In my case, I had to test an application which was inside our corporate network, while working from home through a VPN and a firewall. Normally that is no problem, but jMeter has this funny construction where the slave (jMeter server) wants to connect back to the master (jMeter gui). It took some fiddling with iptables, the jMeter configuration and ssh tunneling to get it to work. Here’s my setup:

Getting this all to behave nicely wasn’t trivial. All the information can be found, but it took me quite some time to piece it all together. If your setup looks like the one above, or if your setup involves having your jMeter master machine on a different subnet and/or behind a firewall, here’s what you need to do.

Step 1: Configuring the slaves
In most jMeter setups, the slaves are Linux machines (any flavor will do, mine are redhat). Create a “jmeter” user on each of the machines so that you don’t have to run the jMeter process as a root user.

  1. Download jmeter as a zip file and unzip it on each of the jMeter slave machines. At the time of this writing I am using jMeter 2.6. Make sure that the files in the bin directory are executable with
    chmod +x *.sh jmeter jmeter-server
  2. On the first machine, edit the jmeter.properties file in the /bin folder of the unpacked jMeter installation and uncomment the line containing the server.rmi.localport setting. I changed mine to read:
    server.rmi.localport=50000
  3. Save the jmeter.properties file and copy it over to all the other machines (in the correct location).
  4. As "root" user, edit the iptables configuration in /etc/sysconfig/iptables and add the following lines, just above the first "REJECT" line (the order is important here):
    # Start jmeter ports
    -A RH-Firewall-1-INPUT -p udp -m udp --dport 1099 -j ACCEPT
    -A RH-Firewall-1-INPUT -p tcp -m tcp --dport 1099 -j ACCEPT
    -A RH-Firewall-1-INPUT -p udp -m udp --dport 50000 -j ACCEPT
    -A RH-Firewall-1-INPUT -p tcp -m tcp --dport 50000 -j ACCEPT
    # End jmeter ports
  5. Save the file and while still being root, restart iptables with
    service iptables restart

This covers the work you need to do on the slaves. To test your setup, start the "jmeter-server" application on one of the slaves. Note the address and port the server reports listening to:

Created remote object: UnicastServerRef [liveRef: 
[endpoint:[10.91.10.11:50000](local),
objID:[-24679ca9:13581440206:-7fff, -1472515273052223306]]]

From another slave, you can see if the iptables configuration works by telnetting to this slave on port 50000. You'll get connected to the jmeter-server process. Use Ctrl-] to exit telnet (there is nothing useful you could type here). If you get a "Connection refused", your iptables configuration is not working. Check your steps again.

For now, stop any running jMeter processes on the slaves.

Step 2: Configuring the master
The master is the system where your jMeter GUI will be running, and it is the system that tells the slaves what to do, and where to deliver their results. In order for the master to do that, it needs to be reachable by the slaves. In the network picture above you see a pesky firewall and a VPN server with mis-matching addresses. In order to make this work we're going to tell the master to listen at a fixed port for the slaves, and then do reverse port forwarding to the slaves so they can deliver content to a local port. Here's how:

  1. Take the same zipfile you used for the slaves. and unpack it on your master machine. In my case, this is a Macbook, running OSX 10.7.2
  2. Edit the jmeter.properties file in the bin folder and list the ip addresses of the jMeter slaves in the remote_hosts property (remove the 127.0.0.1 address) so that it looks like:
    remote_hosts=10.91.10.10,10.91.10.11,10.91.10.12,10.91.10.13

    Then look for the client.rmi.localport line and change that into:

    client.rmi.localport=51000
This is all the configuration you need to do on the master. The trick is in starting it all up, read on.

Step 3: Starting your tests
Now that your infrastructure is in place, let's put it to the test. First, from your jMeter master machine, connect to each of the machines using SSH, and setup a reverse tunnel to port 51000 on your master, like so:

ssh -R 51000:localhost:51000 jmeter@10.91.10.11

Keep each of these sessions open, as the jMeter slave needs to be able to deliver it's test results to your jMeter master. On your master machine, open a command prompt and type the following:

export JVM_ARGS="-Djava.rmi.server.hostname=localhost"

By doing this, jMeter will tell the slaves to connect to "localhost:51000" for delivering their results. This of course ends up being tunneled back to your jMeter master machine.

Then, on the master start jmeter.sh, load up your test script and use the "Run" Menu to run the script on one slave, or all slaves together. Remember that the master pushes your test script to each slave and tells it to run it. So if you have a script with a thread group running 10 threads, each slave will do 10 threads, resulting in 40 threads hitting your application.

Tips and Tricks
During this whole adventure I collected a few handy tips:

  • Document the whole setup, cookbook style, and put that document in an easy to find spot. There will be a time you have to pass this on to a colleague, and it will help a great deal if he can pick up your document and hit the ground running.
  • Put as much of the command line stuff you see above in scripts. In my setup, I have a "connect-slave.sh" script which contains the reverse tunnel trick, a "start-master.sh" which starts the jMeter master with the correct JVM arguments, and a "start-slave.sh" script which starts the slave with a bit more memory. Each slave has 2GB of heap, because I added the following line to the script:
    export JVM_ARGS="-Xmx2028m"
  • As much as you need to pay attention to the performance of your application, also make sure that your test script is lean and mean. Some listeners are very expensive to use. The "View Results Tree" and "View Results in Table" are very memory intensive. Great for debugging a single thread, you'll quickly run out of memory for 100 Threads or longer tests. Once you've got your scripts working, disable as much of the listeners as possible.
  • jMeter debugging is fidgety. When debugging your script, always do 1 thread and 1 iteration on 1 server. Once that works, you can scale up (don't forget to disable the "View Results Tree" listeners, see above).
  • jMeter quickly runs out of memory. Increase the default heap sizes of the master and the slaves by using the -Xmx JVM option.

Happy Testing!

About these ads

14 Responses to Distributed jMeter through VPN and SSL

  1. rusty says:

    Hi Rolfje, I followed your instructions and they work perfectly — we have quite a similar situation to yours and this really helped out, so thanks for that!

    One question as I’m trying to understand *why* this works — it seems like we never tell jmeter-server anything about port 51000, so how does it know to connect there? We’re also not really telling the master about jmeter-server’s listening on port 50000 so I’m kind of confused about how the two parts know about each other’s ports :)

    Can you provide any explanation?

    Thanks again!

    rusty

    • rolfje says:

      Hi Rusty,

      Good question.

      In step 2, item 2, we tell the master to listen on external port 51000.

      In Step 3, we tell the client to forward all local connects to port 5100 to be forwarded to the master.

      When the master uploads the jmx script to the slave, it also pushes local configuration to the slave. So the slave knows of port 51000 because the master passed that to the slave. What the master does not push however, is it’s hostname. That’s why the slave will connect to a local port, and we have to forward that port back to the master.

  2. rusty says:

    Ah, that makes sense — the client passes the connection info along with the test setup… thanks!

    rusty

  3. Thanks so much, this was a super easy-to-use reference to get this working. JMeter slaves across subnets does not work out of the box, and many references tell you it can’t be done. It was easy to implement using this guide in less than an hour’s time.

    Nice write-up!

  4. Deepali says:

    I am using GUI on my machine as master machine.
    I followed your steps but id do not see /etc/sysconfig/iptables on my server but when I start the jmeter-server i get as below

    Using local port: 4000
    Created remote object: UnicastServerRef [liveRef: [endpoint:[130.173.73.121:4000](local),objID:[-3d608ebf:14247b2458f:-7fff, 5062524123169916800]]]

    On the mater machine I have done all the steps you mentioned.
    Now when i wanted to run the step 3 I cannot run the following command on my master machine as it is windows machie i have to run it on the command prompt.but when i run i get error as below

    command: ssh -R 4000:localhost:4000 jmeter@192.168.179.202
    Error:
    C:\Users\mibdmm0>ssh -R 4000:localhost:4000 jmeter@192.168.179.202
    ‘ssh’ is not recognized as an internal or external command,
    operable program or batch file.

    And if i run this command on my slave machine which is linux i get error as
    ssh: connect to host 192.168.179.202 port 22: Connection timed out

    Also when i run my test plan from the jmeter gui on my local machine i get the same eror as
    ERROR: java.net.ConnectException: Connection timed out: connect

    I am not sure why the port is taken as 22.Can you please help me figure this out why it is working this way.

  5. Frans says:

    Great tutorial! Thanks so much.

  6. Jose Velazquez says:

    Hi everyone , I hope anyone of you can help on this… I’m need to record an oracle application with JMeter, the application is hosted outside the company and we reach the web-app through a VPN.

    Following the basic web recoding steps :

    - I cannot setup the proxy on the web server, when I setup a proxy on firefox or IE the application doesn’t work.. does it make sense?, though a VPN I cannot setup a proxy on the browser?

    - So, if I do not use a proxy on the browser Jmeter doesn’t record any action on the browser..

    My URL is like: http://mxapp.tfjfa.com.mx:8081/portalweb

  7. Are you sure about having to open the ports for udp as well?

  8. Slade says:

    Hi,

    I have the same problem. I need to run Jmeter server from different Subnet.

    I tried to do steps described in this post, but the problem is that I’m on Windows OS and I don’t know how to set Step 3 (ssh -R 51000:localhost:51000 jmeter@10.91.10.11)

    Please Help

    • rolfje says:

      Please have a look at CygWin, and don’t forget to install openssh in it. That should fix your problem.

      • Slade says:

        Hello rolfje,

        Thank you a lot for your reply. But unfortunately it doesn’t work anyway. So let me describe all my actions, maybe you will detect what mistake I make:

        First I would like to describe my environment:
        – Master machine is on Windows 8.1
        – Slave machines are on Windows Server 2008 R2
        – I’m using latest version of Jmeter (2.11) on all machines

        Actions done by me:
        1. On all machines I’ve changed jmeter.proprietes “server.rmi.localport=50000″

        2. I could NOT manage to “edit the iptables configuration” as my slave machines are on Windows OS so I skiped this step

        3. When I run jmeter.server on slaves it appears: “Created remote object: UnicastServerRef [liveRef: [endpoint:
        [173.15.30.142:50000](local),objID:[6ad53571:146d1dcd0f: -7fff, 786409434421533333]]]” – I suppose it’s OK.

        4. I’ve configured Master machine remote_hosts=173.15.30.142 then changed client.rmi.localport=51000

        5. I’ve downloaded CygWin and installed openssh

        6. On Master I run CygWin and wrote there
        ssh -R 51000:localhost:51000 jmeter@173.15.30.142
        as response I have: “ssh: connect to host 173.15.30.142 port 22: No error ”

        7. Now I started on Slaves jmeter.server after that I run jmeter.BAT and there “Remote Start”

        As result I have next error: ” Connection refused to host: 173.15.30.142; nested exception is: java.net.ConnectException: Connection timed out: connect ”

        Please help, I would be very grateful.
        Thank you in advance

      • Slade says:

        Thank you a lot !!!

        I’ve found the issue. I didn’t turn off firewall on Slave machines.

      • rolfje says:

        Awesome! Glad to see it works. Have fun testing!

  9. Slade says:

    Hello again Rolfje,

    First of all sorry for so much spam.

    All my previous actions I’ve done without VPN.

    Now I have a problem when I’m working throw VPN.

    Master machine is starting Slaves BUT no data come back to Master.
    Slave machines are making my Load, but on Master I can’t see anything in Aggregate report or any listener. Also in the right upper corner is not displayed that test are running, but on Slave it appears message that “Starting test ..”

    I’ve written in CygWin next command : “export JVM_ARGS=”-Djava.rmi.server.hostname=localhost” But it didn’t help, also I’ve written it in Command Line but ‘export’ is not recognized as command.

    Any idea why could this happen and how could I resolve this issue?

    Best regards,
    Slade

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 29 other followers