[SAC] High load "geotools" job on osgeo6: cryptonight at work

neteler · May 8, 2018, 9:54pm

Hi,

does anone know what this “j” job does which leads to load average: 12.04 for several weeks on osgeo6?

I noticed it a while ago:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23401 geotools 30 10 1219628 25240 1988 S 1200 0.0 811738:16 j

Any cryptomining ongoing there?

strace -p 23401
Process 23401 attached
epoll_wait(3, {}, 1024, 343) = 0
epoll_wait(3, {}, 1024, 478) = 0
epoll_wait(3, {}, 1024, 20) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 857872753}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 857898791}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 857914653}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 857930257}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 857946934}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 857962774}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 857978770}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 857994805}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 858010207}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 858025653}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 858041233}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 858058371}) = 0
epoll_wait(3, {}, 1024, 500) = 0
epoll_wait(3, {}, 1024, 477) = 0
epoll_wait(3, {}, 1024, 21) = 0
epoll_wait(3, {}, 1024, 500) = 0
epoll_wait(3, {}, 1024, 404) = 0
clock_gettime(CLOCK_REALTIME, {1525815798, 768184366}) = 0
clock_gettime(CLOCK_REALTIME, {1525815798, 768222411}) = 0
…

lsof -p 23401
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
j 23401 geotools cwd DIR 253,0 4096 192 /
j 23401 geotools rtd DIR 253,0 4096 192 /
j 23401 geotools txt REG 253,6 786544 7400 /var/tmp/ /j
j 23401 geotools mem REG 253,0 1738176 895459 /lib/x86_64-linux-gnu/libc-2.19.so
j 23401 geotools mem REG 253,0 1051056 895469 /lib/x86_64-linux-gnu/libm-2.19.so
j 23401 geotools mem REG 253,0 31784 895513 /lib/x86_64-linux-gnu/librt-2.19.so
j 23401 geotools mem REG 253,0 137384 820348 /lib/x86_64-linux-gnu/libpthread-2.19.so
j 23401 geotools mem REG 253,0 140928 820349 /lib/x86_64-linux-gnu/ld-2.19.so
j 23401 geotools 0r CHR 1,3 0t0 2052 /dev/null
j 23401 geotools 1w CHR 1,3 0t0 2052 /dev/null
j 23401 geotools 2w CHR 1,3 0t0 2052 /dev/null
j 23401 geotools 3u 0000 0,11 0 13535 anon_inode
j 23401 geotools 4r FIFO 0,10 0t0 1498595664 pipe
j 23401 geotools 5w FIFO 0,10 0t0 1498595664 pipe
j 23401 geotools 6r FIFO 0,10 0t0 1498606412 pipe
j 23401 geotools 7w FIFO 0,10 0t0 1498606412 pipe
j 23401 geotools 8u 0000 0,11 0 13535 anon_inode
j 23401 geotools 9r CHR 1,3 0t0 2052 /dev/null
j 23401 geotools 10u IPv4 1600207795 0t0 TCP osgeo6.osgeo.osuosl.org:40720->89.163.135.118:http (ESTABLISHED)

I don’t quite know what it tries to do.

It comes from an “invisible” (!) directory:

root@osgeo6:/var/tmp# ls -la /var/tmp/
total 198116
drwxr-xr-x 2 geotools users 32 Mar 22 14:56 <<----!!
drwxrwxrwt 4 root root 70 May 8 12:03 .
drwxr-xr-x 12 root root 138 Jul 19 2015 …
drwxr-xr-x 9 geotools users 4096 Sep 23 2015 geotools
-rw-r–r-- 1 geotools users 202861176 Sep 23 2015 geotools.tar.xz
-rw-r–r-- 1 geotools users 149 Sep 23 2015 README.txt

root@osgeo6:/var/tmp# tree
.
├──
│ ├── config.json
│ └── j

Here the magic happens:

root@osgeo6:/var/tmp# cd " "
root@osgeo6:/var/tmp/ # ls -la
total 776
drwxr-xr-x 2 geotools users 32 Mar 22 14:56 .
drwxrwxrwt 4 root root 70 May 8 12:03 …
-rw-r–r-- 1 geotools users 558 Mar 22 14:56 config.json
-rwxr-xr-x 1 geotools users 786544 Mar 18 09:42 j

Weird??

More forensic:

root@osgeo6:/var/tmp/ # file j
j: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.18, BuildID[sha1]=28ed31a04ec9c0f9e35c536cdbb6dfff922e9df3, stripped

root@osgeo6:/var/tmp/ # head -n 10 config.json
{
“algo”: “cryptonight”,
“av”: 0,
“background”: true,
“colors”: true,
“cpu-affinity”: null,
“cpu-priority”: null,
“donate-level”: 0,
“log-file”: null,
“max-cpu-usage”: 100,

Gotcha!

I suggest that we take a series of countermeasures now.

Markus

···

Markus Neteler, PhD
http://www.mundialis.de - free data with free software
http://grass.osgeo.org
http://courses.neteler.org/blog

robe · May 9, 2018, 9:25am

Hmm that does look very suspicious. Not sure why we would be cryptomining. I guess it could be intentional for some kind of testing thing.

Definitely need to do something about this like kill it and delete the files.

The j file seems relatively new too.

And target seems to be going to Germany somewhere

12 116 ms 154 ms 134 ms ve556.ipcar.dus3.myloc.de [62.141.47.106]

13 99 ms 109 ms 99 ms 89.163.135.118

From: Sac [mailto:sac-bounces@lists.osgeo.org] On Behalf Of Markus Neteler
Sent: Tuesday, May 08, 2018 5:54 PM
To: OSGeo-SAC sac@lists.osgeo.org
Subject: [SAC] High load “geotools” job on osgeo6: cryptonight at work

Hi,

does anone know what this “j” job does which leads to load average: 12.04 for several weeks on osgeo6?

I noticed it a while ago:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23401 geotools 30 10 1219628 25240 1988 S 1200 0.0 811738:16 j

Any cryptomining ongoing there?

strace -p 23401
Process 23401 attached
epoll_wait(3, {}, 1024, 343) = 0
epoll_wait(3, {}, 1024, 478) = 0
epoll_wait(3, {}, 1024, 20) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 857872753}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 857898791}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 857914653}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 857930257}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 857946934}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 857962774}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 857978770}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 857994805}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 858010207}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 858025653}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 858041233}) = 0
clock_gettime(CLOCK_REALTIME, {1525815796, 858058371}) = 0
epoll_wait(3, {}, 1024, 500) = 0
epoll_wait(3, {}, 1024, 477) = 0
epoll_wait(3, {}, 1024, 21) = 0
epoll_wait(3, {}, 1024, 500) = 0
epoll_wait(3, {}, 1024, 404) = 0
clock_gettime(CLOCK_REALTIME, {1525815798, 768184366}) = 0
clock_gettime(CLOCK_REALTIME, {1525815798, 768222411}) = 0
…

lsof -p 23401
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
j 23401 geotools cwd DIR 253,0 4096 192 /
j 23401 geotools rtd DIR 253,0 4096 192 /
j 23401 geotools txt REG 253,6 786544 7400 /var/tmp/ /j
j 23401 geotools mem REG 253,0 1738176 895459 /lib/x86_64-linux-gnu/libc-2.19.so
j 23401 geotools mem REG 253,0 1051056 895469 /lib/x86_64-linux-gnu/libm-2.19.so
j 23401 geotools mem REG 253,0 31784 895513 /lib/x86_64-linux-gnu/librt-2.19.so
j 23401 geotools mem REG 253,0 137384 820348 /lib/x86_64-linux-gnu/libpthread-2.19.so
j 23401 geotools mem REG 253,0 140928 820349 /lib/x86_64-linux-gnu/ld-2.19.so
j 23401 geotools 0r CHR 1,3 0t0 2052 /dev/null
j 23401 geotools 1w CHR 1,3 0t0 2052 /dev/null
j 23401 geotools 2w CHR 1,3 0t0 2052 /dev/null
j 23401 geotools 3u 0000 0,11 0 13535 anon_inode
j 23401 geotools 4r FIFO 0,10 0t0 1498595664 pipe
j 23401 geotools 5w FIFO 0,10 0t0 1498595664 pipe
j 23401 geotools 6r FIFO 0,10 0t0 1498606412 pipe
j 23401 geotools 7w FIFO 0,10 0t0 1498606412 pipe
j 23401 geotools 8u 0000 0,11 0 13535 anon_inode
j 23401 geotools 9r CHR 1,3 0t0 2052 /dev/null
j 23401 geotools 10u IPv4 1600207795 0t0 TCP osgeo6.osgeo.osuosl.org:40720->89.163.135.118:http (ESTABLISHED)

I don’t quite know what it tries to do.

It comes from an “invisible” (!) directory:

root@osgeo6:/var/tmp# ls -la /var/tmp/
total 198116
drwxr-xr-x 2 geotools users 32 Mar 22 14:56 <<----!!
drwxrwxrwt 4 root root 70 May 8 12:03 .
drwxr-xr-x 12 root root 138 Jul 19 2015 …
drwxr-xr-x 9 geotools users 4096 Sep 23 2015 geotools
-rw-r–r-- 1 geotools users 202861176 Sep 23 2015 geotools.tar.xz
-rw-r–r-- 1 geotools users 149 Sep 23 2015 README.txt

root@osgeo6:/var/tmp# tree
.
├──
│ ├── config.json
│ └── j

Here the magic happens:

root@osgeo6:/var/tmp# cd " "
root@osgeo6:/var/tmp/ # ls -la
total 776
drwxr-xr-x 2 geotools users 32 Mar 22 14:56 .
drwxrwxrwt 4 root root 70 May 8 12:03 …
-rw-r–r-- 1 geotools users 558 Mar 22 14:56 config.json
-rwxr-xr-x 1 geotools users 786544 Mar 18 09:42 j

Weird??

More forensic:

root@osgeo6:/var/tmp/ # file j
j: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.18, BuildID[sha1]=28ed31a04ec9c0f9e35c536cdbb6dfff922e9df3, stripped

root@osgeo6:/var/tmp/ # head -n 10 config.json
{
“algo”: “cryptonight”,
“av”: 0,
“background”: true,
“colors”: true,
“cpu-affinity”: null,
“cpu-priority”: null,
“donate-level”: 0,
“log-file”: null,
“max-cpu-usage”: 100,

Gotcha!

I suggest that we take a series of countermeasures now.

Markus

–

Markus Neteler, PhD
http://www.mundialis.de - free data with free software
http://grass.osgeo.org
http://courses.neteler.org/blog

neteler · May 9, 2018, 7:20pm

Hi,

for now I put te job to "sleep" using

kill -SIGSTOP 23401

Like that the traces are still there while it cannot continue to mine coins.

I suggest to
- force password reset of all logins on osgeo6 (how?)
- check who was on the machine
May 8 2018, 23:07 server time to install the thing
- eventually get rid of it

Opinions?

Markus

Alex_M · May 9, 2018, 11:52pm

On 05/09/2018 12:20 PM, Markus Neteler wrote:

Hi,

for now I put te job to "sleep" using

kill -SIGSTOP 23401

Like that the traces are still there while it cannot continue to mine coins.

I suggest to
- force password reset of all logins on osgeo6 (how?)
- check who was on the machine
May 8 2018, 23:07 server time to install the thing
- eventually get rid of it

Opinions?

Markus

The most likely entry point is via a website, what php based websites
are being run on osgeo6 right now? What web services run under the
geotools user?

Thanks,
Alex

jef · May 10, 2018, 6:57am

Hi Markus,

On Tue, 08. May 2018 at 23:54:14 +0200, Markus Neteler wrote:

It comes from an "invisible" (!) directory:

root@osgeo6:/var/tmp# ls -la /var/tmp/
total 198116
drwxr-xr-x 2 geotools users 32 Mar 22 14:56 <<----!!

On Wed, 09. May 2018 at 21:20:07 +0200, Markus Neteler wrote:

for now I put te job to "sleep" using

kill -SIGSTOP 23401

Like that the traces are still there while it cannot continue to mine coins.

I suggest to
- force password reset of all logins on osgeo6

I'd also expect that is was brought in via the (geotools) website and didn't
have access to anything else. So we probably don't need a password reset.

(how?)

I'd try: members users | xargs -n1 passwd -e

- check who was on the machine
May 8 2018, 23:07 server time to install the thing

You mean Mar 22:

$ ps -p 23401 -o pid,lstart,comm,cmd
PID STARTED COMMAND CMD
23401 Thu Mar 22 22:56:33 2018 j [ksoftirqd]

Which predates most if not all relevant logs we still have, right? Bacula also
keeps backups only for 1 months AFAICS.

- eventually get rid of it

+1

Jürgen

--
Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31
Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50
Software Engineer D-26506 Norden http://www.norbit.de

strk · May 10, 2018, 9:35am

On Thu, May 10, 2018 at 08:57:04AM +0200, Jürgen E. Fischer wrote:

I'd try: members users | xargs -n1 passwd -e

Isn't access on osgeo6 mediated by either LDAP or SSH ?

--strk;

jef · May 10, 2018, 11:46am

Hi Sandro,

On Thu, 10. May 2018 at 11:35:10 +0200, Sandro Santilli wrote:

On Thu, May 10, 2018 at 08:57:04AM +0200, Jürgen E. Fischer wrote:

> I'd try: members users | xargs -n1 passwd -e

Isn't access on osgeo6 mediated by either LDAP or SSH ?

Are you saying PAM doesn't forward the expiry information? I wasn't sure -
hence "I'd try".

Jürgen

--
Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31
Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50
Software Engineer D-26506 Norden http://www.norbit.de

neteler · May 10, 2018, 1:41pm

On Thu, May 10, 2018 at 8:57 AM, Jürgen E. Fischer <jef@norbit.de> wrote:

On Tue, 08. May 2018 at 23:54:14 +0200, Markus Neteler wrote:

...

I'd try: members users | xargs -n1 passwd -e

osgeo6:~$ members users | tr -s ' ' '\n' | wc -l
183

... truly a multi-user system (*cough*) including funny users like
"osgeotest123" etc. Hope all come with strong passwords.

Markus

strk · May 10, 2018, 1:59pm

On Thu, May 10, 2018 at 03:41:08PM +0200, Markus Neteler wrote:

On Thu, May 10, 2018 at 8:57 AM, Jürgen E. Fischer <jef@norbit.de> wrote:
> On Tue, 08. May 2018 at 23:54:14 +0200, Markus Neteler wrote:
...
> I'd try: members users | xargs -n1 passwd -e

osgeo6:~$ members users | tr -s ' ' '\n' | wc -l
183

... truly a multi-user system (*cough*) including funny users like
"osgeotest123" etc. Hope all come with strong passwords.

It looks like osgeotest123 was registered by Frank Warmerdam
in early October 2007. It is a LDAP entry. I dubt it has a strong
password. The corresponding email is osgeotest123@gmail.com.

Frank, do you still have access to that mailbox ?
Can we drop that LDAP user ?

Note that there's a known bug that does not allow _removing_
shell access from LDAP users:
https://trac.osgeo.org/osgeo/ticket/1786

Jurgen: locking a password in LDAP requires an extension (an "overlay")
and there's currently nobody assigned to that:
https://trac.osgeo.org/osgeo/ticket/1680

--strk;

neteler · September 16, 2018, 2:32pm

Hi SAC,

I just discovered:
on osgeo6 the cryptominer “j” running as “geotools” user is back

Tasks: 522 total, 1 running, 520 sleeping, 1 stopped, 0 zombie
%Cpu(s): 50.3 us, 0.1 sy, 0.0 ni, 49.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 13193000+total, 13111448+used, 815528 free, 8140 buffers
KiB Swap: 15622140 total, 18180 used, 15603960 free. 11951499+cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2845 geotools 20 0 1219628 25032 2048 S 1200 0.0 100848:13 j

… stealing some of our resources.

It runs since:

root@osgeo6:~# ps -p 2845 -o pid,lstart,comm,cmd
PID STARTED COMMAND CMD
2845 Mon Sep 10 11:11:51 2018 j [ksoftirqd]

Clearly, “geotools” was connected that time, for 3 seconds:

root@osgeo6:~# last

…
root pts/0 … Thu Sep 13 13:33 - 13:34 (00:00)
jmckenna pts/0 … Tue Sep 11 05:37 - 20:25 (14:48)
jmckenna pts/0 … Mon Sep 10 11:15 - 13:27 (02:11)
geotools pts/0 104.239.230.121 Mon Sep 10 11:09 - 11:12 (00:03)
root pts/0 … Mon Sep 10 11:07 - 11:08 (00:00)

I suggest that we put stronger measures this time:
Can we now agree that a password reset may be a good idea? Or, ssh key only.

Markus

strk · September 17, 2018, 8:43am

On Sun, Sep 16, 2018 at 04:32:10PM +0200, Markus Neteler wrote:

Clearly, "geotools" was connected that time, for 3 seconds:

root@osgeo6:~# last
...
root pts/0 ............. Thu Sep 13 13:33 - 13:34 (00:00)
jmckenna pts/0 ............. Tue Sep 11 05:37 - 20:25 (14:48)
jmckenna pts/0 ............. Mon Sep 10 11:15 - 13:27 (02:11)
geotools pts/0 104.239.230.121 Mon Sep 10 11:09 - 11:12 (00:03)
root pts/0 ............... Mon Sep 10 11:07 - 11:08 (00:00)

I suggest that we put stronger measures this time:
Can we now agree that a password reset may be a good idea? Or, ssh key only.

Please go ahead with a a password reset, but maybe don't send the
password by mail (the password reset script should print the temporary
password to stdout for you to send securely to a trusted party).

--strk;

jef · September 17, 2018, 1:06pm

Hi Sandro,

On Mon, 17. Sep 2018 at 10:43:48 +0200, Sandro Santilli wrote:

> root pts/0 ............. Thu Sep 13 13:33 - 13:34 (00:00)
> jmckenna pts/0 ............. Tue Sep 11 05:37 - 20:25 (14:48)
> jmckenna pts/0 ............. Mon Sep 10 11:15 - 13:27 (02:11)
> geotools pts/0 104.239.230.121 Mon Sep 10 11:09 - 11:12 (00:03)
> root pts/0 ............... Mon Sep 10 11:07 - 11:08 (00:00)
>
>
> I suggest that we put stronger measures this time:
> Can we now agree that a password reset may be a good idea? Or, ssh key only.

Please go ahead with a a password reset, but maybe don't send the
password by mail (the password reset script should print the temporary
password to stdout for you to send securely to a trusted party).

Apparently they did have a key:

Sep 10 11:09:10 osgeo6 sshd[1150]: Accepted publickey for geotools from 104.239.230.121 port 33281 ssh2: RSA cf:34:61:aa:48:e1:f2:20:16:6e:dc:c1:80:e1:8e:fe

Jürgen

--
Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31
Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50
Software Engineer D-26506 Norden http://www.norbit.de

Pflichtangaben (206 Bytes)

Harrison_Grundy · September 17, 2018, 1:46pm

Or added one whenever they got in... would we notice?

Sent from my iPhone

On Sep 17, 2018, at 9:06 PM, Jürgen E. Fischer <jef@norbit.de> wrote:

Hi Sandro,

On Mon, 17. Sep 2018 at 10:43:48 +0200, Sandro Santilli wrote:

root pts/0 ............. Thu Sep 13 13:33 - 13:34 (00:00)
jmckenna pts/0 ............. Tue Sep 11 05:37 - 20:25 (14:48)
jmckenna pts/0 ............. Mon Sep 10 11:15 - 13:27 (02:11)
geotools pts/0 104.239.230.121 Mon Sep 10 11:09 - 11:12 (00:03)
root pts/0 ............... Mon Sep 10 11:07 - 11:08 (00:00)

I suggest that we put stronger measures this time:
Can we now agree that a password reset may be a good idea? Or, ssh key only.

Please go ahead with a a password reset, but maybe don't send the
password by mail (the password reset script should print the temporary
password to stdout for you to send securely to a trusted party).

Apparently they did have a key:

Sep 10 11:09:10 osgeo6 sshd[1150]: Accepted publickey for geotools from 104.239.230.121 port 33281 ssh2: RSA cf:34:61:aa:48:e1:f2:20:16:6e:dc:c1:80:e1:8e:fe

Jürgen

--
Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31
Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50
Software Engineer D-26506 Norden http://www.norbit.de
norBIT Gesellschaft fuer Unternehmensberatung und Informationssysteme mbH
Rheinstrasse 13, 26506 Norden
GF: Jelto Buurman, HR: Amtsgericht Aurich HRB 100827
Datenschutzerklaerung: https://www.norbit.de/83/
_______________________________________________
Sac mailing list
Sac@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/sac