Birdwatching
I've been looking into the Mysql authentication protocol for my thesis, in this research I looked into the implementation used in Open Canary. This is a honeypot written in Python created by Thinkst. During testing I noticed a small difference in the error returned by Mysql and by Open Canary which made me curious.
Initial Finding
When you try to login to a Mysql server with wrong credentials, an error is returned together with an SQL state. As is defined in the developer documentation here this section of the packet should start with hashtag (#) and should then be followed by exactly five bytes describing the SQL state. However, an error in the implementation by Open Canary put two hashtags there. Something you'd miss if you weren't comparing error messages to each other.
Standard Mysql error:
1 ➜ mysql -h 127.0.0.1 -P3306
2ERROR 1045 (28000): Access denied for user 'niels'@'127.0.0.1' (using password: NO)
Open Canary error (the hashtag changes the syntax highlighting):
1 ➜ mysql -h 127.0.0.1 -P3306
2ERROR 1045 (#2800): Access denied for user 'niels'@'127.0.0.1' (using password: NO)
The Code Behind It
So the code powering this mock-mysql service is quite simple:
1def error_pkt(self, seq_id, err_code, sql_state, msg):
2 data = b"\xff" + struct.pack("<H", err_code) + b"\x23#" + sql_state + msg
3 return self.build_packet(0x02, data)
The error is essentially in the \x23#, as ASCII character 23 already is a
hashtag. There was another error with the access denied state which was
defined as 2800 while it should've been 28000. This is also the reason for
why the double hashtags didn't mess with the format as code was one character
too short making the effective length six again.
Turning It Into an Attack
A honeypot can be used for multiple purposes, it's a device that acts as if it were a real server making it a potential target for hackers. A honeypot's purpose however, is to be attacked. This has a couple of advantages:
- You know there are hackers poking around your network leaving you able to act on them.
- You can potentially figure out what exploits they're using by examining the data sent to the honeypot.
- They waste time on your honeypot instead of attacking real production servers in you network.
To put it simply: if your honeypot is attacked, it will be able to notify you and you can stop the attackers dead in their tracks. So it's pretty essential that your honeypot isn't found out. The error in the implementation shows us that we're not dealing with a genuine Mysql server. We do need to interact by logging into the server however, which would make the honeypot notifiy the system administrators. So we'd know that we were dealing with a honeypot, but the owners of the network are alerted to you presence which is something attackers want to avoid.
So is there a way to look at this error message without triggering the alarm? As it turns out: yes! Mysql's protocol requires packets to be numbered. When a connection is established the Mysql server will initiate the conversation by sending a packet with its banner and capabilities. This is packet zero:
The client will then send an answer back to the server with its own capabilities and the username it wants to login with. The packet number has to be one now:
Whenever the client sends a packet that doesn't have the right packet number
Mysql will return an error saying Got packets out of order with an SQL state
of 08S01. But due to the double hashtag in Open Canary, we can notice two
things:
- The double hashtags.
- The error message starts with the last digit of the full error code
The code behind this one is the following:
1elif seq_id != 1:
2 # error on wrong seq_id, even if payload hasn't arrived yet
3 self.transport.write(self.unordered_pkt(0x01))
4 self.transport.loseConnection()
5 return
6elif payload is not None:
7 # seq_id == 1 and payload has arrived
8 username, password = self.parse_auth(payload)
9 if username:
10 logdata = {'USERNAME': username, 'PASSWORD': password}
11 self.factory.canaryservice.log(logdata, transport=self.transport)
12 self.transport.write(self.access_denied(0x02, username, password))
13 self.transport.loseConnection()
This code is part of an if statement, it checks whether the received packet has the correct sequence id. If not, an out of order error will be returned. The problem is that the code for logging a login attempt is after that check. So when a packet is sent out of order, no logging will be done but a error message will be returned. So an attacker is able to scan a compromised network and detect canaries whithout raising suspicion.
Mitigation
I created a patch which consisted of 2 small changes to the mysql module:
1--- a/opencanary/modules/mysql.py
2+++ b/opencanary/modules/mysql.py
3@@ -16,7 +16,7 @@ class MySQL(Protocol, TimeoutMixin):
4 HEADER_LEN = 4
5 ERR_CODE_ACCESS_DENIED = 1045
6 ERR_CODE_PKT_ORDER = 1156
7- SQL_STATE_ACCESS_DENIED = b"2800"
8+ SQL_STATE_ACCESS_DENIED = b"28000"
9 SQL_STATE_PKT_ORDER = b"08S01"
10
11 # https://dev.mysql.com/doc/internals/en/connection-phase-packets.html#packet-Protocol::Handshake
12@@ -106,7 +106,7 @@ class MySQL(Protocol, TimeoutMixin):
13 MySQL.SQL_STATE_PKT_ORDER, msg)
14
15 def error_pkt(self, seq_id, err_code, sql_state, msg):
16- data = b"\xff" + struct.pack("<H", err_code) + b"\x23#" + sql_state + msg
17+ data = b"\xff" + struct.pack("<H", err_code) + b"#" + sql_state + msg
18 return self.build_packet(0x02, data)
19
20 def connectionMade(self):
21
An extra zero was added to SQL_STATE_ACCESS_DENIED and the \x32 version of
the hashtag was removed from the data byte string.
I reached out to the people at thinkst and the patch was published within 2 days.
Conclusion
A way was found in Open Canary running the mysql module to detect that they were honeypots. This detection could be executed without raising an alarm and therefore bypassing the benefits of using a honeypot.
The team over at Thinkst responded quickly to my mail and we had the issue resolved in two days. It's best to upgrade to Open Canary 0.6.1 or if that's not possible for you at this time, disable the Mysql module.