API doesn't work for querying IPv6? (0 hits)

Lexus · 2022-02-22 11:02 am

I'm trying to use the API to query for IPv6 addresses, but it doesn't find a match, even though the normal SFS search page does.

The IP address is "2a0b:f4c2::7".
(It's a TOR exit node, even though SFS doesn't recognize it as an exit node, but that's a different problem.)
Using stopforumspam.com/search results in:

Found 391 entries for "2a0b:f4c2::/64"

So it generalizes the IP address from "2a0b:f4c2::7" to "2a0b:f4c2::/64" to find matches. All is fine so far.

I'm trying to get the same result with the API, but that doesn't work. I tried:

https://api.stopforumspam.org/api?ip=2a0b:f4c2::7&json (frequency=0)
https://api.stopforumspam.org/api?ip=2a0b:f4c2::7/64&json (invalid ip)
https://api.stopforumspam.org/api?ip=2a0b:f4c2::&json (frequency=0)
https://api.stopforumspam.org/api?ip=2a0b:f4c2:0:0::&json (frequency=0)
https://api.stopforumspam.org/api?ip=2a0b:f4c2:0000:0000:0000:0000:0000:0007&json (frequency=0)
https://api.stopforumspam.org/api?ip=2a0b:f4c2:0000:0000:0000:0000:0000:0000&json (frequency=0)

(I also tried this with POST requests with url form encoding, but I'm always getting 0 hits.)

How can I query the API for IPv6 addresses? Or is this not supported?

Lexus · 2022-02-22 11:18 am

I found an IPv6 address that does work: "2a0b:f4c0:16c:13::1". But it's unclear why "2a0b:f4c2::7" doesn't work.

Alex Kemp · 2022-02-22 12:22 pm

Welcome to SFS, Lexus, and thanks for your report.

Something seems wrong with SFS API-results for your IPv6 queries from many points of view. I first tried the web-search & got the following:-

IP search for '2a0b:f4c2::7' (NOT exact OR exact match) wrote:

Found 391 entries for "2a0b:f4c2::/64"

IP search for '2a0b:f4c2' (NOT exact match) wrote:

Found 1000 entries for "2a0b:f4c2"

IP search for '2a0b:f4c2' (exact match) wrote:

Found 0 entries for "2a0b:f4c2"

I tried one of the 1,000 results:

results for https://www.stopforumspam.com/ipcheck/2a0b:f4c2:1:: wrote:

2a0b:f4c2:1::/64 appears in our database 390 times

I also looked at Evidence for one of the listed spammers:

results for evidence 203542663 wrote:

IP = 2a0b:f4c2::

That seems about right, although I suspect that the exact/not-exact may be muddled. PS: that ISP is riddled with spammers:

$ whois 2a0b:f4c2:1::/64
% Information related to '2a0b:f4c2:1::/48'

% Abuse contact for '2a0b:f4c2:1::/48' is 'abuse@relayon.org'

inet6num:       2a0b:f4c2:1::/48
netname:        RELAYON
remarks:        -----------------------------------
remarks:        This network is used for Tor Exits.
remarks:        We do not have any logs at all.
remarks:        For more information please visit:
remarks:        https://www.torproject.org
remarks:        -----------------------------------
…
organisation:   ORG-CTSL7-RIPE
org-name:       CIA TRIAD SECURITY LLC
org-type:       OTHER
address:        2701 Centerville Road
address:        New Castle County
address:        Wilmington
address:        Delaware 19808
address:        USA

Next was to use IPv6 as reported above & try within the port-80-API (next post):
tl;dr : the port-80 API is completely broken for IPv6

Alex Kemp · 2022-02-22 2:08 pm

To qualify my last remark in my previous post, the port-80 API seems to be broken for certain ranges of IPv6. I'm trying many wrinkles of IPv6, some in illegal format, just to see what kind of response the API will give:

First is 2a0b:f4c2:1::/64 (broken addressing):

<response success="true">
	<type>ip</type>
	<error>invalid ip address</error>
</response>

2a0b:f4c2:1:: (wrong result):

<response success="true">
	<type>ip</type>
	<appears>no</appears>
	<frequency>0</frequency>
</response>

2a0b:f4c2:1 (broken addressing):

<response success="true">
	<type>ip</type>
	<error>invalid ip address</error>
</response>

2a0b:f4c2::7 (wrong result):

<response success="true">
	<type>ip</type>
	<appears>no</appears>
	<frequency>0</frequency>
</response>

2a0b:f4c2::/64 (broken addressing):

<response success="true">
	<type>ip</type>
	<error>invalid ip address</error>
</response>

2a0b:f4c2:: (wrong result):

<response success="true">
	<type>ip</type>
	<appears>no</appears>
	<frequency>0</frequency>
</response>

2a0b:f4c2 (broken addressing):

<response success="true">
	<type>ip</type>
	<error>invalid ip address</error>
</response>

Then, at long last, using the value from Lexus' last post:
2a0b:f4c0:16c:13::1 (valid!):

<response success="true">
	<type>ip</type>
	<appears>yes</appears>
	<lastseen>2022-02-02 11:13:11</lastseen>
	<frequency>31</frequency>
</response>

I do think that the result should include the value searched upon (as sanity confirmation), but there is zero point in doing that first if the algorithm is broken. I'll PM + email pedigree to fix it (the only guy that can do so).

Lexus · 2022-02-22 7:18 pm

The non-default response format also shows additional info, including a "normalized" field, which could be the address that the API uses for the query. For example:
"2a0b:f4c2::7" has a "normalized" value of "2a0b:f4c2:0:0::" and
"2a0b:f4c0:16c:13::1" has a "normalized" value of "2a0b:f4c0:16c:13::".

Here are some more examples:
(Those with "frequency=0" are missed from the API.)

2a0b:f4c0:16c:11::1	Normalized=2a0b:f4c0:16c:11::	Frequency=50
2a0b:f4c0:16c:12::1	Normalized=2a0b:f4c0:16c:12::	Frequency=15
2a0b:f4c0:16c:13::1	Normalized=2a0b:f4c0:16c:13::	Frequency=31
2a0b:f4c0:16c:14::1	Normalized=2a0b:f4c0:16c:14::	Frequency=32
2a0b:f4c0:16c:15::1	Normalized=2a0b:f4c0:16c:15::	Frequency=49
2a0b:f4c0:16c:16::1	Normalized=2a0b:f4c0:16c:16::	Frequency=35
2a0b:f4c0:16c:1:0:0:0:1	Normalized=2a0b:f4c0:16c:1::	Frequency=59
2a0b:f4c0:16c:1::1	Normalized=2a0b:f4c0:16c:1::	Frequency=59
2a0b:f4c0:16c:2:0:0:0:1	Normalized=2a0b:f4c0:16c:2::	Frequency=30
2a0b:f4c0:16c:2::1	Normalized=2a0b:f4c0:16c:2::	Frequency=30
2a0b:f4c0:16c:3::1	Normalized=2a0b:f4c0:16c:3::	Frequency=45
2a0b:f4c0:16c:4::1	Normalized=2a0b:f4c0:16c:4::	Frequency=31
2a0b:f4c0:16c:5::1	Normalized=2a0b:f4c0:16c:5::	Frequency=14
2a0b:f4c0:16c:6::1	Normalized=2a0b:f4c0:16c:6::	Frequency=10
2a0b:f4c0:16c:7::1	Normalized=2a0b:f4c0:16c:7::	Frequency=37
2a0b:f4c0:16c:8::1	Normalized=2a0b:f4c0:16c:8::	Frequency=26
2a0b:f4c0:16c:9::1	Normalized=2a0b:f4c0:16c:9::	Frequency=6
2a0b:f4c1:0:0:0:0:0:6	Normalized=2a0b:f4c1:0:0::	Frequency=0
2a0b:f4c1:0:0:0:0:0:7	Normalized=2a0b:f4c1:0:0::	Frequency=0
2a0b:f4c1:0:0:0:0:0:8	Normalized=2a0b:f4c1:0:0::	Frequency=0
2a0b:f4c1:2::240	Normalized=2a0b:f4c1:2:0::	Frequency=0
2a0b:f4c1:2::241	Normalized=2a0b:f4c1:2:0::	Frequency=0
2a0b:f4c1:2::242	Normalized=2a0b:f4c1:2:0::	Frequency=0
2a0b:f4c1:2::243	Normalized=2a0b:f4c1:2:0::	Frequency=0
2a0b:f4c1:2::244	Normalized=2a0b:f4c1:2:0::	Frequency=0
2a0b:f4c1:2::245	Normalized=2a0b:f4c1:2:0::	Frequency=0
2a0b:f4c1:2::247	Normalized=2a0b:f4c1:2:0::	Frequency=0
2a0b:f4c1:2::248	Normalized=2a0b:f4c1:2:0::	Frequency=0
2a0b:f4c1:2::249	Normalized=2a0b:f4c1:2:0::	Frequency=0
2a0b:f4c1:2::250	Normalized=2a0b:f4c1:2:0::	Frequency=0
2a0b:f4c1:2::252	Normalized=2a0b:f4c1:2:0::	Frequency=0
2a0b:f4c1:2::253	Normalized=2a0b:f4c1:2:0::	Frequency=0
2a0b:f4c1:2::254	Normalized=2a0b:f4c1:2:0::	Frequency=0
2a0b:f4c1::4	Normalized=2a0b:f4c1:0:0::	Frequency=0
2a0b:f4c1::6	Normalized=2a0b:f4c1:0:0::	Frequency=0
2a0b:f4c1::7	Normalized=2a0b:f4c1:0:0::	Frequency=0
2a0b:f4c1::8	Normalized=2a0b:f4c1:0:0::	Frequency=0
2a0b:f4c2:1::1	Normalized=2a0b:f4c2:1:0::	Frequency=0
2a0b:f4c2:2::1	Normalized=2a0b:f4c2:2:0::	Frequency=0
2a0b:f4c2:2::32	Normalized=2a0b:f4c2:2:0::	Frequency=0
2a0b:f4c2:2::33	Normalized=2a0b:f4c2:2:0::	Frequency=0
2a0b:f4c2:2::35	Normalized=2a0b:f4c2:2:0::	Frequency=0
2a0b:f4c2:2::36	Normalized=2a0b:f4c2:2:0::	Frequency=0
2a0b:f4c2:2::40	Normalized=2a0b:f4c2:2:0::	Frequency=0
2a0b:f4c2:2::41	Normalized=2a0b:f4c2:2:0::	Frequency=0
2a0b:f4c2:2::42	Normalized=2a0b:f4c2:2:0::	Frequency=0
2a0b:f4c2:2::46	Normalized=2a0b:f4c2:2:0::	Frequency=0
2a0b:f4c2:2::50	Normalized=2a0b:f4c2:2:0::	Frequency=0
2a0b:f4c2:2::53	Normalized=2a0b:f4c2:2:0::	Frequency=0
2a0b:f4c2:2::55	Normalized=2a0b:f4c2:2:0::	Frequency=0
2a0b:f4c2:2::56	Normalized=2a0b:f4c2:2:0::	Frequency=0
2a0b:f4c2:2::58	Normalized=2a0b:f4c2:2:0::	Frequency=0
2a0b:f4c2:2::60	Normalized=2a0b:f4c2:2:0::	Frequency=0
2a0b:f4c2:3::82	Normalized=2a0b:f4c2:3:0::	Frequency=0
2a0b:f4c2:3::84	Normalized=2a0b:f4c2:3:0::	Frequency=0
2a0b:f4c2:3::86	Normalized=2a0b:f4c2:3:0::	Frequency=0
2a0b:f4c2:3::87	Normalized=2a0b:f4c2:3:0::	Frequency=0
2a0b:f4c2:3::93	Normalized=2a0b:f4c2:3:0::	Frequency=0
2a0b:f4c2:3::94	Normalized=2a0b:f4c2:3:0::	Frequency=0
2a0b:f4c2::1	Normalized=2a0b:f4c2:0:0::	Frequency=0
2a0b:f4c2::11	Normalized=2a0b:f4c2:0:0::	Frequency=0
2a0b:f4c2::12	Normalized=2a0b:f4c2:0:0::	Frequency=0
2a0b:f4c2::14	Normalized=2a0b:f4c2:0:0::	Frequency=0
2a0b:f4c2::16	Normalized=2a0b:f4c2:0:0::	Frequency=0
2a0b:f4c2::18	Normalized=2a0b:f4c2:0:0::	Frequency=0
2a0b:f4c2::2	Normalized=2a0b:f4c2:0:0::	Frequency=0
2a0b:f4c2::20	Normalized=2a0b:f4c2:0:0::	Frequency=0
2a0b:f4c2::22	Normalized=2a0b:f4c2:0:0::	Frequency=0
2a0b:f4c2::5	Normalized=2a0b:f4c2:0:0::	Frequency=0
2a0b:f4c2::7	Normalized=2a0b:f4c2:0:0::	Frequency=0
2a0b:f4c2::9	Normalized=2a0b:f4c2:0:0::	Frequency=0

Maybe those zero/empty groups are causing some problems in the API IP address lookup code, because they can be represented in multiple ways.

Alex Kemp · 2022-02-22 7:25 pm

I think that the problem is attempting to use just 2 Quartets. However, what you have provided is incredibly useful.

I have previously both PM'd & emailed pedigree. He is the only one that can fix this, so we now need to wait.

pedigree · 2022-02-23 9:33 pm

thanks for all this info. I'll have a look at this today and see where my code is broken. Its nice to see someone post actual data instead of the usual "computer says no", so thanks again

pedigree · 2022-02-25 6:58 am

>I found an IPv6 address that does work: "2a0b:f4c0:16c:13::1". But it's unclear why "2a0b:f4c2::7" doesn't work.

IPv6 addresses are normalised at the /64

2a0b:f4c0:16c:13::1 = 2a0b:f4c0:16c:13::0/64
2a0b:f4c2::7 = 2a0b:f4c2::0/64

https://www.stopforumspam.com/ipcheck/2a0b:f4c2::

2a0b:f4c2::7 is included in that /64 but as you say, it's not showing in the API results. This isnt a bug in the API, its a bug in the code that parses the data into the API....

The hunt begins

The API normalises that IP to 2a0b:f4c2:0:0::

This is a sha1 of 8b624c1601b17a528e2ba80951ed902aec7e29e5

the data bucket named 8b62 (top 16 bytes) does not contain this hash so its missing in the API data store.

so looking at the DB and the value 2a0b:f4c2::7 is normalised in a different format of 2a0b:f4c2::7 with a hash of e04ca8ec207ec47d7c216e0d0236c0364e339b10, putting its data in bucket e04c

and there is the problem.....

updating the database to normalise with a ::0 suffix would be a major piece of work, no small undertaking at all

this fix is simple in the API. here is how the IPv6 was built in a different format

        str = addr.parts[0].toString(base16) + ":" + 
            addr.parts[1].toString(base16) + ":" +
            addr.parts[2].toString(base16) + ":" +
            addr.parts[3].toString(base16) + "::";

the code to handle this does produce a valid IPv6 string however in a different format from the code processing. It doesn't compress in the same way.

This fix wasn't hard, it's a change to the logic so that it produces the same string

{"success":1,"ip":{"value":"2a0b:f4c2::7","appears":1,"frequency":392,"lastseen":"2022-02-22 22:11:34","normalized":"2a0b:f4c2::","confidence":98.87}}

{"success":1,"ip":{"value":"2a0b::7","frequency":0,"appears":0,"normalized":"2a0b::"}}

{"success":1,"ip":{"value":"2a0b::7:7:7:77","frequency":0,"appears":0,"normalized":"2a0b::"}}

{"success":1,"ip":{"value":"2a0b:7:7:7:77::","frequency":0,"appears":0,"normalized":"2a0b:7:7:7::"}}

{"success":1,"ip":{"value":"2a0b:0:0:0:77::","frequency":0,"appears":0,"normalized":"2a0b::"}}

and for non normalised IP addresses

{"success":1,"ip":{"value":"2a0b:0:0:77::","frequency":0,"appears":0}}

I have some more unit tests to write for this before I can deploy the fix. It will change the format of the normalised IPv6 address in the result, but its still a valid compressed IPv6 address. Do you think that this rarely used format change would be an acceptable change? It's not a huge bit of work to not change anything

but I have a fix for this.... now just the TOR flag for IPv6 networks.

pedigree · 2022-02-26 11:13 pm

And an update with support for ipv6 exit nodes

{"success":1,"ip":{"value":"2a0b:f4c2::7","appears":1,"frequency":391,
"lastseen":"2022-02-22 22:11:34","torexit":1,"normalized":"2a0b:f4c2:0:0::",
"confidence":98.86}}

Alex Kemp · 2022-02-26 11:49 pm

Well, that is excellent, pedigree. Now, when are you going to implement it?

Currently:

A web-search for 2a0b:f4c2::7 shows 391 entries
The web ipcheck for 2a0b:f4c2::7 shows 391 entries, but
The web api for 2a0b:f4c2::7 shows 0 entries.

Whoops.

pedigree · 2022-02-27 4:06 am

I'm writing the tests for it now. I'm being very careful around the internal change to normalising IPv6 addresses and how this will impact blacklist subnet/range tests

The damned tests are taking longer than the fix... typical

Alex Kemp · 2022-02-27 10:44 am

Thanks.

tmo · 2022-10-09 8:29 am

Just curious: Have the improvements been implemented by now?

pedigree · 2022-10-09 8:35 pm

this was deployed a while ago now

#1 2022-02-22 11:02 am

API doesn't work for querying IPv6? (0 hits)

#2 2022-02-22 11:18 am

Re: API doesn't work for querying IPv6? (0 hits)

#3 2022-02-22 12:22 pm

Re: API doesn't work for querying IPv6? (0 hits)

#4 2022-02-22 2:08 pm

Re: API doesn't work for querying IPv6? (0 hits)

#5 2022-02-22 7:18 pm

Re: API doesn't work for querying IPv6? (0 hits)

#6 2022-02-22 7:25 pm

Re: API doesn't work for querying IPv6? (0 hits)

#7 2022-02-23 9:33 pm

Re: API doesn't work for querying IPv6? (0 hits)

#8 2022-02-25 6:58 am

Re: API doesn't work for querying IPv6? (0 hits)

#9 2022-02-26 11:13 pm

Re: API doesn't work for querying IPv6? (0 hits)

#10 2022-02-26 11:49 pm

Re: API doesn't work for querying IPv6? (0 hits)

#11 2022-02-27 4:06 am

Re: API doesn't work for querying IPv6? (0 hits)

#12 2022-02-27 10:44 am

Re: API doesn't work for querying IPv6? (0 hits)

#13 2022-10-09 8:29 am

Re: API doesn't work for querying IPv6? (0 hits)

#14 2022-10-09 8:35 pm

Re: API doesn't work for querying IPv6? (0 hits)

Board footer