Inconsistencies

[pieterhartel]

Before I make lots of changes I'd like to ask what the reason is that not all domains return similar information. For example comparing class WhoisAU(WhoisEntry) and class WhoisRs(WhoisEntry) (in whois/parser.py) one class returns the registrant name as "name" and the other as "registrant_name". I have many historical whois outputs that I would like to analyse, but for consistency I need the same keys.

[richardpenman]

Basically these parsers were contributed over time by various people and I was not diligent at keeping the fields consistent. If someone is motivated to change them to consistent fields, that would be great! I would be keen to merge such a PR.

[pieterhartel]

I am tempted but how do I debug existing or new REs?

[richardpenman]

Sorry not sure what you mean. There are test cases for the current names that would need to be adjusted.

[pieterhartel]

To explain my question, I have added a parser class for Belgium as follows:

class WhoisBe(WhoisEntry):
    """Whois parser for .be domains"""
            
    regex: dict[str, str] = {
        "domain_name": r"Domain: *(.+)",
        "status": r"Status: *(.+)",
        "registrar": r"Registrar:\s+Name: (.+)", 
        "creation_date": r"Registered: *(.+)",
        "name_servers": r"Nameservers:\s((?:\s+?[\w.]+\s)*)",  # list of name servers
    }   
            
    def __init__(self, domain: str, text: str):
        if "Status: AVAILABLE" in text:
            raise WhoisDomainNotFoundError(text)
        else:
            WhoisEntry.__init__(self, domain, text, self.regex)

My test program below cannot match the RE for the registrar (It should be Nameshield). It prints null instead:

{
  "domain_name": [ "BE", "bnpparibas.be" ],
  "status": [ "ACTIVE", "NOT AVAILABLE" ],
  "registrar": null,
  "creation_date": "2001-06-12 00:00:00",
  "name_servers": "ns3.domivesta.net\n\tns4.domivesta.com\n\tns1.bnpparibas.com\n\tns2.bnpparibas.com"
}

Here is my test program:

import whois
text = """
% IANA WHOIS server
% for more information on IANA, visit http://www.iana.org
% This query returned 1 object

refer:        whois.dns.be

domain:       BE
...
status:       ACTIVE
...
Domain:	bnpparibas.be
Status:	NOT AVAILABLE
Registered:	Tue Jun 12 2001
...
Registrar:
	Name:	Nameshield
	Website:	http://www.nameshield.net
...
"""
domain_name = "bnpparisbas.be"
print( whois.parser.WhoisEntry.load( domain_name, text ) )

[pieterhartel]

By trying a few things I found the error in the RE. r"Registrar:\s+Name: (.+)" should be r"Registrar:\s+Name: *(.+)". However, trial-and-error is a little bit difficult, hence my question how to debug the REs.

[richardpenman]

When debugging regular expressions I'm not familiar with a better approach than trail-and-error. Maybe you would find something like regex101.com helpful to highlight the match?

[pieterhartel]

Ok. Another question: Do you want the changes to be non-breaking? For example looking like this:

[richardpenman]

Sorry for delay responding - yes can be breaking. I would release these breaking changes under a new version.

[pieterhartel]

I have opened a pull request for my changes.